Join the waitlist

Dataset

The private values a round is scored against, and how to keep them current.

A game's answers are data a fan must not be able to see. This is where they live, how you get them there, and what it costs you to keep them current.

The shape, in one picture

Three things, and only the middle one is secret.

  Studio            what a fan reads:   "Michael Jordan"
     |              the board, the labels, every word
     |
     |  entry_key   the join:            jordami01
     v
  Dataset    what a fan must not read:  32292
     |              a private file, yours, never served to a browser
     v
  The Engine        holds what provisioning joined,
                    and never lets a value out

The producer types a name. You supply what the name is worth. Provisioning puts them together at the moment a round is built, the Engine holds the result, and the number never reaches the page.

Which formats can use one

Push and hunt, today.

FormatUses a dataset
PushAlways. The game names one, and a fan's picks add up its values.
HuntWhen a round's Answer from data field names a measure. The board holds dataset keys, Which player is right says highest, lowest, only_nonzero or only_zero, and the answer is worked out when the round is provisioned. Leave the field empty and you type the answer yourself.
GroupNot yet. A group round carries its whole answer on the element.
PredictNot yet. A prediction is settled from the result you set in Studio.

Build a group or predict game against a dataset and nothing reads it: the round provisions from its own fields and the dataset is ignored.

Why it is not a live feed

A round must not change while somebody is playing it. If a value moves between one fan's pick and another's, the two of them played different games, the leaderboard compares values that were never comparable, and a dispute cannot be settled.

So a dataset is fresh when a round is provisioned and frozen while it is played. Push whenever you like. A round already open keeps the values it opened with; the next one picks up your change. That is a property of the Engine, not something you have to arrange.

The file

JSON, one document, one or more measures. A measure is a thing you can count a choice on.

{
  "corpus": "1",
  "measures": [
    {
      "key": "career_points",
      "label": "Career Points",
      "values": { "jordami01": 32292, "birdla01": 21791 }
    }
  ]
}

values maps a key to a number. The key is the join, and it is the only part that has to agree with anything: whatever a producer types as a choice's reference in Studio must appear here, spelled the same way.

Use identifiers your source already has. Inventing your own means maintaining a second mapping for ever.

What the file has to satisfy, all of it checked before anything is uploaded:

  • corpus is the string "1". It is the document's schema version rather than yours. The file is yours and changes whenever you push a new one, with no deploy, so the one thing we rely on is that it says what it is.
  • At least one measure, and no two sharing a key. A round names the measure it is counted on, so two measures under one key is a round that cannot say what scored it.
  • Every value is a whole number, zero or above. Not a decimal, not negative, not a string. A total that arrives as 21.5 is refused.
  • label is optional. It exists so a refusal can say the dataset holds Career Points and Career Assists rather than pts and ast.

Two optional fields sit beside measures, and reference build writes both when you pass --label and --source:

{
  "corpus": "1",
  "source": "Basketball Reference, career regular season",
  "entries": { "jordami01": { "label": "Michael Jordan" } },
  "measures": [ ... ]
}
  • entries names who each key is, so a producer choosing players in Studio reads "Michael Jordan" rather than jordami01. It scores nothing and never reaches a fan. Each entry may also carry team and position.
  • source is one line saying where the figures came from, shown when a round reveals its numbers.

A dataset with one measure needs nothing more. One with several needs the round to name the one it counts on, which a producer sets on the round's Measure field.

Your Dataset field holds a dataset name. The element's Content source field, source_url, is not read yet. If you set it on a round Gamestage builds from Studio, your deploy is refused. A round that is already live keeps playing, and the refusal is logged against the game. A game still taking its round from the manifest does not read the element at all, so the field is ignored there. Clear the field and enter the content in the element's own fields. See Game formats.

Scoring a push round by how close it came

By default, a push round scores its raw total and a bust scores the overshoot as a negative number: The Limit depends on this. Add a scoring block to a measure to score by proximity instead:

{
  "key": "career_points",
  "label": "Career Points",
  "values": { "jordami01": 32292, "birdla01": 21791 },
  "scoring": { "mode": "proximity", "pointsPer": 1000, "exactBonus": 500 }
}

Points are floor(total / pointsPer), a bust scores 0, and a total landing exactly on the target adds exactBonus and marks the play exact, which the leaderboard shows as a badge. Against a £1,000,000 target with the defaults above, a squad at £877,000 scores 877 and a perfect £1,000,000 scores 1,500. Leave scoring out and nothing changes.

A measure can also set its own rank bands, replacing The Limit's defaults (2%, 5%, 10%, 18%, 28% of the target):

{
  "key": "career_points",
  "tiers": [
    { "within": 0.05, "label": "Hall of Fame" },
    { "within": 0.15, "label": "Contender" },
    { "within": 0.3, "label": "Also Ran" }
  ]
}

within is a fraction of the target, ordered closest first, and at least one tier is required if you set tiers at all. The names a fan reads still come from a producer's Rank names in Studio when they set any; these labels are only what shows when that is empty.

Building it from a file or an API

You rarely start with a file in this shape. Start with what you have, a JSON file, a CSV export or an API address, and ask what is in it:

$ gamestage reference inspect ./players.json

It finds the lists of records inside, describes each field, and suggests which field is the key, which holds a name a producer reads, and which hold numbers a round could count. The measures come best first: fields on every record, named like something a game counts (a price, points, goals, minutes), and bounded. Transfer counts, selections, this week's figure and forecasts come last. The suggestion is a starting point, so pick the measures your game is about. Then build the dataset from those choices:

$ gamestage reference build ./players.json --key id --measure goals --label name --out dataset.json

Both run on your own machine and need no account. build applies the rules above for you: decimals are stored as whole numbers with a scale, a key that appears twice is refused, and a measure holding a negative number is refused by name. The CLI reference has every option.

Pushing it

$ gamestage reference push the-limit/careers.json ./careers.json

The first argument is the name a producer types in Studio, in the game's Dataset field. The second is the file on your machine.

push needs an account, as do list, status, run, download and delete below. Run gamestage login first, and the account has to be approved before it will go through. check, inspect and build run entirely on your own machine and need neither.

It validates before anything leaves, so a malformed file fails here rather than through a round refusing to provision hours later. Then it writes two objects: a pointer at the name, and an immutable copy keyed by the digest of exactly these bytes. A round records that digest, so there is always an answer to "what was this round scored on".

The output says what the push replaced: the digest the name held before, and any measure that version had and this file does not. Push a price file over a dataset of goals and assists and it says Replaces 375bdcdf...; goals and assists are no longer in it. It names measures and never a value.

Check your keys before a producer meets them:

$ gamestage reference check ./careers.json jordami01 birdla01

A key the file does not hold is left out of its group: that player is not offered, and the round's log names who went. It never counts as nothing, because a choice silently worth zero reads as a hard round rather than a broken one. If dropping it leaves a group with one player where the producer wrote several, or with none, the round is refused. This command finds those keys first, run by the person who can fix them.

Keeping it current

Run the push again. Each one is a new version, and old versions stay readable, so a round pinned to last week's values can still say what they were.

For data you push, put it on your own schedule rather than asking us for one. You know when your source updates and we do not, the credentials are yours, and a one-line command in your CI is smaller than a scheduler somebody has to operate on your behalf.

A dataset a source pulls for you keeps itself current on the source's schedule. To pull one now, for example right after you connect the source, run gamestage reference run <name>. It says how many entries landed and their digest, never a value, and a dataset can be pulled this way once every 15 minutes.

A push that would drop a measure's scoring is refused. Each push replaces the whole file, so pushing from an older copy quietly removes anything added since, including a scoring block: a live round would go back to scoring a bust as a large negative number instead of 0. push compares against what is live and refuses, naming the measure:

$ gamestage reference push wages/players.json ./players.json
This push would stop career_points scoring rounds.

Add the block back, or pass --drop-scoring if removing it is the point.

Seeing, downloading and removing it

$ gamestage reference list
$ gamestage reference status wages/players.json

list shows every dataset your workspace holds and which games read each one. status shows one in full, with the rounds built from it. Neither prints a value.

$ gamestage reference download wages/players.json

download gives you back a file you uploaded, byte for byte, to check what rounds are scored against or to edit and push again. Data Gamestage pulls from a source for you cannot be downloaded: it is licensed for scoring games, not for taking away.

$ gamestage reference delete wages/old-players.json

delete removes a dataset no game reads. It is refused while a game that is not archived reads it, because that game's next round would have nothing to be scored against. Rounds already open keep their numbers either way.

A worked example, with free data

Fantasy Premier League publishes every player with a current price, free, with no key and no registration. One request, and it is a whole dataset.

The shape below is ours, not theirs. We test what we accept and not what anybody else serves, so read the payload and check the field names before running this: it is a public API nobody promised us and it is free to move things. The part worth copying is everything after const reference =.

const bootstrap = await fetch(
  "https://fantasy.premierleague.com/api/bootstrap-static/",
).then((r) => r.json());

const reference = {
  corpus: "1",
  measures: [
    {
      key: "player_price",
      label: "Price, in tenths of a million",
      // Their own player id is the key, so nothing has to be mapped.
      //
      // `now_cost` is 155 for a player worth 15.5m, and it stays in tenths.
      // A value has to be a whole number, and a budget compared in tenths is
      // exact where one compared after dividing is a rounding argument on a
      // game you lose by going a penny over.
      values: Object.fromEntries(
        bootstrap.elements
          .filter((p) => typeof p.now_cost === "number")
          .map((p) => [p.id, p.now_cost]),
      ),
    },
  ],
};

Or, without writing any code, the same figures from the command line, with the measure named after FPL's own field, now_cost:

$ gamestage reference build https://fantasy.premierleague.com/api/bootstrap-static/ \
    --list '$.elements' --key id --measure now_cost --label web_name --out prices.json

Note the filter. A player with no price is left out rather than given a zero, and build does the same. Zero is a free pick in a game about a budget, and a fan who picks one finds a round that looks hard rather than broken. Leaving them out means the round refuses to build until somebody notices, which is the failure you want.

Then in Studio, a choice on the board carries the same id as its reference:

Slate:   Forward
Choice:  Haaland            reference 411

FPL gives every player a new id each season, so take the ids from this season's feed with reference inspect, never from last season's game.

A fan sees "Haaland". The Engine knows what he costs. Nothing in the page carries the number.

Catalogue images and video

$ gamestage reference push club-archive ./catalogue.csv --kind media
club-archive: 2 catalogued, 0 unchanged, 1 refused
  Studio Space       5d1c0b7e-…
  - goal-1998-final: The 1998 final goal (asset 2fa8e707-…)
  - kit-home: Home kit (asset e60577f9-…)

Refused
  - Row 4 (badge): Local files are not supported yet; give an https address.

--kind media puts each picture or video into your workspace's own Studio Space asset library, names it, and tags it with the catalogue's name and the row's key. A producer then picks it from the asset library in Studio, filtered by that tag, instead of uploading the file again. Gamestage keeps a record of each item beside it: its key, label, Studio asset, where it came from and who owns it.

The catalogue is a CSV with a header line. Columns can come in any order:

key,label,url,tags,rights,source_id,alt
goal-1998-final,The 1998 final goal,https://cdn.club.com/archive/goal.jpg,goal|1998|final,Club archive; club games only,dam-77,A striker scores from the edge of the box
kit-home,Home kit,https://cdn.club.com/kit/home.png,kit,Club archive,,
ColumnRequiredWhat it is
keyYesWhat a producer and a round use to name the item. Letters, digits, dots, dashes and underscores, up to 128 characters, different on every row
labelYesThe name a producer reads in Studio. Set as the asset's name
urlYesAn https:// address that serves the file itself, readable without signing in
rightsYesWho owns it and what it may be used for. Studio has no field for this, so Gamestage keeps it with the item
tagsNoExtra tags, separated by `\`. The catalogue's name and the key are always added
source_idNoThe item's id in your own asset system, kept so a later connector updates the item rather than adding it again
altNoAlt text for a screen reader. Set as the asset's alt text

Pictures can be JPEG, PNG, WebP or GIF; Studio stores them as WebP. Video can be MP4 or WebM. The largest file is 32 MB, which is Studio's own limit for an upload by hand.

Every row is checked before anything leaves your machine, then the good rows go up eight at a time. A refused row never stops the others. Each refusal names the row and what to do:

RefusalWhat to do
No key, label or urlFill in the cell
No rights givenSay who owns it and what it may be used for
A local file, such as ./photos/goal.jpgLocal files are not supported yet; give an https address
Not an https addressServe the file over https and give that address
The same key twiceGive each row its own key
The address cannot be readCheck it opens in a browser without signing in
Not a picture or videoGive the address of the file itself, not a page about it
Over 32 MBGive a smaller version

The command exits 1 when any row was refused, so a script running it notices. Fix those rows and push the same file again: a row already catalogued from the same address is left alone and uploads nothing, and a new label, alt text or tags update the item in place.

$ gamestage reference status club-archive
$ gamestage reference check club-archive goal-1998-final

status lists the items with their rights. check shows what each key became: its Studio asset, tags, rights, source id and the address it came from, and names any key the catalogue does not hold.

Media needs your own Studio Space. Everybody in a Space can see its asset library, so a workspace still in the Space every new account shares is refused before anything is uploaded: Media needs your own Studio Space; ask Monterosa to set one up. A workspace gets its own Space on its first deploy, and a customer organisation's workspaces use the organisation's Space.

Not yet: uploading files from your machine, and Studio's picture fields opening the asset library filtered by a catalogue's tag. Until that lands, a producer finds the items in the Space's asset library by searching for the tag.

Where the data comes from

Decide this before you design the game. Your own data costs nothing to license. A provider's can take months.

Where the data comes from covers the options, the sources we offer, who makes each request, and what happens when a step fails.

What never goes in the file

Only what a choice is worth. Not a round, not a board, not the words a fan reads: those are the producer's and they live in Studio, so they can be changed without a deploy.

And nothing about a person beyond what the game needs. A dataset is copied onto our infrastructure and kept, so a file holding more than the game reads is a liability with no upside.