# Dataset A game's answers are data a fan must not be able to see. This is where they live, how you get them there, and what it costs you to keep them current. ## The shape, in one picture Three things, and only the middle one is secret. ``` Studio what a fan reads: "Michael Jordan" | the board, the labels, every word | | entry_key the join: jordami01 v Dataset what a fan must not read: 32292 | a private file, yours, never served to a browser v The Engine holds what provisioning joined, and never lets a value out ``` The producer types a name. You supply what the name is worth. Provisioning puts them together at the moment a round is built, the Engine holds the result, and the number never reaches the page. ## Why it is not a live feed A round must not change while somebody is playing it. If a value moves between one fan's pick and another's, the two of them played different games, the leaderboard compares values that were never comparable, and a dispute cannot be settled. So a dataset is **fresh when a round is provisioned and frozen while it is played**. Push whenever you like. A round already open keeps the values it opened with; the next one picks up your change. That is a property of the Engine, not something you have to arrange. ## The file JSON, one document, one or more measures. A measure is a thing you can count a choice on. ```json { "corpus": "1", "measures": [ { "key": "career_points", "label": "Career Points", "values": { "jordami01": 32292, "birdla01": 21791 } } ] } ``` `values` maps a key to a number. The key is the join, and it is the only part that has to agree with anything: whatever a producer types as a choice's reference in Studio must appear here, spelled the same way. Use identifiers your source already has. Inventing your own means maintaining a second mapping for ever. What the file has to satisfy, all of it checked before anything is uploaded: - **`corpus` is the string `"1"`.** It is the document's schema version rather than yours. The file is yours, hosted by you and changed without a deploy, so the one thing we rely on is that it says what it is. - **At least one measure, and no two sharing a `key`.** A round names the measure it is counted on, so two measures under one key is a round that cannot say what scored it. - **Every value is a whole number, zero or above.** Not a decimal, not negative, not a string. A total that arrives as `21.5` is refused. - **`label` is optional.** It exists so a refusal can say the dataset holds Career Points and Career Assists rather than `pts` and `ast`. A dataset with one measure needs nothing more. One with several needs the round to name the one it counts on, which a producer sets on the round's **Measure** field. Dataset is not a round's **Content source**. A Content source is a URL to the round JSON itself, including its required shape and access rules, in [Game formats](/docs/game-formats#what-content-source-takes). ## Scoring a push round by how close it came By default, a push round scores its raw total and a bust scores the overshoot as a negative number: The Limit depends on this. Add a `scoring` block to a measure to score by proximity instead: ```json { "key": "career_points", "label": "Career Points", "values": { "jordami01": 32292, "birdla01": 21791 }, "scoring": { "mode": "proximity", "pointsPer": 1000, "exactBonus": 500 } } ``` Points are `floor(total / pointsPer)`, a bust scores 0, and a total landing exactly on the target adds `exactBonus` and marks the play exact, which the leaderboard shows as a badge. Against a £1,000,000 target with the defaults above, a squad at £877,000 scores 877 and a perfect £1,000,000 scores 1,500. Leave `scoring` out and nothing changes. A measure can also set its own rank bands, replacing The Limit's defaults (2%, 5%, 10%, 18%, 28% of the target): ```json { "key": "career_points", "tiers": [ { "within": 0.05, "label": "Hall of Fame" }, { "within": 0.15, "label": "Contender" }, { "within": 0.3, "label": "Also Ran" } ] } ``` `within` is a fraction of the target, ordered closest first, and at least one tier is required if you set `tiers` at all. The names a fan reads still come from a producer's Rank names in Studio when they set any; these labels are only what shows when that is empty. ## Pushing it ``` $ gamestage reference push the-limit/careers.json ./careers.json ``` The first argument is the name a producer types in Studio, on the game's **Reference data** field: that is the name on screen, though this book calls the same thing a dataset. The second is the file on your machine. `push` is the one command in this chapter that needs an account. Run `gamestage login` first, and the account has to be approved before it will go through. `check` below runs entirely on your own machine and needs neither. It validates before anything leaves, so a malformed file fails here rather than through a round refusing to provision hours later. Then it writes two objects: a pointer at the name, and an immutable copy keyed by the digest of exactly these bytes. A round records that digest, so there is always an answer to "what was this round scored on". Check your keys before a producer meets them: ``` $ gamestage reference check ./careers.json jordami01 birdla01 ``` A key the file does not hold refuses the whole round rather than counting as nothing. That is deliberate, because a choice silently worth zero reads as a hard round rather than a broken one, and nothing downstream would notice. This command is the same check, run by the person who can fix it. ## Keeping it current Run the push again. Each one is a new version, and old versions stay readable, so a round pinned to last week's values can still say what they were. Put it on your own schedule rather than asking us for one. You know when your source updates and we do not, the credentials are yours, and a one-line command in your CI is smaller than a scheduler somebody has to operate on your behalf. **A push that would drop a measure's `scoring` is refused.** Each push replaces the whole file, so pushing from an older copy quietly removes anything added since, including a `scoring` block: a live round would go back to scoring a bust as a large negative number instead of 0. `push` compares against what is live and refuses, naming the measure: ``` $ gamestage reference push wages/players.json ./players.json This push would stop career_points scoring rounds. ``` Add the block back, or pass `--drop-scoring` if removing it is the point. ## A worked example, with free data Fantasy Premier League publishes every player with a current price, free, with no key and no registration. One request, and it is a whole dataset. **The shape below is ours, not theirs.** We test what we accept and not what anybody else serves, so read the payload and check the field names before running this: it is a public API nobody promised us and it is free to move things. The part worth copying is everything after `const reference =`. ```js const bootstrap = await fetch( "https://fantasy.premierleague.com/api/bootstrap-static/", ).then((r) => r.json()); const reference = { corpus: "1", measures: [ { key: "player_price", label: "Price, in tenths of a million", // Their own player id is the key, so nothing has to be mapped. // // `now_cost` is 155 for a player worth 15.5m, and it stays in tenths. // A value has to be a whole number, and a budget compared in tenths is // exact where one compared after dividing is a rounding argument on a // game you lose by going a penny over. values: Object.fromEntries( bootstrap.elements .filter((p) => typeof p.now_cost === "number") .map((p) => [p.id, p.now_cost]), ), }, ], }; ``` **Note the filter.** A player with no price is left out rather than given a zero. Zero is a free pick in a game about a budget, and a fan who picks one finds a round that looks hard rather than broken. Leaving them out means the round refuses to build until somebody notices, which is the failure you want. Then in Studio, a choice on the board carries the same id as its reference: ``` Slate: Forward Choice: Haaland reference 351 ``` A fan sees "Haaland". The Engine knows what he costs. Nothing in the page carries the number. ## Where the data comes from Decide this before you design the game. Your own data costs nothing to license. A provider's can take months. [Where the data comes from](/docs/data-sources) covers the options, the sources we offer, who makes each request, and what happens when a step fails. ## What never goes in the file Only what a choice is worth. Not a round, not a board, not the words a fan reads: those are the producer's and they live in Studio, so they can be changed without a deploy. And nothing about a person beyond what the game needs. Dataset is copied onto our infrastructure and kept, so a file holding more than the game reads is a liability with no upside.