{"doc":"data","title":"Dataset","markdown":"# Dataset\n\nA game's answers are data a fan must not be able to see. This is where they\nlive, how you get them there, and what it costs you to keep them current.\n\n## The shape, in one picture\n\nThree things, and only the middle one is secret.\n\n```\n  Studio            what a fan reads:   \"Michael Jordan\"\n     |              the board, the labels, every word\n     |\n     |  entry_key   the join:            jordami01\n     v\n  Dataset    what a fan must not read:  32292\n     |              a private file, yours, never served to a browser\n     v\n  The Engine        holds what provisioning joined,\n                    and never lets a value out\n```\n\nThe producer types a name. You supply what the name is worth. Provisioning\nputs them together at the moment a round is built, the Engine holds the\nresult, and the number never reaches the page.\n\n## Why it is not a live feed\n\nA round must not change while somebody is playing it. If a value moves between\none fan's pick and another's, the two of them played different games, the\nleaderboard compares values that were never comparable, and a dispute cannot be\nsettled.\n\nSo a dataset is **fresh when a round is provisioned and frozen while it is\nplayed**. Push whenever you like. A round already open keeps the values it\nopened with; the next one picks up your change. That is a property of the\nEngine, not something you have to arrange.\n\n## The file\n\nJSON, one document, one or more measures. A measure is a thing you can count a\nchoice on.\n\n```json\n{\n  \"corpus\": \"1\",\n  \"measures\": [\n    {\n      \"key\": \"career_points\",\n      \"label\": \"Career Points\",\n      \"values\": { \"jordami01\": 32292, \"birdla01\": 21791 }\n    }\n  ]\n}\n```\n\n`values` maps a key to a number. The key is the join, and it is the only part\nthat has to agree with anything: whatever a producer types as a choice's\nreference in Studio must appear here, spelled the same way.\n\nUse identifiers your source already has. Inventing your own means maintaining a\nsecond mapping for ever.\n\nWhat the file has to satisfy, all of it checked before anything is uploaded:\n\n- **`corpus` is the string `\"1\"`.** It is the document's schema version rather\n  than yours. The file is yours, hosted by you and changed without a deploy, so\n  the one thing we rely on is that it says what it is.\n- **At least one measure, and no two sharing a `key`.** A round names the\n  measure it is counted on, so two measures under one key is a round that\n  cannot say what scored it.\n- **Every value is a whole number, zero or above.** Not a decimal, not\n  negative, not a string. A total that arrives as `21.5` is refused.\n- **`label` is optional.** It exists so a refusal can say the dataset holds\n  Career Points and Career Assists rather than `pts` and `ast`.\n\nA dataset with one measure needs nothing more. One with several needs the\nround to name the one it counts on, which a producer sets on the round's\n**Measure** field.\n\nDataset is not a round's **Content source**. A Content source is a URL\nto the round JSON itself, including its required shape and access rules, in\n[Game formats](/docs/game-formats#what-content-source-takes).\n\n## Scoring a push round by how close it came\n\nBy default, a push round scores its raw total and a bust scores the overshoot\nas a negative number: The Limit depends on this. Add a `scoring` block to a\nmeasure to score by proximity instead:\n\n```json\n{\n  \"key\": \"career_points\",\n  \"label\": \"Career Points\",\n  \"values\": { \"jordami01\": 32292, \"birdla01\": 21791 },\n  \"scoring\": { \"mode\": \"proximity\", \"pointsPer\": 1000, \"exactBonus\": 500 }\n}\n```\n\nPoints are `floor(total / pointsPer)`, a bust scores 0, and a total landing\nexactly on the target adds `exactBonus` and marks the play exact, which the\nleaderboard shows as a badge. Against a £1,000,000 target with the defaults\nabove, a squad at £877,000 scores 877 and a perfect £1,000,000 scores 1,500.\nLeave `scoring` out and nothing changes.\n\nA measure can also set its own rank bands, replacing The Limit's defaults\n(2%, 5%, 10%, 18%, 28% of the target):\n\n```json\n{\n  \"key\": \"career_points\",\n  \"tiers\": [\n    { \"within\": 0.05, \"label\": \"Hall of Fame\" },\n    { \"within\": 0.15, \"label\": \"Contender\" },\n    { \"within\": 0.3, \"label\": \"Also Ran\" }\n  ]\n}\n```\n\n`within` is a fraction of the target, ordered closest first, and at least one\ntier is required if you set `tiers` at all. The names a fan reads still come\nfrom a producer's Rank names in Studio when they set any; these labels are\nonly what shows when that is empty.\n\n## Pushing it\n\n```\n$ gamestage reference push the-limit/careers.json ./careers.json\n```\n\nThe first argument is the name a producer types in Studio, on the game's\n**Reference data** field: that is the name on screen, though this book calls\nthe same thing a dataset. The second is the file on your machine.\n\n`push` is the one command in this chapter that needs an account. Run\n`gamestage login` first, and the account has to be approved before it will go\nthrough. `check` below runs entirely on your own machine and needs neither.\n\nIt validates before anything leaves, so a malformed file fails here rather than\nthrough a round refusing to provision hours later. Then it writes two objects: a\npointer at the name, and an immutable copy keyed by the digest of exactly these\nbytes. A round records that digest, so there is always an answer to \"what was\nthis round scored on\".\n\nCheck your keys before a producer meets them:\n\n```\n$ gamestage reference check ./careers.json jordami01 birdla01\n```\n\nA key the file does not hold refuses the whole round rather than counting as\nnothing. That is deliberate, because a choice silently worth zero reads as a\nhard round rather than a broken one, and nothing downstream would notice. This\ncommand is the same check, run by the person who can fix it.\n\n## Keeping it current\n\nRun the push again. Each one is a new version, and old versions stay readable,\nso a round pinned to last week's values can still say what they were.\n\nPut it on your own schedule rather than asking us for one. You know when your\nsource updates and we do not, the credentials are yours, and a one-line command\nin your CI is smaller than a scheduler somebody has to operate on your behalf.\n\n**A push that would drop a measure's `scoring` is refused.** Each push\nreplaces the whole file, so pushing from an older copy quietly removes\nanything added since, including a `scoring` block: a live round would go back\nto scoring a bust as a large negative number instead of 0. `push` compares\nagainst what is live and refuses, naming the measure:\n\n```\n$ gamestage reference push wages/players.json ./players.json\nThis push would stop career_points scoring rounds.\n```\n\nAdd the block back, or pass `--drop-scoring` if removing it is the point.\n\n## A worked example, with free data\n\nFantasy Premier League publishes every player with a current price, free, with\nno key and no registration. One request, and it is a whole dataset.\n\n**The shape below is ours, not theirs.** We test what we accept and not what\nanybody else serves, so read the payload and check the field names before\nrunning this: it is a public API nobody promised us and it is free to move\nthings. The part worth copying is everything after `const reference =`.\n\n```js\nconst bootstrap = await fetch(\n  \"https://fantasy.premierleague.com/api/bootstrap-static/\",\n).then((r) => r.json());\n\nconst reference = {\n  corpus: \"1\",\n  measures: [\n    {\n      key: \"player_price\",\n      label: \"Price, in tenths of a million\",\n      // Their own player id is the key, so nothing has to be mapped.\n      //\n      // `now_cost` is 155 for a player worth 15.5m, and it stays in tenths.\n      // A value has to be a whole number, and a budget compared in tenths is\n      // exact where one compared after dividing is a rounding argument on a\n      // game you lose by going a penny over.\n      values: Object.fromEntries(\n        bootstrap.elements\n          .filter((p) => typeof p.now_cost === \"number\")\n          .map((p) => [p.id, p.now_cost]),\n      ),\n    },\n  ],\n};\n```\n\n**Note the filter.** A player with no price is left out rather than given a\nzero. Zero is a free pick in a game about a budget, and a fan who picks one\nfinds a round that looks hard rather than broken. Leaving them out means the\nround refuses to build until somebody notices, which is the failure you want.\n\nThen in Studio, a choice on the board carries the same id as its reference:\n\n```\nSlate:   Forward\nChoice:  Haaland            reference 351\n```\n\nA fan sees \"Haaland\". The Engine knows what he costs. Nothing in the page\ncarries the number.\n\n## Where the data comes from\n\nDecide this before you design the game. Your own data costs nothing to license.\nA provider's can take months.\n\n[Where the data comes from](/docs/data-sources) covers the options, the sources\nwe offer, who makes each request, and what happens when a step fails.\n\n## What never goes in the file\n\nOnly what a choice is worth. Not a round, not a board, not the words a fan\nreads: those are the producer's and they live in Studio, so they can be changed\nwithout a deploy.\n\nAnd nothing about a person beyond what the game needs. Dataset is copied\nonto our infrastructure and kept, so a file holding more than the game reads is\na liability with no upside.\n"}