{"doc":"data-sources","title":"Data sources","markdown":"# Data sources\n\n[Dataset](/docs/data) is the file of hidden values a round is scored\nagainst. This chapter covers where those values come from: how a source is\nturned into that file, who makes the request, and what happens when a step\nfails.\n\n## The pipeline\n\n```\n  A SOURCE            a feed you call, a feed we call, or a file you write\n     |\n     v\n  TRANSFORM           one function per source. It maps the source's fields\n                      onto our measure keys and normalises the units.\n     |\n     v\n  A DATASET           the same shape, whatever the source was\n     |\n     v\n  PUBLISHED           you push it, into a private prefix no browser reads\n     |\n     v\n  A PRODUCER'S ROUND  the words a fan sees, each choice carrying a key\n     |\n     v\n  JOINED              when the round is built, on that key\n     |\n     v\n  THE ENGINE          holds the values. The browser gets labels only.\n```\n\nEvery source ends in the same shape. Everything after the transform is the same\ncode, whatever the values came from.\n\nOnce a source is connected and pulling, [`gamestage round seed`](/docs/cli#round-seed-game-count-n)\nbuilds candidate rounds straight from it and creates them in Studio, rather\nthan you typing a board by hand.\n\n## Three parts\n\n**A transform** is one function per source. It takes a payload and returns a\ndataset. It does no fetching. `supplier-fpl.ts` and `supplier-nba.ts` in\n`packages/schema` are the two we have.\n\n**A measure key** is our name for a quantity, such as `player_price` or\n`career_points`. The transform maps the source's field onto it and normalises\nthe units. Fantasy Premier League prices stay in tenths of a million, because a\ndataset holds whole numbers and dividing here would round a budget a fan\ncan lose by a penny.\n\n**An entry key** is what a producer types against each choice in Studio, in the\n`entry_key` field. It joins a label a fan reads to a value they never see. Use\nthe source's own id, not a name. Names repeat, differ between feeds, and change.\n\n## Measure keys are not a shared vocabulary\n\nThere is no canonical list of measure keys. Each transform holds its own mapping\ntable and each source declares what it produces, but nothing checks a key\nagainst a list. Two sources can use one key for different quantities.\n\nThis is safe while a workspace's dataset comes from one source, because a\nround names the measure it is counted on. It stops being safe when one game\ndraws on two sources. Tell us before you build that, not after.\n\n## Four sources\n\nThe register is an array in `packages/schema/src/suppliers.ts`. Adding a source\nis a code change and a release.\n\n| Source | Standing | Who calls | Needs a key | Available to |\n| --- | --- | --- | --- | --- |\n| Your own data | creator | you | No | Everyone, on every plan |\n| Fantasy Premier League | unofficial | us | No | Everyone, on every plan |\n| ESPN | unofficial | us | No | Everyone who switches it on |\n| Sportmonks football | creator | us, with your key | Yes | Everyone who connects a Sportmonks key |\n\nThe table says who may use a source, not what runs today. `gamestage sources`\nprints the same list with its state for your workspace: connected, available,\navailable once you add a key, or not available with the reason.\n\nYour own data works now. Fantasy Premier League runs now: a scheduled pull\nfetches every Premier League player's price, season points and points per\ngame once a day, with each player's name, club and position, so a producer\npicks players by name rather than by id. The Points sample game plays on it.\nESPN runs once you [switch it on](#switch-on-espn): a scheduled pull fetches\nseason statistics for every player in the league you choose, twice a day.\nSportmonks runs once you [connect your own key](#connect-your-own-key): a\nscheduled pull fetches season statistics for whichever competitions your plan\ncovers, with each player's name, team and position.\n\nWhat a creator sees before connecting:\n\n- **Your own data**: a file you push with `gamestage reference push` from the\n  machine you signed in on.\n- **Fantasy Premier League**: free, with no contract behind it. Prices move\n  weekly, and a round in play keeps the ones it opened with.\n- **ESPN**: free, with no contract behind it. Player season statistics for\n  the NBA, WNBA, NFL, MLB and NHL. It carries no Premier League or other\n  association football.\n- **Sportmonks football**: your own Sportmonks plan and key. Their lowest,\n  no-cost plan covers a few smaller leagues; the Premier League needs a paid\n  one. Check your plan's terms cover this use before a game with a prize.\n\nA source you cannot use is still listed, with the reason. Hiding it would mean\nyou never learn it exists.\n\n## Switch on ESPN\n\nESPN needs no key. Switch it on for your workspace, one league at a time:\n\n```\ngamestage sources connect espn --league nba --measures points,rebounds,assists\n```\n\nOr open **Datasets** in Stage, find ESPN under Sources and choose **Switch\non**: pick a league, tick the measures you want, and name the dataset.\n\n| Option | Type | Required | Default | What it does |\n| --- | --- | --- | --- | --- |\n| `--league` | `nba`, `wnba`, `nfl`, `mlb` or `nhl` | Yes | none | The league to fetch |\n| `--measures` | comma-separated measure keys | No | a starting set for the league | The numbers a round can be scored on |\n| `--season` | year | No | the current season | Which season's statistics |\n| `--name` | dataset name | No | `espn-<league>` | What a game names in Studio to play on it |\n\n`gamestage sources status espn --league nba` lists the measure keys a league\noffers. `gamestage sources status espn` lists what you have switched on and\nhow often each is fetched.\n\nThe first fetch comes within the hour. After that it is fetched every 12\nhours, and you cannot ask for more often: if a game needs fresher numbers, ask\nMonterosa, who can set a shorter interval for one dataset.\n\n`gamestage sources disconnect espn --name espn-nba` stops the fetching. The\ndata already fetched stays, so a game playing on it keeps working.\n\nESPN is unofficial. Read [what that means for you](#what-an-unofficial-source-means-for-you)\nbefore building on it: use it to build and test, and do not launch a public\ngame on it without the data owner's permission.\n\nA switch-on the CLI refuses says why: a league ESPN does not cover, a measure\nthe league does not offer, or a dataset name already used by another source.\n\n## Connect your own key\n\nSportmonks needs your own key, because the arrangement is yours: your plan,\nyour terms, your limits. Connect it without it passing through a chat or a\nterminal:\n\n```\ngamestage sources connect sportmonks\n```\n\nThat prints a link to the Data area in Stage, where you paste the key into a\nmasked field. It is checked with Sportmonks and stored in AWS Secrets Manager,\nnever shown again. `gamestage sources status sportmonks` says whether it is\nconnected and what your plan covers, and `gamestage sources disconnect\nsportmonks` removes it; the datasets you already have stay.\n\nOnce it is connected, we call Sportmonks on a schedule using your key: your\nplan's limits are the ones spent, and your terms are what bind the call. What\nis ours is the machine the schedule runs on. [The CLI reference](/docs/cli) has\nthe full command options, including reading the key from your own 1Password at\nyour own terminal with `--from-op`, which an agent must never run for you.\n\n## Standing\n\nStanding describes the arrangement behind a source, not the quality of its\ndata.\n\n- **`licensed`**: a contract exists. Suitable for a game with something at stake.\n- **`unofficial`**: no contract, no support, and it can change without notice.\n  Suitable for a prototype.\n- **`creator`**: your own data, or your own licensed feed. Your rights, your\n  call.\n\nFantasy Premier League looks licensed and is not. Nothing is promised about that\nendpoint, and a fantasy price is a number from their game rather than a\ncompetition record. It stays `unofficial`.\n\n### What an unofficial source means for you\n\nEvery unofficial source, Fantasy Premier League today, carries this, and\n`gamestage sources`, `verify`, `deploy` and `promote` all say it:\n\n> Requests to this source are made on behalf of you, the game builder. You are responsible for the data you consume, and for securing the legal rights and licence to use it. We recommend using it for testing only, and not launching with it unless you have explicit permission from the data owner.\n\nIt is a recommendation, so nothing refuses a game over it. It is said at each\nof those steps because each is a step closer to fans.\n\n## Bring your own licensed feed\n\nThe `creator` standing covers a feed you license as well as a file you write.\nThe arrangement is yours: your contract, your terms, your key. Gamestage is not\na party to it, which is the property a rights-holder with a data licence needs.\nThe values are fetched server-side and held by the Engine, and no response a\nbrowser receives carries them, so the game never exposes what the licence\nprotects.\n\nWhat that means today: you transform the feed into a dataset and push it with\n`gamestage reference push`, on whatever schedule your source updates.\n\nOpen and public data sources sit under `unofficial`, each with its own licence\nto respect. They are right for a prototype or a taster. A game with something\nat stake needs a source with a contract behind it.\n\n## Who makes the request\n\nEach source also records who makes the call.\n\nA source we call is bound by terms that bind us, and if they refuse us it\naffects every game on the platform. A source you call is bound by your terms,\nruns from your machine, and cannot affect anybody else.\n\nSportmonks is a third shape: we make the call, on a schedule, but with your\nkey. Your plan's terms and limits are the ones spent; what is ours is the\nmachine the schedule runs on. Disconnecting your key stops the pull without\naffecting anybody else's game.\n\nThis is why changing source later is a row in the register rather than a\nrewrite, and why pushing your own file is the lowest-risk option.\n\n## Choosing a source\n\nDecide this before you design the game.\n\n**Your own data.** A club knows its squad, a broadcaster knows its schedule, a\npublisher owns its archive. No licence and no expiry.\n\n**A feed you already pay for.** Most broadcasters hold a match-data licence, so\nthe work is a transform of data you are entitled to. Check that your licence\ncovers this use.\n\n**A provider you would need to license.** A real cost and a real lead time.\nFind out now, not after the game is built.\n\n**Anything scraped.** We cannot help, and you should assume a game built on it\ncannot carry a prize.\n\n### Player wages, as an example\n\nWages look like an easy game and are a hard case. They are usually not in the\nmatch-data feeds a broadcaster licenses, and the published figures are estimates\ncompiled by third parties under their own terms. Two consequences:\n\n- A prize decided by an estimate can be disputed. If a fan can argue with the\n  number, they can argue with the result.\n- A licence that forbids redistribution may forbid this use, even though the\n  values never reach a fan.\n\nDesign for the data you have. \"Whose career total is higher\" needs no feed.\n\"Whose wage is higher\" needs a licence and a refresh.\n\n## Errors\n\nEach failure below has a defined behaviour.\n\n- **A dataset names a source that does not exist.** Refused, and the source is\n  named. A run that silently did four of five looks like a run that had four to\n  do.\n- **A workspace may not use the source.** Refused before the request is made, so\n  it costs the source no traffic. The same check decides what Stage lists and\n  what a write allows, so the two cannot disagree.\n- **A fetch fails.** No data is written; the pointer is marked as failing.\n  What is published stays published, and a game being played carries on with\n  the values it has.\n- **An entry has no value.** It is named and left out, never set to zero. A\n  player worth nothing is a free pick in a game about a budget.\n- **The data has not changed.** The digest-keyed copy is not rewritten; the\n  pointer's timestamp moves so the next run knows this dataset was checked,\n  rather than looking permanently overdue.\n- **What we fetched is not a valid dataset.** It is validated before it is\n  compared, so a run cannot report \"unchanged\" about bytes it never checked.\n- **Two measures share a key.** Refused. A round names the measure it is counted\n  on, so two measures under one key cannot say what scored it.\n- **A choice has a key the file does not hold.** The whole round is refused and\n  the missing keys are named. A choice silently worth zero looks like a hard\n  round rather than a broken one.\n- **Dataset changes while a round is live.** The round is left alone and\n  the reason is logged against the game once. Fans keep the values they\n  started on, and a deploy or a refresh applies the change.\n- **The digest-keyed copy is immutable by convention.** Nothing in the publish\n  path rewrites one, and the copy is written before the pointer, so a name cannot\n  resolve to bytes nobody can fetch. Whether the bucket would refuse an overwrite\n  is a storage setting rather than a promise made here.\n\n## Adding a source we do not have\n\nThere are two options, and neither is a screen.\n\nWrite the file yourself and push it. This is the `creator` row. It needs nothing\nfrom us and is on every plan.\n\nOr we write a transform: a row in the register and a function that returns a\ndataset. That is a code change and a release. There is no self-serve path\nonto a new feed. Fantasy Premier League runs for every workspace because the\nrelationship with the Premier League is ours; a new source on that footing is\nthe same, a code change and a release. A source that runs on your own key,\nsuch as Sportmonks, is different: you connect it yourself with `gamestage\nsources connect`, and the scheduled pull starts once it is connected. Ask us\nbefore building on a source neither route covers.\n\n## Managing what you have\n\nPushing a dataset is CLI-only, with `gamestage reference push`. The Data\nscreen in Stage lists the sources available to your workspace and, in its own\ntable, the datasets a workspace already holds: name, source, which games use\nit, its measures, and when it was last checked. Opening a row shows more\ndetail. It never shows a value.\n\nSigning in is a device grant confirmed in a browser, and the session stays on\nthat machine, so a push from CI runs with a session placed there rather than\nby signing in. Run the push on your own schedule: you know when your source\nupdates and we do not, the credentials are yours, and one command in your CI\nis smaller than a scheduler somebody has to operate for you.\n"}