Data sources

Dataset is the file of hidden values a round is scored against. This chapter covers where those values come from: how a source is turned into that file, who makes the request, and what happens when a step fails.

The pipeline

  A SOURCE            a feed you call, a feed we call, or a file you write
     |
     v
  TRANSFORM           one function per source. It maps the source's fields
                      onto our measure keys and normalises the units.
     |
     v
  A DATASET           the same shape, whatever the source was
     |
     v
  PUBLISHED           you push it, into a private prefix no browser reads
     |
     v
  A PRODUCER'S ROUND  the words a fan sees, each choice carrying a key
     |
     v
  JOINED              when the round is built, on that key
     |
     v
  THE ENGINE          holds the values. The browser gets labels only.

Every source ends in the same shape. Everything after the transform is the same code, whatever the values came from.

Once a source is connected and pulling, `gamestage round seed` builds candidate rounds straight from it and creates them in Studio, rather than you typing a board by hand.

Three parts

A transform is one function per source. It takes a payload and returns a dataset. It does no fetching. supplier-fpl.ts and supplier-nba.ts in packages/schema are the two we have.

A measure key is our name for a quantity, such as player_price or career_points. The transform maps the source's field onto it and normalises the units. Fantasy Premier League prices stay in tenths of a million, because a dataset holds whole numbers and dividing here would round a budget a fan can lose by a penny.

An entry key is what a producer types against each choice in Studio, in the entry_key field. It joins a label a fan reads to a value they never see. Use the source's own id, not a name. Names repeat, differ between feeds, and change.

Measure keys are not a shared vocabulary

There is no canonical list of measure keys. Each transform holds its own mapping table and each source declares what it produces, but nothing checks a key against a list. Two sources can use one key for different quantities.

This is safe while a workspace's dataset comes from one source, because a round names the measure it is counted on. It stops being safe when one game draws on two sources. Tell us before you build that, not after.

Four sources

The register is an array in packages/schema/src/suppliers.ts. Adding a source is a code change and a release.

The table says who may use a source, not what runs today. gamestage sources prints the same list with its state for your workspace: connected, available, available once you add a key, or not available with the reason.

Your own data works now. Fantasy Premier League runs now: a scheduled pull fetches every Premier League player's price, season points and points per game once a day, with each player's name, club and position, so a producer picks players by name rather than by id. The Points sample game plays on it. ESPN runs once you switch it on: a scheduled pull fetches season statistics for every player in the league you choose, twice a day. Sportmonks runs once you connect your own key: a scheduled pull fetches season statistics for whichever competitions your plan covers, with each player's name, team and position.

What a creator sees before connecting:

• Your own data: a file you push with gamestage reference push from the machine you signed in on.
• Fantasy Premier League: free, with no contract behind it. Prices move weekly, and a round in play keeps the ones it opened with.
• ESPN: free, with no contract behind it. Player season statistics for the NBA, WNBA, NFL, MLB and NHL. It carries no Premier League or other association football.
• Sportmonks football: your own Sportmonks plan and key. Their lowest, no-cost plan covers a few smaller leagues; the Premier League needs a paid one. Check your plan's terms cover this use before a game with a prize.

A source you cannot use is still listed, with the reason. Hiding it would mean you never learn it exists.

Switch on ESPN

ESPN needs no key. Switch it on for your workspace, one league at a time:

gamestage sources connect espn --league nba --measures points,rebounds,assists

Or open Datasets in Stage, find ESPN under Sources and choose Switch on: pick a league, tick the measures you want, and name the dataset.

gamestage sources status espn --league nba lists the measure keys a league offers. gamestage sources status espn lists what you have switched on and how often each is fetched.

The first fetch comes within the hour. After that it is fetched every 12 hours, and you cannot ask for more often: if a game needs fresher numbers, ask Monterosa, who can set a shorter interval for one dataset.

gamestage sources disconnect espn --name espn-nba stops the fetching. The data already fetched stays, so a game playing on it keeps working.

ESPN is unofficial. Read what that means for you before building on it: use it to build and test, and do not launch a public game on it without the data owner's permission.

A switch-on the CLI refuses says why: a league ESPN does not cover, a measure the league does not offer, or a dataset name already used by another source.

Connect your own key

Sportmonks needs your own key, because the arrangement is yours: your plan, your terms, your limits. Connect it without it passing through a chat or a terminal:

gamestage sources connect sportmonks

That prints a link to the Data area in Stage, where you paste the key into a masked field. It is checked with Sportmonks and stored in AWS Secrets Manager, never shown again. gamestage sources status sportmonks says whether it is connected and what your plan covers, and gamestage sources disconnect sportmonks removes it; the datasets you already have stay.

Once it is connected, we call Sportmonks on a schedule using your key: your plan's limits are the ones spent, and your terms are what bind the call. What is ours is the machine the schedule runs on. The CLI reference has the full command options, including reading the key from your own 1Password at your own terminal with --from-op, which an agent must never run for you.

Standing

Standing describes the arrangement behind a source, not the quality of its data.

• licensed: a contract exists. Suitable for a game with something at stake.
• unofficial: no contract, no support, and it can change without notice. Suitable for a prototype.
• creator: your own data, or your own licensed feed. Your rights, your call.

Fantasy Premier League looks licensed and is not. Nothing is promised about that endpoint, and a fantasy price is a number from their game rather than a competition record. It stays unofficial.

What an unofficial source means for you

Every unofficial source, Fantasy Premier League today, carries this, and gamestage sources, verify, deploy and promote all say it:

It is a recommendation, so nothing refuses a game over it. It is said at each of those steps because each is a step closer to fans.

Bring your own licensed feed

The creator standing covers a feed you license as well as a file you write. The arrangement is yours: your contract, your terms, your key. Gamestage is not a party to it, which is the property a rights-holder with a data licence needs. The values are fetched server-side and held by the Engine, and no response a browser receives carries them, so the game never exposes what the licence protects.

What that means today: you transform the feed into a dataset and push it with gamestage reference push, on whatever schedule your source updates.

Open and public data sources sit under unofficial, each with its own licence to respect. They are right for a prototype or a taster. A game with something at stake needs a source with a contract behind it.

Who makes the request

Each source also records who makes the call.

A source we call is bound by terms that bind us, and if they refuse us it affects every game on the platform. A source you call is bound by your terms, runs from your machine, and cannot affect anybody else.

Sportmonks is a third shape: we make the call, on a schedule, but with your key. Your plan's terms and limits are the ones spent; what is ours is the machine the schedule runs on. Disconnecting your key stops the pull without affecting anybody else's game.

This is why changing source later is a row in the register rather than a rewrite, and why pushing your own file is the lowest-risk option.

Choosing a source

Decide this before you design the game.

Your own data. A club knows its squad, a broadcaster knows its schedule, a publisher owns its archive. No licence and no expiry.

A feed you already pay for. Most broadcasters hold a match-data licence, so the work is a transform of data you are entitled to. Check that your licence covers this use.

A provider you would need to license. A real cost and a real lead time. Find out now, not after the game is built.

Anything scraped. We cannot help, and you should assume a game built on it cannot carry a prize.

Player wages, as an example

Wages look like an easy game and are a hard case. They are usually not in the match-data feeds a broadcaster licenses, and the published figures are estimates compiled by third parties under their own terms. Two consequences:

• A prize decided by an estimate can be disputed. If a fan can argue with the number, they can argue with the result.
• A licence that forbids redistribution may forbid this use, even though the values never reach a fan.

Design for the data you have. "Whose career total is higher" needs no feed. "Whose wage is higher" needs a licence and a refresh.

Errors

Each failure below has a defined behaviour.

• A dataset names a source that does not exist. Refused, and the source is named. A run that silently did four of five looks like a run that had four to do.
• A workspace may not use the source. Refused before the request is made, so it costs the source no traffic. The same check decides what Stage lists and what a write allows, so the two cannot disagree.
• A fetch fails. No data is written; the pointer is marked as failing. What is published stays published, and a game being played carries on with the values it has.
• An entry has no value. It is named and left out, never set to zero. A player worth nothing is a free pick in a game about a budget.
• The data has not changed. The digest-keyed copy is not rewritten; the pointer's timestamp moves so the next run knows this dataset was checked, rather than looking permanently overdue.
• What we fetched is not a valid dataset. It is validated before it is compared, so a run cannot report "unchanged" about bytes it never checked.
• Two measures share a key. Refused. A round names the measure it is counted on, so two measures under one key cannot say what scored it.
• A choice has a key the file does not hold. The whole round is refused and the missing keys are named. A choice silently worth zero looks like a hard round rather than a broken one.
• Dataset changes while a round is live. The round is left alone and the reason is logged against the game once. Fans keep the values they started on, and a deploy or a refresh applies the change.
• The digest-keyed copy is immutable by convention. Nothing in the publish path rewrites one, and the copy is written before the pointer, so a name cannot resolve to bytes nobody can fetch. Whether the bucket would refuse an overwrite is a storage setting rather than a promise made here.

Adding a source we do not have

There are two options, and neither is a screen.

Write the file yourself and push it. This is the creator row. It needs nothing from us and is on every plan.

Or we write a transform: a row in the register and a function that returns a dataset. That is a code change and a release. There is no self-serve path onto a new feed. Fantasy Premier League runs for every workspace because the relationship with the Premier League is ours; a new source on that footing is the same, a code change and a release. A source that runs on your own key, such as Sportmonks, is different: you connect it yourself with gamestage sources connect, and the scheduled pull starts once it is connected. Ask us before building on a source neither route covers.

Managing what you have

Pushing a dataset is CLI-only, with gamestage reference push. The Data screen in Stage lists the sources available to your workspace and, in its own table, the datasets a workspace already holds: name, source, which games use it, its measures, and when it was last checked. Opening a row shows more detail. It never shows a value.

Signing in is a device grant confirmed in a browser, and the session stays on that machine, so a push from CI runs with a session placed there rather than by signing in. Run the push on your own schedule: you know when your source updates and we do not, the credentials are yours, and one command in your CI is smaller than a scheduler somebody has to operate for you.