An Experimental Turn: Measuring Predictability, Not Value

Every model up to this point answers a version of the same question: how much is this action worth? I wanted to try answering a genuinely different one: how expected was it, given the patterns present in real match play — not value, but predictability.

Why sequence, not space#

Football is played as a sequence of decisions, each one shaped by what just happened — structurally the same as a sentence, where each word depends on the words before it. That observation is the whole idea: treat a possession as a sequence of tokens (what kind of action, in what part of the pitch), and fit a model of how often one token typically follows another. Mechanically, this reuses the exact transition-probability machinery already built for xT — same tool, aimed at a different question.

The per-action metric borrows the actual term from information theory for this exact quantity: surprisal — the negative log-probability of an observed event, derived from three simple properties any measure of "how surprising is this" should have (rarer things are more surprising, certain things are unsurprising, and the surprise of two independent events should add together). Those three properties turn out to pin down one specific formula, uniquely: -log2(P(event)). The per-possession aggregate is perplexity, the standard measure of how well a language model predicts real text, here measuring how well a model predicts real football.

Proving it actually measures something#

Before trusting any of this, the model needed a sanity check: a model that's learned nothing real should be equally "surprised" by real football and by the same actions in randomly shuffled order. It wasn't — real possessions consistently scored meaningfully lower average surprisal than shuffled versions of themselves, confirmation the model had learned genuine structure in how football is actually played.

Three bugs, in sequence, each teaching something different#

Building this exposed a chain of increasingly subtle sparsity problems, each one worth describing on its own:

First, the model's smoothing (necessary so that never-before-seen action sequences don't produce an undefined score) meant that any transition that had literally never been observed in training got tied to every other never-observed transition from the same starting point — regardless of how unusual the real outcome actually was. Fixed by only ranking transitions that had been observed at least once.

Second, and much subtler: contexts with few total observations — goalkeeper passes, by simple volume — produced artificially inflated surprisal scores, because a small denominator let the smoothing term dominate the real data. This surfaced concretely as several real, recognisable goalkeepers unexpectedly topping the "most surprising actions" list, for reasons that had nothing to do with tactical unpredictability. Fixed by requiring a minimum number of real observations behind any context before trusting its score.

Third, and still an open, documented limitation: among genuinely rare transitions from an identical, common starting context, several different real outcomes can still tie exactly, since the current formula only depends on how often something happened, not on how far the specific outcome deviated from what's typical there. A complete version would need a secondary, geometric measure of deviation layered on top.

What it actually found#

Cross-referencing each action's surprisal against its xT value revealed a specific shape, not a simple straight-line relationship: predictable actions cluster tightly around zero added value — both low-risk and low-reward. Unpredictable actions spread much further in both directions. Predictable, common actions are consistently moderate; the genuinely rare ones are a mixed bag — some are the moment of individual brilliance that breaks a defence, some are pure overambition that gives the ball away.

That's a more precise and more useful finding than "surprising actions are more valuable" on its own, which the data only weakly supports — the average value barely moves with predictability (correlation around 0.20). What actually moves is the spread. That distinction matters practically: it reframes the question from "should players just be more unpredictable" to "when is trading a safer floor for a higher ceiling actually the right call" — a much closer match to a real coaching decision than a blanket recommendation would be.

I tested this pattern properly before trusting it, rather than presenting a single scatter plot as proof: it survived being checked separately within each event type (passes and carries independently, not just pooled together), and it survived a formal statistical test against the possibility that it was just an artifact of having less data at higher surprisal levels.

Objections and limitations#

The funnel result above survived three checks before I was willing to trust it: a shuffled-sequence baseline, stratification by event type, and a formal bootstrap test against the possibility that the variance increase was a sample-size artifact rather than a real effect. What follows is what that scrutiny did not cover — stated precisely, not softened into narrative.

1. Causal direction is unresolved. The tests above rule out two specific confounds (event-type mixing, sample-size noise) but not a third: a lurking "phase-of-play instability" variable could independently drive both higher surprisal and higher value variance, with no causal link between the two at all. This isn't testable with event data alone — it would require a proxy for defensive organisation or possession chaos that isn't present in the schema. The defensible claim is that surprisal and value variance are robustly associated; "predictability reduces variance" is a causal claim this data cannot support.

2. Surprisal is corpus-relative, not an intrinsic property of the action. A score of -log2 P(token | context) is only ever "how rare was this transition, in this specific pooled training set" — five leagues, 2018–2024. The identical real-world action, scored against a model trained on a different competition, era, or level, would produce a different number. Any claim phrased as if surprisal measures something intrinsic to the action, independent of the training distribution, overstates what's actually computed.

3. The model is order-1 and therefore context-blind beyond one step. P(next | previous) has no mechanism to condition on scoreline, match minute, or opponent identity. A give-and-go in minute 3 of a 0–0 and the identical zone-transition in minute 93 of a game chasing an equaliser score identically, because the model literally cannot distinguish them. Extending to order-2 was considered and rejected: the token space (event_type × zone) already produces a vocabulary in the hundreds; an order-2 model's state space grows combinatorially, and the sparsity problems already documented above would compound sharply without a more sophisticated smoothing scheme (Kneser–Ney-style backoff) than Laplace smoothing supports.

4. Errors compound across three sequentially dependent models. The xT-delta cross-reference isn't surprisal against ground truth — it's surprisal against xT, which is itself computed from the xG model's shot-value estimates. Each of the three carries its own documented limitations (xG's incomplete feature set, xT's turnover_value = 0 simplification, the grid-resolution choice needed to make xT and the sequence model spatially comparable at all). The funnel finding's reliability is bounded by the weakest link in that chain, not by the surprisal model in isolation.

5. Statistical significance is not practical significance. The bootstrap test returned p < 0.0005 at every surprisal level beyond the reference bin. With a sample in the hundreds of thousands, this threshold does not require a large effect to clear — it confirms the pattern is not noise, and says nothing about whether the magnitude is large enough to change an actual coaching decision. That question remains open and untested.

6. Scope is narrower than the framing sometimes implies. Every model here is trained on five professional women's domestic leagues over a six-year span, under an untested assumption that tactical structure is comparable enough across them to pool. No international football, no men's football, no lower divisions are represented. Every finding is a claim about this specific population, not football unqualified.

7. No out-of-time validation exists. Every train/test split — for xG, xT, and the sequence model alike — is a random match-level split drawn from the same pool of seasons. None of it tests generalisation to a genuinely future season the model has never seen in any form, which matters if tactical trends shift meaningfully year to year, as they plausibly do over a span this long.

8. Several parameters were chosen adaptively, not pre-registered. Grid resolution, the Laplace smoothing constant, the context_total >= 50 reliability threshold, and the decision to pool five leagues were each set partly in response to how earlier results looked, not fixed in advance of seeing any data. This is normal for exploratory work, but it introduces a real, named risk in applied statistics — researcher degrees of freedom — worth disclosing rather than presenting each choice as if it had a clean a priori justification.

9. The action vocabulary is blind to defensive play entirely. Tokenisation covers Pass, Carry, Shot, and Dribble — interceptions, blocks, recoveries, and pressure events are invisible to the model. A possession's real unpredictability plausibly depends partly on what the defense did in response to it, which this representation cannot capture in any form.