PSDG

Appearance

Learning the wrong state ​

PSDG tests whether an agent has learned the game, or merely learned a lossy surface representation of the game.

What this page is for ​

This is research documentation: the trained-agent complement to the structural tops-only aliasing exhibit. That page proves, by oracle enumeration, that a tops-only view is not a sufficient state. This page shows what happens when an actual learner is trained on PSDG — and the result is cleaner than “tops-only is imperfect.” It dissociates two failure modes that surface metrics conflate:

A full-state learner generalises to optimal play across openings. A tops-only learner fails across almost every opening. Removing the Exchange aliasing does not rescue it, because the wrong representation already corrupts the Draft.

Crucially, “train more” fixes the objective failure but not the representation failure. The training-proof anchor is the enumerated aliasing floor, not any one loss percentage — the trained loss rate is budget-sensitive. Trained to 8× the converged budget, the tops-only learner still never reaches the full-state’s stable optimum (see Is this just undertraining?).

Setup (pinned & scoped) ​

The discipline is solver-as-examiner: the oracle is not a teacher during training; it grades afterward.

  • Learner: tabular Q-learning for player A, terminal reward only ({−1, 0, +1}), no oracle labels during training; γ = 1 (finite horizon).
  • Observation encoders: full (tops + facings + ownership — Markov for v1.13) vs tops-only (facings dropped — deliberately aliased).
  • Opponents B (fixed, non-learning): random-legal and optimal (oracle).
  • Audit (after training): oracle-grounded regret, split Draft vs Exchange, plus win/draw/loss; regret in oracle win/loss units (±1).
  • Scope: a tabular learning demonstration on the YouTube-demo opening and a seeded opening family with fixed crystals; converged budget (5×10⁵ episodes, robustness checked to 4×10⁶ = 8×); generalisation measured on A-win openings (so “wins → losses” is meaningful). This is not a claim that “all tops-only agents lose,” nor a deployment/protocol claim (those live elsewhere).

Internal dev-tree code (public psdg has no private/): see Reproduce.

Result 1 — win rate hides regret; the objective failure is fixable ​

Against a weak (random-legal) opponent, the full-state learner wins ~98% (baselines: random-A 60.8%, grab-the-6 78.0%) — yet it is not oracle-optimal. Across 10 seeds only 1/10 plays the optimal opening draft, and mean Draft regret is ~0.49. More training does not fix this: the suboptimal draft still wins ~98% against a weak opponent, so terminal reward provides almost no gradient toward optimal drafting.

Swap in an optimal opponent and that regret vanishes — the full-state learner plays perfectly (100% win, Draft regret 0). The proxy/objective failure is opponent-limited: fix the objective (a punishing opponent) and the learner recovers. This is the empirical, trained version of the parable’s “optimally wrong” story.

Result 2 — the representation failure is irreducible ​

This result has two layers, and it matters which one you quote.

(a) Training-proof core (structural). By oracle enumeration the tops-only view has a non-zero aliasing floor on the demo opening — Exchange 0.021 and Draft 0.0097 in oracle win/loss units (worked example). This is a property of the input, independent of any learner, opponent, or budget: there exist positions with identical tops that need opposite optimal actions, so no memoryless tops-only policy is right in both.

(b) What a trained agent actually pays (on-trajectory, vs optimal B). Here the honest result is not a single plateau. We trained to 8× the converged budget (5 seeds each):

budget (vs optimal B)tops-only wintops-only losstotal regretseeds fully solved (0 regret)
500k (converged)94.0%6.0%0.106—
1M98.9%1.1%0.0203 / 5
2M98.9%1.0%0.0611 / 5
4M (8×)97.9%2.1%0.0860 / 5

(full-state is 0% loss / 0 regret at every seed and every budget, out to 8M.)

Read it carefully — it cuts both ways:

  • More training does shrink the raw loss. The 6% at the converged 500k budget falls to ~1–2% at 2–4×. So the loss rate is budget-sensitive, and quoting “5.5%” as a fixed number (or as “matching the 0.021 floor”) would overclaim — trained Exchange regret actually wanders over 0.013–0.078 across budgets.
  • But the gap to full-state never closes — and is unstable. Even at 8× the plateau, 0 of 5 seeds reach the full-state’s clean optimum; the solved-seed count is non-monotone (3/5 → 1/5 → 0/5), and the residual error just migrates between Draft and Exchange rather than vanishing. There is no stable tops-only optimum to converge to. The full-state learner, by contrast, converges to 0 and stays there at every seed and budget.

So the defensible statement is: a sufficient state lets the learner solve PSDG completely and stably; the tops-only state does not, and 8× more training does not recover the difference. Anchor the claim to the enumerated floor (a), not to any one loss percentage in (b).

Result 3 — it generalises, and it is broader than the Exchange ​

Across a seeded suite of A-win openings, the full-state learner wins 100% on every opening, while tops-only converts wins → losses on 15/16 openings that have a structural Exchange floor.

Two findings sharpen this:

  • Trained cost exceeds the structural floor. Mean trained excess regret ~0.13 vs structural floor ~0.05: optimising around a lossy representation makes the damage worse than the irreducible minimum. The floor is a lower bound; a trained aliased agent generally pays more.
  • The cost is not confined to the Exchange. Control openings with a zero Exchange-aliasing floor still lose — through Draft-stage aliasing. Confirmed two independent ways:
    • Trained decomposition: on controls the excess regret is overwhelmingly in the Draft (+0.168) not the Exchange (+0.026).
    • Learner-independent enumeration: a structural Draft floor > 0 on every control opening (and on 14/16 test openings) — the same oracle-enumeration method used for the Exchange floor, so it cannot be a training-dynamics artifact.
    • Across random openings: the same enumeration over 200 random 6-dice openings finds a structural Draft floor in 96% of them (98% with the published fixed crystals) — 97–99% among A-win openings — with a small, reported zero-floor class. The floor is a broad property of the game, not the curated set. (floor_distribution.py)

Dropping facings induces self-blindness across the game, not at one trick node. The demo opening’s clean Exchange-localisation was partly special; in general the insufficiency bites at the Draft too.

Why this matters ​

PSDG separates two things that win-rate-style metrics blur together:

  1. Learning ability — can the method learn the game at all? (Full-state → perfect, everywhere.)
  2. State-representation adequacy — is the agent allowed to see what governs outcomes? (Tops-only → losses, everywhere.)

And the two have different remedies: the objective failure yields to “train against a stronger signal”; the representation failure does not — the only handle is fixing what the agent observes. (AI safety — three handles.) That is why “looks capable” (high win rate vs a weak baseline) can uncouple from robust play under the true rules — measured here, not asserted.

Is this just undertraining? Would a bigger model (e.g. AlphaZero) help? ​

Short answer: partly fair about the loss number, but no about the conclusion — and a heavier learner mostly wastes effort on this particular claim. Be precise about what is and isn’t training-proof.

The training-proof part is a lower bound, not a training outcome. The tops-only cost is pinned to an oracle-enumerated floor (Exchange 0.021; Draft floor > 0 on every control opening). That floor is information-theoretic: if two true positions collapse to one observation yet need different optimal actions, then any memoryless policy of that observation — tabular, deep net, MCTS, future-SOTA — is wrong on at least one. You cannot optimise below a bound that lives in the input, not the optimiser.

We tested the “train more” objection directly — and report it honestly. Training tops-only to 8× the converged budget (4M episodes, 5 seeds) against the optimal opponent (Result 2b): the raw loss does shrink (6% → ~1–2%), so anyone quoting a fixed “5.5%” is overclaiming — that number is budget-sensitive. But the gap never closes and never stabilises: at 8× the plateau 0/5 seeds reach the full-state’s clean optimum, the solved-seed count is non-monotone (3/5 → 1/5 → 0/5), and the residual error merely migrates between Draft and Exchange. The full-state learner, same algorithm and budget, is 0 regret / 100% win at every seed and budget. So the failure is the state, not the schedule — but the right thing to quote is the enumerated floor and the never-solved gap, not a loss percentage.

Why AlphaZero specifically does not help — and can mislead:

  • Fed the tops-only observation as a memoryless policy/value net, depth and width buy nothing: the facings are not in the input, so no function class recovers the distinction — the same floor binds. (This is the rock-solid form of the claim: a memoryless function of the tops-only observation. History-conditioned or true-model-search agents are a different case — see the trap below and “where a heavier method would help.”)
  • Fed the full state, it would reach optimal play — but so does the tabular learner here already; AlphaZero would merely reproduce “full-state → optimal” at far greater compute.
  • The trap: AlphaZero’s MCTS plans with a true simulator. Expanding the game tree reads the true facings during search and de-aliases the agent — quietly smuggling back the very information we removed. That makes it a worse instrument for isolating representation sufficiency, not a better one. The clean test is precisely the observation-restricted, model-free setting used here.
  • It also runs into the project’s own thesis — nobody out-minimaxes the oracle (PSDG for ML). Training a stronger learner to “win more” is the wrong story; the oracle is a ruler, not an opponent to beat.

Where a heavier method would add real value (genuine future work, not AlphaZero):

  • Function approximation (a neural policy) on the same observations — to show the effect is not a tabular artifact. It would attack the magnitude above the floor, never the floor itself.
  • Memory / history-conditioned agents (recurrent or sequence models). Facings are determined by the draft history, so an agent conditioning on the full action history could reconstruct the dropped state and de-alias. That would not refute the thesis — it confirms it: a sufficient state must be carried somehow, whether fed in the observation or recovered from history.

What this is not ​

  • Not “tops-only agents always lose.” Outside the conflicting subset a tops-only Gift/draft is fine; the claim is that a sufficient state must carry facings, at both phases.
  • Not a deployment/protocol result. The static-vs-re-solving and Exchange-timing splits (~8.5% / 6.9% / 5.7%) are a separate axis (snapshot).
  • Not hidden information. After setup PSDG is perfect-information; facings are on the table. The tops-only learner simply isn’t fed them — aliasing, not secrecy (FAQ).
  • Scope: tabular Q-learning, one opening family / fixed crystals, oracle ±1 regret units. Transfer to function approximation, other opening families, and sequential-Exchange variants is a separate (open) claim.

Reproduce ​

Public repo — ml/ (standard library only; clone and run):

  • sweep_matrix.py — observation × opponent matrix (Result 2a).
  • robustness_budget.py — budget sweep to 8× (Result 2b: the “train more” test).
  • step3_cross_opening.py — cross-opening generalisation; logs the Draft/Exchange split (Result 3).
  • structural_floors_cross.py — learner-independent Draft vs Exchange structural floors.
  • floor_distribution.py — Draft-floor distribution across random openings (prevalence + magnitude).
  • aliasing_exchange.py — the enumerated Exchange floor, no training (~10s).

Development tree only (private/psdg/ml/): sweep_phase5.py (Result 1 convergence sweep) and STATUS.md (handoff with the full tables and decisions log).