Learning faster with the symmetry
Reinforcement learning and its inverse, with and without the imbalance equivalence — and what happens when the symmetry is only approximately true. Companion to On a Simple Relationship Between Order Imbalance, Skew and Width in Over-The-Counter Trading, §5.
The flow regime cycles through four imbalances (q = 0.40, 0.50, 0.60, 0.70, switching every 400 enquiries). Identical tabular Q-learners quote into this market. The baseline learns a separate policy per regime; each table sees a quarter of the data. The symmetry learner keeps one policy in the balanced frame and maps it through the skew shift δ(q) = (w/2) log(q/(1−q)): every regime’s experience trains the same table. That is a heuristic even under the paper’s model: the exact symmetry sends regime q to the balanced problem at carrying cost M(q) c, so regimes with different M(q) have different optimal policies. Across these four regimes M(q) is at most 1.09. The robust learner adds a small per-regime residual table on top of the shared one, so it can absorb whatever the symmetry gets wrong. The oracle quotes the exact optimal policy for the true generative model.
The win curve selector changes the generative model of the best competing quote. Exponential is the paper’s model: constant hazard, under which the symmetry is exact once M(q) is accounted for. Logistic is an illustrative non-exponential curve with increasing hazard, plus competitors who crowd the flow-heavy side, so the symmetry becomes only approximately true. The δ the learners use is deliberately NOT recalibrated: it is the misspecified, exponential-model formula.
Under the exponential model both shared-table learners settle within about a tenth of the oracle, well ahead of the per-regime baseline. Under the logistic model they still beat the baseline by a wide margin, and the gap to the oracle widens to about a seventh.
The inverse problem, where robustness earns its keep
Now watch instead of act. A dealer quotes optimally across the same regimes, and an observer sees her quotes with noise. The target is her quote skew at each inventory, half the difference between her markdown and markup. Under the exponential model this is the slope S(x) of her inventory cost wherever neither quote is at the zero floor. Three estimators: per-regime uses only the target regime’s observations. Pooled de-tilts everything by δ(q) and averages. It is biased even under the exponential model, because the regimes differ in M(q) and δ removes only the first-order part. Unlike the forward learner, an estimator cannot absorb its own bias. Shrunk is the robust design: random-effects partial pooling, in which the between-regime dispersion of the de-tilted means estimates the size of the symmetry violation and sets the pooling weight automatically — the same reliability-sets-the-dial mechanism as the Schur damping γ*.
Under the exponential model pooled is best at small samples, then stalls at its bias while per-regime keeps improving. Under the logistic model with these four regimes the de-tilted skews happen to agree closely, and pooled stays competitive. Shrunk sets its own pooling weight in both cases.
Environment and solver: mm_core.js, rl_core.js. Roadmap entries P3, P7 and P9 in papers/ROADMAP.md.