Abstract

A production forecasting head kept collapsing: trained on frozen Moirai-MoE embeddings of BTCUSDT perpetual futures, it converged to a near-constant median and a directional accuracy indistinguishable from a coin flip. This study was built to find out whether that was a bug in the head or a property of the data. It is the latter. On a 158,528-window frozen embedding store spanning 2019–2026, under purged and embargoed chronological splits, short-horizon direction is not predictable to a degree that survives realistic transaction cost — not from the frozen embeddings, not from causal bar features, not under vol-scaled or triple-barrier targets, and not after confidence gating or horizon extension. The same pipeline reaches a test R² of 0.40 predicting realized volatility, which is what makes the directional null trustworthy rather than a plumbing failure.

A follow-up study reframed the question as a proper-scoring one: make the unconditional distribution an explicit object and require a head to earn out-of-sample skill over it. The strongest baseline turned out to be a ten-bucket causal volatility table (+3.84% validation pinball skill, +3.31% on the untouched test split), and no learned head deforming the frozen tokens matched it. Permutation controls localize that skill precisely: shuffling the embeddings barely moves it, while shifting the target in time destroys it. An error review then found the binding constraint — the heads had never been given the causal tabular features, and the backbone's per-window standardization structurally removes the volatility level from its tokens. With twelve tabular features wired in and a volatility-distillation auxiliary loss, a learned head finally beat the bucket, reaching +6.04% validation skill over the unconditional marginal. The center remained inert throughout. Every result here is scale, not direction.

A note on this page. Unlike the other reports in this series, this study was never written up as a manuscript — it ran as an internal engineering investigation with its evidence in dated experiment logs. This page was written from those logs and the emitted run artifacts. All figures are generated directly from the study's runs/conditional_distribution artifacts; numbers quoted from the probe phases come from the experiment log and are attributed where they appear.

The native C++/CUDA runtime carried a monotone quantile forecasting head that behaved pathologically in training. Its pinball loss improved over the unconditional baseline, which looked like learning, but every diagnostic that mattered for trading stayed flat: the predicted median barely moved between windows, directional accuracy sat at 0.50, and a cost-aware backtest produced essentially no trades. The working assumption was a defect in the head or its optimization — a plausible one, since the head was slow to iterate on, requiring full MSVC/nvcc rebuilds and kernel parity checks for each change.

The head lab was built to test that assumption cheaply. The C++ trainer already cached, per window, the frozen backbone's output tokens; exporting that cache to a Python-native store turns head research into a seconds-per-epoch GPU loop with no backbone forward pass and no rebuilds. The question it was built to answer was deliberately narrow and falsifiable: can any head or loss, on these frozen representations, produce a real out-of-sample directional edge that survives realistic transaction cost?

The store holds 158,528 windows of BTCUSDT USD-M perpetual 15-minute bars from September 2019 to April 2026. Each window carries the frozen Moirai-MoE output tokens as a 90 × 768 tensor — three input scales of thirty patches each, or 69,120 dimensions per window — alongside its forward targets at horizons of 1, 4, and 16 bars (15 minutes, 1 hour, 4 hours) and the funding rate. Quantile work throughout uses the levels 0.05, 0.25, 0.50, 0.75, and 0.95. Target standard deviations on the training split are 35, 70, and 137 basis points at the three horizons.

Leakage control was implemented before any result was taken seriously, because it gates everything downstream. Splits are chronological with a purge: from each earlier split, windows whose anchor falls within the embargo of the next split's start are dropped, and the embargo is required to be at least the maximum horizon, since labels span forward in time. Critically, the embargo is measured in time rather than in row count — store anchors are not uniformly spaced, and only about 69% of consecutive anchors are actually fifteen minutes apart. The embargoed splits are 126,815 training, 15,837 validation, and 15,852 test windows. The earlier non-embargoed splitter was retained only as a deliberately labeled leaky reference.

The final test split is treated as a scarce resource throughout: selection happens on validation, and test is read sparingly and deliberately, a discipline that becomes load-bearing in Section 6.

The first task was to reproduce the production failure in the fast harness, which it does exactly. Over twenty epochs the baseline head's validation pinball converges from 0.1551 to 0.1512 against an unconditional marginal of 0.1563, and reaches 0.1550 on test against a marginal of 0.1595. So it genuinely beats the marginal. But directional accuracy stays at roughly 0.50 throughout, per-epoch values ranging only from 0.494 to 0.509, and median dispersion sits between 0.01 and 0.025 in standardized units — the predicted median varies by one or two percent of a target standard deviation across windows, which is near-constant. The median magnitude clears a realistic cost band in about 1% of windows, so the costed backtest is flat.

The diagnosis follows from those numbers together: the head beats the marginal only by fitting a better-calibrated spread, while the center stays put. That is not a broken optimizer. It is a head correctly reporting that, conditional on its inputs, the distribution's location is approximately the unconditional location and only its width is informative.

Baseline	Split	Pinball (std. units)	Directional accuracy	Median dispersion
Unconditional marginal	val	0.1563	0.500 / 0.506 / 0.513	0
Unconditional marginal	test	0.1595	0.503 / 0.502 / 0.500	0
Funding-regime marginal	val	0.1555	0.500 / 0.506 / 0.513	~1e−5
Linear probe (ridge, mean-pooled)	val	—	0.498 / 0.495 / 0.494	~5e−4
Quantile head (the collapsing baseline)	val	0.1512	~0.50	~0.015
Quantile head (the collapsing baseline)	test	0.1550	0.503	~0.024

Two reference points bound the problem. Always-long earns +0.75 basis points per window on validation and −1.34 on test before cost, falling to −6.3 on test at five basis points of round-trip cost: Bitcoin's drift does not survive cost at these horizons on this sample. And the linear probe on mean-pooled embeddings finds nothing at all, with negative standardized R² at every horizon.

A linear probe on pooled tokens is a weak test, so the search was widened along three axes at once: targets (raw return, volatility-scaled return, triple-barrier labels), feature sets (nine causal exogenous features; embedding mean, max, and standard-deviation reductions; both together), and model class (ridge for linear structure, a two-layer MLP for nonlinear structure), across horizons from 15 minutes to 3 days.

The frozen embeddings return 0.49 to 0.51 directional accuracy for every target, pooling, and model class. Adding them to the exogenous features does not help and often hurts through overfitting: at the 4-hour horizon, exogenous features alone reach 0.5288 validation and 0.5240 test, while the same features plus embeddings reach 0.5228 validation but fall to 0.5023 on test. The tabular features do carry a small, genuinely nonlinear edge — ridge misses it entirely at roughly 0.50, so it is interaction and regime structure rather than a linear factor — worth about 0.52 to 0.53 directional accuracy on both validation and test, strongest at 4 hours.

That edge does not survive contact with execution. Under a realistic non-overlapping backtest holding one position at a time for the full horizon, with the confidence threshold chosen on validation and applied to test, the 4-hour lane sits near breakeven at two basis points of cost (+0.17 basis points per trade) and turns clearly negative at five (−2.8). The 1-hour lane's validation-tuned threshold loses at every cost level on test. Nor does the edge concentrate where a real signal should: at 4 hours the top-50% confidence bucket scores 0.537 against the top-5% bucket's 0.534, which is the signature of noise rather than of a signal with a confident tail. Extending horizons to 12 hours, 1 day, and 3 days to amortize cost, and enriching the feature set, reproduces the same pattern in sharper form — strongly positive on validation at every horizon and cost, and negative essentially everywhere on test. That is textbook validation-threshold overfitting under non-stationarity.

A null result is only worth as much as the pipeline that produced it. If the harness could not detect signal that was genuinely present, the directional null would be uninformative. So the same machinery was pointed at a target known to be learnable: log realized volatility over the next h bars.

It reaches a test R² of 0.36 to 0.40 at the 4-hour horizon. That validates the pipeline and makes the directional null trustworthy. It also produced the study's first constructive finding: the frozen embeddings carry real volatility information, and it is partly orthogonal to the tabular features, since the combination beats either input alone.

Target	Exogenous features	Embeddings	Embeddings + exogenous
Next 4-hour realized volatility	0.31–0.34	0.27–0.31	0.36–0.40
Next 24-hour realized volatility	0.24–0.29	0.22–0.28	0.26–0.33

This reframes what the frozen representation is for. It is not a source of directional alpha at these horizons, but it does encode volatility and regime structure usable for sizing, risk targeting, and volatility-scaled strategies.

The probe phases judged models on whether their predictions varied. That is a weak and slightly circular standard — a head can be made to produce non-constant output by loss engineering without carrying any information. A follow-up study replaced it with a stricter one: make the unconditional distribution an explicit object, and require any learned head to earn out-of-sample proper-scoring skill over it. Skill is reported as one minus the ratio of model pinball to baseline pinball, so zero is a tie.

The first question is what the strongest non-learned baseline actually is. Five causal regime schemes were fit as per-bucket empirical quantile tables on the training split and scored against the global unconditional marginal on validation. The winner is not close: a ten-bucket table on realized-volatility deciles beats the global marginal by 3.84%, and generalizes to +3.31% on the untouched test split.

Against that bar, the learned heads were swept broadly: four head families (additive residual, location-scale, per-gap deformation, and a regime mixture), three token readouts (mean pooling, recency weighting, and attention), five loss terms, and two target families. None of them, deforming the global marginal from the frozen tokens, reached the hand-specified volatility bucket. The best managed +1.77% against the bucket's +3.84%.

Three independent lines of evidence establish that whatever skill exists here is scale, not center. The center deformation the head learns is inert, correlating with the realized target at only 0.02 to 0.07 out of sample, with a magnitude of roughly 4% of the baseline interquartile range. Directional accuracy stays between 0.50 and 0.52 with a Matthews correlation coefficient near zero. And on a volatility-scaled target, which removes the regime scale by construction, the head's skill falls to approximately zero — taking away the volatility structure takes away the only thing it was exploiting.

The permutation controls are the most informative artifact the study produced, because they attribute the skill rather than merely measuring it. Shuffling the embeddings against their targets barely moves the finalist's skill, from +3.95% to +3.85%. Shifting the target in time, which breaks the causal alignment while leaving everything else intact, collapses it to −0.57%. The skill is therefore the causal link from the explicit volatility regime to the target, and essentially none of it lives in the frozen representation.

Two synthetic controls confirm the harness itself is trustworthy in both directions. On a target constructed to be independent of all features, out-of-sample skill is +0.000 and the center correlation is about 0.01 — the head shrinks to the baseline and invents nothing. On a target constructed as a decodable projection of the embeddings plus noise, skill is +0.24 and the center correlation is about 0.65 — it recovers a signal that is genuinely present. A harness that neither hallucinates nor misses is what licenses reading the main result as a real null.

Before accepting "no learned head beats a ten-bucket table" as final, the pipeline was audited empirically for defects that could be suppressing results. Six checks were run against the data rather than the source: the join used to attach features preserves row order; a cross-identity check confirms feature–target alignment is bit-exact, with maximum error 0.0 across 108,673 pairs; no training window's label window crosses a split boundary at the current split placement; and adjacent-token cosine similarity confirms the readouts group tokens correctly, breaking at positions 30 and 60 exactly as a scale-major layout predicts. No signal-suppressing bug exists.

The audit did, however, find a decisive omission. The residual heads had been conditioned only on frozen tokens and a funding scalar — they were never given the causal tabular features. Meanwhile the baseline beating them is literally a coarse bucketing of trailing realized volatility. The heads were being asked to beat a feature they could not see.

There is a structural reason they could not recover it on their own. Moirai-MoE standardizes each window and variate before embedding, which removes the absolute volatility level — precisely the quantity the winning baseline conditions on — from the tokens by construction. The tokens can encode volatility dynamics and shape, but never level. Expecting them to beat a realized-volatility baseline was structurally impossible, and the per-window scaler statistics are the missing input.

Wiring a twelve-feature causal tabular block into the residual heads, and adding an auxiliary loss that distills the proven volatility signal directly into the head's scale output, changes the conclusion. A learned head now beats the strongest baseline: +1.62% over the volatility bucket, or +5.40% over the global marginal, rising to +2.28% and +6.04% with the distillation term. Twelve tabular features did what 69,120 frozen token dimensions could not.

Two qualifications belong with that number. The time-shift control passes for the new candidate, with skill against the global marginal collapsing from +5.40% to −0.56%, so it is causal. But the single sanctioned test evaluation had already been spent on the earlier embeddings-only finalist, and this candidate is validation-selected only. Reading test again is a second query on a held-out set, and the recommendation of record is to run rolling walk-forward refits first and then spend one test read on the final configuration. The +6.04% is therefore an honest validation number, not a confirmed out-of-sample one.

The study's purpose was to decide what belongs in the native runtime, and it answers that cleanly in three parts.

Port the volatility-conditioned baseline. A causal per-regime marginal quantile table on realized-volatility deciles, fit on train and shrunk toward the global marginal, is simple, interpretable, leakage-safe, trivially parity-tested against Python on fixed batches, and delivers +3.31% out-of-sample pinball skill with a calibrated volatility-conditioned interval. Alongside it, the twelve-feature location-scale residual head with volatility distillation is small enough to be a genuine port candidate, subject to the test read above.

Do not port a learned head for direction or center. There is no center or sign edge in the frozen representation to port: the center-to-target correlation is approximately zero and directional accuracy is 0.50. Adding a learned center pathway would add complexity for no measurable gain.

Do not unfreeze the backbone for this objective. The frozen representations carry no directional signal, and their scale signal is already dominated by an explicit volatility bucket, so adapter or LoRA fine-tuning is high-cost and low-expected-value here. If the volatility level is what the tokens lack, the cheaper repair is to feed realized volatility into the backbone as an additional input variate and re-export — Moirai-MoE is any-variate, so this repairs the structural gap without fine-tuning anything. If direction is the goal, the lever is new information — order-book depth, cross-asset lead-lag — or a different execution framing, not a new head on this store.

The directional conclusion is scoped to the exported frozen embeddings plus canonical bar features on a single symbol. It says nothing about order-book or microstructure data, cross-asset relationships, or alternative data, and if a directional edge exists at these horizons it most plausibly lives in data this store does not contain. All directional numbers assume taker cost; a maker-side or funding-carry framing is a different strategy class and was not evaluated.

The evaluation also rests on a single static split under strong non-stationarity: training spans the high-volatility 2019–2024 period while validation and test fall in the calmer 2024–2026 stretch, and the baselines, target standardizer, and head are each fit once. The validation-to-test collapse seen in the directional gating work is exactly what static fits produce under drift, and rolling purged walk-forward refits — specified but not yet run — would strengthen the evaluation and would likely help the model side more than the baseline side.

One reporting artifact is worth flagging for anyone reading the raw logs: training pinball exceeds validation pinball (0.20 against 0.15), which looks inverted. It is a scale effect, not overfitting in reverse — targets are standardized by the training standard deviation while the later validation period is genuinely calmer, so its standardized targets are smaller.

The probe phases run as standalone scripts over the exported store: baselines and leakage diagnostics, causal feature and alternative target construction, the target-by-feature-by-model probe, confidence gating with the non-overlapping backtest, the longer-horizon extension, and the volatility probe. The conditional-distribution suite runs in four staged phases — synthetic controls, baseline audit, the residual-head matrix, and the finalist with its permutation controls and single test read — behind a six-case unit-test suite. Every run writes a self-contained artifact folder holding its resolved config, split manifest, baseline registry, per-split and per-regime metrics, directional scorecard, calibration report, delta diagnostics, permutation controls, training log, checkpoint, and prediction export, with a global leaderboard across runs.

The harness improvements that made the study feasible were left in place and are reusable: the time-based purged and embargoed splitter, an in-RAM float16 preload path with a memory budget that automatically raises the training stride to fit, and a lazy per-worker memmap open that resolves a Windows dataloader spawn failure. Together these took the training loop to roughly 4.7 seconds per epoch on a single RTX 4070.