AI Prediction-Market Backtesting: Test LLM Strategies Without Future Leakage

Why AI backtesting is a separate problem

A deterministic rule can be reviewed line by line. An LLM can change its reasoning with the prompt, model version, context, tool results and sampling settings. It may also know facts that occurred after the simulated date because its training data or an attached search tool reaches beyond the historical clock. A normal trading backtest does not automatically control those risks.

Prediction markets make the leakage especially severe. The market question and final resolution are public after settlement, so a model can appear to forecast an event while recalling or inferring its answer from later information. The test must preserve both market-time integrity and model-time integrity.

Two AI backtesting workflows

DepthFeed's currently public workflow supports the first path: ask a model to propose a clear hypothesis, translate it into a preset or custom rule, freeze the parameters, and run it through Backtest Lab. This separates idea generation from evaluation and makes the replay repeatable.

A model-in-the-loop replay requires additional controls: a closed information environment, sequential portfolio state, durable model-call records and a point-in-time tape for every cycle. Do not describe a chat transcript over resolved markets as this stronger form of backtest.

Build the historical information boundary

The safest implementation uses an as-of boundary: source timestamp less than or equal to decision time. Every feature, summary and tool response must inherit that boundary. A single unbounded convenience field, such as today's market status or a final outcome label, can invalidate the full run.

• Set a decision timestamp and include only market rows observed at or before it.
• Bound any trailing lookback explicitly; never use a nearest row that can come from the future.
• Exclude settlement results, final prices, later headlines and current web search.
• Record the exact market universe shown to the model, including contracts it skipped.
• Hash or version the input tape so the same decision can be reconstructed later.
• Report missing data as missing instead of filling it from a later observation.

Prevent answer and prompt leakage

Freeze the system prompt, customer instructions, model identifier, sampling mode, structured-output schema and available tools. Remove language that reveals the outcome indirectly, including final market status, settlement wording, retrospective article summaries or filenames created after resolution.

For historical event questions, assume that pretrained model memory may contain later facts even when browsing is disabled. Use cut-off-safe questions where possible, evaluate leakage controls separately, and compare performance on fresher or privately held-out periods. A high score on well-known resolved events is not by itself evidence of forecasting skill.

Make the decision protocol auditable

Ask for structured actions rather than free-form recommendations. A useful decision includes market identity, side, estimated probability, confidence, maximum entry price, requested notional and a short thesis. Server-side rules should then accept, resize or reject the proposal according to cash, exposure, confidence, edge and price limits.

Store passes and rejected trades, not only fills. If the final report contains only winners the model chose, there is no denominator for selectivity, coverage or failure analysis. The durable record should connect each action to its exact input boundary and resulting portfolio state.

Replay the portfolio sequentially

Portfolio cycles must run in chronological order because one decision changes the cash and open exposure available to the next. Parallelize market discovery inside a timestamp if needed, but synthesize one final portfolio decision against one frozen tape. Do not let independent model calls spend the same cash.

Carry open positions, realized cash and risk limits forward. Then report P&L by category and venue as well as in aggregate. This reveals whether an apparently intelligent portfolio is one concentrated directional bet repeated across correlated contracts.

Use executable fills, not the model's quoted price

The model may state a maximum price; it should not be allowed to invent its own fill. Add a defined execution delay, locate the first recorded book at or after that simulated time, and walk available asks or bids only up to the limit. Store VWAP, filled fraction, observation time and the book used.

Mark the final portfolio conservatively. A midpoint mark assumes liquidation without crossing the spread or consuming size. A depth-aware executable liquidation better answers what the portfolio could have realized; stale or absent books should remain an explicit failure condition.

Worked workflow: from an LLM hypothesis to a replay

Start with a bounded prompt: ask the model for one falsifiable Polymarket or Kalshi hypothesis, the observable inputs required, a precise entry rule, position size, exit or settlement behavior and the conditions under which it should pass. Prohibit profit claims and request no retrospective examples.

Translate the answer into a Backtest Lab preset or custom rule without changing its thresholds after seeing results. Select the intended venue and market family, use a chronological sample, and compare midpoint, fixed slippage and recorded depth. Inspect every trade and preserve a later holdout. If it survives, deploy the frozen rule to paper trading and evaluate it on outcomes the model could not have known when the rule was created.

Metrics that distinguish forecasting from trading

Forecast quality and trading performance are related but not identical. A calibrated model can lose by paying prices that already reflect its information. A poorly calibrated model can make money briefly through luck or one concentrated position. Report both layers and avoid reducing the evaluation to a leaderboard headline.

Compare models without moving the goalposts

• Give every model the same frozen market tape, tool set, output schema and portfolio limits.
• Use the same decision timestamps, execution latency, fill engine and final marking rule.
• Separate provider failures, invalid output and risk-rule rejections from market losses.
• Record token and model cost alongside performance; a marginal gain may not justify much higher inference cost.
• Repeat stochastic runs or use deterministic settings where supported, and report the variance.
• Keep named-model pages and claims current because model versions and availability change.

Classify failures instead of hiding them

These outcomes mean different things. Treating a provider timeout as a confident pass inflates selectivity; dropping an unfilled order overstates executable performance; and silently repairing malformed model output gives one model human help the others may not receive. The evaluation should define each class before the run and apply it mechanically.

Publish the full funnel: eligible markets, markets shown, proposed actions, passes, invalid outputs, guardrail rejections, execution rejections, partial fills and completed positions. That denominator makes it possible to distinguish a cautious model from an unreliable one.

The minimum reproducibility record

For every run, preserve the model provider and exact model slug, the prompt templates, tool definitions, structured-output schema, sampling configuration and customer instructions. Record the historical start and end, decision interval, lookback, venues, categories, starting cash, exposure limits, execution latency and final marking method.

For every decision cycle, preserve the decision timestamp, tape version, tape as-of time, eligible market count, input hash, portfolio before and after, model usage and error state. For every proposed trade, store its thesis and probability alongside the observed market probability, confidence, limit price, requested size, rejection code and the historical book used for any fill.

Provider models can be updated behind a stable name, so an exact rerun may still differ later. A durable record cannot eliminate that platform variance, but it can reveal it. Where deterministic seeds or pinned versions are unavailable, repeat the same test enough times to report run-to-run dispersion instead of presenting one favorable sample.

• Model slug, provider, prompt, tools, schema and sampling settings.
• Historical boundary, market universe, decision interval and lookback.
• Portfolio limits, execution latency, fill engine and final mark.
• Input and book hashes tied to every decision and fill.
• Token usage, model cost, invalid outputs and all rejection reasons.
• Repeated-run variance when the model or provider is stochastic.

Forward-test before making an AI performance claim

A historical AI test can still benefit from model memory, researcher prompt tuning and repeated experimentation. The cleanest next step is a locked forward paper period. Freeze the prompt or extracted rule, begin after the configuration is final, price every action from the live displayed book, and publish the full denominator.

DepthFeed Paper Trading provides the available forward path for deterministic rules and external bot signals. It uses virtual cash and does not place live exchange orders. That distinction should remain visible anywhere an AI workflow is described.

What DepthFeed offers today

Today, researchers can use an AI model to create or refine an explicit strategy, replay that frozen rule in Backtest Lab against recorded prediction-market books, compare three fill assumptions, inspect risk and individual trades, and move compatible survivors to live paper trading. The historical data API and MCP server also support custom agent research outside the dashboard.

DepthFeed does not currently advertise a public autonomous model-in-the-loop backtest API or live AI trade execution. The narrower workflow is intentional and testable: the model proposes; recorded market evidence evaluates; paper trading supplies the forward record.