Backtesting & Validation

Backtesting and Validation is a skill for CLI Trader that provides a rigorous methodology for testing trading strategies against historical data and validating them before risking real capital. Every profitable-looking strategy needs to survive a gauntlet of statistical validation before it deserves your money, and this skill gives your agent the knowledge to run that gauntlet properly. It covers the complete pipeline from raw data handling through simulation, bias detection, and the disciplined transition from backtest to paper trading to live deployment.

The skill structures backtesting as a multi-stage pipeline. Data preparation includes handling splits, dividends, survivorship bias, and timestamp alignment. Strategy simulation accounts for realistic execution assumptions including slippage, commissions, partial fills, and market impact. Bias detection is a first-class concern, with explicit checks for look-ahead bias in feature construction, survivorship bias in asset selection, and overfitting via parameter sensitivity analysis. Walk-forward analysis with rolling optimization and out-of-sample testing windows provides the most realistic performance estimates. The skill covers key performance metrics — Sharpe ratio, Sortino ratio, Calmar ratio, maximum drawdown, profit factor, and win rate — and teaches the agent which metrics matter most for different strategy types. Integration patterns for Freqtrade and other open-source backtesting frameworks are included for practical implementation.

Within a CLI trading workflow, this skill acts as the quality gate between strategy development and capital deployment. After developing a strategy idea using other skills (technical analysis, mean reversion, sentiment), pipe it through the backtesting and validation pipeline to stress-test its assumptions. The agent will flag statistical weaknesses, suggest robustness checks, and recommend whether the strategy is ready for paper trading or needs further refinement. This systematic approach prevents the expensive mistake of deploying strategies that looked great in-sample but fail out-of-sample.

For the complete methodology — including formulas, decision trees, worked examples, and validation checklists — see the full Backtesting & Validation skill guide.