Ask an AI assistant to design an ETF strategy for retirement. It can explain the investment logic, write the code, and help produce a chart that looks ready to publish. That is useful. It also makes it easier to reach a convincing result before checking whether the result was possible.
My first question is simple: can I follow one trade from the information available at the time to the price at which it could actually have been executed?
A strategy can fail that test even when its code runs and its explanation sounds sensible. For someone with accumulated savings, the consequences extend beyond a disappointing chart. The same capital may need to support spending through a market decline.
I write OnyxValue , where I examine systematic ETF investing. The five checks below concern the evidence behind an investment process. They apply whether AI writes the code, proposes the strategy, or interprets the news. The numerical example is invented to make an accounting error visible; it is not an OnyxValue trading rule.
1 Turn the recommendation into a specification
“Choose strong ETFs and avoid large losses” describes an ambition. It does not specify a strategy.
Before evaluating a chart, write down the investable universe, the selection and exit rules, when decisions occur, how orders are executed and what happens to uninvested cash. Define the comparison portfolio on the same dates, in the same currency, with consistent treatment of distributions and costs.
A useful specification also explains why the process might work. Is it seeking a return associated with a particular risk? Responding to persistent price behaviour? Improving how an investor manages cash needs? The explanation should exist before a search through dozens of settings produces an attractive chart.
AI has different roles here. Translating a fixed rule into code is different from proposing the rule. A model that reads news introduces another set of questions about what it knows. Keep those roles visible so that each can be checked appropriately.
Once the specification is written, ask someone else to interpret it. If two readers would place different trades, the ambiguity belongs in the research record. A working program may simply have made an unresolved choice on your behalf.
2 Follow one trade through time
Imagine an ETF closes at $100 on Friday and $110 on Monday. The strategy starts Monday in cash. It uses Monday’s closing price to calculate a buy signal, with the first permitted purchase at Tuesday’s open.
An incorrect backtest might assign Monday’s return to the newly selected ETF:
$110 ÷ $100 − 1 = 10%
On a $1,000 allocation, that creates $100 of apparent profit before the decision to buy existed. The strategy was still in cash during Monday’s move. Tuesday’s purchase price, and everything earned after it, remain unspecified.
This is look-ahead bias: using information before it was available. Trace the timestamps of the data, the signal, the order, and the assumed fill. Check which position earns each period’s return. Moving a signal column by one row can change the apparent economics of an entire strategy.
Language models introduce a related complication. Asking a model to behave as though it were 2018 does not remove knowledge acquired during training. If it identifies companies, interprets events, or selects securities using later information, an apparently historical exercise may contain future knowledge.
Glasserman and Lin examine this problem in stock return predictions based on GPT sentiment analysis. Their result is more nuanced than a blanket warning that AI inflates backtests: within the training window they studied, anonymized headlines performed better than the originals. General company knowledge could distract the model as well as introduce look-ahead concerns. [1]
Record the model version, prompts, retrieval sources, and the disclosed training period, where available. A stated knowledge cutoff helps describe the setup; it does not establish that every input is free of contamination.
3 Inspect the research history behind the winner
AI reduces the effort required to try another ETF universe, lookback period, or exit rule. The successful configuration can be presented in a paragraph. The unsuccessful attempts may disappear from view.
That matters because repeatedly selecting the best result from historical data increases the chance of selecting noise. Bailey and coauthors formalize this problem as backtest overfitting and explain why familiar validation methods can be unreliable in investment research. [2]
Keep a research log: what was tried, why it changed, and which data informed the decision. A final chart without that history leaves the reader unable to judge how much searching preceded it.
Reserve later observations for evaluation before using them to refine the process. Once a test period has helped choose a setting, its results are part of development. Renaming it “out of sample” does not restore its independence.
Also examine nearby settings. A broad region of reasonable outcomes is more reassuring than a single exceptional combination surrounded by poor results. But those neighbouring tests share data. They show sensitivity, not independent confirmation.
The practical aim is to make the search visible enough to assess whether the conclusion survives beyond the conditions that produced it.
4 Calculate what implementation asks of the investor
A backtest must describe a portfolio an investor could have held.
Check whether the securities existed throughout the period and whether discontinued funds have disappeared from the dataset. Mark any substitute used before a fund’s inception. Establish whether prices include distributions and whether fund expenses are already reflected, so they are not deducted twice.
Then account for commissions, bid and ask spreads, slippage, and any financing costs relevant to the strategy. A rule based on the closing price needs an execution assumption consistent with when that closing price becomes known.
Taxes require a separate specification. Account type, jurisdiction, realized gains, losses, and the timing of sales can change the comparison. If the question is how much capital remains after both portfolios are sold, apply that final sale consistently. If the comparison ends without liquidation, keep the remaining tax liabilities visible rather than treating deferred tax as money already available to spend.
Arnott, Harvey and Markowitz’s backtesting protocol addresses the broader discipline of quantitative research, including the risks of repeatedly extracting apparent discoveries from limited financial data. Its relevance here is methodological; it is not a validation of a particular chatbot or ETF strategy. [3]
For the investor, this check turns an attractive percentage into a more useful question: what capital, trading activity and implementation assumptions are required to obtain it?
5 Separate reproducibility from evidence of an edge
Reproducing a calculation is necessary. Two implementations can nevertheless share the same flawed timing assumption or contaminated dataset.
Preserve the data snapshot, code, assumptions, and relevant AI prompts so that another person can rebuild the result. Ask the reviewer to inspect the accounting and the research choices as well as the arithmetic.
Then record future decisions before their outcomes are known. Keep historical tests, timestamped strategy records, and actual account results clearly identified. They answer different questions. A personal investment confirms that money is at risk; the evidence attached to a published return still needs to specify how that return was measured.
Prospective observation reduces the scope to rewrite decisions after the event. It does not make a short favourable period conclusive, especially when a rising market could explain much of the result.
Kong and coauthors’ position paper argues that financial evaluations of language models should explicitly address biases such as look-ahead effects, survivorship, and omitted trading costs. Its proposed framework helps organize scrutiny; it is not a certificate that a strategy works. [4]
For someone funding withdrawals, the next question is personal: could the portfolio and spending plan tolerate a longer or deeper setback than the test contains? Even a correctly calculated historical result cannot settle that on its own.
Before the next chart
Ask these five questions before allowing an AI-generated strategy to influence an investment decision:
Can I state the investment rationale, data sources, and comparison assumptions before looking at the result?
Can I trace one trade from available information to an executable order and the correct return period?
Could the model, dataset or security selection contain information unavailable at the time?
Can I see the unsuccessful research choices and identify which observations were genuinely reserved for evaluation?
Can another person reproduce the accounting, and what evidence exists beyond the development sample?
These questions make an investment claim inspectable. Passing them does not guarantee an advantage, but failing them identifies work that needs doing before the chart becomes a portfolio decision.
AI can help write tests, expose inconsistencies, and document assumptions. Used that way, it can strengthen the research process. The investor still has to decide whether the evidence is sufficient for the money involved.
About the author
Dr. Kay Dirting is a physician and holds US and European patents covering the use of hyaluronic acid in antiviral treatment. He invests his own capital using systematic ETF rules and writes OnyxValue , where he examines how investment strategies can be tested and implemented. The publication provides monthly ETF selections, risk updates, and dated results reports.
Further reading from OnyxValue
A Good Backtest Was Only the Beginning
The 15-Minute-a-Month ETF System Behind the Result
Sources
[1] Glasserman, P., and Lin, C. (2023). Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis. arXiv:2309.17322.
https://arxiv.org/abs/2309.17322
[2] Bailey, D. H., Borwein, J. M., López de Prado, M., and Zhu, Q. J. (2015). The Probability of Backtest Overfitting. Author manuscript dated February 27, 2015.
https://www.davidhbailey.com/dhbpapers/backtest-prob.pdf
[3] Arnott, R., Harvey, C. R., and Markowitz, H. (2019). A Backtesting Protocol in the Era of Machine Learning. The Journal of Financial Data Science, 1(1), 64–74. Link to the October 2018 working paper version.
https://people.duke.edu/~charvey/Research/Published_Papers/SSRN-id3275654.pdf
[4] Kong, Y., et al. (2026). Position: Evaluating LLMs in Finance Requires Explicit Bias Consideration. Proceedings of Machine Learning Research, 306, 171089–171110.
https://proceedings.mlr.press/v306/kong26i.html
This article is educational and does not provide individualized investment or tax advice. The numerical illustration is hypothetical. Historical and simulated results do not guarantee future investment outcomes.
Thank you, Meng, for welcoming this contribution. If you’ve used AI to build or assess an investment strategy, what was hardest to verify before you felt comfortable relying on it? I’d be especially interested in any difficulty tracing a signal through to a trade that could actually have been executed.
Onyx