Why Your Backtests Lie to You

Recently, someone sent me the results of a five-year backtest and asked how much faith to put in them.

The numbers looked reasonable: roughly 0.28R per trade, positive expectancy, and around 100 trades a year. No 100% win rate and no vertical equity curve.

My answer was that the result could be real. But something else caught my attention.

The author had tested different periods, widened the coin universe, added timeframes, and started building a regime detector. At Ronin, we have been down exactly the same road.

The more variations you run through the same history, the more likely you are to find a combination that happens to explain the past perfectly.

And that is only one way to fool yourself.

The other arrives even earlier: a simulator can show you trades that would never have existed in the real market.

First, ask whether those trades could have filled

Most people start with TradingView.

Drop in a few indicators, hit Strategy Tester, get a green curve. Fast and convenient.

TradingView is useful for checking a raw idea. But inside it is a broker emulator. On historical data it sees chart prices and infers how price moved within a bar.

Say an hourly candle touched both your stop and your take-profit. OHLC data cannot tell you which happened first. The emulator chooses a path according to its own rules.

Bar Magnifier adds lower-timeframe data and improves the estimate. It still cannot see your real queue position, the available size ahead of your order, or the state of the book at the time of execution. TradingView documents these limitations in its strategy documentation.

The central problem with limit orders is simple: touching a price does not mean getting filled.

There may have been $2 million resting ahead of you. $200,000 trades at the level, price reverses, and the simulator records your entire entry.

In the market, you would have had no position at all.

Partial fills have the same problem. There is $30,000 resting at a level, your order is $100,000, and the test fills the lot at a single price.

With market orders, slippage begins to matter. You consume several levels of the book, so the average fill price depends on position size and current depth.

Then add commissions, funding, and the delay between the signal and order acknowledgement. A small edge can disappear right there.

How we model this at Ronin

We collect Bybit data and work with tick trades and order-book updates.

Bybit sends a snapshot first, then streams delta updates over WebSocket. The messages carry a sequence number and matching-engine time, which lets us align book changes with the trade stream. The mechanism is documented in the Bybit order-book feed.

The working set stays in RAM. Redis is a hot cache, so each iteration does not reread the same data from disk.

The current stack is:

• Python 3.14 for research, statistics, EV calculations, and the risk module;
• Go for connectors, WebSocket, REST, and execution services;
• Redis for hot data and intermediate run state.

We built our own core to control the execution model. It needs to account for queue position, partial fills, dynamic slippage, fees, funding, and latency.

You cannot reconstruct an exact queue position without order-level data. Where the data does not resolve the answer, we use a conservative estimate. If several fill paths are possible, the engine does not pick the most favourable one.

Latency deserves the same treatment.

We are not doing HFT or trying to beat a market maker by one millisecond. We still model delay.

If moving from 1 ms to 50 ms destroys expectancy, the system depends on an ideal entry. If it holds up at 50, 100, and 200 ms, a server closer to the exchange will not materially change the result.

Bybit also converts a market order into an IOC limit order with a slippage guard. If there is not enough liquidity inside the allowed range, part of the order will not execute. That behaviour is in the exchange documentation.

So “I sent a market order” and “I received the full size at the best price” are not the same statement.

Assume we modelled execution correctly

We reconstructed the book, estimated the queue, and added fees, funding, latency, and partial fills.

Now the backtest honestly describes how the system would have traded then.

The remaining question is: how much did that market resemble today's?

The person in the message put the problem very accurately:

As soon as the narrative changes, I feel like a blind kitten. The setup still looks fine, but I cannot understand why the result is negative.

The setup on the chart still looks familiar. The conditions underneath it have changed.

Volatility changed. Depth changed. Stops began to behave differently. After liquidity is swept, price used to return; now it carries on. Different participants, different leverage, different capital flows.

The strategy keeps finding the old picture, but the old edge inside it is gone.

How a long backtest hides that problem

Imagine 1,000 trades.

The old regime contains 800 trades with an average result of +0.20R.

The current regime contains 200 trades with an average result of −0.35R.

The aggregate result remains positive:

0.8 × 0.20R + 0.2 × (−0.35R) = +0.09R

Over the full period, the system is profitable. Over its most recent 200 trades, it loses consistently.

If the next trades resemble the current regime, the historical +0.09R will not help. You will be trading a −0.35R process.

Across a long sample, the old profits simply outweigh the fresh losses. The curve stays green even though the system has stopped working.

That does not mean old data should be discarded. It means I do not want to blend different markets into one average number.

Five years of data is not automatically a good sample

At 100 trades a year, five years gives you about 500 trades. At first glance, that sounds adequate.

But 500 rows in a report do not necessarily mean 500 independent bets.

Suppose BTC is falling and the system opens shorts in 20 altcoins at once. The journal now shows 20 trades. In reality, all of them depend on the same Bitcoin move.

Widening the coin universe increased frequency, but added very little independent information.

Timeframes have the same issue. Signals on 1H and 4H can be expressions of one impulse. The backtest records two trades, even though the system placed the same bet twice.

Then there is survivorship bias.

If you take today's list of surviving coins and test it on history, the backtest already knows which projects survived earlier cycles. Dead and delisted projects will not be in the sample, and their potential losses disappear with them.

That is why I look beyond trade count. What matters is how many independent market situations the test contained, and which instruments were actually available at the time.

Then hypothesis shopping begins

Suppose you tested:

• 10 moving-average periods;
• 8 stop variations;
• 5 coin universes;
• 4 timeframes;
• 3 regime filters.

That is 4,800 combinations, before exits, targets, and position management even enter the picture.

Among 4,800 tests, a beautiful result is almost guaranteed to appear.

The problem is that the final Sharpe ratio displays only the winner. The other 4,799 attempts disappear from the report.

This is backtest overfitting and multiple testing. The more variations you try, the less you should trust the best result unless you adjust for the number of attempts.

Probability of Backtest Overfitting and the Deflated Sharpe Ratio exist for exactly this reason. They do not make a test perfect, but they at least recognise how many times the researcher tried to find a winner. Bailey and López de Prado offer a useful explanation in their article.

There is another important point: once you have looked at data, it is no longer new to you.

You see a drawdown and add a filter. You see too few trades and widen the universe. You dislike the 1H long setup and keep only the short.

Every decision may be reasonable. But the history has already influenced the system. The final result is no longer a fully out-of-sample test.

Bitcoin as an example of a regime shift

In 2017, the market was thinner. Retail participants, spot trading, forums, and ICOs played an outsized role.

By 2021, developed perpetual-futures markets, high leverage, and liquidation cascades had arrived.

In January 2024, the SEC approved the listing and trading of spot bitcoin ETP shares in the United States. By early 2026, ETPs and large public companies together held close to 12% of BTC supply. That is a different composition of capital. SEC, Fidelity

The ticker stayed the same. Its behaviour changed.

Bitcoin first exceeded its previous all-time high before the 2024 halving. In October 2025, price reached roughly $126,000. That is about 2x from the halving price, materially weaker than prior cycles.

BTC later lost more than 50% from its peak. Yet by July 5, 2026, nearly 15 million BTC had not moved for at least 155 days. The market has grown, the share held by long-term investors has risen, and volatility and cycle amplitude have declined. Glassnode, Fidelity

In timing, the four-year cycle roughly held: the peak came about a year and a half after the halving.

But the calendar said nothing about amplitude, liquidity, or who was participating.

If that happened to crypto's most familiar model, it can happen just as easily to our strategy.

Does that mean a shorter backtest is better

Sometimes. But not because it is shorter.

A short test is closer to the current regime. It contains fewer old conditions that no longer relate to today's market.

At the same time, a short sample is more exposed to randomness. With 20 trades, you can get an excellent result or a disaster without either proving a durable edge.

That is why I now separate the jobs.

Long history is for finding where the system broke. I look at crises, volatility shifts, widening spreads, rising slippage, and rare tail events.

A recent window tells us whether the system fits the current regime.

New data that did not participate in development is for validating the result.

I no longer see the point in collapsing all of that into one total return.

How we test now

First, we articulate where the EV comes from.

If the explanation ends with “the indicators lined up nicely,” the hypothesis is not ready. You need to understand whose behaviour creates the opportunity and why it survives fees.

Then we split history by time:

1. Train for development.
2. Validation for selecting parameters.
3. Test, left untouched until the system is fixed.

Then we run walk-forward tests. We tune on the previous window, freeze the parameters, test the next segment, shift the window, and repeat.

We also measure results by market regime.

For the regime itself, we monitor realised volatility, trend character, spread, depth, slippage, funding, basis, open interest, and rolling correlations.

A regime detector does not need to catch the exact reversal candle. I do not ask it to. It needs to notice that conditions have left the working range and reduce risk.

We keep every run, not just the winning configuration. If one point works while nearby parameters lose money, I consider the system fragile.

Then comes shadow trading and the smallest real size. We compare model price with actual fills, slippage, fill quality, and live expectancy.

Only a forward test can reveal what the developer has not already seen.

Why I rebuilt Ronin

At the end of July, I rebuilt the system completely.

The old version worked. Then the market changed its behaviour, and the old logic started losing trade after trade.

I deliberately tested the period containing my own drawdown. I wanted to learn precisely where the edge had disappeared: volatility, market structure, execution, or the entry logic itself.

But once I changed the system, that period could no longer count as an independent test. I had seen it and used it for tuning.

So the rules of the new version are fixed. The next trades must supply the evidence.

Not an old green curve. New data.

The takeaway

When someone shows me a backtest now, I look at four things first:

• could those trades actually have filled;
• how many hypotheses were tried before the result appeared;
• which market regime generated most of the profit;
• what happened on data the developer had not seen.

Total return comes after that.

A long test shows where a system broke. A recent test shows whether it fits the current market. Forward testing checks whether the model matches real execution.

You cannot find a working strategy once and trade it for years without revisiting it. The market changes its participants, liquidity, and character of movement.

A system must know more than when to enter.

It must understand when its edge has disappeared.

This material is educational and not investment advice.