Price Action Lab Blog

Premium Market Analysis

Quantitative trading

Two Empirical Rules About Backtesting

Most backtests in financial blogosphere and even academic papers are wrong or ambiguous not only due to data-mining bias and over-fitting but also due to errors. Backtesting correctly requires understanding of data, software and markets. Knowing how to program is necessary but not sufficient for generating realistic backtests.

A few years ago I joined a website that provided curated links to quant articles. Soon my website was ranked in top ten contributors. This gave me the opportunity to get exposed to a large volume of work by various other authors. To make a long story short, more than 60% or even close to 80% of backtests results were either due to data-mining bias or wrong application of backtesting. Some were even hopelessly wrong, for example a backtest of a momentum strategy plagued by look-ahead bias by a hedge fund manager who actually presents himself as an expert and trades OPM.

After I started publishing articles trying to correct some of the mistakes, the hate emails kept coming in. So much hate it was hard to pack in a few lines of email. The peak in hate came with an article by a failed fund manager who promotes himself as a trend-following expert where he insulted me for just repeating what a very successful hedge fund manager has already said several times. I soon realized that this was mostly a group of losers that wanted to take advantage of the work of a few competent individuals to get exposure. I withdrew and immediately the hate emails stopped. This was one of the most unpleasant experiences I have ever had.

Occasionally, one of my followers retweets some articles posted by that link aggregator website and I see the same errors repeated by some wannabes. A few days ago there was an article of a hypothetical edge in S&P 500 futures when in fact there is no edge and the results are due to misunderstanding of data.

In a nutshell, backtests can be wrong or misleading due to the following:

  1. Data-mining bias: many alternative hypotheses are considered until one is found that satisfies some objectives. Validation is useless if the out-of-sample is used repeatedly because it has become part of the in-sample.
  2. Data issues: If the data is not clean or adjustments are made, as in split-adjusted, dividend adjusted, continuous adjusted futures (Panama method), etc, some edges may disappear and some new edges that are in reality flukes may appear depending on how the strategy logic treats the data. We are not going in details here but a whole book can be written about this subject. Unfortunately, naive quants and wannabes do not understand even 1% of this.
  3. Look-ahead bias: This is not as uncommon as one may assume given the fact that it went unnoticed even in academic publications for a long time.
  4. Errors and assumptions in backtesting programs: In 1990s, two of the most popular backtesters had flaws that affected a large class of trading strategies. I documented them in detail but I never published them for obvious reasons. In addition, some backtesters make some assumptions about entries, exits, etc. Those are usually not understood because almost no one reads manuals.
  5. Use of statistical or general purpose languages can easily lead to mistakes. Examples are R and Python. Backtesting goes beyond programming and requires understanding of markets. Anyone who thinks that backtesting is the job of a programmer without any other knowledge and experience about the markets may pay a price. A quant must first learn markets well and have skin-in-the-game and then start backtesting. Trying to learn markets through backtesting usually does not end well.

Rule 1: If you assume a backtest is wrong, the probability that you are correct is about 70%. It is a reasonable assumption to make and not paying attention to backtests is sound protection over the longer-term.

Rule 2: Always be skeptical of too good to be true results. Usually any annualized return in excess of 12% to 13% is either suspicious or ignores significant risks by applying hidden assumptions and filters. There is no free lunch in the markets.

If you found this article interesting, I invite you follow this blog via any of these methods: RSS or Email, or follow us on Twitter

If you have any questions or comments, happy to connect on Twitter: @mikeharrisNY

Charting and backtesting program: Amibroker


Quantitative analysis of Dow-30 stocks and 30 popular ETFs is included in our Weekly Premium Report. Market signals for longer-term traders are offered by our premium Market Signals service. Mean-reversion signals for short-term SPY traders are provided in our Mean Reversion report.

Copyright Notice

Leave a Reply

Theme by Anders Norén