Premium Market Analysis

Quantitative trading, Trader education

Perils of Using Exchange Traded Products in Backtests

Backtests with Exchange Traded Products (ETPs) can be misleading because these securities were developed after the stock market crash of 1987 and the bulk of them after the dot com bear market.

Due to small samples and lack of exposure to extreme market conditions, backtest results using ETP historical data may not reflect potential risks from market left tail events.

ETPs go back to January 1993 with the development of the successful and popular  SPDR S&P 500 exchange traded fund from State Street Global Advisors. But most ETPs were developed after the dot com bear market top and historical data do not include the risks traders faced in 1987 crash, 1998 (Rubble crisis and LTCM bailout.) and dot com bear market.

Most quants understand these issues with ETPs but there is a large volume of publications with examples of strategies and portfolios using ETPS that mainly target investors and retail traders who may not be familiar with the perils of backtesting. But a picture is worth a thousand words.

Below is a backtest of a strategy that buys SPY at the close after three consecutive losing days with a combined loss of 6% or more. The strategy exits at the next close (Financial blogosphere was overloaded with similar backtests in the last two days.)

The equity line above shows a winning strategy with just a temporary setback in 2008 with a loss of about 7% that was immediately followed by a gain of 14%.

Win rate is about 64%  although the sample of trades is small at 22.

Next, instead of SPY we use S&P 500 index.  The results are even more hypothetical now because the index is not tradable but we are not interested in exact performance but in analyzing the risks of the strategy by increasing the sample to include data since 1940.

It may be seen from the equity line that this strategy got caught in the 1987 crash. Specifically, the strategy lost 20.5% in one trade due to the crash although the win rate increased to 70% overall and the sample from 22 to 40 trades.

For most portfolios presented in books and articles using ETPs it is not trivial task to extend the sample size; it can be done but the process is often complicated.

Charting and backtesting program: Amibroker

Data provider: Norgate Data

Technical and quantitative analysis of major stock indexes and 34 popular ETFs are included in our Weekly Premium Reports. Market signals for position traders are offered by our premium Market Signals service

If you found this article interesting, you may follow this blog via RSS or Email, or in Twitter