The SPY2p5 system was machined designed on April 4, 2012 by Price Action Lab using an in-sample from SPY inception (01/29/1993) to December 31, 2008. Then the system was validated on unseen data, called the the out-of-sample, from January 2, 2009 to April 5, 2012 and the results were published in this blog two days later. By the end of last year the SPY2p5 system had outperformed the ETF in forward trading. This post poresents a study of the significance of this trading system.
The steps for using Price Action Lab to design the SPY2p5 systems were outlined in the April 7, 2012 post. The same results will be always generated by the program when using the same in-sample data. Price Action Lab is based on a proprietary algorithm that produces the same output each time it encounters the same conditions and this determinism is in compliance with the standards of scientific testing and analysis. This is very important because a high percentage of claims in the trading blogosphere cannot be reproduced by anyone.
Below is the equity curve of the Amibroker backtest of the SPY2p5 system from the ETF inception to the close of yesterday (05/10/2013):
The in-sample, out-of-sample and forward testing periods are indicated on the above chart. Note that in the combined period the SPY buy and hold return with dividends included was 8.66% and the maximum drawdown was about -55.20%.
Summary of SPY2p5 system performance:
|Avg. bars in trade||6.63|
Is the SPY2p5 system better than random?
This is an important question and a positive result answers partly the significance issue. We are interested to know whether the performance of this system was obtained by chance and in reality the system has no intelligence in pairing market returns with its own trading signals. In order to answer this question we will run a simulation of 10,000 random system trading SPY. The systems generate signals by flipping a coin at the first bar that it is flat. If heads come up, a long position is initiated at the next open and held for 2.5% profit target or stop-loss, the same as the original system. If tails come up, a short position is initiated at the next open and held for 2.5 profit target or stop-loss. Long and short signals do not interact to keep the profit target and stop-loss in effect. Essentially, we are testing to see whether the entry part of the system that was generated by Price Action Lab has no intelligence and the performance is basically due to the exit part, as it is often the case with moving average crossover systems for trend-following. Below is the distribution of the Compound Annual Return CAR for the random system:
It may be seen that no random system generated a CAR above 10% and the SPY2p5 system CAR of 17.81% is about 6 standard deviations away from the mean of the above distribution, something that makes it a highly unlikely event for a random system, a sort of a black swan.
(1) If the coin is biased to generate a higher proportion of long trades as in the original system, the maximum CAR from the simulation rises to about 12% but still far below of the system CAR performance.
(2) If instead of a profit target and stop-loss the random system is let to exit and reverse positions, the maximum CAR of the simulation increases to less than 16% as was shown in another post. The resulting distribution is shown below for 20,000 simulation runs:
The maximum CAR of the above distribution is still much below that of the SPY2p5 system.
Is the SPY2p5 system curve-fitted?
The analysis just presented applies for rejecting trading systems and not for declaring them acceptable meaning, that if a system appears no intelligent it should be discarded but if it does not fail the test (null hypothesis rejected) it could be that it was created by a very intelligent curve-fitting mechanism. This distinction has not been made clear in the trading literature as far as I know and often random simulation tests are used to rule out a curve-fit, something that cannot be done by that type of analysis. Price Action Lab minimizes in advance the adverse effects of curve-fitting by generating TYPE-III systems only, i.e. systems whose entry signals cannot be fitted by changing parameter values but only the exits can be changed. Regardless, this issue must be investigated and portfolio backtests may provide the answer if properly applied. In other words, the system must also show positive performance in forward tests for a variety of correlated and non-correlated symbols. This type of study was already performed in the case of the SPY2p5 system for the period 04/09/2012 to 03/15/2013 in this post. Since inception, the SPY2p5 system has been profitable in several ETFs, some of which are listed below, including a marginally positive performance in TLT, an ETF that is highly anti-correlated with SPY:
No curve-fitting and an intelligent system may still not be enough
Even if the system is intelligent and it is not curve-fitted it can fail spectacularly. This is a reality that makes system trading hard. The failure can be due to market conditions that were not encountered in any of the tests. This is why a comprehensive portfolio backtest is essential to account for as many conditions as possible. The market always tries to defeat all algorithms that try to profit and system traders should understand that system development and improvement should be a daily task if they want to succeed. In this respect, tools like Price Action Lab that create systems in a deterministic way without utilizing data snooping and while minimizing data-mining bias and curve-fitting are quite useful but eventual success depends on the skills and work put by the user. Anyone who claims to have a program that can create automatically final systems that guarantee profitability under actual trading conditions and sells it to the public is in the best case economically non-rational (if not a crook).
Disclosure: no relevant position at the time of this post and no plans to initiate any positions within the next 72 hours..
Charts created with AmiBroker – advanced charting and technical analysis software. http://www.amibroker.com/”