Out-Of-Sample Performance Variations

In this article we look at variations between past and recent results generated by the Quantopian platform for the same out-of-sample period of a specific strategy . Despite the discrepancies, our strategy outperforms the benchmark since it joined the contest on August 24, 2016, by about 12%.

The DIAT4S3 strategy was developed by an earlier version of Deep Learning Price Action Lab (DLPAL) software on April 2013, with in-sample data from DIA inception to 12/31/2009. Below is the output of the program:

Below is the out-of-sample performance of the strategy from 01/04/2010 to 08/16/2016, as determine by a Quantopian backtest on 1-minute data on August 16, 2016:

The return is 147.5%, Sharpe is 1.78, maximum drawdown is 19.4% and alpha is 0.17. The strategy outperformed SPY buy and hold by more than 25% in the out-of-sample period from 01/04/2010 to 08/16/2016.

We went back to Quantopian a few days ago to test again the performance of the above strategy in the same out-of-sample period from 01/04/2010 to 08/16/2016. This is the result:

To our surprise, the performance was significantly different although still positive. Specifically, the return, Sharpe and alpha were lower. Drawdown is a little lower by about 1%. Below is a comparison table:

Parameter Old backtest New backtest
Return 147.5% 128.07%
Alpha 0.17 0.10
Sharpe 1.78 1.12
Max. DD -19.4% -18.12%
Benchmark 121.7% 121.7%

There are large discrepancies between the old and new backtests. Sharpe is lower by nearly 40% although the benchmark return remains the same, which probably means that the differences are not due to data adjustments but due to some internal execution simulation algorithm. Is this algorithm realistic or it underestimates strategy returns? I obviously have no idea and only Quantopian hopefully knows the answer.

By the way, a modified long/short version of this strategy that also trades TLT short when it takes a long position in DIA, has outperformed SPY total return since it joined the Quantopian contest, on August 24, 2016:

Return for the modified strategy is 37.44% versus 25.52% for SPY as of 03/26/2018. We are certainly proud of this strategy and DLPAL S. We are not aware of any other vendor of software that automatically discovers trading strategy that has put any of them under scrutiny by an independent and respectable platform such as Quantopian.

Finally, Quantopian is a free platform and there is always possibility that its developers implement/update certain standards that may affect previous results. When using any platform or software, trading strategy developers should always try to understand its advantages and limitations.


You can download a demo of DLPAL S from this link. For more articles about DLPAL S click here.

If you have any questions or comments, happy to connect on Twitter: @priceactionlab

Subscribe via RSS or Email, or follow us on Twitter.

Disclaimer:  No part of the analysis in this blog constitutes a trade recommendation. The past performance of any trading system or methodology is not necessarily indicative of future results. Read the full disclaimer here.

Copyright Notice

This entry was posted in Trading Strategies and tagged , , , , . Bookmark the permalink.

Leave a Reply