Cross-validation is a method for establishing criteria for model selection. In its simplest form, it is based on dividing the data sample into two subsamples: the development sample and the validation sample. In practice, proportions of 75% and 25% are usually used for the development and the validation samples. This type of tests are also known as out_of_sample tests and provide a means of identifying significant patterns and minimizing data snooping bias. In this study we will consider patterns from the search results for SPY ETF for 2% profit target and 2% stop loss. The development sample is daily data from 01/02/1997 to 12/29/2006. The validation sample is also daily data from 01/03/2007 to 11/17/2010. It can be seen that the development sample covers 10 years worth of daily data and the validation sample about 5 years worth of daily data and they are close to a 67% to 33% proportion.
Price Action Lab discovered 106 patterns with Profit Factor > 1.5 as shown in the figure below. The patterns are sorted according to their number of trades
Each line in the results corresponds to a price pattern that satisfies the performance parameters specified by the user. P is success rate of the pattern, PF is the profit factor, Trades is the number of historical trades, CL is the maximum number of consecutive losers, Type is LONG for long patterns and SHORT for short patterns , Target is the profit target, Stop is the stop-loss and C indicates whether % or points for the exits. Last Date and First Date are the last and first historical data file.
Example 1: Testing of all patterns from the development phase on out_of_sample data and then selecting best performers according to some criteria.
Price Action Lab offers the capability of testing all the patterns in the results with a mouse click. This is accomplished by the "Test Patterns" option:
The results of the test are shown below (sorted for highest win rate P):
From the above results, one may select patterns that have profit factor > a, a > 1, while at the same time possibly satisfying other criteria, like for example Trades > T, T = 20, etc. For example, below are the results for patterns with PF > 2:
Example 2: Select patterns from the development step and then use the "Test Patterns" option to determine out_of_sample performance:
In the following screenshot, patterns with more that 50 trades were selected from the original results (development sample):
Next, the selected patterns are tested on out_of_sample data to determine their performance:
It may be seen that only 4 out of the 17 patterns have a profit factor less than 1, i.e. they are not profitable. The net profit (in points) can be found after backtesting each pattern and it is equal to 74.9264. In the same period, SPY buy-and-hold gain was 118.22 -141.37 =
- 23.15 points. Thus, the buy-and-hold performance was negative.
