Below is a diagram of the general use of DLPAL
The training and testing step uses the data and parameters provided by the user to identify strategies and systems of strategies. The program generates code for the strategies and systems internally and for use with popular trading platforms.
The next step involves the validation of the output to minimize data-mining bias for the purpose of in turn minimizing the probability of a Type-I error (false discovery). This bias arises from various sources but primarily from over-fitting, selection bias and data-snooping bias.
Out-of-sample tests are not very effective in reducing Type-I error due to the multiple comparisons in machine learning processes. Multiple comparisons involve testing a large number of features and rejecting those that fail to contribute to performance while keeping those that improve performance. This process has selection bias and data-snooping bias if the out-of-sample is used repeatedly to evaluate new results. In addition, when many features are involved in the development of strategies and systems, there is always danger of overfitting to noise.
Although DLPAL provides tools the user can use to validate results out-of-sample, it is recommended instead to use the out-of-sample in the train and testing phase and validate the results using data from a portfolio of comparable securities.
If out-of-sample tests are desirable, Robustness tests on out-of-sample performance may be used to evaluate the significance of the results. DLPAL provides tools for Robustness analysis of the identified strategies.
It should be clear from the above diagram that validation is a user task. Validation is both an art and a science. Any claims that validation can be accomplished in automated mode are motivated by a lack of understanding of the complexity of this process. As a result, DLPAL does not may any claims of identifying “final strategies” for use in trading. This is a task of the user. DLPAL provides the machine learning and analysis tools only to generate candidate strategies for validation by the user. Part of a trading edge comes from a robust process of identifying anomalies in price series and part comes from the trader ability to validate the results.