Below is a diagram of the general use of DLPAL DQ
The training and testing step uses the data and parameters provided by the user to identify strategies and generate signals. The program generates code for the strategies for use with popular trading platforms.
The next step involves the validation of the strategies and signals to minimize data-mining bias for the purpose of in turn minimizing the probability of a Type-I error (false discovery). This bias arises from various sources but primarily from over-fitting, selection bias and data-snooping bias.
Out-of-sample tests cannot be used with a scanner where the objective is to use the whole price history to find repeatable patterns. Other tests, such as Monte Carlo simulations, are not very effective in reducing Type-I error due to the multiple comparisons in machine learning processes. Multiple comparisons involve testing a large number of features and rejecting those that fail to contribute to performance while keeping those that improve performance. This process is plagued selection bias and data-snooping bias if data is used repeatedly to evaluate new results. In addition, when many features are involved in the identification of strategies, there is always danger of overfitting to noise.
DLPAL DQ allows testing for significance by validating the results using data from a portfolio of comparable securities, called portfolio backtesting. In addition, robustness tests may be used to evaluate the significance of the results. DLPAL DQ provides tools for robustness analysis of the identified strategies.
It should be clear from the above diagram that validation is a user task. Validation is both an art and a science. Any claims that validation can be accomplished in automated mode are motivated by a lack of understanding of the complexity of this process. As a result, DLPAL DQ does not may any claims of identifying “robust strategies” for use in trading. This is a task of the user. DLPAL DQ provides the machine learning and analysis tools only to generate candidate strategies for validation by the user. Part of a trading edge comes from a robust process of identifying anomalies in price series and part comes from the trader ability to validate the results.