Developing trading strategies is a tedious and time consuming task with a low success rate. Significance of results often cannot be inferred from historical tests and validation samples due to data snooping, especially when many trials are involved. As a result, one way of testing strategies is with real money. This is what your hedge fund may be doing.
Let us start with a simple question: How is trading strategy significance determined? One practical answer is via the use of Calmar ratio (annualized return divided by maximum drawdown); if the strategy delivers sufficiently high risk-adjusted returns when compared to the performance of a benchmark, then it is a significant result.
For example, Calmar for SPY since inception based on total return is about 0.16 (annualized return is about 8.9% and maximum drawdown is close to 55.2%.) If a hedge fund delivers a higher Calmar ratio for a period of about 3-5 years, then this is significant performance. Obviously, a Calmar ratio of 0.4, could be achieved by a strategy with lower drawdown and annualized return, for example 6% annualized return and 15% maximum drawdown. Some investors look only at the absolute return and ignore the drawdown. They should answer this question: Would they have stayed invested in the market after a 55% drawdown to realize the high long-term return or would have they sold everything and register a big loss? This is where an investor should be honest.
A fundamental problem of trading strategy developers is that it takes time and money to test a strategy live. This is simply not practical and as a result a theoretical measure of significance is required to serve as a filter of candidate strategies.
Next, we consider the theoretical answer to significance. A strategy is usually considered significant when it offers a good forecast of future returns. But this is a very abstract notion of significance and it has to be quantified. One approach is via the use statistics. Statistics allow us to make inferences about population parameters from samples. This is the main goal of this art and science. Actually it is more of an art than a science. How can this apply to theoretical testing of significance of trading strategies?
A simple test is to try to find evidence against the null hypothesis that the returns of the strategy come from a population with zero mean. But all we have is a sample of historical data. What complicates this task is the possibility that the sample is not representative of the population because the test did not include a wide range of market conditions. Then, at another complication level, the sample may be the result of data snooping bias. This bias arises from the dangerous practice of reusing historical data to test alternative hypotheses with a large number of predictors, also known as features in machine learning.
Machine learning has gained again popularity but its effectiveness depends on the quality of features used and its results may be plagued by data-snooping bias because it involves testing many alternative hypotheses with different combinations of features and different classifiers. It is exceptionally hard to test the null hypothesis with results derived from machine learning. Several methods have been proposed for adjusting statistics for multiple trials but risk of false rejections and false positives is still too high.
Machine learning can be very useful and successful in identifying the best alternatives among a group of good options but if the options are not good (economically significant) the results are usually random and over-fitted to noise. A degree in Computer Science is not sufficient to develop significant trading strategies. In-depth understanding of markets and trading is also required. When volatility returns to the markets, some quant funds may face large losses if their strategies fail to adjust to abrupt changes in price action because they were developed with data from a period of low volatility.
The conclusion is that determining theoretical significance is a difficult task and the probability of a false positive (Type-I) error is always high not only due to the fact that the signal to noise ratio in the markets is low but primarily due to multiple trials and data snooping bias.
As a result, it is possible that your hedge fund is testing its strategies live with your money. Here is a list of questions you could ask its management:
- Is the hedge fund using machine learning?
- Is there a claim of a quantitative approach to strategy development?
- Is there any reference to backtesting, historical testing or simulations?
- Has the fund been in operation for more than 5 years?
- Is the fund operated by experienced traders or only by tech enthusiasts?
Note that not all funds that use backtesting and machine learning are in the process of testing random strategies with your money but it is always good to keep that possibility in mind. I hope this brief article will help but professional help will come from your financial adviser. There are good, well-informed, financial advisers that can guide investors in this area and help them understand the risks of investing in quantitative hedge funds.
If you have any questions or comments, happy to connect on Twitter: @mikeharrisNY
Technical and quantitative analysis of Dow-30 stocks and 30 popular ETFs is included in our Weekly Premium Report. Market signals for longer-term traders are offered by our premium Market Signals service. Mean-reversion signals for short-term SPY traders are provided in our Mean Reversion report.