Ruined by Hypothesis Testing [Premium Articles]

Hypothesis testing is often a naive way of assessing the significance of trading strategies because results are conditioned on historical data and can be plagued by multiple comparisons. This is the story of Don, an ambitious fund manager who got into trouble after applying hypothesis testing to a trend-following strategy.

For access to premium content, you must be a subscriber. Please login if you are already a subscriber or subscribe to continue reading...

This entry was posted in Premium Content and tagged , , . Bookmark the permalink.

11 Responses to Ruined by Hypothesis Testing [Premium Articles]

  1. Maverick says:

    Dear Mike,
    Isn't your price action system based strictly on data mining too? Or maybe put it differently, pattern mining?

    • Dear Maverick,

      Everything is data-mining. Even your brain does data-mining when you are thinking about a system or anything else. There is no escape from data-mining. Note this carefully.

      Unlike other software packages, mine includes warnings about data-mining and suggestions:

      http://www.priceactionlab.com/Literature/pal70manual/pal70manual.html#process69

      My blog contains many articles about this problem and suggestions.

      One objective of this article was to show that hypothesis testing cannot solve data-mining problems.

      Take care and good luck to you.

  2. Stock-Chartist says:

    Part of the fallacy in your argument is that you ended the test on 12/2012.

    If you had let the data run to current and beyond, they would have recovered much of the draw down and, given the recent market downturn and the correction that's likely to continue, the trading system would have worked.

    The 2011 correction was unusually steep and swift and caught everyone off guard both in exiting and re-entering.

    • dmicross says:

      Sorry but your thinking is fallacious.Harris has passed all tests of fallacious thinking. The test ended on 12/2012 because the 20% or so loss and 35% maximum drawdown was more than the investors could withstand.

      Like most chartists you think based on hindsight. On 12/2012 no one knew the future. You know now but this did not help the fictional character Don.

      Edit

      Sorry Mike for the attack on your visitor but I cannot stand these chart guys.

      Adrian

    • What fallacy and what argument?

      This is analysis Stock-Chartist. There is no fallacy or argument. If they let the system trade it would have broken even by the end of April 2013. The issue is that they did not expect a 34% drawdown and a 20% loss. You know now based on hindsight of the recovery but what about if the Fed did not institute QE2 or QE3? The fact of the matter is that the system failed to work as expected and there was no point in assuming more risk with it or that it would recover. Usually in professional trading there is a cumulative stop-loss of about 20%. Small account traders are not constrained by this as they can ruin their accounts to zero without any repercussions. Usually funds establish a maximum loss after which they are liquidated.

      I suggest you make a step beyond hindsight.

    • "given the recent market downturn and the correction that's likely to continue, the trading system would have worked."

      Do you get all that from charts? 🙂

      Seriously, do you think that would erase the 34% drawdown? What about if we get action similar to the action in 2011? I do not have a crystal ball.

  3. Bill says:

    Excellent article Mike. Thanks for sharing your insight with us.

    Bill

  4. Emlyn Flint says:

    Hi,

    Interesting article but one which I think unfairly judges hypothesis testing as a poor inferential tool. Assume that a strategy will only be traded if one thinks that it has a reasonable chance of making money, assuming some acceptable risk and ruin probability boundaries (amongst other criteria). Thus, the decision to go with a particular strategy is thus based on an hypothesis formed about that strategy's validity. Seen this way, hypothesis testing is a useful and well-suited tool.

    I don't think a single example of a very specific hypothesis test is enough to debunk the entire platform. To my mind, it speaks more to the importance of formulating the correct hypothesis, testing in a suitable manner for the problem at hand and finally, understanding what the output is telling you and knowing the inferential limits thereon.

    Multiple testing can be handled efficiently, the sampling issue is valid but can also be controlled for in many ways, and although structural breaks as you have highlighted above are clearly an issue given that they significantly affect the underlying market distribution, their effect is also to some extent controllable or at least understandable and therefore able to be modelled.

    Given that your hypothesis test is using a data mined strategy, what you are actually analysing is the statistical power of a specific testing formulation. Smarter choice of testing hypothesis coupled with techniques to minimise the type of error that most needs to be controlled will go a long way to removing the above issues.

    Hypothesis testing is routinely and successfully used in demography, ecology and environmental studies; systems that I would contend can be and usually are far more complex than financial markets. If done properly, there is no ex ante reason that it can not work just as well with trading systems.

    Regards,
    Emlyn

    • Hello Emlyn,

      This was not a paper for a peer-reviewed journal but only a blog to show the perils of using hypothesis testing with curve-fitted systems through a simple example.

      Hypothesis testing may be successful in many fields but the fact is that the population parameters there do not change much as it is the case with the markets.

      Please note that the 1987 crash was a 20-sigma event that was a signal of a big change in the markets and also an outcome of big changes. The practical question is: is the sampling distribution standard deviation realistic? If not, the hypothesis test is useless and even misleading. Can you assume that the sampling distribution variance will remain constant?

      More importantly, how do longer-term averages relate to short-term moves? In my story, Don got ruined before the system had a chance to recover. In other fields you do not deal with limited capital, leverage, stop-losses and related risks. You just make inferences about population parameters from samples and the financial risks are not high.

      There is also a lot of controversy in social science research about the value of hypothesis tests. This is far and beyond the scope of short blogs that only present specific examples.

      "Given that your hypothesis test is using a data mined strategy, what you are actually analysing is the statistical power of a specific testing formulation. Smarter choice of testing hypothesis coupled with techniques to minimise the type of error that most needs to be controlled will go a long way to removing the above issues."

      Yes, I agree and it is one of the points made in the blog. However, after making all corrections for errors you will find out that the required significance level is prohibitive in most cases.

      Best.

Leave a Reply