Price Action Lab Blog

Premium Market Analysis

Market Statistics

On Investment Guru Accuracy

In a post yesterday with the title “Gurus Achieve An Astounding 47.4% Accuracy!”, Rick Ferri discusses results about the accuracy of 68 investment gurus in a study by the CXO Advisory Group. Contrary to claims made about the significance of the results, I argue that a better assessment should take into account the reward/risk profile of each guru. I argue that the 47.4% result is not that bad when viewed in the proper context.

I follow Rick Ferri and his blog although he is a proponent of passive investing and I am a short-term quant trader. Rick’s investment approach has great value for a large class of investors who cannot and actually should not try to analyze the markets and make their own decisions but they should rely instead on the performance of a low cost passive index funds. Thus, I agree with the notion that some people should not even try to time their investment decisions. However, it is true that even the decision to invest passively or add to a passive investment implicitly involves timing. Obviously, those who added to their passive investments at the bottom of the market in 2009 have done a lot better that those who did near the top of 2007. Therefore, as far as I am concerned, there is always a timing component in all passive investments. However, I understand and I respect the arguments of the passive indexing crowd and I believe they are valid for a large class of investors.

In his recent post about guru accuracy Rick Ferri wrote:

The results are in and they are bad. After tracking 68 experts and 6,582 market forecasts, CXO Advisory Group has concluded that the average market prediction offered by experts has been below 50% accuracy. Flip a coin and your odds for predicting the market are better. It’s hard to imagine that the average market expert isn’t able to at least match the track record of a coin flip, but it’s true. Figure 1 has, by name, the relative performance accuracy of every guru that CXO Guru Grades has tracked.

To start with, there is no such thing as an average market expert. These are descriptive statistics and an average of reality, like there is no average family with 1.62 kids. Take for example the futures market: the average gain is exactly zero on closed contracts because this is a zero-sum game by definition (in reality it is a slightly negative sum game when trading friction is added). Can we claim based on that that investments in commodity futures are worthless?

Actually, there are several CTAs with many years of profitability. If one averages all futures CTAs the result should be a mean return of close to zero. But this is a descriptive statistic that reveals the nature of the game and not the potential of this asset class or the ability of some CTAs to perform better than others. Descriptive statistics measure some value of a random variable based on assumptions about the values that this variable takes.

The above is one reason that a comparison to a coin toss in the case of gurus is not relevant. When one flips a coin the outcome of this experiment can be either heads or tails, but not both. If the coin is fair we know that the frequency of heads will be equal to that of tails after a sufficient large number of tosses. However, these are the frequencies of the events and they are equal by definition as there is some circularity involved. The statistics here just reflect the fact that the coin is fair and the frequency of heads and tails will be 50%. But when we assign a random variable X to the events, then we have a different game altogether. If when heads shows up the payout is $2 and when tails shows up it is -$1 then this game has a positive longer-term expectation of $0.5 per toss even if the probability of heads is 50%. In a similar way, in the markets we should not only be looking at the win rate of forecasts but also at the values of the random variables that are assigned to them. For example, a given guru can be 50% correct in the longer-term but he makes twice as much as he losses on each call. Overall, this guru is profitable in the longer-term (although he may be ruined in the meantime due to volatility of returns but this is another chapter) with a profit factor (sum of winners divided by sum of losers) of 2, which denotes a good performance.

Below is a copy of an email I sent to Rick Ferri:

Actually 47.4% may not be a bad number at all. In a coin flip you assume an equal reward:risk for heads and tails. But gurus often assume a large target and a small stop-loss. If for example the guru has a 50% hit rate but a 2:1 reward:risk ratio then his profit factor (sum of winners over sum of losers) is 2, which is pretty good. In my opinion a reasonable upper limit for experienced gurus on reward:risk is 3:1 in which case the profit factor will be close to 3 on the average. The break-even for that ratio is a win rate of 25%. Therefore, one must also look at the reward:risk ratios and take the joint distribution. The study is hopelessly simplistic. The win rate, reward:risk and profit factor are connected by a unique function: 

To conclude, it sounds like 47.4% is pretty good if one assumes that these gurus would take a stop-loss half of the profit objective on the average. Nevertheless, this is one reason traders say that hit rate is meaningless unless one knows the reward:risk ratio. Actually, the person who performed the study only confirmed the Central Limit Theorem and nothing else.

Maybe you should take a fresh look.

If you disagree with the above, I will be happy to hear from you.


Comments are Closed

Theme by Anders Norén