Believe Quantitative Research At Your Own Peril

Almost every day research papers and articles are published in the financial blogosphere that make false or unfounded claims. Usually, no one will check the results. I sometimes do and in many cases I discover errors and false assumptions.

The most recent blow to quantitative research claims came last month with a paper by Valerly Zakaamulin challenging the most widely quoted research paper in the financial blogosphere by Glabadanidis about the outperformance of moving average rules. Zakamulin showed that the alleged outperformance was due to look-ahead bias, which is a basic error, in my opinion due coding errors. Of course, those who have quoted the debunked results are keeping quiet with no apologies issued to their readers.

Just a couple of weeks before the release of the paper by Zakamulin challenging common beliefs about moving average rules, an article was posted in a popular quant compendium praising Glabadanidis’ work. The article included backtest results for a 24-month period moving average in a monthly SPY chart, since inception. The entry and exit rules are simple:

Buy if price > MA(24)
Sell  if price < MA(24)

I call this rule “SMA24”. Two extraordinary things occurred:

(1) The author of the quant article possibly reproduced the look-ahead bias error of Glabadanidis, claiming a high CAGR of more than 16% for the timing strategy! He also reported a close to 15% annualized return for SPY buy and hold!

(2) When I contacted the author of the article and pointed to the recent paper by Zakamulin, he responded with a tweet accusing me of trying to attract visitors to my website! I leave judgment of this response to the reader. Below, I include results that show why the results reported by Glabadanidis and the quant artcile are wrong.

Backtesting the SMA24 rule in R

I used R code developed by Ilya Kipnis, who is an expert in this area. I made some slight modifications (mov. avg. period, table columns, labels, etc.)

Below are the results for Adjusted SPY with monthly data from inception to 04/20/2016:

SPY_20160420_SMA24_perf

SPY_20160420_SMA24

The annualized return of the moving average rule is nowhere close to 16% but actually about 9.3%. The buy and hold annualized return is also about 9.3%. This does not justify a switch from a passive to an active strategy for the majority of investors, even in the presence of a significantly lower drawdown.

Below are the results for S&P 500 with monthly data from 01/1950 to 04/20/2016:

SPX_1950_20160420_SMA24_perf

SPX_1950_20160420_SMA24

The annualized return of the timing strategy is about 150 basis points less than the annualized return of buy and hold, i.e., the SMA24 underperforms buy and hold by a wide margin. Although the maximum drawdown of buy and hold is about -53%, that of the timing strategy is also large at -35.3%.

Could my analysis be wrong instead? Below are results from a backtest in Amibroker:

SPX_1950_20160420_SMA24_AMI

All metrics calculated in Amibroker are close to those calculated using the R code. CAGR of buy and hold is 7.54% and of the timing strategy it is 5.54%. Maximum drawdown levels match. There are some slight differences with Sharpe calculations due to the method followed.

I was not surprised by these results because I have been doing backtests of all kinds of systems in the past 25 years. The errors usually come from either a graduate student who is in a rush to finish a dissertation or from some paper or article author with little experience with real life results. But also those who provide high visibility to these articles without reviewing them properly may be responsible for misleading the public because the authors are usually raised to the status of an expert due to their actions. Therefore, a lot of emphasis must be put in peer review and in the use of multiple platforms and languages for backtesting. It is just ludicrous that a whole financial community has been fooled by a few papers until Zakamulin took the time to check them out because they appeared extraordinary. On the other hand, extraordinary is what the sales departments of some funds are looking for.

Note that same story is true for the SMA10 rule that is widely used by fund managers. Below are the results for S&P 500 in the period 01/1950 to 04/20/2016:

SPX_1950_20160420_SMA10_perf

SPX_1950_20160420_SMA10

The timing rule underperforms buy and hold but on a risk-adjusted basis it performs better due to a much lower maximum drawdown. However, the outperformance of the CAFR of the timing strategy is not confirmed here.  It is also not confirmed when the timing rule is tested in Amibroker:

SPX_1950_20160420_SMA10_AMI

In this case also the results are much closer for annualized returns, max. drawdown volatility, Sharpe and Calmar ratio. Buy and hold outperforms the moving average rule by about 50 basis points in the case of annualized returns.

The error made by Glabadanidis was repeated in a recent article dealing with quantitative strategy research. Can anyone guess how many articles with similar errors have been written and published? I spot errors almost every time I decide to scrutinize the results of some article or paper, some superficial and some serious. I some cases I contact the author. I remember this incident when I spotted a basic math error in one of these blogs that are promoted in a compendium that aggregates articles that are not peer reviewed. I left a comment in the blog suggesting the change. The change was made, a received a thank you note by email, but the comment was deleted! Surprised? I was not.

Journal papers that include empirical results must be scrutinized more carefully before published. The mass production of papers has increased risks for the public because of the status that their authors achieve due to the financial blogosphere. Articles published in blogs should be also peer-reviewed before included in aggregator feeds or websites. Especially when journals or aggregators become aware of errors in a paper or article, they should withdraw it and/or ask the author for a response or apology.

As the rush to gain personal recognition in the finance field becomes more intense due to increasing competition and no barrier to entry, there will be more errors, plagiarism and even some scattered deliberate efforts to mislead the public. Results that look too good to be true are usually not true. Maybe it is a good idea for those that publish such articles in journals or aggregator websites to ask authors to justify their claims or issue an apology when proven wrong.

Subscribe via RSS or Email, or follow us on Twitter.

Charting and backtesting program: Amibroker
Disclaimer

Detailed technical and quantitative analysis of Dow-30 stocks and popular ETFs can be found in our Weekly Premium Report.

Copyright Notice

This entry was posted in Quantitative trading and tagged , , , . Bookmark the permalink.

Leave a Reply