Excel Mistakes are the Least of our Worries

If you follow economics outside of this blog you will no doubt have seen this week’s big hoo-har about Reinhart-Rogoff (R-R). In case you haven’t I’ll explain what happened.

About three years ago, two Harvard Economists published a paper that concluded that when a country’s debt to GDP ratio exceed 90% it caused the country to experience significantly lower economic growth (actually on average a recession). Paul Krugman describes the subsequent events here but essentially there were two immediate problems with the paper:

Correlation and Causation

A very common problem in statistics is confusing correlation and causation. Correlation between two factors means that as one changes the other does too. Causation between two factors means that as one changes it causes the other to change. This might seem like a small difference but it isn’t. It is very important.

This is more easily explained with an example. If you were to plot on a map of Great Britain the location of every mobile phone mast and then were to plot the locations that children were conceived, you would get a striking correlation. This does not mean that mobile phone masts cause conception and it doesn’t mean that conception causes mobile phone masts. What is actually going on is that both conceptions and mobile phone masts are more likely to occur in areas of high population density. However, if you didn’t know the difference between correlation and causation you could easily, (and wrongly) conclude from your research that mobile phone masts increased fertility.

There is another problem in implying causation from correlation. Even if there is a causal relationship it could run either way. Do mobile phone masts cause increased birth rates or do increase birth rates cause mobile phone masts? In the case of the R-R paper, not only did they decide there was a causal relationship in the correlation, they decided that it was high debt that caused low growth without entertaining the possibility that low growth might cause higher debt.

This was problem number one.

Peer Review

Problem number two was that with the same set of data, no one else could reproduce the result showing some special tipping point at 90%. Lots of people tried it but no matter what they did, they couldn’t reach the same conclusion that R-R managed and because R-R had not published their workings no one knew how they’d reached that conclusion.

This was problem number two.

So move on three years to the current day and what has happened? Well in the interim, pretty much every austerity advocate had used this paper as proof of why austerity must be pursued at all costs. But this week after many requests, Reinhart and Rogoff released the Excel spreadsheet they’d made and on which they had based their findings. Oops.

In addition to the previous problems there were three more very obvious problems with their workings.

Cherry-Picked Data

The analysis that R-R did was not over all of the available data. Oddly they had chosen to exclude specific countries’ performance in specific years with no explanation as to why they had done it. Worse still, if you included them the magic 90% threshold disappeared.

Bizarre Weighting

In coming to their overall conclusion, R-R weighted certain results higher than other results using a method behind which no one can quite understand the motivation. Weight the results using any conventional means and guess what? The magic 90% threshold disappears.

Excel Error

Most embarrassing of all was that there was a basic Excel error in the spreadsheet – they’d averaged over a column of numbers and got the range wrong so missed a bunch of the numbers out of the average. Guess what happens if you correct it? Yep.

As Paul Krugman concludes in his article on all of this:

So will toppling Reinhart-Rogoff from its pedestal change anything? I’d like to think so. But I predict that the usual suspects will just find another dubious piece of economic analysis to canonize, and the depression will go on and on.

And this is really the important point. The R-R paper only ever became as famous as it did because it told the “usual suspects” what they wanted to hear. It gave them some kind of economic credibility on which to base the policies that they wanted to implement anyway. R & R shouldn’t be blamed for all of the ways in which their paper was used – politicians and the media should take some of that, especially when there were so many question marks over it from the start.

But the real thing I conclude from the whole debacle is simply this. When the crisis struck, governments had the option to follow the economic lessons we had learnt from the past. Lessons which had solved previous crises like The Great Depression of the 1930s.

But they chose not to.

They chose instead to pursue the things they wanted to do anyway and then cherry-pick any bit of research they could find to support it, irrespective of how many glaring problems it might have hanging over it. In all of this, until this week, the R-R paper was really their most coveted prize.

So what have we seen in the intervening days since the paper’s public demolition? We have seen the stimulus crowd publicly claiming a massive victory over the austerity crowd and while that might well be the case, I again worry we might miss the important point here. What we’re doing here is arguing over one piddly little paper that gained fame because it formed a new economic theory that supported the policies of austerity. But that paper really should never ever have become as big as it did.

We never needed “new cutting edge research” to solve this problem. The problem we have and the solutions to it are all covered in the first year of an undergrad economics degree. Yes, if a couple of Harvard economists come up with a new idea we should have a look at it but when it clearly had such big, unexplained problems from the outset, how could governments (ours included) end up basing their fiscal policy on it?

I mean – would it really be so ridiculous to go with the things we actually know?

RedEaredRabbit