That Way Madness Lies
Friday, June 1, 2012 at 11:56AM
Lindsay Cohn

Prof. Jacqueline Stevens of Northwestern has written an opinion piece for the New York Times in which she says she is sympathetic to Congress's attempts to block the National Science Foundation from funding Political Science research, because Political Scientists have turned out to be very bad at predicting major international events such as the Arab Spring or the fall of the Soviet Union. Here is my response.

This article is ridiculous. As Prof. Stevens quotes Popper saying, it's possible to make accurate predictions only in closed, simple, controlled systems, and we know the world isn't one of those. She's not the only person besides Popper smart enough to have noticed that. There's no way any one person could ever know enough about a particular situation to say "I think there will be a revolution there if a young person chooses to set himself on fire. Otherwise, things will just keep going." Guess who else can't make predictions like that? Biologists. When is this species going to evolve, and how? YOU DON'T KNOW? What the hell is the point of your "science" then?

The "scientific benchmark" is NOT predicting the future. Scientific prediction is about whether your hypotheses "predict" what happens in your data. The point of political science research is to help us understand why things happen the way they do, and to help us produce better policy. Sometimes that works, sometimes it doesn't. Some research really is just confirming trivialities, but it is the nature of research to fail far more often than it succeeds. I believe I've read somewhere that chimps throwing darts are actually BETTER at prediction than economists, meteorologists, and sportscasters, so it looks like we're doing pretty well.

Furthermore, her solution is ludicrous. So sometimes the people handing out money influence what the people receiving the money can study. Wow. What a newsflash. So we're supposed to do what? HAVE A LOTTERY. So instead of there possibly being some inappropriate influence, there will be no vetting process at all.

I'm fine with people criticizing political science. I have some problems with it myself. By all means, let's talk about disciplinary biases towards quantitative research methods; let's talk about how bad a lot of qualitative research is; let's talk about how lots of political scientists don't study anything that would be useful to policy-makers; let's talk about how the peer review system has some major flaws. But as far as criticisms go, the one that we didn't predict big stuff happening and therefore are a bunch of failures is pretty stupid. For one thing, most political science research is probabilistic, not deterministic, which means that even if we wanted to make predictions they'd all be in terms of what's more or less likely to happen, not what WILL happen. Plus, there is a big difference between polling political scientists about what they think will happen, and doing a systematic study of political science research to see how accurate its generalized predictions would be in particular situations. Prof. Stevens doesn't bother with that, but I'd be willing to bet we wouldn't do that badly.

It is important to have government funding for political science research. Why? First, because research is inherently inefficient, thus unlikely to be funded by profit-motivated entities unless there is the possibility of a huge return (e.g. chemistry/pharmaceuticals). Second, because our only client is the government/public. Oh, and which kind of research is less capital-intensive and needs far less grant money? The quantitative numbers-crunching kind that Prof. Stevens thinks is so pointless. So: less government money = EVEN MORE QUANT BIAS.

I know Prof. Stevens is a smart lady, and she is in intelligent company when she complains that Political Scientists aren't predicting big stuff (John Lewis Gaddis is no dummy). To claim in one breath, though, that prediction is impossible and that political scientists are failures for not predicting accurately, is ... unhelpful.

 

I had a thoughtful response to the above. Here it is, with my further response:

@Lindsay, you stated, "The 'scientific benchmark' is NOT predicting the future." Then followed it with, "Scientific prediction is about whether your hypotheses "predict" what happens in your data. The point of political science research is to help us understand why things happen the way they do, and to help us produce better policy."

Help us produce better policy. I am stuck on that. So what does the scientific benchmark do if it does not "predict the future?" How does it help us produce better policy? Aren't we producing better policy so we can deal better with the future?

I will side with Prof. Stevens. Political Science has risen to become sacrosanct around Washington, yet the greatest policy failure—there is no articulated Grand Strategy driving US foreign policy, we bumble from one surprise to the next—is because a bunch of mountebanks have gone to Washington and promised to have the cure, advising from without from think tanks, academia, or newspapers while hiding behind fancy degrees and peer-reviewed papers (a stark exception being the neocons). They have no skin in the game, and when proven wrong claim that it is unfair to be expected to "predict the future" with a model never intended to do that. Guess what? Their models shouldn't be used to guide the formulation of policy either.

Political science has a place, and it has great value. But we should be careful to not ascribe to it a status it does not deserve, or a role it cannot fulfill.

 

LPC: in order to make good policy, we need to know what works and what doesn't. That is the purpose of scientific research: to determine the conditions/factors that seem to affect the outcomes we care about. We then create policy in an attempt to manipulate those factors in order to produce the outcomes. E.g. if we know that non-violent protest movements are far more likely to succeed than violent ones, we can encourage non-violence, oppose protests turning into armed rebellion, etc. If we know from research that intervention in a civil war tends to produce a lasting peace only if the intervenor is willing and able to commit to a significant presence in-country for a significant amount of time, that tells us something about what we may or may not want to do if there's a civil war going on that we care about. That doesn't mean that we can predict when a protest movement is going to happen, or whether it will turn violent, or when a civil war is going to break out. It means we have a POLICY when it happens. I didn't see anyone claiming that epidemiology was pointless when they failed to tell us all that swine flu was going to happen. What epidemiology helps us do is try to put measures in place to prevent epidemics, and to deal with epidemics when they happen.

I disagree with your assessment of the problem. Throughout its entire history the US has only rarely had something that could be called a grand strategy, and this is hardly the result of political scientists as a group (a discipline that has existed only since the 1950s) claiming to have a grand strategy, hoodwinking everyone in Washington with it, and then being wrong. Different presidents have listened to different advisors. That has always been the case. Some of those advisors have been political scientists, and some have not. Some presidents have listened to political scientists, and some have not. Some political scientists have been right, and others wrong. The idea that our current problems are either new or created by the members of a single academic discipline seems a little ... overwrought.

I completely agree that we shouldn't ascribe Political Science magical status ... the same way we shouldn't ascribe doctors magical powers to know what's wrong with us or geologists magical powers to predict earthquakes (can they do that? Not usually). I also agree that it should not be asked to fulfill a role it cannot. What I find ridiculous is the dual claim that prediction is impossible and that Political Scientists are failures because they cannot predict. Like military experts, we advise. Anyone wandering around claiming to make major predictions about wildly complex situations is free to do so, but that is his/her educated opinion. Not science.

One last thing: I'm not saying that prediction is impossible or that we should never attempt it. I'm saying that our ability to predict is limited, should be used carefully and with humility, and is emphatically NOT the benchmark for whether our research is useful to society.

Article originally appeared on lpcohn (http://lpcohn.squarespace.com/).
See website for complete article licensing information.