In 2002 in the run-up to the USA’s second invasion of Iraq, when he was challenged about the allegations made by the Bush administration about Iraq’s weapons of mass destruction (WMD) arsenal, Donald Rumsfeld made a memorable logical statement: “The absence of evidence is not evidence of absence. … Simply because you do not have evidence that something exists does not mean that you have evidence that it doesn’t exist.”

A parody of Rumsfeld from the (massively under-appreciated) comic series “The Boondocks”. Warning: coarse language.

In terms of twisted logic, Rumsfeld was right: the fact that intelligence couldn’t find conclusive proof for WMDs in Iraq didn’t necessarily mean they weren’t there; their available methods simply weren’t good enough to find them. But empirically, of course, he was wrong: as we now know, the reason why no proof was found for the WMDs was, they simply weren’t there.

Fast-forward to 2011, to a debate about the evidence for positive impacts of microfinance. Six British researchers recently published an exhaustive study (actually a Systematic Review, S.R.); as I explained on this blog, they pulled together all existing 2,643 publications about microfinance’s impact and looked in depth at the best 58.

Their conclusions – which are too complex and fine-grained to really present in a nutshell – effectively: (1) raised doubts about the research designs used so far, (2) re-iterated that the available evidence could “neither support nor deny the notion that microfinance is pro-poor and pro-women”, and (3) suggested that there has been an “inappropriate optimism towards microfinance”. And finally, they suggested that pursuing microfinance without proof that it works bears the risk of not running other programmes which could work better – opportunity cost.

The reaction from two prominent impact evaluators, Jonathan Morduch and David Roodman, has been chilly, which is somewhat understandable given their own studies being critically reviewed repeatedly mentioned (and David Roodman says, with reference to his & Morduch’s work, somewhat caricatured) in the S.R. Both are essentially retorting to the S.R.’s authors: the absence of evidence is not the evidence of absence.

Jonathan Morduch expresses this, and the implications for him, clearly. He says the S.R. is causing “confusion” (among whom actually?) between proving that something doesn’t work, and not being able to prove that it works. His suggestion is for impact assessors is

stick to your guns and re-double efforts to do better evaluations. The state of the literature on microfinance impacts is better characterized as “not being able to prove that it works” than as “proving that it doesn’t work.” I’ve long argued that, because of the lack of clarity on big impacts, we need to cut the hype—and, based on the available evidence, proceed with cautious optimism.

David Roodman’s reaction is characteristically less subtle. On the one hand he writes “with a couple of exceptions, [the S.R.] concisely corroborates my thinking”, but on the other hand he finds it “problematic in certain ways, even mildly offensive”. He claims that “in naming intellectual allies and opponents, the report appears to pick sides in a way that departs from the evidence it so thoroughly critiques” (a comment which mystifies me, since I can’t see how saying that there are sides in a debate equals choosing sides). He even sees the S.R. authors seemingly making “common cause” with the “nihilism” of Milford Bateman, a comment I can only read as trying to establishing guilt by association.

But the real upshot of his reaction is the same as Morduch’s: “lack of evidence means lack of evidence of help and lack of evidence of harm”. A lively and refreshing debate has ensued in the comments on Roodman’s blog piece, in which Maren Duvendack, one of the S.R. authors, clarifies that “while agreeing … that the evidence of impacts … is inconclusive we suggest that [many others] continue to believe in its (MF) beneficence … while we suggest it might be more appropriate to conclude that the evidence is more or less equally consistent with the hypothesis of little (direct) beneficence for the poorest and that therefore what one concludes depends on other evidence and arguments.” I have heard that there might be a more extensive response by the S.R. authors in the pipeline.

While I took off my economics hat a few years ago and I’m certainly no logician, I can’t help being struck by, on the one hand, the technical sophistication of the debate; and on the other hand the simple logical confusedness of it. A quick look on Wikipedia could help:

there are only two possibilities, given a null result:

  1. Nothing detected, and X is not present.
  2. Nothing detected, but X is present (Option eliminated by careful research design).

What the S.R. therefore does is challenge statement #2. The S.R. proposes: if your research design is actually good, but does not find anything, then that is because nothing is there. (Note that the authors are actually very careful in statements about nothing being there, proposing that this might be an explanation.)

I also understand (from Stephen Hales) that you can prove a negative – that is, if you are looking for it. Microfinance RCTs aren’t.

This is where Rumsfeld comes in again. In his view, the lack of proof for WMDs was a sign of insufficient methods, and not a sign of them not being there. At least, however, he didn’t claim to be agnostic about those WMDs. He knew they were there, and acted accordingly with conviction, regardless of what any doubting Thomases might say. (Which reminds me of a joke at the time: “You ask how we know Iraq has WMDs? We kept the receipts.”)

I see a parallel with the behaviour of many members of the microfinance research community; not because they profess faith in microfinance’s benevolent impacts or because they like wars (neither is the case!), but because they acknowledge there being no proof of impact, yet do not (openly) doubt impact. Their feigned agnosticism is, in my opinion, incompatible with the idea of “proceeding with cautious optimism”. If Rumsfeld had proceeded cautiously on the assumption that WMDs were there, perhaps a terrible war would have been averted, but the maxim “in dubio pro reo” (≈guilty until proven innocent) would still have been violated. If microfinance programmes are considered worthwhile until proven ineffective, the logic is equally skewed.

The reason I say this is I haven’t heard any RCT researchers saying “hold your horses, stop your microfinance programming, we impact experts really haven’t got a clue what this microfinance thing does; it could really be harmful but we don’t know” (though I will stand corrected if someone from the RCT community has said things along those lines – in dubio pro researcher). Proceeding with caution should not simply be to lament the persistent hype around microfinance (which many have done) but to express actual serious concern about microfinance existing and growing at its current level – the with global loan portfolio set to double roughly every two years – and to call for a halt until there is evidence of positive impact.

Actually, and here is my challenge, the cautious thing to do would be to stop microlending except to a few (randomly selected) test populations until it can be reasonably established that it works; a move which, after all, according to the present state of evidence, should neither harm nor benefit the poor. My reasoning is that, if researchers really are agnostic about microfinance’s impact, there should be no objection.

(phil)