Maren Duvendack, Richard Palmer-Jones, James G Copestake, Lee Hooper, Yoon Loke, Nitya Rao
London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.
A new systematic review of the evidence on microfinance, published last week, is dynamite for the world’s most popular development policy. Madeleine Bunting of the Guardian has already referred to it as “microfinance’s sober reckoning”, likening the findings to a “hangover after a big party”. Bangladeshi news calls it a “damning report”.
Being co-published by the UK Department for International Development (DFID), previously a strong microfinance supporter, this “study of studies” comes from deep within the policy community – a first for a truly critical study of microfinance. The authors (economists and medical researchers mainly based at the University of East Anglia) looked at thousands of existing studies on microfinance. Their conclusions are anything but minced words:
Our report shows that almost all impact evaluations of microfinance suffer from weak methodologies and
inadequate data, thus adversely affecting the reliability of impact estimates. Nevertheless authors often draw strong policy conclusions generally supportive of microfinance. This may have lead to misconceptions about the actual effects of programmes, thereby diverting attention from the search for perhaps more pro-poor interventions and more robust evaluations. (from the Policy Brief)
So, after 30-odd years and $ hundreds of billions of lending, there still is no proof that microfinance actually works.
(See also, a follow-up post: here)
In itself, this finding isn’t new; it’s what microfinance researchers have been debating for years, and particularly strongly since the two first Randomised Control Trials (RCTs) in 2009 found no evidence. But to have systematically evaluated all known studies on microfinance is a heroic achievement, a herculean task, and Duvendack (who completed her PhD last year), Palmer-Jones, Copestake, Hooper, Loke and Rao deserve quite a bit of credit for it.
A herculean task completed
Duvendack et. al.’s review is exhaustive. The authors trawled through multiple databases to find all studies ever done on microfinance, leaving them with a final list of 2,643 publications, which they systematically whittled down to a shortlist of the 58 highest-quality by research design and methods of analysis. The studies they reviewed studied all types of microfinance: individual, group and mixed lending, and credit-only programs as well as those offering a range of other services. They included studies looking for economic outcomes, social outcomes and women’s empowerment.
Disastrously, the authors of the review found that most studies relied on methods far too weak to support their conclusions; and those with strong methods found nothing conclusive:
There are only two RCTs of relevance to our objectives; neither has appeared in peer-reviewed form, and our judgement is that one has low-moderate and the other high risk of bias. Neither finds convincing impacts on well-being. We found nine pipeline studies, which have been reported in ten papers. All pipeline studies were based on non-random selection of location and clients, and most have only ex-post cross-sectional data, some with retrospective panel data information allowing (low validity) impact estimates of change in outcome variables. Thus, we deal with a set of relatively low validity papers, from which it would be unwise to draw strong conclusions. In contrast to some recent reviews, this is the conclusion we wish to emphasise, in large part perhaps because of a preference for avoiding type 1 errors. That is, we come down on the side of ‘there is no good evidence for’, rather than ‘there is no good evidence against the beneficent impact of microfinance’. (Review, p. 72)
I can only read the last sentence as a remark directed at the authors of the two major 2009 RCT studies, who declined to interpret their results as evidence against microfinance, and have subtly changed their conclusions in successive versions to appear less negative, and have indefinitely kept their studies from final publication in any journals. Indeed, the biggest surprise in Duvendack et. al.’s review is their deeply critical conceptual treatment of RCTs, which are currently considered something of a gold standard in microfinance evaluation. I’m really glad about this, since I have felt very uneasy about the RCT method for some time.
The dangerously flawed medical analogy
RCTs took the experimental design used in most medical research and applied it to microfinance: take a normal population afflicted by a certain problem (cancer/poverty), and randomly select a treatment group and a control group. Then administer the treatment to one group, and statistically compare the change in dependent variables between the two groups (cancer cells/income, etc.).
The idea was that, by applying a sophisticated and internally valid research design to a large enough randomly-selected sample, the impact of microfinance could finally conclusively be proven. But the new sophisticated studies couldn’t find anything. Even so, this translation of medical reasoning into the world of microfinance is incomplete and flawed for several reasons – here would be my (non-exhaustive) list:
- The placebo effect is not accounted for, and is possibly even reversed. In medicine, both the treatment group and the control group are given a pill (figuratively); either one with an active ingredient, or a sugar pill. But in microfinance RCTs, one group gets a loan, while the other gets nothing. This process is not double-blind, as in medical testing; both the “patients” and the “doctors” know who is being treated and who isn’t. It is likely, therefore that the placebo effect even runs in reverse: a loan recipient is likely to believe she must objectively be more entrepreneurial than her (identical) neighbour who was denied a loan. The neighbour in turn may be discouraged and underperform accordingly, even though both were just randomly selected.
- Programme placement bias is a problem, as Duvendack et. al. briefly point out. That is, microfinance programmes are not randomly distributed. The fact that the two major RCT trials were administered in Hyderabad and Manila makes a potentially huge difference; these are growing cities with entrepreneurial opportunities. And furthermore: in politics and economics, local context matters, while in medicine illnesses should be more or less uniform – cholera in India is the same as cholera in Ghana, but poverty in Hyderabad can be expected to be very different from poverty in rural Ghana in terms of causes and consequences.
- Microfinance RCT’s don’t test for medical equivalence. In medicine, a new treatment is always evaluated against the best known existing treatment. But microfinance RCTs act as if the only alternative to microfinance were to do nothing. Perhaps ontologically this makes sense for organisations like the World Bank, but as scientific practice, it is disingenuous. The long-run effects of, say, investment in education funded via public borrowing should, for example, be compared with the effects of investment in microenterprises. This sort of comparative cost-benefit perspective is systematically lacking.
- Finally, the ethics are troubling. Economists are often frustrated by the impossibility of manipulating environments to create an experimental setting in the real world. Microfinance RCTs are one rare exception where economists have practically found a dream case in which they can manipulate entire populations according to their methodological needs. While internally logically valid, there is something disturbing about research designs which depend on manipulating key variables in the lives of economically and socially imperiled precarious populations. This type of research on a hypothesised tool for empowerment is only possible thanks to the powerlessness of the subjects.
Reminder: despite all this, the RCTs could not prove any impact. Who knows what they would find if placebos could be implemented, placement were random, and the results were compared against other (perhaps less market-friendly) development project options.
But worse yet, as Duvendack et. al. argue, the RCT studies didn’t even digest their own medicine. Rather than acknowledge that a lack of evidence indicates a lack of positive impact, the RCT authors chose to portray the lack of evidence as proof that more research is needed (for proof to finally be found). Duvendack et. al. say,
Failing to contradict the alternate hypothesis encourages one to believe there is a positive effect and therefore to tend to (continue to) reject the null (no effect) hypothesis even though it (no effect) may be true. […] Even for critics of these evaluations the absence of robust evidence rejecting the null hypothesis of no impact has not led to a rejection of belief in the beneficent impacts of microfinance (Armendáriz de Aghion and Morduch 2010, p310; Roodman and Morduch 2009, p39-40), since it allows the possibility that more robust evidence (from better designed, executed and analysed studies) could allow rejection of this nul. However, given the possibility that much of the enthusiasm for microfinance could be constructed around other powerful but not necessarily benign, from the point of view of poor people, policy agendas (Bateman 2010, Roy 2010), this failure to seriously consider the limitations of microfinance as a poverty reduction approach, amounts in our view to a failure to take seriously the results of appropriate critical evaluation of evaluations. (72-73)
“Enthusiasm built on foundations of sand”
Given the lack of real evidence for microfinance’s impacts, Duvendack et. al. are right to raise the question of opportunity cost.
[I]t might have been more beneficial to explore alternative interventions that could have better benefitted poor people and/or empowered women. Microfinance activities and finance have absorbed a significant proportion of development resources, both in terms of finances and people. Microfinance activities are highly attractive, not only to the development industry but also to mainsteam financial and business interests with little interest in poverty reduction or empowerment of women, as pointed out above. There are many other candidate sectors for development activity which may have been relatively disadvantaged by ill-founded enthusiasm for microfinance. […]
However, it remains unclear under what circumstances, and for whom, microfinance has been and could be of real, rather than imagined, benefit to poor people. Unsurprisingly we focus our policy recommendations on the need for more and better research. [… But w]hile there is currently enthusiasm for RCTs as the gold standard for assessing interventions, there are many who doubt the universal appropriateness of these designs. Indeed there may be something to be said for the idea that this current enthusiasm is built on similar foundations of sand to those on which we suggest the microfinance phenomenon has been based. (75)
The microfinance community, with its strong tendency to disqualify contradictory evidence via processes of groupthink, has yet to respond to Duvendack et. al.’s review. Back in 2009, the RCT studies were at worst friendly fire, and were already greeted with considerable hostility from microfinance practitioners. Duvendack et. al.’s review is, at the very least, a real challenge.
PS: On a side-note, I have no idea why David Roodman sees Duvendack et. al.’s work making “common cause with someone who views with nihilism the work to which they are devoting their careers” – Milford Bateman that is. Bateman’s book is political economy, while Duvendack et. al. are mainly about methodology. Not all criticisms of microfinance are the same.