What is the evidence of the impact
of microfinance on the well-being of
poor people?
Maren Duvendack, Richard Palmer-Jones, James G Copestake, Lee Hooper, Yoon Loke, Nitya Rao
August 2011
London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.
A new systematic review of the evidence on microfinance, published last week, is dynamite for the world’s most popular development policy. Madeleine Bunting of the Guardian has already referred to it as “microfinance’s sober reckoning”, likening the findings to a “hangover after a big party”. Bangladeshi news calls it a “damning report”.
Being co-published by the UK Department for International Development (DFID), previously a strong microfinance supporter, this “study of studies” comes from deep within the policy community – a first for a truly critical study of microfinance. The authors (economists and medical researchers mainly based at the University of East Anglia) looked at thousands of existing studies on microfinance. Their conclusions are anything but minced words:
Our report shows that almost all impact evaluations of microfinance suffer from weak methodologies and
inadequate data, thus adversely affecting the reliability of impact estimates. Nevertheless authors often draw strong policy conclusions generally supportive of microfinance. This may have lead to misconceptions about the actual effects of programmes, thereby diverting attention from the search for perhaps more pro-poor interventions and more robust evaluations. (from the Policy Brief)
So, after 30-odd years and $ hundreds of billions of lending, there still is no proof that microfinance actually works.
(See also, a follow-up post: here)
In itself, this finding isn’t new; it’s what microfinance researchers have been debating for years, and particularly strongly since the two first Randomised Control Trials (RCTs) in 2009 found no evidence. But to have systematically evaluated all known studies on microfinance is a heroic achievement, a herculean task, and Duvendack (who completed her PhD last year), Palmer-Jones, Copestake, Hooper, Loke and Rao deserve quite a bit of credit for it.
A herculean task completed
Duvendack et. al.’s review is exhaustive. The authors trawled through multiple databases to find all studies ever done on microfinance, leaving them with a final list of 2,643 publications, which they systematically whittled down to a shortlist of the 58 highest-quality by research design and methods of analysis. The studies they reviewed studied all types of microfinance: individual, group and mixed lending, and credit-only programs as well as those offering a range of other services. They included studies looking for economic outcomes, social outcomes and women’s empowerment.
Disastrously, the authors of the review found that most studies relied on methods far too weak to support their conclusions; and those with strong methods found nothing conclusive:
There are only two RCTs of relevance to our objectives; neither has appeared in peer-reviewed form, and our judgement is that one has low-moderate and the other high risk of bias. Neither finds convincing impacts on well-being. We found nine pipeline studies, which have been reported in ten papers. All pipeline studies were based on non-random selection of location and clients, and most have only ex-post cross-sectional data, some with retrospective panel data information allowing (low validity) impact estimates of change in outcome variables. Thus, we deal with a set of relatively low validity papers, from which it would be unwise to draw strong conclusions. In contrast to some recent reviews, this is the conclusion we wish to emphasise, in large part perhaps because of a preference for avoiding type 1 errors. That is, we come down on the side of ‘there is no good evidence for’, rather than ‘there is no good evidence against the beneficent impact of microfinance’. (Review, p. 72)
I can only read the last sentence as a remark directed at the authors of the two major 2009 RCT studies, who declined to interpret their results as evidence against microfinance, and have subtly changed their conclusions in successive versions to appear less negative, and have indefinitely kept their studies from final publication in any journals. Indeed, the biggest surprise in Duvendack et. al.’s review is their deeply critical conceptual treatment of RCTs, which are currently considered something of a gold standard in microfinance evaluation. I’m really glad about this, since I have felt very uneasy about the RCT method for some time.
The dangerously flawed medical analogy
RCTs took the experimental design used in most medical research and applied it to microfinance: take a normal population afflicted by a certain problem (cancer/poverty), and randomly select a treatment group and a control group. Then administer the treatment to one group, and statistically compare the change in dependent variables between the two groups (cancer cells/income, etc.).
The idea was that, by applying a sophisticated and internally valid research design to a large enough randomly-selected sample, the impact of microfinance could finally conclusively be proven. But the new sophisticated studies couldn’t find anything. Even so, this translation of medical reasoning into the world of microfinance is incomplete and flawed for several reasons – here would be my (non-exhaustive) list:
- The placebo effect is not accounted for, and is possibly even reversed. In medicine, both the treatment group and the control group are given a pill (figuratively); either one with an active ingredient, or a sugar pill. But in microfinance RCTs, one group gets a loan, while the other gets nothing. This process is not double-blind, as in medical testing; both the “patients” and the “doctors” know who is being treated and who isn’t. It is likely, therefore that the placebo effect even runs in reverse: a loan recipient is likely to believe she must objectively be more entrepreneurial than her (identical) neighbour who was denied a loan. The neighbour in turn may be discouraged and underperform accordingly, even though both were just randomly selected.
- Programme placement bias is a problem, as Duvendack et. al. briefly point out. That is, microfinance programmes are not randomly distributed. The fact that the two major RCT trials were administered in Hyderabad and Manila makes a potentially huge difference; these are growing cities with entrepreneurial opportunities. And furthermore: in politics and economics, local context matters, while in medicine illnesses should be more or less uniform – cholera in India is the same as cholera in Ghana, but poverty in Hyderabad can be expected to be very different from poverty in rural Ghana in terms of causes and consequences.
- Microfinance RCT’s don’t test for medical equivalence. In medicine, a new treatment is always evaluated against the best known existing treatment. But microfinance RCTs act as if the only alternative to microfinance were to do nothing. Perhaps ontologically this makes sense for organisations like the World Bank, but as scientific practice, it is disingenuous. The long-run effects of, say, investment in education funded via public borrowing should, for example, be compared with the effects of investment in microenterprises. This sort of comparative cost-benefit perspective is systematically lacking.
- Finally, the ethics are troubling. Economists are often frustrated by the impossibility of manipulating environments to create an experimental setting in the real world. Microfinance RCTs are one rare exception where economists have practically found a dream case in which they can manipulate entire populations according to their methodological needs. While internally logically valid, there is something disturbing about research designs which depend on manipulating key variables in the lives of economically and socially imperiled precarious populations. This type of research on a hypothesised tool for empowerment is only possible thanks to the powerlessness of the subjects.
Reminder: despite all this, the RCTs could not prove any impact. Who knows what they would find if placebos could be implemented, placement were random, and the results were compared against other (perhaps less market-friendly) development project options.
But worse yet, as Duvendack et. al. argue, the RCT studies didn’t even digest their own medicine. Rather than acknowledge that a lack of evidence indicates a lack of positive impact, the RCT authors chose to portray the lack of evidence as proof that more research is needed (for proof to finally be found). Duvendack et. al. say,
Failing to contradict the alternate hypothesis encourages one to believe there is a positive effect and therefore to tend to (continue to) reject the null (no effect) hypothesis even though it (no effect) may be true. […] Even for critics of these evaluations the absence of robust evidence rejecting the null hypothesis of no impact has not led to a rejection of belief in the beneficent impacts of microfinance (Armendáriz de Aghion and Morduch 2010, p310; Roodman and Morduch 2009, p39-40), since it allows the possibility that more robust evidence (from better designed, executed and analysed studies) could allow rejection of this nul. However, given the possibility that much of the enthusiasm for microfinance could be constructed around other powerful but not necessarily benign, from the point of view of poor people, policy agendas (Bateman 2010, Roy 2010), this failure to seriously consider the limitations of microfinance as a poverty reduction approach, amounts in our view to a failure to take seriously the results of appropriate critical evaluation of evaluations. (72-73)
“Enthusiasm built on foundations of sand”
Given the lack of real evidence for microfinance’s impacts, Duvendack et. al. are right to raise the question of opportunity cost.
[I]t might have been more beneficial to explore alternative interventions that could have better benefitted poor people and/or empowered women. Microfinance activities and finance have absorbed a significant proportion of development resources, both in terms of finances and people. Microfinance activities are highly attractive, not only to the development industry but also to mainsteam financial and business interests with little interest in poverty reduction or empowerment of women, as pointed out above. There are many other candidate sectors for development activity which may have been relatively disadvantaged by ill-founded enthusiasm for microfinance. […]
However, it remains unclear under what circumstances, and for whom, microfinance has been and could be of real, rather than imagined, benefit to poor people. Unsurprisingly we focus our policy recommendations on the need for more and better research. [… But w]hile there is currently enthusiasm for RCTs as the gold standard for assessing interventions, there are many who doubt the universal appropriateness of these designs. Indeed there may be something to be said for the idea that this current enthusiasm is built on similar foundations of sand to those on which we suggest the microfinance phenomenon has been based. (75)
The microfinance community, with its strong tendency to disqualify contradictory evidence via processes of groupthink, has yet to respond to Duvendack et. al.’s review. Back in 2009, the RCT studies were at worst friendly fire, and were already greeted with considerable hostility from microfinance practitioners. Duvendack et. al.’s review is, at the very least, a real challenge.
(phil)
PS: On a side-note, I have no idea why David Roodman sees Duvendack et. al.’s work making “common cause with someone who views with nihilism the work to which they are devoting their careers” – Milford Bateman that is. Bateman’s book is political economy, while Duvendack et. al. are mainly about methodology. Not all criticisms of microfinance are the same.
15 comments
Comments feed for this article
August 20, 2011 at 04:40
David Roodman
Hi Phil,
A few comments.
One of the two RCTs (Karlan & Zinman on the Philippines) actually has been published, in Science:
http://www.sciencemag.org/content/332/6035/1278.abstract?sid=bf6d21ca-5fb3-4e9e-9404-90ae6992d148
I don’t know the publication status of the other, but I would be surprised if the authors were intentionally holding back from publishing it because of doubts about its validity. Possibly, they are waiting to analyze the three-year follow-up data. Or possibly it is just going through the normal process. Or they are just busy (they are extremely productive and in demand).
It seems relatively undisputed that a) RCTs are the most reliable study design that has been applied to study the average impacts of microcredit on the poor and b) they have limitations, mostly with external validity (generalizability). Accepting these points, what practical conclusion does an inventory of their limitations lead us to? To the conclusion that we shouldn’t do RCTs? I don’t think so, because that would mean just giving on the academic approach to impact evaluation, which seems unwarranted. Rather, it should lead us to be cautious in overgeneralizing from RCTs. That’s fine, but not new, and well understood by the RCT authors.
The last sentence of your first block quote may actually be directed less at RCT authors and more at Roodman & Morduch (see my post at http://j.mp/qvB1PX) who have never done and may never do an RCT. Certainly the second full block quote is. To me it seems strange to set up a choice between the positive and negative implications of a conclusion that we lack evidence. It is simultaneously true that we have little evidence of harm and little evidence of help. Roodman & Morduch say both, roughly in balance, but the lit review chooses to quote us emphasizing the lack of evidence of harm.
Why I wrote that Bateman views the literature that is the subject of this review with nihilism: the section of his book whose job is seemingly to prove the premise that microfinance doesn’t work, or anyway review what the research literature has to offer on the premise, is very short, ignores what everyone agrees are the most important studies (the RCTs), and consists mainly of a misinterpretation of my work with Jonathan Morduch. In addition, he commented on my blog that his book has little time for RCTs. Duvendack et al., in contrast, have a great deal of time for them.
–David
August 21, 2011 at 12:42
8/21/2011 Blogs Update « AbolishPoverty
[…] The Mothership of Microfinance Impact Studies has landed […]
August 22, 2011 at 19:17
philmader
Hi David, thanks for your comments. Karlan & Zinman in “Science”: I assumed, since Duvendack et. al. actually mentioned the non-peer-reviewedness of both major RCTs, that it was still true (which I guess at the time of their writing it was). I hadn’t seen that (relatively recent) publication – now I know where to cite it in future, thanks. I still find the fact that the Banerjee et. al. study remains unpublished a bit puzzling given how it has made more waves than more published papers ever have, but let’s see what comes of it; your explanations make sense.
My point about the limitations of RCTs is, I guess, more subtle than you appreciated. I’m not saying “stop doing RCTs”. What I’m saying is they’re not as good as most have made them out to be. Some of the factors which make RCTs valid in medical testing are not there in microfinance. And it appears to me that the risk of a reversed placebo effect and the need to consider equivalent options (rather than doing nothing) are very serious concerns which have hardly, if at all, been addressed by RCT enthusiasts; the ethics problem is of course a separate issue.
At the end of the day, what to make of a lack of evidence is as old a question as the science itself. But it’s disingenuous to act as if a non-refutation of a hypothesis were a success instead of a good reason to doubt the hypothesis. It isn’t like the RCT testers went out into the field to prove once and for all that microfinance causes harm to borrowers, and returned with data that couldn’t conclusively show a lack of harm. RCTs were supposed to finally prove, instead, the miracle of microfinance against all logical and theoretical doubts; and the research community was surprised at the lack of evidence for something which they had never questioned (your stainless steel rule notwithsanding). Of course the lack of evidence raises a need for more testing (assuming the design is valid), and maybe third time is lucky. And surely “no effects found” doesn’t constitute proof of harmful effects. But it should make one rethink the hypothesis. Next time around it should be “microfinance makes no discernible difference to household incomes, girls’ schooling, etc…”, and if that hypothesis were corroborated by the data, it should indeed prompt a rethink, shouldn’t it?
As for different types of critical thinking on microfinance, I can’t see why writing about RCTs should be the only (or predominant way) of engaging with the question of whether microfinance works. I don’t see your book mainly reviewing RCTs, either. What Milford Bateman did in his book was make a number of logical arguments why microfinance was unlikely to work (fallacy of composition, displacement, saturation) which I have yet to see logically refuted; but, true, it’s a completely different type of work from an exhaustive review of existing studies. Therefore, I still don’t see them making “common cause”, except for their conclusions both being critical. That in itself should be no crime.
Phil
August 23, 2011 at 23:55
Caitlin McShane
Debating how to measure the impact of microfinance and what conclusions to draw is a heady and fiery debate. Roodman, Karlan, Dunford and Morduch battle over the complexities in this video recording from “The Promise & Peril of Microfinance Impact Evaluations” at the 2011 Microfinance USA conference in New York.
http://www.microfinanceusaconference.org/videos-2011/session-01-promise-and-peril.php
Tom Murphy summarizes the debate in this blog entry:
http://www.aviewfromthecave.com/2011/06/thoughts-on-rcts-from-microfinance-usa.html
August 24, 2011 at 17:06
philmader
Caitlin, thank you for sending this. I should have mentioned Tom Murphy’s insightful blog entry, even if only for his quotation about the “measurement mafia”. The video I was not aware of, so thank you very much – it is definitely worth watching!
August 29, 2011 at 18:23
The Rumsfeldian Logic of Microfinance Impact Assessment: A Challenge to Agnostics «
[…] researchers recently published an exhaustive study (actually a Systematic Review, S.R.); as I explained on this blog, they pulled together all existing 2,643 publications about microfinance’s […]
August 30, 2011 at 14:24
Monitoring and Evaluation NEWS » Blog Archive » Systematic review: What is the evidence of the impact of microfinance on the well-being of poor people?
[…] The Mothership of Microfinance Impact Studies has landed at http://www.governancexborders.com, 19 August 2011 […]
January 23, 2012 at 16:50
Save the date: «
[…] of microfinance crises, growing overindebtedness, borrower suicides, disappointing impact findings, and a prize-winning Norwegian documentary contributing to Muhammad Yunus being removed from […]
January 23, 2012 at 16:51
Save the Date: Bateman-Roodman Debate «
[…] of microfinance crises, growing overindebtedness, borrower suicides, disappointing impact findings, and a prize-winning Norwegian documentary contributing to Muhammad Yunus being removed from […]
September 21, 2012 at 13:03
Ahmad Jazayeri
The debate is not about the discovery of “how things really are” but what Rorty called “alternative self-images” of the various sides of this debate each trying to justify a position to allocate capital. Just because there is no evidence using a specific medical methodology it does not mean that they have proven the opposite point.
The microfinance community would benefit from reading Rorty, Berlin, Alfred Schutz, Polanyi and Vico to know that in social science the search for so-called “universal truth” is nothing but a chimera. A pragmatic definition of truth is “what is useful” and “the consensus in the community”a la Kuhn involving intention, motivation, personal experience, path-dependence and institutional interests. We need much more qualitative and conversational “reflections” and less quantitative randomized modelling and imitations of medical approaches. I think Roodman’s conversational book is the kind of studies we need much more of.
The market economy and the neoclassical modellings based on the assumptions of formal economics captures only a small fraction of the real substantive economy in poor countries. People are not viruses to be tested based on medical methods; focus on what matters in social science which is understanding people in their historical and institutional and experiential context and how and why they make decisions.
September 21, 2012 at 14:17
philmader
Ahmad, very interesting thoughts, which I’ll have to think about. But particularly your last point I can only fully agree upon. Conceptualising poverty as if it were a disease (with no history, no factors which actively re-/produce it, and prone to simple “cures”) is misleading and perhaps even dangerous. As for the question of “universal truths” versus Kuhnian paradigm consensuses, my intention throughout has not been to justify RCTs as the ultimate measure (which certainly they shouldn’t be), but rather to point out how the microfinance community fails to “take its own medicine”: first justifying microfinance on the claim of “so and so many people escape poverty, and this is empirically demonstrable”, but when these empirics fail to materialise, sticking to the claim nonetheless. Phil
September 21, 2012 at 15:54
Ahmad Jazayeri
Yes Phil,
I agree that critical self-examination is not their forte, a weakness in the micro-finance industry that is lead by “best practice gurus”. Sounds a bit evangelistic and borders on the religious!
One methodological issue in my view is that the DFID study is committing an error of wanting to validate some kind of “universal truth” for or against microfinance based on the “scientific method”. Is this really unattainable in social science? All we have are approximations and “a slow and painful choice between alternative self-images” that somehow justify what we are trying to achieve.
So where do we go from here?
I am an advocate for more resources to be put at the service of evaluations based on “conversations” with the beneficiaries in a systematic way and a “dialogue with history” to gather evidence on the individual and institutional experience and learn from success and failure stories.
Consider this as a methodological recall based on the insights of “conversational philosophy” and “economic sociology”, disciplines that are rarely considered as sufficiently “rigorous” by the academia doing impact studies. What is called a “low validity paper” for the authors could be the biography of a single borrower which can teach us quite a lot on who is benefiting and who is loosing out. Why should the biography of a poor borrower be less instructive than that of Jack Welch?
Ahmad
September 21, 2012 at 16:51
philmader
In that case, Ahmad, I would be interested to hear your opinion on “Portfolios of the Poor”, which deliberately tried to study (in a more fine-grained individual way) the biographies of poor people – albeit, only their financial biographies.
Of course you are right, when you ask people to reflect upon their relationship with the creditor you get completely different insights, and not necessarily ones the creditor wants to hear…
March 11, 2013 at 10:42
Microfinance and European Crisis Management | thecurrentmoment
[…] of microfinance-saturated countries like Bangladesh or Bolivia show and numerous scientific studies have underscored; but they have certainly proven their worth at tempering redistributional demands while […]
August 25, 2015 at 15:17
Out now: The Political Economy of Microfinance: Financializing Poverty |
[…] and how?”, it offers new insights into which have particular significance in light of the continually unresolved issues around poverty impact. More than 35 years into the microfinance experiment, the […]