How To Measure Microfinance's Social Impact

This is an interesting piece offered by the people from Wharton on microfinance which policymakers should delve deeper into. There is always a concern that large allocations to institutions like Amanah Ikhtiar Malaysia may be altruistic but, suffers the risk that noble objectives are sidetracked by poor understanding of microfinance and its implementation.

Two recent studies have sparked debates over the benefits of giving out small loans to the world's poor.

For three decades, microfinance institutions have given out small loans to the world's poor--mostly women--and amassed hundreds if not thousands of case studies showing that the loans help alleviate poverty, improve health, increase education and promote women's empowerment. Skeptics, however, have argued there is not enough hard data to prove that microfinance transforms lives on a large scale, and they have called for more rigorous analysis.

Now two new studies have raised doubts about long-held beliefs in microfinance. The studies--which used randomized, controlled trial methodologies--did find that microloans helped poor entrepreneurs boost profits in their businesses. However, the studies found little impact on health, education, average consumption, women's decision making or self-reported well being.

The unexpected results, which were quickly picked up by The Economist and other media, created a stir in the microfinance community. "They don't capture what we believe is the real impact or dimensions of microfinance, which sometimes takes years to demonstrate," says Bruce MacDonald, senior vice president of communications at Accion International, a Boston-based non-profit involved in microfinance since 1973. "We would definitely disagree that microfinance doesn't have an impact on poverty because we have seen the impact over the past 30 years."

One problem with previous research, microfinance's critics respond, is self-selection--that is, studies only survey clients who successfully take out loans. The new studies differ from the past in that both attempt to measure the impact of microfinance by comparing two groups: one with access to microcredit and one without.

In the first study, researchers from the Massachusetts Institute of Technology (MIT) worked with a microfinance lender in India to look at 104 slums in Hyderabad. Half of the slums were randomly selected to receive a loan branch, while the other half were not. Fifteen to 18 months later, researchers interviewed 6,850 households in the community--about 65 per slum--in an attempt to gauge changes in economic and personal well-being.

The second study, by Dean Karlan of Yale University and Jonathan Zinman of Dartmouth College, used a similar method to analyze microfinance in the Philippines, but instead of communities they looked at individual borrowers, tracking a total of 1,601 marginally credit-worthy loan applicants in Manila. Half were randomly offered a loan; half were denied. Researchers interviewed the sample group 11 to 22 months later to see if anything had changed. In both cases, results were mixed.

"While microcredit succeeds in affecting household expenditure and creating and expanding businesses, it appears to have no discernible effect on education, health or women's empowerment," states the India study. "Of course, after a longer time, when the investment impacts (may) have translated into higher total expenditure for more households, it is possible that impacts on education, health or women's empowerment would emerge.

However, at least in the short term (within 15-18 months), microcredit does not appear to be a recipe for changing education, health or women's decision making. Microcredit therefore may not be the miracle that is sometimes claimed on its behalf, but it does allow households to borrow, invest, and create and expand businesses."

The Empowerment of Women
Alex Counts, president of Washington-based Grameen Foundation USA, calls the studies "preliminary" and believes the timeframe is too limited to be meaningful. He cautions against making sweeping generalizations about microfinance on the basis of two studies, given how environments can vary from country to country.

A better measure, he argues, is research his organization commissioned in 2005 that analyzed 90 major impact studies and found that microfinance had positive effects. Counts also questions the Manila results based on the lender involved. "They studied a microfinance institution that is in no way typical of good microfinance practice in the Philippines," Counts says.

For microfinance practitioners who focus on women's empowerment, the results of the studies were particularly surprising. The India study found no impact on women's decision making, while the Manila study found some evidence that microcredit helped men more than women.

"It surprised me but I also felt that it was leaving something out," says Mary Ellen Iskenderian, president and CEO of Women's World Banking. The 30-year-old New York-based institution, which works with 40 microfinance lenders in 28 countries, has "solid data" from numerous studies around the world that shows women who take out microloans experience a decrease in domestic violence. Women's empowerment is "infinitely harder to measure but every bit as important as the economic change," says Iskenderian, who sees the empowerment factor come up in many studies. "It's borne out by data as well as [by] speaking to the women themselves."

Indeed, it may not be practical--or even possible--to measure poverty alleviation or profound social changes with randomized controlled trials, some say. Microfinance is too complex, its programs too varied. Loans are all different sizes for different types of businesses, and villages and families are never exactly the same. As microfinance grows and expands, it has become more difficult to isolate a "control group," especially in urban areas, because communities not served by one microfinance institution may still have access to another. Iskenderian recalls hearing of one attempted study that had to regroup after researchers discovered women in the control group (the village denied microfinance) were secretly sending their daughters to apply for loans in the next village, where credit was available.

Despite controversy over methodology, the two studies reflect a recent push in the microfinance industry to better measure social performance, says Wharton management professor Keith Weigelt. In recent years, as an increasing number of for-profit lenders have entered a market once dominated by non-profits and aid agencies, experienced practitioners have become concerned that microfinance is straying from its original mission of social development--what those in the industry call the "double-bottom line."

Meanwhile, microfinance institutions and their donors have stepped up efforts in the past few years to improve transparency and impact assessment. The Grameen Foundation and the Consultative Group to Assist the Poor (CGAP), the microfinance research arm at The World Bank, commissioned the development of the Progress out of Poverty Index (PPI) to help microfinance institutions measure social impact.

In 2008, the MasterCard Foundation, a large microfinance donor, gave $740,000 to the Microfinance Information eXchange (MIX) to help launch a web platform to track performance. And large donors such as The Bill and Melinda Gates Foundation have financed increasing numbers of studies on microfinance. "It's an evolution of the industry that we are seeing more studies being done," according to Weigelt. Finding an accurate measure of social impact is "the holy grail."

Rachel Glennerster, executive director of the Abdul Latif Jameel Poverty Action Lab at MIT and co-author of the India study, believes her work attracted so much attention in the press because it filled "a gaping hole" in microfinance research. "There were lots of questions being asked: Does it work? What does it really achieve? And there just wasn't really good evidence to answer the questions," she said. "So you fill a gap. You answer a question that people desperately want answered."

The Complexity of Poverty

While most microfinance practitioners say they welcome the data and look forward to more studies, not everyone agrees how and what those studies should measure next. Some acknowledge that they always knew the limitations of microfinance, and never claimed it would work miracles. They say that 30 years of surveys provide ample support for microfinance's social benefits, and should not be dismissed because of two studies. Others find flaws in the new studies themselves: The timeframe is too short, the scope is too limited, and randomized controlled trials are more suited to measuring something very specific--like the effect of a drug--than something as complex as poverty. Still, others say this is a blessing in disguise--an opportunity to rethink and regroup as microfinance matures into its next phase.

Although inconclusive on their own, the recent studies "mark a milestone" in microfinance research, states David Roodman, research fellow at the Center for Global Development. Roodman, who admits he has "spent way too much time taking apart studies" on the impact of microfinance, says he has been "almost universally unconvinced" by studies of the past. In June 2009, Roodman co-authored a working paper titled, "The Impact of Microcredit on the Poor in Bangladesh: Revisiting the Evidence," with New York University professor Jonathan Morduch that appears to upend one of the industry's most oft-quoted bodies of research--field surveys done in Bangladesh in the 1990s that Grameen Bank founder and 2006 Nobel Peace Prize winner Muhammad Yunus often cites as proof that 5% of the bank's borrowers pull themselves out of poverty each year.

Roodman insists he isn't skeptical of microfinance itself, but believes the data gathered up to now have not conclusively proven any benefit on average. "The real problem is that this data doesn't answer what we want it to answer," he says. "My stance has always been 'show me.'" By using a randomized controlled trial method, considered "the gold standard in drug trials," the new studies offer more credible data than microfinance has had before, Roodman says. "An intellectual revolution has arrived at the doorstep of microfinance."

What purpose the data actually serve is not easily answered. Michael Chu, a management professor at Harvard Business School, an expert in microfinance and former president and CEO of Accion, points out that health, education, finance and poverty are often tightly intertwined. A microloan might help a family start a business, for example, but if a single family member suddenly falls ill, the borrower might be unable to keep the business going--and will probably sink every penny the family has into emergency health care, he says.

Using randomized controlled trials to test something very exact--such as effectiveness of working capital--could be useful, according to Chu. Aside from that, he questions if so much precision is necessary. "What are you trying to do? Are you trying to get to the precision of academia? Or decision making? Because for certain parts of the academy, it may make sense to see if it was 53% or 57%, the impact of X.

But for a decision maker, whether it's somebody in the government trying to act against poverty, somebody on the front lines managing microfinance institutions, or somebody in a philanthropic organization trying to decide where to deploy the dollars, that's another level of decision making. In that case, a) you need to act, and b) do something that's intelligent. ... What level of precision do you need? We're discussing poverty. Urgency matters. Of course you can sit on the sidelines and wait for all these trials, but if you need that kind of precision to act, you're never going to be at the forefront of social action."

Others say microfinance has never been a silver bullet to alleviate poverty, and that financial access itself is enough of a goal. "It's somewhat narrow to define the value of microfinance intervention just as poverty alleviation," says Eliza Erikson, who manages the microfinance portfolio for Calvert Foundation, a nonprofit in Bethesda, Md., that focuses on socially responsible investment. "The benefit of microfinance isn't necessarily poverty alleviation but extending access to financial services, which allows people to remain stable and not live such precarious lives. There's a very tenuous line between being poor and being destitute."

The Next Wave of Research
Most people who live in poverty actually have extremely sophisticated financial lives, in part because their incomes fluctuate wildly from one day to the next, says Morduch, the New York University professor, in his new book, Portfolios of the Poor: How the World's Poor Live on $2 a Day. Having access not only to credit but also to savings, insurance and remittances would bring benefits to the poor by helping them stabilize their lives, according to Morduch. A secure savings account or an insurance policy might not lift a person out of poverty, for example, but it might alleviate an enormous amount of stress and help him or her avoid financial catastrophe.

Those benefits are unlikely to show up in the results of a randomized, controlled trial, however. "Portfolios of the Poor shows that people are looking to better manage their resources--not to grow a business necessarily, but to cope better with the ups and downs," says Morduch. "They're looking for things that are not so easily picked up by these kinds of studies."

Others say more studies are needed on the different types of microfinance and how it can be used in conjunction with other programs. "Microfinance is only one tool," says Donna Katzin, executive director of Shared Interest, a New York-based nonprofit that facilitates microloans in South Africa. "It's most productively used when it's combined with other strategies to build social capital as well as financial capital." For example, microfinance programs can be paired with education or health programs to achieve greater social benefits than if they focused on business alone, she adds. The studies "will really become useful to us when they help us discern which microfinance strategies have the most impact."

Waiting in line to add to the new research is Wharton business and public policy professor Santosh Anagol, who is currently planning a study that might help answer such questions. Anagol is setting up a randomized, controlled trial in Uttar Pradesh, a province in northern India, to study about 70 rural villages split into three groups. One village will receive access to microcredit, most likely through a loan officer from a for-profit microfinance institution who will visit the villages on a regular basis. The second group will receive microcredit in conjunction with some type of training program focusing on women--possibly a program about health, literacy or empowerment. The final group will receive neither service.

Anagol hopes the study will yield findings not only about the impact of microfinance on poverty but also on gender empowerment. "One of the things that both the aid community and academics are thinking about is, 'Is there a way to leverage microfinance to [promote] other types of services?'" Anagol says. The project is still in the early stages and will likely take at least three years to complete. "That's one of the weaknesses of these randomized evaluations. They do take a lot of time and resources."

Katie Torrington, who facilitates studies about the impact of microfinance on clients for Washington-based FINCA International, is looking forward to the next wave of research. She says it's probably time for the industry to step back and take a fresh look at its work after years of rapid growth. In the 1980s, simple success stories were enough to fuel the industry, and in the 1990s, the focus turned to refining financial performance standards.

The next logical step is social performance. "Now, as the industry is maturing, quite rightly we're also refining our measurement processes," she says. "This is a jumping off point, and we need to replicate studies like these in more locations. ... Every 10 years or so you want to stop doing business as usual and say, 'How are things different now?' We don't view it with fear. It's an opportunity."