Research for Haiti

Evaluation Resources: Free Introduction and Guidance on Impact Evaluations

The Network of Networks on Impact Evaluation (NONIE) has a very readable and fairly brief introductory text, Guidance on Impact Evaluation, available for free on its web site. (Note: Keep in mind that impact evaluation is not the only kind of evaluation, so this is not a review of the broader field of evaluation.)

From the introduction:

The purpose of NONIE is to promote more and better impact evaluations among its members. Issues relating to evaluations in general are more effectively dealt with in the parent networks, and are thus not the primary focus of NONIE. NONIE will focus on sharing of methods and learning-by-doing to promote the practice of IE (impact evaluation). The current Guidance document was developed for supporting those purposes.

The Guidance is structured around nine key issues in impact evaluation:

1. Identify the (type and scope of the) intervention
2. Agree on what is valued
3. Carefully articulate the theories linking interventions to outcomes
4. Address the attribution problem
5. Use a mixed methods approach: The logic of the comparative advantages of methods
6. Build on existing knowledge relevant to the impact of interventions
7. Determine if an impact evaluation is feasible and worth the cost
8. Start collecting data early
9. Front-end planning is important

Impact Evaluation Reading (free online): “Handbook on Impact Evaluation”

You can read “Handbook on Impact Evaluation” by Khandker, Koolwal, and Samad for free online.

The book focuses on quantitative methods (using examples from developing nations and even some advice on using the statistical software program Stata). Quantitative methods may not appeal to many (same with reading a book online) but the first chapters alone appear to be well worth reading if you are not sure what the heck “evaluation” is or how various kinds of evaluations differ.

Update: Tracking the $3 Billion and Now Private Funds

A prior post provided some early thoughts on where the money is going and where it is coming from. I know see an interesting "make a table" function at Relief Web that presents data from the Financial Tracking System (FTS) of the UN. The page and the instructions are a little confusing, but here's the table I produced (I move to the second tab some text the table generates, sorted by recipient (called "appealing agency" awkwardly fy FTS, and I hid some columns to make the table tidier, but not date was removed). The figures are updated from the prior post and it has more details (as well as fewer acronyms). Please try out the make a table function and let me know if you get something different.

I think it is urgent that larger recipient groups begin re-granting to small NGOs in Haiti, especially in the countryside, to jump start relief efforts that are focused on the planting season, on decentralizing population, and on putting people to work. This should be tracked, too, so the grants are transparent and that we know where the money is finally going and for what.

The Center for Economic and Policy Research (CEPR) has a Haiti Relief and Reconstruction Watch blog set up. They note that the Federal Procurement Data System has set up a special page for accessing data on US government contracts in Haiti (note: you have to set up a free account to access the data). I have not had a chance to play with it yet, but CEPR discusses what they have learned on their page. Press and advoactes will be making lots of heat around for-profit contractors (especially those that have parent companies with financial interests in Haiti, or that have had scandals in their past) receiving money for work in Haiti.

Where is the $3 Billion to Haiti Going?

Relief Web has posted some tables on the amounts INGOS and donor nations are giving to Haiti. I had an analysis posted here, but the formatting was going haywire for some reason. The observations I made are now in this file.

Aid for Haiti: What kind? Planned by whom? For Whom?

The editor of Wired shares on their popular new blog Haiti Rewired some honest reflections on how hard it is for people to assess what it is Haiti will need:

Perhaps naturally, as the editor of Wired.com, I've been consciously and unconsciously framing the reconstruction in terms of delivering telecom service over mesh networks, bootstrapping data mashups to bring a Web 3.0 layer to the relief, delivering green tech and so on. Sexy.

Here's what the Lambi Funds wants: Electric grain mills for grinding corn and millet, a portable irrigation pump, ox ploughs, a goat breeding program, a fishing boat and reforestation.

Bad aid starts with ignorance and condescension.

(As an aside: technophiles may wish to take a look at this list of cautionary tips by the people at Catapult Design.) In addition to the plan from Lambi Fund, one can look at this rough plan that a priest in the rural community Fondwa made. Fondwa's agriculture school (one of only a few in the countryside in Haiti) and other community buildings suffered damage from the earthquake. Note that these plans, general as they are, focus on very concrete and immediate needs rooted in the experience of projects under the control of local leaders.

On a completely different scale then the mistakes that some individual donors or observers may make, International Organizations (IOs) and International NGOs (INGOs) run the risk of selecting goals that favor their own organizational needs (the literature on this is mixed but the concern is certainly fair). As I have posted before, albeit regarding peacekeeping operations, large international programs can have economic impacts that are not easy to discern from the outset. Moreover, there are important questions about the transparency and accountability of the planning process for the expenditure of hundreds of millionsof dollars that have been raised by INGOs or committed by donor nations (pledges which often never materialize). Who will be selecting the values and outcomes the projects aim for? Who will evaluate them and manage them? Will they improve or reduce self-determination in Haiti? (see Paul Farmer's testimony to the US Senate on this last point.)

On a related note, I want to get off my chest a pet peeve: It has been popular in the press recently to complain about the billions in international aid funds that "Haiti wasted." But for decades aid to Haiti was controlled by donors not Haitians, or at least not many Haitians. (I swear some journalists make it sound like they think all of Haiti got together in a soccer stadium and voted to spend aid money on lottery tickets.) While it is true that the Haitian government (at all levels) has grave weaknesses, the opportunity in Haiti now,it seems to me, calls for strengthening and developing Haitian government, not acting like the Duvaliers are still hanging around. The fact is there is an associational life in Haiti that is very rich, indicating democratic skills rivaling those at which US citizens are famously believed to excel. (For examples: See this article by Media Global for also about Fondwa. And, to finish off your dose on the interesting work that has been going on in Fondwa see this documentary; the portion on development begins about half way through).

 

 

Evaluation: Textbooks

Do you have introductory books (or articles) on evaluation to suggest? I will try to post some online resources later, of course, but here are two books with a wealth of introductory information on evaluation. The books below are texts for graduate students and professionals. Thus, while readable, the size and breadth of these books may not make them useful for boardmembers or organizational membership, so I will look for some workbooks and other materials that address those audiences. If people have materials in French or Creole they wish to identify for others, please send it in (see top post for contact information).

1. Evaluation: A Systematic Approach by Rossi, Lipsey, and Freeman. Often considered THE introductory text to the field of evaluation. The link takes you to Google Books where much of the book is availabe for examination. Even with a few pages missing, many of the chapters are well worth reading or consulting on-line if that is your only form of access.
   
2. Handbook of Practical Program Evaluation edited by Wholey, Hatry, and Newcomer. Each chapter introduces a topic or innovation in evaluation; this is not an overal introduction but readable reviews of essential topics by experts. Again, much of the book can be reviewed at Google Books which is where the link above takes you.