Impact Evaluation in Practice
Impact Evaluation in Practice
Impact Evaluation in Practice
Chapter Four. Randomized Assignment
Randomized Assignment of Treatment sixty-four
Chapter Ten. Evaluating Multifaceted Programs one hundred seventy-five
Evaluating Programs with Varying Treatment Levels one hundred seventy-six
Chapter Twelve. Managing an Impact Evaluation
Kinds of Data That Are Needed Using Existing Quantitative Data two hundred ninety-four Collecting New Survey Data two hundred ninety-nine
Twelve point one. Guiding Principles for Engagement between the Policy and Evaluation Teams
Twelve point three. Examples of Research-Policy Team Models
Fourteen point one. The Policy Impact of an Innovative Preschool Model in Mozambique
Fourteen point three. Disseminating Impact Evaluations Effectively two hundred fifty-five
Fourteen point five. Impact Evaluation Blogs
Sixteen point one. Constructing a Data Set in the Evaluation of Argentina's Plan Nacer two hundred ninety-seven
Sixteen point three. Designing and Formatting Questionnaires
Sixteen point five. Data Collection for the Evaluation of the Atención a Crisis Pilots in Nicaragua
B two point two point one. Identifying a Mechanism Experiment from a Longer Results Chain thirty-seven
Two point two. The HISP Results Chain forty
Three point two. A Valid Comparison Group fifty-three
Four point one. Characteristics of Groups under Randomized Assignment of Treatment sixty-eight
Four point three. Steps in Randomized Assignment to Treatment seventy-six
Four point five. Estimating Impact under Randomized Assignment eighty-one
Five point two. Estimating the Local Average Treatment Effect under Randomized Assignment with Imperfect Compliance ninety-seven
Five point four. Estimating the Local Average Treatment Effect under Randomized Promotion one hundred six
Six point two. Rice Yield, Smaller Farms versus Larger Farms (Follow-Up) one hundred seventeen
Six point four. Manipulation of the Eligibility Index one hundred twenty
Six point six. Participation in HISP, by Baseline Poverty Index one hundred twenty-two
Seven point one. The Difference-in-Differences Method one hundred thirty-two
Eight point one. Exact Matching on Four Characteristics one hundred forty-four
Eight point three. Matching for HISP: Common Support one hundred fifty-three
Ten point one. Steps in Randomized Assignment of Two Levels of Treatment one hundred seventy-seven
Ten point three. Crossover Design for a Program with Two Interventions one hundred eighty-one
Eight point three. Evaluating HISP: Matching on Baseline Characteristics and Regression Analysis one hundred fifty-four
B ten point one point one. Summary of Program Design one hundred seventy-eight
Eleven point two. Comparing Impact Evaluation Methods one hundred ninety-four
Twelve point two. Disaggregated Costs of a Selection of World Bank-Supported Impact Evaluations two hundred eighteen
Thirteen. Ensuring Reliable and Credible Information for Policy through Open Science two hundred thirty-eight
Fifteen point one. Examples of Clusters
Road Map to Contents of the Book
Complementary Online Material
Development of Impact Evaluation in Practice
Evidence-Based Policy Making
Box one point two: The Policy Impact of an Innovative Preschool Model: Preschool and Early Childhood Development in Mozambique
What Is Impact Evaluation?
Prospective versus Retrospective Impact Evaluation
Efficacy Studies and Effectiveness Studies
Box one point six: Informing National Scale-Up through a Process Evaluation in Tanzania
Cost-Benefit and Cost-Effectiveness Analysis
Box one point seven: Evaluating Cost-Effectiveness: Comparing Evaluations of Programs That Affect Learning in Primary Schools
Ethical Considerations Regarding Impact Evaluation
Impact Evaluation for Policy Decisions
Box one point eight: Evaluating Innovative Programs: The Behavioural Insights Team in the United Kingdom
Box one point nine: Evaluating Program Design Alternatives: Malnourishment and Cognitive Development in Colombia
Box one point ten: The Impact Evaluation Cluster Approach: Strategically Building Evidence to Fill Knowledge Gaps
Deciding Whether to Carry Out an Impact Evaluation
Constructing a Theory of Change
Box Two point One: Articulating a Theory of Change: From Cement Floors to Happiness in Mexico
Developing a Results Chain
Specifying Evaluation Questions
Box two point two: Mechanism Experiments
Box two point three: A High School Mathematics Reform: Formulating a Results Chains and Evaluation Question
The Health Insurance Subsidy Program: An Introduction
Selecting Outcome and Performance Indicators
Checklist: Getting Data for Your Indicators
Causal Inference and Counterfactuals
Estimating the Counterfactual
Two Counterfeit Estimates of the Counterfactual
Counterfeit Counterfactual Estimate One: Comparing Outcomes Before and After a Program
Evaluating the Impact of HISP: Doing a Before-and-After Comparison of Outcomes
Counterfeit Counterfactual Estimate two: Comparing Enrolled and Nonenrolled (Self-Selected) Groups
Evaluating the Impact of HISP: Comparing Enrolled and Nonenrolled Households
Randomized Assignment of Treatment
Box four point one: Randomized Assignment as a Valuable Operational Tool
Why Does Randomized Assignment Produce an Excellent Estimate of the Counterfactual?
External and Internal Validity
Box four point two: Randomized Assignment as a Program Allocation Rule: Conditional Cash Transfers and Education in Mexico
Box four point three: Randomized Assignment of Grants to Improve Employment Prospects for Youth in Northern Uganda
Box four point four: Randomized Assignment of Water and Sanitation Interventions in Rural Bolivia
Box four point five: Randomized Assignment of Spring Water Protection to Improve Health in Kenya
Box four point six: Randomized Assignment of Information about HIV Risks to Curb Teen Pregnancy in Kenya
When Can Randomized Assignment Be Used?
How Do You Randomly Assign Treatment?
At What Level Do You Perform Randomized Assignment?
Estimating Impact under Randomized Assignment
Evaluating the Impact of HISP: Randomized Assignment
Evaluating Programs When Not Everyone Complies with Their Assignment
Types of Impact Estimates
Randomized Assignment of a Program and Final Take-Up
Estimating Impact under Randomized Assignment with Imperfect Compliance
Box five point two: Using Instrumental Variables to Deal with Noncompliance in a School Voucher Program in Colombia
Interpreting the Estimate of the Local Average Treatment Effect
Randomized Promotion as an Instrumental Variable
The Randomized Promotion Process
Estimating Impact under Randomized Promotion
Box five point three: Randomized Promotion of Education Infrastructure Investments in Bolivia
Evaluating the Impact of HISP: Randomized Promotion
Limitations of the Randomized Promotion Method
Checklist: Randomized Promotion as an Instrumental Variable
Evaluating Programs That Use an Eligibility Index
Fuzzy Regression Discontinuity Design
Box six point two: Social Safety Nets Based on a Poverty Index in Jamaica
Checking the Validity of the Regression Discontinuity Design
Box six point three: The Effect on School Performance of Grouping Students by Test Scores in Kenya
Evaluating the Impact of HISP: Regression Discontinuity Design
Limitations and Interpretation of the Regression Discontinuity Design Method
Checklist: Regression Discontinuity Design
Regression Discontinuity Design
Difference-in-Differences
The Difference-in-Differences Method
Box seven point two: Using Difference-in-Differences to Study the Effects of Police Deployment on Crime in Argentina
The "Equal Trends" Assumption in Difference-in-Differences
Testing the Validity of the "Equal Trends" Assumption in Difference-in-Differences
Box seven point three: Testing the Assumption of Equal Trends: Water Privatization and Infant Mortality in Argentina
Box seven point four: Testing the Assumption of Equal Trends: School Construction in Indonesia
Evaluating the Impact of HISP: Using Difference-in-Differences
Limitations of the Difference-in-Differences Method
Propensity Score Matching
Combining Matching with Other Methods
Matched Difference-in-Differences
Box eight point one: Matched Difference-in-Differences: Rural Roads and Local Market Development in Vietnam
Box eight point two: Matched Difference-in-Differences: Cement Floors, Child Health, and Maternal Happiness in Mexico
The Synthetic Control Method
Evaluating the Impact of HISP: Using Matching Techniques
Limitations of the Matching Method
Addressing Methodological Challenges
Unintended Behavioral Effects
Box Nine point One: Folk Tales of Impact Evaluation: The Hawthorne Effect and the John Henry Effect
Addressing Methodological Challenges
Addressing Methodological Challenges
Designing an Impact Evaluation That Accounts for Spillovers
Addressing Methodological Challenges
Addressing Methodological Challenges
Box nine point four: Evaluating Spillover Effects: Conditional Cash Transfers and Spillovers in Mexico
Box nine point five: Attrition in Studies with Long-Term Follow-Up: Early Childhood Development and Migration in Jamaica
Timing and Persistence of Effects
Box nine point six: Evaluating Long-Term Effects: Subsidies and Adoption of Insecticide-Treated Bed Nets in Kenya
Addressing Methodological Challenges
Evaluating Multifaceted Programs
Evaluating Programs with Varying Treatment Levels
Evaluating Multifaceted Programs
Box ten point one: Testing Program Intensity for Improving Adherence to Antiretroviral Treatment
Box ten point two: Testing Program Alternatives for Monitoring Corruption in Indonesia
Evaluating Multiple Interventions
Determining Which Method to Use for a Given Program
How a Program's Rules of Operation Can Help Choose an Impact Evaluation Method
Principles for Well-Defined Program Assignment Rules
Deriving Comparison Groups from Operational Rules
Prioritizing Beneficiaries
A Comparison of Impact Evaluation Methods
A Backup Plan for Your Evaluation
Finding the Smallest Feasible Unit of Intervention
Box eleven point one: Cash Transfer Programs and the Minimum Level of Intervention
Managing an Evaluation's Team, Time, and Budget
Roles and Responsibilities of the Research and Policy Teams
The Policy Team: Policy Function and Program Management Function
Who Cares about the Evaluation and Why?
The Research and Policy Team Partnership during the Evaluation
Box twelve point one: Guiding Principles for Engagement between the Policy and Evaluation Teams
Establishing Collaboration
The Fully Integrated Model
Outsourcing Evaluations at the Millennium Challenge Corporation
Integration at Innovations for Poverty Action
Partnership Models at the World Bank
Choosing a Research Team Partner
How to Time the Evaluation
How to Budget for an Evaluation
Budgeting for an Impact Evaluation
Options for Funding Evaluations
Managing Ethical and Credible Evaluations
The Ethics of Running Impact Evaluations
The Ethics of Assignment to Treatment and Comparison Groups
Protecting Human Subjects during Data Collection, Processing, and Storage
Ensuring Reliable and Credible Evaluations through Open Science
Publication Bias and Trial Registries
Data Mining, Multiple Hypothesis Testing, and Subgroup Analysis
Checklist: An Ethical and Credible Impact Evaluation
A Solid Evidence Base for Policy
Box fourteen point one: The Policy Impact of an Innovative Preschool Model in Mozambique (continued from chapter one)
Tailoring a Communication Strategy to Different Audiences
Box fourteen point two: Outreach and Dissemination Tools
Box fourteen point three: Disseminating Impact Evaluations Effectively
Box fourteen point four: Disseminating Impact Evaluations Online
Box fourteen point five: Impact Evaluation Blogs
Part four HOW TO GET DATA FOR AN IMPACT EVALUATION
Sampling and Power Calculations
Box fifteen point one: Random Sampling Is Not Sufficient for Impact Evaluation
Deciding on the Size of a Sample for Impact Evaluation: Power Calculations
The Rationale for Power Calculations
Estimating Average Outcomes for the Treatment and Comparison Groups
Comparing the Average Outcomes between the Treatment and Comparison Groups
Two Potential Errors in Impact Evaluations
Why Power Calculations Matter for Policy
Power Calculations Step by Step
Evaluating the Impact of HISP: Deciding How Big a Sample Is Needed to Evaluate an Expanded HISP
Power Calculations with Clusters
Evaluating the Impact of HISP: Deciding How Big a Sample Is Needed to Evaluate an Expanded HISP with Clusters
Moving Beyond the Benchmark Case
Kinds of Data That Are Needed
Using Existing Quantitative Data
Box sixteen point one: Constructing a Data Set in the Evaluation of Argentina's Plan Nacer
Box sixteen point two: Using Census Data to Reevaluate the PRAF in Honduras
Collecting New Survey Data
Determining Who Will Collect the Data
Developing and Piloting the Data Collection Instrument
Box sixteen point three: Designing and Formatting Questionnaires
Box sixteen point four: Some Pros and Cons of Electronic Data Collection
Conducting Fieldwork and Undertaking Quality Control
Processing and Storing the Data
Box sixteen point five: Data Collection for the Evaluation of the Atención a Crisis Pilots in Nicaragua
Box sixteen point six: Guidelines for Data Documentation and Storage
Checklist: Core Elements of a Well-Designed Impact Evaluation
Checklist: Tips to Mitigate Common Risks in Conducting an Impact Evaluation