Instrumental Variables Method for Impact Evaluation

1 Instrumental Variable Method for Impact Evaluation SA. Quimbo 1 September 2014 Asian Development Bank Why IE? To estimate the treatment effect of an intervention T on an outcome Y Examples: What is the effect of a credit program participation on consumption? On health? What is the effect of a conditional cash transfer program on consumption? education? maternal outcomes? What is the effect of a subsidy on health insurance premiums on enrolment? The Main Challenge: Finding the Counterfactual Key Question: What would have happened to the beneficiaries if the program had not existed? Ideal Strategy Source: Khandker (2006) Ideal Strategy Randomized evaluations: Allocate a program randomly across a sample of observations Two groups: control (counterfactual) and treatment groups Control and treatment groups have the same expected outcome in the absence of the program Challenges: ethical considerations, finding the equivalent group, costs Typical Case Reality: most programs require participation among a group of eligible individuals or households Example: Micro-credit program Targeted to eligible households Voluntary participation among the eligible or non-eligible Less-than-ideal Strategy Time Source: Khandker (2006) Projection for beneficiaries, without the policy start Evaluator s Problem.

2 Self-selection Program participation is non-random or a choice variable or endogenous Both observed and unobserved characteristics of participants make them different from those who did not participate Unobserved preferences and motivations Unobserved ability Unobserved opportunity costs of participating Estimation Equation To compare the outcomes of individuals that chose to participate and not: T=1 if enrolled in the program T=0 if not enrolled in the program X = vector of exogenous characteristics that also affect Y e = error term, , unobserved characteristics affecting the outcome Y=a0X+a1T+e The Estimation Problem when Self-Selection is Present Corr (T,e) 0 Treatment variable of interest is endogenous in the equation This violates a key assumption of Ordinary Least Squares (OLS) Recall: OLS yields Best Linear Unbiased estimates Estimate of 1 is inconsistent (does not tend toward the population parameter being estimated as sample size increases) Instrumental Variables Method provides a simple solution to the problem Two Requirements for IVs Let T be the potentially endogenous regressor.

3 We need a variable z (not among the Xs) that satisfies two conditions: Condition 1: Corr(z,T) 0 or z is correlated with T Condition 2: Corr(z,e) = 0 or z must be uncorrelated with e Y=a0X+a1T+eRequirements for IVs Graphically z (1) T (2) y (3) (4) e OLS requires that arrow (3) does not exist If it does, one needs an instrument z but there should not be an arrow from e to z (condition 2); no route from z to y except those through arrows (1) and (2), and arrow 1 must exist (condition 1) There must be as many instruments are there are endogenous regressors Estimation technique: Two Stage Least Squares Stage 1. Regress T on Z and X. Obtain the predicted value of T, T-hat (predicted values of T) reduced form Stage 2.

4 Regress Y on X and T-hat Treatment effect is T=b0X+b1Z+uY=a0X+a1T^+ea^1 Implementing this in stata Option 1 reg t z X predict that reg y X that (problem: reported standard errors are incorrect) Option 2 ivreg y X (t=z) Remarks on the Use of IV IV is used when the unobserved characteristics (captured by e) affecting T vary over time Another Method DID can be used if these unobserved characteristics do not vary over time Example Microcredit The Impact of Group-based Credit Programs on Poor Households in Bangladesh: Does the Gender of Participants Matter? (Pitt and Khandker 1998, Journal of Political Economy) Impact Evaluation : Instrumental Variable Method (presentation by Khandker in June 2006) Background Impact of the 3 credit programs in Bangladesh including that of the Grameen Bank All 3 programs offer credit to the landless rural poor (owning less than acre of land), using peer monitoring as a substitute for collateral Grameen Bank provides credit to members who form self-selected groups of 5 Loans are given to individual group members, but the whole group become ineligible for further loans if any member defaults Groups meet weekly to may payments on their loans Estimation Approach The paper estimates the Impact of program participation, by gender, on consumption, assets, expenditure, labor supply, and schooling The outcome equation: Y outcome.

5 X individual characteristics, C credit, j village, i individual, and are unobserved characteristics Estimation Approach The credit equation (measures program participation): Z individual characteristics affecting demand but have no direct effect on the outcome Endogenous Program Placement Error terms in the C equation are correlated with the error terms in the Y equation Unobserved village characteristics can affect both demand for credit and outcomes Unobserved household characteristics can affect both demand for credit and outcomes Non-random placement of credit programs (poorer and more flood prone areas; areas whose officials requested for the program) Remedial Measure Household-level endogenous program placement can be resolved by Instrumental Variables In the C equation, Z represents Instrumental Variables In Search of Z Solution (2-step): 1.

6 Use a survey design that is quasi-experimental Households are sampled from program and non-program villages Eligible and non-eligible households are sampled from both types of villages Participant and non-participant households are sampled from eligible households In Search of Z 2. Need an exogenous rule for program participation: Landholding criteria: < acre of landholdings Gender: Male (female) members of landless households cannot participate in program if village does not have a male (female) program group Impacts of Grameen Bank Women s Credit on Household Per Capita Consumption Example Dams (Duflo and Pande, 2007) Background and study objectives Lots of dams in India (4k large dams, 3rd largest builder of dams) Good effects: increased irrigation, hydroelectricity Bad effects: displaced people, altered cropping patterns, increased salination and waterlogging of arable land (reduced land productivity) To evaluate the Impact of dams in agricultural production and poverty reduction Estimation issue: dams are not exogenous.

7 Dam construction is determined by regional economic growth and returns to investments; dams are subject to endogenous program placement Dams are constructed in fast growing areas spurious positive relationship between dam construction and poverty reduction OLS estimates would yield inconsistent estimates of dam effects on poverty reduction: Identification of the instrument River gradient affects ease of dam construction: Low gradient: suitable for irrigation dams High gradient: suitable for hydroelectric power Identifying assumption: The nonmonotonic relationship between the likelihood of dam construction and river gradient forms the basis for identification river gradient -> DAMS -> poverty Definition of river gradient District gradient characterizes the steepness of the ground surface and is measured as the tangent of the surface.

8 Measure used for the analysis: Fraction of district area falling into four gradient categories: flat ( percent), moderate ( percent), steep (3-6 percent), and very steep (above 6 percent). General description of the estimation approach First stage: Number of dams per district is regressed on river gradient (interacted with dam incidence in the state), and other Variables Second stage: Y = f(predicted number of dams, inter alia) Conclusion Our poverty results also suggest a worsening of living standards in the district where the dam is Exercise The Philippine Conditional Cash Transfer Program Pantawid Pamilya is a human development program of the national government that invests in the health and education of poor households, particularly of children aged 0-18 years old. Patterned after the conditional cash transfer scheme implemented in other developing countries, the Pantawid Pamilya provides cash grants to beneficiaries provided that they complywith the set of conditions required by the program.

9 Pantawid Pamilya operates in 79 provinces covering 1484 municipalities and 143 cities in all 17 regions nationwide. The program has 4,090,667 registered households as of 25 June 2014. The Philippine Conditional Cash Transfer Program To avail of the cash grants beneficiaries should comply with the following conditions: 1. Pregnant women must avail pre- and post-natal care and be attended during childbirth by a trained health professional; 2. Parents must attend Family Development Sessions (FDS); 3. 0-5 year old children must receive regular preventive health check-ups and vaccines; 4. 6-14 years old children must receive deworming pills twice a year. 5. All child beneficiaries (0-18 years old) must enroll in school and maintain a class attendance of at least 85% per month. The Philippine Conditional Cash Transfer Program Health Grant: P500 per month per household or P6000 per year Education Grant: P300 per month for 10 months for children ages 3-14 years old, up to a maximum of 3 children per household Each household receives P1,400 per month (500 pesos per month for health and 900 pesos per month for education) for 5 years as long as conditions are satisfied Evaluation Questions are the outcome indicators?

10 There an endogenous program placement problem? Why or why not? yes (in No. 2), how can this be addressed using IV? What instruments can be used to identify program Impact ?