Improved and Appropriate Research Approaches and Tools for Malaria SBCC

The recent Strengthening Measurement and Evaluation for Malaria SBCC webinar featured three presentations and two new M&E tools, as well as a number of improved and appropriate research approaches and tools.

Propensity score matching (PSM) and ideation were mentioned, but not described in-depth, so more information has been included below.

What is propensity score matching (PSM)? It’s a statistical technique that allows researchers to more accurately measure social and behavior change communication (SBCC) program impact and to make a strong case for causal attribution. PSM helps researchers determine whether the program was actually responsible for the changes in knowledge, attitudes and behaviors that occurred. Impact evaluation of SBCC programs requires comparison between what happened as a result of a program and what would have happened in the absence of the program (a counterfactual). Randomized control trial designs (RCTs) accomplish this by randomly assigning some people to get a treatment and others to not get the treatment, and comparing the results between the two groups. In large-scale SBCC programs, such as entertainment-education programs that use mass media to reach a national audience, it can be impossible or undesirable to prevent some people from receiving the messages simply in order to create a counterfactual condition for the purposes of evaluation. Unfortunately for program evaluators, people who hear or see program messages may be different from those who are not exposed (those that Unexposed to the intervention matched pairs based on selected variables exposed to the intervention do not hear or see the program messages). Those people that are exposed to messages may be more affluent, better educated, more motivated or otherwise predisposed to participate in the program and respond to it. PSM provides a way to take those differences and predispositions – individuals’ propensity for exposure – into consideration and to control for them when calculating program impact.

If a sample of survey respondents is drawn in a way that is representative of the intended SBCC audience, then the PSM analysis can be used to estimate the actual number of people in the population who changed as a result of the program. The survey data used for PSM should contain measures of things that make a person more likely to be exposed to the program, such as age, gender, language preference, household income, place of residence, prior behavior, access to media and other demographic and lifestyle factors. Multiple regression analysis is used to identify which of these characteristics is most strongly related to program exposure. PSM then matches people in the survey sample who have the same characteristics that make them more or less likely to be exposed to the intervention. It then compares the extent of behavior change among similar people who were exposed (the treatment group) and those not exposed (the matched comparison group).

PSM gives us confidence that the only difference between the matched persons is the one we want to examine: exposure to a specific SBCC intervention. This allows researchers to evaluate behavior change while controlling for the variables that predispose some people to be exposed and to change. This way, without assigning some people to receive the program and denying it to others, researchers can be certain that the predisposing variables are not the reason that an individual responded positively to an SBCC program – rather, it was the program itself that had an effect on the individual’s behavior.

For an example of how propensity score matching has been used successfully, read this discussion, initiated on the online social network Springboard for Health Communication Professionals.