The recent Strengthening Measurement and Evaluation for Malaria SBCC webinar featured three presentations and two new M&E tools, as well as a number of improved and appropriate research approaches and tools.
Propensity score matching (PSM) and ideation were mentioned, but not described in-depth, so more information has been included below.
What is propensity score matching (PSM)? It’s a statistical technique that allows researchers to more accurately measure social and behavior change communication (SBCC) program impact and to make a strong case for causal attribution. PSM helps researchers determine whether the program was actually responsible for the changes in knowledge, attitudes and behaviors that occurred. Impact evaluation of SBCC programs requires comparison between what happened as a result of a program and what would have happened in the absence of the program (a counterfactual). Randomized control trial designs (RCTs) accomplish this by randomly assigning some people to get a treatment and others to not get the treatment, and comparing the results between the two groups. In large-scale SBCC programs, such as entertainment-education programs that use mass media to reach a national audience, it can be impossible or undesirable to prevent some people from receiving the messages simply in order to create a counterfactual condition for the purposes of evaluation. Unfortunately for program evaluators, people who hear or see program messages may be different from those who are not exposed (those that Unexposed to the intervention matched pairs based on selected variables exposed to the intervention do not hear or see the program messages). Those people that are exposed to messages may be more affluent, better educated, more motivated or otherwise predisposed to participate in the program and respond to it. PSM provides a way to take those differences and predispositions – individuals’ propensity for exposure – into consideration and to control for them when calculating program impact.
If a sample of survey respondents is drawn in a way that is representative of the intended SBCC audience, then the PSM analysis can be used to estimate the actual number of people in the population who changed as a result of the program. The survey data used for PSM should contain measures of things that make a person more likely to be exposed to the program, such as age, gender, language preference, household income, place of residence, prior behavior, access to media and other demographic and lifestyle factors. Multiple regression analysis is used to identify which of these characteristics is most strongly related to program exposure. PSM then matches people in the survey sample who have the same characteristics that make them more or less likely to be exposed to the intervention. It then compares the extent of behavior change among similar people who were exposed (the treatment group) and those not exposed (the matched comparison group).
PSM gives us confidence that the only difference between the matched persons is the one we want to examine: exposure to a specific SBCC intervention. This allows researchers to evaluate behavior change while controlling for the variables that predispose some people to be exposed and to change. This way, without assigning some people to receive the program and denying it to others, researchers can be certain that the predisposing variables are not the reason that an individual responded positively to an SBCC program – rather, it was the program itself that had an effect on the individual’s behavior.
For an example of how propensity score matching has been used successfully, read this discussion, initiated on the online social network Springboard for Health Communication Professionals.
What is ideation? It refers to how new ways of thinking (or new behaviors) are diffused through a community by means of communication and social interaction among individuals and groups. Behavior is influenced by multiple social and psychological factors, as well as skills and environmental conditions that facilitate behavior. SBCC can affect all of these factors.
Ideational factors are grouped into three categories: cognitive, emotional and social.
- Cognitive factors address an individual’s beliefs, values and attitudes (such as risk perceptions), as well as how an individual perceives what others think should be done (subjective norms), what the individual thinks others are actually doing (social norms) and how the individual thinks about him/herself (self-image).
- Emotional factors include how an individual feels about the new behavior (positive or negative) as well as how confident a person feels that they can perform the behavior (self-efficacy).
- Social factors consist of interpersonal interactions (such as support or pressure from friends) that convince someone to behave in a certain way, as well as the effect on an individual’s behavior from trying to persuade others to adopt the behavior as well (personal advocacy).
These elements combine to affect behavior similarly to how multiple risk factors affect the probability of getting a disease. When the ideation model is used in programs, researchers measure the factors that apply to a person, then combine them to create an ideation score that predicts how likely a person is to adopt a behavior. The more ideational factors that apply to someone, the greater the likelihood that they will adopt a health behavior. When ideational factors are summed in this way, they are highly predictive of health behaviors.