A/B testing the effect of active listening

In 2020, Katikati supported Africa’s Voices Foundation to run an A/B test to improve how they handled conversations. This is how they did it in Katikati and what they found.

Background

During July to November 2020, Africa’s Voices Foundation (AVF) used Katikati to understand and respond to radio audiences in Somalia who expressed rumour, stigma and misinformation in regards to the COVID-19 pandemic in their SMS messages to their radio shows (see use case here). 

Given the sensitive nature of such topics, building a trusted channel of two-way communication is central to shifting ideas, norms and practices. As a result, AVF wanted to explore whether taking a more conversational approach to respond to people who shared rumour, stigma and misinformation would lead to more active engagement from the person as an indication of more effective communication of preventative health messages.

Set-up

Katikati is designed to allow you to tailor conversations for individuals as well as groups of people. This makes it easy to run a small live experiment (an “A/B test”) on two different messaging approaches, which is exactly what AVF did for responding to rumour, stigma or misinformation comparing between: (A) a standard health message clarifying the misinformation or rumour shared, and (B) a more conversational engagement aiming to first draw the participant into a short discussion before sharing the same standard health message.

Katikati’s conversation tagging feature was of key use here to select, organise, and treat the two groups of people differently. For this experiment, AVF selected a group of 258 individuals who had sent messages containing evidence of rumour, stigma or misinformation, and then further split them into two groups: the A group (Control) consisted of 166 people, whilst the B group (Treatment) included 92 people.

This more conversational approach took the form of messages to an individual which valued the individual's voice, asked questions to understand their thoughts and valued the conversation in the closing statements. This resulted in conversations for the Control vs Treatment Group like the following (Note: this example is in English but these conversations were held in Somali):

Results

One key aspect AVF was interested in was whether the conversational approach created more active and positive engagement as an indicator of uptake of health messaging. To measure this, they designed specific metrics to track for example whether a participant replied in response to the standard health message, whether their reply was positive or negative, whether the participant replied to next week’s questions and whether their reply was substantive and on topic) and so on.

Katikati’s tagging feature came in useful again. AVF researchers developed a coding frame for the designed metrics, and then reviewed the conversations and labelled the messages according to the developed metrics. Considering the below metrics, AVF learned that a conversational approach leads to a higher response rate which is sustained even two weeks later and results in more positive sentiments as compared to standard protocol messaging.

Whilst the conversational approach increased engagement for both, when the data was disaggregated by gender this effect is much more pronounced for females compared to male as can be seen in this table below.

Conclusion

Africa’s Voices Foundation conducted this conversational experiment whilst continuing to respond and engage other participants at the same time. Katikati makes it possible to have different, scalable and human-led conversations in the same space through a combination of novel interface design and machine assistance that augments human capability rather than replaces it.

This A/B test showed that active listening can help build engagement and potentially provide a better approach to addressing rumour, stigma and misinformation. As a result of this test, AVF revised their standard messaging to start with an active listening component, while maintaining the standard single-reply approach.

Many thanks to Ella Rechter for her help in producing this Use Case.