In 2019, KIN Center for Digital Innovation set up ChatLab to investigate the potential of chatbots for conducting qualitative research. Under the supervision of Assistant Professors, Mahmood Shafeie Zargar and Ella Hafermalz, a group of Vrije Universiteit Masters students explored how chatbots can revolutionize qualitative research. After only five months, the chatbots developed were able to successfully and autonomously complete qualitative interviews. The bots successfully captured interview data from different employees about how they had perceived their onboarding process. ChatLab showed that though the bots were not completely independent researchers, they can contribute to qualitative research.
More sophisticated chatbots may unleash a whole new range of possibilities
Chatbots have been a hot topic for the last couple of years, and they have been implemented across a range of industries. However, chatbots are rarely used to support qualitative research. One reason is that, in qualitative research, data collection is aimed at gathering rich information. Today’s chatbot applications are instead only able to participate in limited, specific conversation topics. These bots are used as funnelling instruments, channelling conversations to a range of pre-specified objectives. A second reason is that qualitative interviewing is regarded as a ‘human’ job which requires context knowledge and empathy. Chatbots often lack the ability to identify nuanced human emotions and cannot build context knowledge without first gathering a huge amount of specific and costly information. Despite these challenges, the students in ChatLab sank their teeth into developing an interview bot.
Chatbots can be designed to autonomously conduct qualitative interviews
ChatLab developed two bots that were able to autonomously complete qualitative interviews with employees on their perception of the onboarding process (often defined ast the first 90-days on a new job). Onboarding proved a great research context for these bots, as it entails both structural, repetitive procedures (e.g. a tour of the building) as well as emotional experiences of employees on their first day (e.g. feeling nervous or lost).
The ChatLab studies resulted in two important preliminary findings:
- The chatbots were quite successful in asking probing questions, which are essential questions in qualitative interviewing. The bots successfully asked follow up questions aimed at evoking more detailed answers, such as “what happened next?” or “how did that make you feel?”.
- Bots tended to reduce social desirability bias compared to face-to-face interviews. Interviewees were less inclined to give socially desirable answers compared to when facing human interviewers. The tendency to make favourable impressions on other humans dropped which resulted in interviewees answering more honestly when being interviewed by a chatbot.
On the short term, chatbots will not be able to mimic all human research skills
Still, the ‘infant’ chatbots built by ChatLab in 2019 experienced some growing pains. Compared to human interviewers, the chatbot interviews resulted in less lengthy answers since the bots captured fewer examples and justifications for answers. Whereas interviews conducted by humans often exceed the original timeslot for an interview, the chatbots rarely took longer than expected. Human interviews can surface and dig deeper into new, interesting topics that were not originally anticipated, whereas chatbots are more rigid and stick to their original interview guide. Secondly, the chatbots struggled to adjust the conversation flow and the follow-up questions based on the content of the previous responses. At times, this resulted in awkward conversations as the interviewees got frustrated when the bot asked them redundant questions. To overcome this limitation, the students argue that a lot of data, e.g. in the form of human to human interview transcripts, will be needed to further train and develop the bots.
To build more sophisticated bots, a multidisciplinary approach is essential
To get to these insights, a group of Master’s thesis students worked together in ChatLab, creating a complete design approach for the development of the chatbots. Each student covered a different perspective around the development and implementation of the chatbots into their research. They all worked together to gather traditional interview data which helped to feed the bot interview protocols and conversation flows. The bots were then deployed into real organizations and used to answer the student’s research questions.
To capture important interviewing skills and to integrate those into the bots, the students conducted both face-to-face, as well as digital interviews and used their insights to design the chatbots. After numerous iterations, the chatbots started to show sufficient abilities to be able to conduct interviews autonomously.
ChatLab’s multidisciplinary approach, combing both domain knowledge and technical design features, is quite unique compared to how chatbots are often researched or built. More often than not, a one-dimensional approach is chosen where technical design dominates. While this may work when building simplistic, funnelling bots, to develop more sophisticated chatbots, it is essential to also incorporate high-level domain knowledge. Perspectives included in the ChatLab included technical design, interview design, differences between human vs chatbot interviews, and the human perception of bot interviewers.
Next steps: toward human-robot collaboration in qualitative research?
In March 2020, a new group of students have started to build on the research conducted by ChatLab 1.0. The continuous nature of ChatLab is essential to develop a deep knowledge base across multiple cohorts. This way, the combination of theses and cohorts will help to build even more sophisticated bots able to generate and capture rich responses from interviewees.
Whether such chatbots may replace human interviewers some day is hard to answer. Traditional human abilities to capture emotions and interpret intonations or body language will remain difficult to mimic in a bot. However, while, on short term, bots may not yet reach the same level of skills as humans in semi-structured interviews, they might prove valuable at certain stages of the research process.
As is seen in other industries as well, robots and humans can very well complement each other. For example, one could think of chatbots conducting the introductory parts of interviews or doing ‘interview sweeps’, finding the most interesting interviewees for human colleagues to follow-up with. This way, chatbots could save precious time by processing more potential candidates and allowing their human counterparts to focus on the cognitively challenging parts. In this way, artificial intelligence can ‘augment’ rather than automate human tasks and skills.
In March 2020, a new group of Master’s students have picked up where the 2019 cohort left off. They will further develop the chatbot and focus on the design of the bot, the interview design, chatbots’ performance vs human interviewers, and human perception.
Deep dive into the content
Are you interested to learn more about this research? Please contact KIN.SBE@VU.NL for more information. We work with organizations to connect and translate these insights into your contexts.
Browse the 2019 cohort theses here:
- Tugce Bilge — Can chatbots make you talk?
- Carla Manquillo Huete — Application of the chatbot technology on interviewing as qualitative research tool
- Alexander Berkhout — Do we care what we say to a chatbot?
- Eleni Diasiti — Face-to-face versus Chatbot Interviews: The Implementation of Probes on the Quality of Answers
- Bas Wolff — Designing a chatbot interview