A-CONECT: Designing AI-based Conversational Chatbot for Early Dementia Intervention

Junyuan Hong*1, Wenqing Zheng*1, Han Meng2, Siqi Liang2, Anqing Chen1, Hiroko H. Dodge3,4, Jiayu Zhou2, Zhangyang Wang1
1University of Texas, Austin, 2Michigan State University, 3Massachusetts General Hospital, 4Harvard Medical School,
*Indicates Equal Contribution

utaustin msu mgh hms

Paper Chatbot

Motivated by the clinical finding that frequent conversational interactions may slow down dementia development, we design the A-CONECT chatbot with an end-to-end pipeline including digital-twin simulation, evaluation and design iterations.

Abstract

Mild Cognitive Impairment (MCI) is a prodrome stage of Alzheimer's Disease and related dementia. Its detection is essential for early intervention and cohort enrichment. A recent clinical trial showed that frequent conversation could be an effective strategy against social isolation and cognitive decline. Though effective, there are challenges for an intervention to be widely deployed due to the involvement of trained human moderators. In this paper, we study an innovative solution that uses an AI-based chatbot to replace human moderators, thus greatly improving the accessibility of this accessible therapeutic approach. We integrate the established clinical trial protocols into the automatic chatbot for stimulating cognitive functions through cognitively demanding, engaging, and user-friendly voice-based conversations. To evaluate the effectiveness, we create MCI digital twins--virtual replicas of MCI patients--offering a scalable and realistic assessment method. With the digital twins, we provide an end-to-end framework for evaluating and iterating the chatbot. Our experiments show the chatbot's proficiency in fostering natural conversations and its potential as a cost-effective, accessible tool in dementia intervention.

A-CONECT Chatbot

Evidence from previous research shows that more participation in cognitive, leisure, and social activities correlates with a decreased chance of being diagnosed with dementia diseases. These findings drive the conversational intervention for cognitive diseases. Our proposed method uses a chatbot to resemble the human moderator in dementia intervention and, therefore, should follow the principles of the I-CONECT clinical protocol (Dodge, et al., 2023).

P1: Cognitively-demanding Conversation Strategies:

  • P1.1: Novel chat experiences.
  • P1.2: Stimulating executive functions.

P2: Patient-Friendly Interaction.

  • P2.1: User-friendly interface that does not have any barriers for the oldest adults to use the software/hardware.
  • P2.2: Engaging conversations that should encourage active patient participation.
  • P2.3: Data privacy should be seriously handled towards building trust with users.

Sample conversation with the A-CONECT chatbot
Sample conversations from clinical trial are available at the I-CONECT project which was used to build Digital Twins.

Development Pipeline

The language functionality lies at the core of the proposed conversational chatbot and therefore should be validated before deployment. In this section, we provide an end-to-end pipeline that evaluates and iterates the LLM-based design. The crux of the pipeline is an interactive test environment in which we can probe the capabilities of the chatbot. As outlined in the below figure, the pipeline includes 6 steps. (1) We curate data of conversations between patients and moderators. (2) Then, we utilize the data to create digital twins for each patient. (3) We design the chatbot based on pre-trained LLMs. (4) We simulate conversations between the digital twin and the chatbot. (5) We evaluate the digital twins by examining the symptom similarity between the conversations by digital twins and those by human patients. (6) With the simulated conversations, we evaluate if the chatbot can realize the design principles. Based on the evaluation results, we can iterate or rank design choices in step (3). Finally, we deploy our chatbot in conversations with humans.

diagram

The development process of building a chatbot for MCI patients. We collect patients' conversation data from human clinical trials and build digital twins to simulate conversations with the designed chatbot.

Evaluation of Chatbot in Simulation

In the below table, we evaluate the chatbot in conversations with 9 digital twins of patients and report the t-test results on the difference between ChatGPT and our chatbot as moderators. Each digital twin is evaluated with 20 conversations. Across all patients, we observe significant changes in Recall Ratio, Patient/Moderator Word Ratio, and Question Ratio. The changes imply that the chatbot is actively involving the patients in conversation probably by asking more questions. The engagement leads to more cognitively demanding recalling of past experiences and more speeches by the patients. In terms of the fluency of patients or the moderator, there are no significant differences. In other words, the chatbot is as fluent as the ChatGPT.

Mod RR ↑ P-FR ↑ WR ↑ M-FR ↑ QR ↑
ChatGPT 0.271 0.462 0.425 0.999 0.213
A-CONECT 0.423 0.421 0.701 0.999 0.616
p-value 0.001 0.212 0.014 0.859 0.000

BibTeX

@article{hong2024aconect,
  title={A-CONECT: Designing AI-based Conversational Chatbot for Early Dementia Intervention},
  author={Hong, Junyuan and Zheng, Wenqing and Meng, Han and Liang, Siqi and Chen, Anqing and H. Dodge, Hiroko and Zhou, Jiayu and Wang, Zhangyang},
  journal={ICLR 2024 Workshop on Large Language Model (LLM) Agents},
  year={2024}
}