Overview
Heart disease remains the leading cause of death for women in the United States. Even with public campaigns, awareness has fluctuated, underscoring the need for scalable education approaches. This analysis compares two approaches to teaching women about heart attack symptoms and response: a human-delivered SMS intervention and a fully automated AI chatbot named HeartBot. The study contributes to our understanding of how delivery mode influences knowledge gains, user experience, and perceived conversational quality.
Background and Rationale
Previous research suggests AI chatbots can support health education, but direct comparisons with human-led interventions are limited. The HeartBot project built content from clinical guidelines and public health materials to educate women without a heart disease history. The human dataset used a cardiovascular nurse interventionist delivering content through two sessions over one week. The secondary analysis aims to determine whether HeartBot can match or approach the impact of human-delivered education on recognizing heart attack signs, differentiating symptoms, calling 911, and reaching the ER within 60 minutes.
Study Design and Datasets
This analysis pooled two non-randomized datasets from the AI Chatbot Development Project conducted between September 2022 and January 2024. Participants were US-based women aged 25 and older, without prior heart disease or stroke, and without health professional status. The Human dataset involved two online conversation sessions with a trained interventionist; the HeartBot dataset involved a single SMS-based session with the AI chatbot. Both arms used the same knowledge questions to assess outcomes before and after the intervention.
Primary Outcome: Knowledge and Awareness
The core outcomes measured participants’ confidence in recognizing heart attack signs and their ability to respond appropriately. Four questions were adapted from a validated scale, each rated from 1 (not sure) to 4 (sure). Across both delivery modes, post-intervention scores rose significantly, indicating enhanced knowledge and readiness to act. However, the magnitude of improvement differed by delivery mode and question.
Secondary Outcomes: User Experience and Conversational Quality
Researchers evaluated message effectiveness, perceived humanness, naturalness, and coherence using established scales. They also assessed LIWC-derived word counts to gauge engagement, and asked participants whether they believed they were texting a human or a chatbot. Overall, participants rated human-delivered conversations higher on conversational quality and related metrics. A sizable portion could not reliably identify the partner’s identity, reflecting similar levels of perceived agency between human and AI in some cases.
Key Findings
- Human-delivered education significantly increased all four knowledge outcomes, with large adjusted effects for recognizing signs (AOR ≈ 15) and differentiating symptoms (AOR ≈ 9).
- HeartBot also improved knowledge, though effect sizes were smaller than the human arm, and statistical significance persisted across most questions.
- Conversational quality and perceived message humanness were higher in the human-delivered arm, likely due to longer, more nuanced interactions.
- HeartBot demonstrated potential as a scalable, cost-efficient alternative, especially where human resources are limited, but may require multiple sessions and ongoing refinement to maximize impact.
Limitations and Cautions
These findings stem from quasi-experimental data, not a randomized controlled trial. Differences in session length, content scope (some topics omitted in HeartBot), and incentive amounts confound direct comparisons. Word counts varied dramatically, complicating dose-response interpretation. Measures relied on self-reported knowledge and perceptions collected shortly after intervention, which may not reflect long-term knowledge retention or behavior.
Implications for Public Health
The study suggests AI chatbots like HeartBot can meaningfully raise heart attack knowledge among women in the US, a critical first step toward timely emergency response. The lower cost and 24/7 availability of AI tools make them appealing for broad dissemination, especially in settings with limited health workforce. To advance this field, future research should pursue randomized designs, longer follow-up, and multi-session chatbot interventions that more closely mirror human-delivered formats.
Future Directions
Recommended next steps include: conducting RCTs to establish causal efficacy; expanding HeartBot’s conversational repertoire and adaptivity; incorporating objective performance measures (simulated scenarios, decision-making tasks); and exploring targeted recruitment to improve generalizability across diverse populations and health literacy levels.
Conclusion
Both human-delivered and AI chatbot interventions can increase knowledge and awareness of heart attack symptoms among women in the United States. While human-delivered conversations yielded stronger improvements and higher user satisfaction in this analysis, AI chatbots hold promise as scalable educational tools. Ongoing refinement and rigorous evaluation will be essential to maximize their impact on women’s heart health.
