Abstract
Artificial intelligence (AI) is increasingly being explored in health and social care to reduce administrative workload and allow staff to spend more time on patient care. This paper evaluates a voice-enabled Care Home Smart Speaker designed to support everyday activities in residential care homes, including spoken access to resident records, reminders, and scheduling tasks. A safety-focused evaluation framework is presented that examines the system end-to-end, combining Whisper-based speech recognition with retrieval-augmented generation (RAG) approaches (hybrid, sparse, and dense). Using supervised care-home trials and controlled testing, we evaluated 330 spoken transcripts across 11 care categories, including 184 reminder-containing interactions. These evaluations focus on (i) correct identification of residents and care categories, (ii) reminder recognition and extraction, and (iii) end-to-end scheduling correctness under uncertainty (including safe deferral/clarification). Given the safety-critical nature of care homes, particular attention is also paid to reliability in noisy environments and across diverse accents, supported by confidence scoring, clarification prompts, and human-in-the-loop oversight. In the best-performing configuration (GPT-5.2), resident ID and care category matching reached 100% (95% CI: 98.86-100), while reminder recognition reached 89.09\% (95% CI: 83.81-92.80) with zero missed reminders (100% recall) but some false positives. End-to-end scheduling via calendar integration achieved 84.65% exact reminder-count agreement (95% CI: 78.00-89.56), indicating remaining edge cases in converting informal spoken instructions into actionable events. The findings suggest that voice-enabled systems, when carefully evaluated and appropriately safeguarded, can support accurate documentation, effective task management, and trustworthy use of AI in care home settings.
Abstract (translated)
人工智能（AI）在健康与社会照护领域的应用日益受到关注，旨在减轻行政工作负担，使工作人员能更专注于患者照护。本文评估了一款面向养老院日常活动的语音智能音箱系统，该系统支持通过语音访问居民记录、提醒事项及任务调度。研究提出了一种以安全为核心的端到端评估框架，结合基于Whisper的语音识别与检索增强生成（RAG）方法（包括混合、稀疏和密集模式）。通过监督式养老院试验与受控测试，我们对11个照护类别中的330条语音转录文本进行了评估，其中包含184条带提醒内容的交互。评估聚焦于：（i）居民身份与照护类别的正确识别；（ii）提醒事项的识别与提取；（iii）不确定环境下端到端调度的正确性（包括安全延迟/澄清机制）。鉴于养老院场景对安全性的高要求，研究特别关注系统在嘈杂环境及多样口音下的可靠性，并通过置信度评分、澄清提示及人在环中监督机制提供支持。在表现最佳的配置（GPT-5.2）中，居民ID与照护类别匹配准确率达100%（95%置信区间：98.86-100），提醒识别率达89.09%（95%置信区间：83.81-92.80），实现零遗漏提醒（100%召回率），但存在部分误报。通过日历集成实现的端到端调度，其提醒数量精确匹配率为84.65%（95%置信区间：78.00-89.56），表明将非正式语音指令转化为可执行事件仍存在边缘案例。研究结果表明，经过严格评估并配备适当防护措施的语音系统，能够在养老院场景中支持准确记录、高效任务管理以及值得信赖的AI应用。
URL
https://arxiv.org/abs/2603.23625
