EACare: Embodied Agent to support elderly mental wellbeing

The main goal of this multidisciplinary project is to develop a robot head with communicative skills capable of interacting with elderly people at their convenience.
The robot will be able to analyze their mental and psychological status via powerful audiovisual sensing and assessing their mental abilities to identify subjects in high risk or possibly at the first stages of depressive or dementing disorders, who would benefit from proper medical consultation. The framework will also provide tools for dementia preventive training.
We propose an ambitious, innovative and integrated approach that combines:
  • an easy-to-use patient-friendly and caregiver-friendly embodied agent to interact with patients via both verbal and non-verbal communication, including dialogue especially designed by psychiatrists to probe, measure, test, and monitor the patients mental state,
  • pioneering technological solutions at both the algorithmic and system level for multimodal information processing to achieve audiovisual human-computer interaction including action recognition, aural and facial expressions, and speech understanding,
  • systematic evaluation of the developed systems in both large and focused social groups, using feedback from each evaluation to guide further scientific research and design.

Consortium

Department of Robotics, Perception and Learning (RPL)
KTH Royal Institute of Technology, Sweden
Project Director: Hedvig Kjellström
Department of Speech, Music and Hearing (TMH)
KTH Royal Institute of Technology, Sweden
Principal Investigator: Joakim Gustafson
Principal Investigator: Jonas Beskow
Department of Neurobiology, Care Sciences and Society (NVS)
Karolinska Institute, Sweden
Principal Investigator: Miia Kivipelto

News

Vad är tidigt stöd när minnet sviktar, Stockholm Demensförenings Samtalsserie Del 1, December, 2020.

Robot kan upptäcka demens tidigt, Framtidens Forskning, June, 2020.

Sverige behöver satsa på AI, IVA Aktuellt, 2:18-23, April, 2018.

Sweden: 'Social Robot' designed to recognise very early stages of Alzheimers, Ruptly TV, August, 2016.

Social robot ska kunna upptäcka demens tidigare, Dagens Nyheter, July, 2016.

Publications

R Nagy*, T Kucherenko*, B Moell, A Pereira, H Kjellström, and U Bernardet. A framework for integrating gesture generation models into interactive conversational agents. In International Conference on Autonomous Agents and Multiagent Systems, demo track, 2021. (*Joint first authors)
Non-verbal behavior production

P Jonell*, B Moell*, K Håkansson*, G Eje Henter, T Kucherenko, O Mikheeva, G Hagman, J Holleman, M Kivipelto, H Kjellström, J Gustafson, and J Beskow. Multimodal capture of patient behaviour for improved detection of early dementia: Clinical feasibility and preliminary results. Frontiers in Computer Science, doi: 10.3389/fcomp.2021.642633, 2021.
Data collection and preliminary analyses

T Kucherenko*, P Jonell*, Y Yoon*, P Wolfert, and G Eje Henter. A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020. In International Conference on Intelligent User Interfaces, 2021. (*Joint first authors)
Non-verbal behavior production

P Jonell, Y Yoon, P Wolfert, T Kucherenko, and G Eje Henter. HEMVIP: Human Evaluation of Multiple Videos in Parallel. arXiv preprint arXiv:2101.11898, 2021.
Evaluation methods

T Kucherenko, D Hasegawa, N Kaneko, G Eje Henter, and H Kjellström. Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation. International Journal of Human-Computer Interaction, doi: 10.1080/10447318.2021.1883883, 2021.
Non-verbal behavior production

K Håkansson, J Beskow, H Kjellström, J Gustafsson, A Bonnard, M Rydén, S Stormoen, G Hagman, U Akenine, K Morales Peres, G Henter, M Sundström, and M Kivipelto. Robot-assisted detection of subclinical dementia: progress report and preliminary findings. Alzheimer's & Dementia 16(S6), e043311, 2020.
Data collection and progress report

P Jonell, T Kucherenko, G Eje Henter, and J Beskow. Let's Face It: Probabilistic Multi-Modal Interlocutor-Aware Generation of Facial Gestures in Dyadic Settings. In ACM International Conference on Intelligent Virtual Agents, 2020.
Adaptive non-verbal behavior production

P Jonell*, T Kucherenko*, I Torre, and J Beskow. Can we trust online crowdworkers? Comparing online and offline participants in a preference test of virtual agents. In ACM International Conference on Intelligent Virtual Agents, 2020. (*Joint first authors)
Evaluation methods

K Håkansson and M Kivipelto. Början till en global livsstilsrevolution? Äldre i Centrum vol 2020 1-2, 2020.
Clinical geriatrics

O Mikheeva, I Kazlauskaite, H Kjellström, and C H Ek. Bayesian nonparametric shared multi-sequence time series segmentation, arXiv preprint arXiv:2001.09886, 2020.
Machine learning

T Kucherenko, P Jonell, S van Waveren, G Henter, S Alexanderson, I Leite, and H Kjellström. Gesticulator: A framework for semantically-aware speech-driven gesture generation, In ACM International Conference on Multimodal Interaction, 2020.
Non-verbal behavior production

P Jonell, P Fallgren, F I Dogan, J Lopes, U Wennberg, and G Skantze. Crowdsourcing a Self-Evolving Dialog Graph.In International Conference on Conversational User Interfaces, 2019.
Dialog systems

P Jonell, T Kucherenko, E Ekstedt, and J Beskow. Learning Non-verbal Behavior for a Social Robot from YouTube Videos. In ICDL-EpiRob Workshop on Naturalistic Non-Verbal and Affective Human-Robot Interactions, 2019.
Non-verbal behavior generation

Z Malisz, P Jonell, and J Beskow. The visual prominence of whispered speech in Swedish. In International Congress of Phonetic Sciences, 2019.
Data collection and analysis

T Kucherenko, D Hasegawa, G E Henter, N Kaneko, and H Kjellströom. Analyzing input and output representations for speech-driven gesture generation, ACM Intelligent Virtual Agents Conference, 2019.
Non-verbal behavior production

P Jonell. Using social and physiological signals for user adaptation in conversational agents. In International Conference on Autonomous Agents and Multiagent Systems, Doctoral Consortium, 2019.
Dialogue system design

T Kucherenko, D Hasegawa, N Kaneko, G E Henter, and H Kjellström. On the importance of representations for speech-driven gesture generation. In International Conference on Autonomous Agents and Multiagent Systems, Extended Abstracts, 2019.
Non-verbal behavior production

P Jonell, M Bystedt, F I Dogan, P Fallgren, J Ivarsson, M Slukova, U Wennberg, J Lopes, J Boye and G Skantze. Fantom: A crowd-sourced social chatbot using an evolving dialog graph. In 2nd Proceedings of Alexa Prize, 2018.
Dialogue system design

P Jonell, C Oertel, D Kontogiorgos, J Beskow, and J Gustafson. Crowd-sourced Multimodal Corpora Collection Tool. In LREC, 2018.
Data collection

T Kucherenko. Data driven non-verbal behavior generation for humanoid robots. In International Conference on Multimodal Interfaces, Doctoral Symposium, 2018.
Non-verbal behavior production

T Ngandu, F Mangialasche, and M Kivipelto. Lifestyle interventions to prevent cognitive impairment and dementia. Nature Neurology, (in press).
Clinical geriatrics

K Håkansson, T Ngandu, and M Kivipelto. The patient with cognitive impairment. In V Hachinsky (Ed), Treatable and Potentially Preventable Dementias, Cambridge University Press, 2018.
Clinical geriatrics

P Jonell, M Bystedt, P Fallgren, D Kontogiorgos, J Lopes, Z Malisz, S Mascarenhas, C Oertel, E Raveh, and T Shore. ARMI: An architecture for recording multimodal interactions. In Language Resources and Evaluation Conference, 2018.
Dialogue system design

P Jonell, C Oertel, D Kontogiorgos, J Beskow, and J Gustafson. Crowdsourced multimodal corpora collection tool. In Language Resources and Evaluation Conference, 2018.
Dialogue system design

O Mikheeva, C H Ek, and H Kjellström. Perceptual facial expression representation. In IEEE International Conference on Automatic Face and Gesture Recognition, 2018.
Machine learning

M Kivipelto and K Håkansson. A rare success against Alzheimer's. Scientific American (April):32-37, 2017.
Clinical geriatrics

P Jonell, J Mendelson, T Storskog, G Hagman, P Östberg, I Leite, T Kucherenko, O Mikheeva, U Akenine, V Jelic, A Solomon, J Beskow, J Gustafson, M Kivipelto, and H Kjellström. Machine Learning and Social Robotics for Detecting Early Signs of Dementia, arXiv preprint arXiv:1709.01613, 2017.
Position paper

P Jonell, C Oertel, D Kontogiorgos, J Beskow, and J Gustafson. Crowd-powered design of virtual attentive listeners. In International Conference on Intelligent Virtual Agents, 2017.
Dialogue system design

C Oertel, P Jonell, D Kontogiorgos, J Mendelson, J Beskow, and J Gustafson. Crowd-sourced design of artificial attentive listeners. In INTERSPEECH: Situated Interaction, 2017.
Dialogue system design

C Oertel, P Jonell, K El Addad, E Szekely, and J Gustafson. Using crowd-sourcing for the design of listening agents: Challenges and opportunities. In ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents, 2017.
Dialogue system design

K Håkansson, A Ledreux, K Daffner, Y Terjestam, P Bergman, R Carlsson, M Kivipelto, B Winblad, A-C Granholm, A K H Mohammed. BDNF responses in healthy older persons to 35 minutes of physical exercise, cognitive training, and mindfulness: Associations with working memory function. Journal of Alzheimer's Disease. 55(2):645-657, 2016.
Clinical geriatrics