TY - JOUR
T1 - Challenges in Observing the Emotions of Children with Autism Interacting with a Social Robot
AU - Erol Barkana, Duygun
AU - Bartl-Pokorny, Katrin D.
AU - Kose, Hatice
AU - Landowska, Agnieszka
AU - Milling, Manuel
AU - Robins, Ben
AU - Schuller, Björn W.
AU - Uluer, Pinar
AU - Wrobel, Michal R.
AU - Zorcec, Tatjana
N1 - © 2024 The Author(s). This is an open access article distributed under the Creative Commons Attribution License (CC BY), https://creativecommons.org/licenses/by/4.0/
PY - 2024/12/30
Y1 - 2024/12/30
N2 - This paper concerns the methodology of multi-modal data acquisition in observing emotions experienced by children with autism while they interact with a social robot. As robot-enhanced therapy gains more and more attention and proved to be effective in autism, such observations might influence the future development and use of such technologies. The paper is based on an observational study of child-robot interaction, during which multiple modalities were captured and then analyzed to retrieve information on a child’s emotional state. Over 30 children on the autism spectrum from Macedonia, Turkey, Poland, and the United Kingdom took part in our study and interacted with the social robot Kaspar. We captured facial expressions/body posture, voice/vocalizations, physiological signals, and eyegaze-related data. The main contribution of the paper is reporting challenges and lessons learned with regard to interaction, its environment, and observation channels typically used for emotion estimation. The main challenge is the limited availability of channels, especially eyegaze-related (29%) and voice-related (6%) data are not available throughout the entire session. The challenges are of a diverse nature—we distinguished task-based, child-based, and environment-based ones. Choosing the tasks (scenario) and adapting environment, such as room, equipment, accompanying person, is crucial but even with those works done, the child-related challenge is the most important one. Therapists have pointed out to a good potential of those technologies, however, the main challenge to keep a child engaged and focused, remains. The technology must follow a child’s interest, movement, and mood. The main observations are the necessity to train personalized models of emotions as children with autism differ in level of skills and expressions, and emotion recognition technology adaptation in real time (e. g., switching modalities) to capture variability in emotional outcomes.
AB - This paper concerns the methodology of multi-modal data acquisition in observing emotions experienced by children with autism while they interact with a social robot. As robot-enhanced therapy gains more and more attention and proved to be effective in autism, such observations might influence the future development and use of such technologies. The paper is based on an observational study of child-robot interaction, during which multiple modalities were captured and then analyzed to retrieve information on a child’s emotional state. Over 30 children on the autism spectrum from Macedonia, Turkey, Poland, and the United Kingdom took part in our study and interacted with the social robot Kaspar. We captured facial expressions/body posture, voice/vocalizations, physiological signals, and eyegaze-related data. The main contribution of the paper is reporting challenges and lessons learned with regard to interaction, its environment, and observation channels typically used for emotion estimation. The main challenge is the limited availability of channels, especially eyegaze-related (29%) and voice-related (6%) data are not available throughout the entire session. The challenges are of a diverse nature—we distinguished task-based, child-based, and environment-based ones. Choosing the tasks (scenario) and adapting environment, such as room, equipment, accompanying person, is crucial but even with those works done, the child-related challenge is the most important one. Therapists have pointed out to a good potential of those technologies, however, the main challenge to keep a child engaged and focused, remains. The technology must follow a child’s interest, movement, and mood. The main observations are the necessity to train personalized models of emotions as children with autism differ in level of skills and expressions, and emotion recognition technology adaptation in real time (e. g., switching modalities) to capture variability in emotional outcomes.
KW - Autism
KW - Automatic emotion recognition
KW - Robot-enhanced therapy
KW - Social robots
UR - http://www.scopus.com/inward/record.url?scp=85208810632&partnerID=8YFLogxK
U2 - 10.1007/s12369-024-01185-3
DO - 10.1007/s12369-024-01185-3
M3 - Article
SN - 1875-4791
VL - 16
SP - 2261
EP - 2276
JO - International Journal of Social Robotics
JF - International Journal of Social Robotics
IS - 11
ER -