Publication
Visuo-auditory Multimodal Emotional Structure to Improve Human-Robot-Interaction
| datacite.subject.fos | Engenharia e Tecnologia::Engenharia Eletrotécnica, Eletrónica e Informática | |
| datacite.subject.sdg | 03:Saúde de Qualidade | |
| datacite.subject.sdg | 07:Energias Renováveis e Acessíveis | |
| datacite.subject.sdg | 11:Cidades e Comunidades Sustentáveis | |
| dc.contributor.author | Prado, José Augusto | |
| dc.contributor.author | Simplício, Carlos | |
| dc.contributor.author | Lori, Nicolás F. | |
| dc.contributor.author | Dias, Jorge | |
| dc.date.accessioned | 2026-01-15T16:25:05Z | |
| dc.date.available | 2026-01-15T16:25:05Z | |
| dc.date.issued | 2011-12-20 | |
| dc.description.abstract | We propose an approach to analyze and synthesize a set of human facial and vocal expressions, and then use the classified expressions to decide the robot’s response in a human-robot-interaction. During a human-tohuman conversation, a person senses the interlocutor’s face and voice, perceives her/his emotional expressions, and processes this information in order to decide which response to give. Moreover, observed emotions are taken into account and the response may be aggressive, funny (henceforth meaning humorous) or just neutral according to not only the observed emotions, but also the personality of the person. The purpose of our proposed structure is to endow robots with the capability to model human emotions, and thus several subproblems need to be solved: feature extraction, classification, decision and synthesis. In the proposed approach we integrate two classifiers for emotion recognition from audio and video, and then use a new method for fusion with the social behavior profile. To keep the person engaged in the interaction, after each iterance of analysis, the robot synthesizes human voice with both lips synchronization and facial expressions. The social behavior profile conducts the personality of the robot. The structure and work flow of the synthesis and decision are addressed, and the Bayesian networks are discussed. We also studied how to analyze and synthesize the emotion from the facial expression and vocal expression. A new probabilistic structure that enables a higher level of interaction between a human and a robot is proposed. | eng |
| dc.description.sponsorship | The authors gratefully acknowledge support from Institute of Systems and Robotics at University of Coimbra (ISR-UC), Portuguese Foundation for Science and Technology (FCT) [SFRH/BD/60954/2009, Ciencia2007, PTDC/SAU-BEB/100147/2008], and Polytechnical Institute of Leiria (IPL). | |
| dc.identifier.citation | Prado, J.A., Simplício, C., Lori, N.F. et al. Visuo-auditory Multimodal Emotional Structure to Improve Human-Robot-Interaction. Int J of Soc Robotics 4, 29–51 (2012). https://doi.org/10.1007/s12369-011-0134-7 | |
| dc.identifier.doi | 10.1007/s12369-011-0134-7 | |
| dc.identifier.issn | 1875-4791 | |
| dc.identifier.issn | 1875-4805 | |
| dc.identifier.uri | http://hdl.handle.net/10400.8/15352 | |
| dc.language.iso | eng | |
| dc.peerreviewed | yes | |
| dc.publisher | Springer | |
| dc.relation.hasversion | https://link.springer.com/article/10.1007/s12369-011-0134-7 | |
| dc.relation.ispartof | International Journal of Social Robotics | |
| dc.rights.uri | http://creativecommons.org/licenses/by-nc-nd/4.0/ | |
| dc.subject | Visual perception | |
| dc.subject | Auditory perception | |
| dc.subject | Emotion recognition | |
| dc.subject | Multimodal interaction | |
| dc.subject | Social behavior profile | |
| dc.subject | Bayesian networks | |
| dc.title | Visuo-auditory Multimodal Emotional Structure to Improve Human-Robot-Interaction | eng |
| dc.type | journal article | |
| dspace.entity.type | Publication | |
| oaire.citation.endPage | 51 | |
| oaire.citation.issue | 1 | |
| oaire.citation.startPage | 29 | |
| oaire.citation.title | International Journal of Social Robotics | |
| oaire.citation.volume | 4 | |
| oaire.version | http://purl.org/coar/version/c_970fb48d4fbd8a85 | |
| person.familyName | Simplício | |
| person.givenName | Carlos | |
| person.identifier.orcid | 0000-0001-6281-4059 | |
| relation.isAuthorOfPublication | 2e15bd33-f7a7-466f-8013-1fdd73b75e7c | |
| relation.isAuthorOfPublication.latestForDiscovery | 2e15bd33-f7a7-466f-8013-1fdd73b75e7c |