#audio-visual data

Abstract: Recent advances in computational and sensing technologies allowed to incorporate different devices into a smart systems, making the ubiquitous or pervasive computing a hot topic for research and commercial projects. One technology, that can help the user to interact with invisible system representing smart environment is spoken dialogue system. Following the success in research on automatic speech recognition and natural language understanding, spoken dialogue systems have significantly improved themselves during the past decade and now bringing the communication between human and machine closer to natural level. Having user as a main subject, both system may benefit from explicit information about his current state and mood, adjusting their behaviour to the certain extent. In this paper we consider the combination of ubiquitous computing, spoken dialogue systems, and emotion recognition technologies, suggest possible ways of information flow, discuss future applications and potential problems. We find, that these technologies can be complementary to each other, increasing their flexibility, robustness and intelligibility when combined. We present the usage of such approach in a smart house environment, continuously tracking the state of the user, interacting with them in real time and reacting to mood changes.

Conference: http://www.percom.org/Previous/ST2019/

Abstract: To design more context-aware systems for smart environments, especially smart cities, the psychological user status such as emotion should be considered in addition to environmental information. In this study, we focus on the tourism domain as a typical use-case, and propose a multimodal tourist emotion recognition method during the sightseeing. We employ behavioural cues (eye and head/body movement) and audio-visual features to recognise emotion. As a result of real-world experiments with tourists, we achieved up to 0.71 of average recall score in 3-class emotion recognition task with feature level fusion.

Conference (UERMMI): https://uermmi2018.wordpress.com/

Conference (UbiComp): http://ubicomp.org/ubicomp2018/

ACM DL: https://doi.org/10.1145/3267305.3267687