Certificate for Best Paper Award of the Human-Computer Interaction Thematic Area
The award has been conferred to
James Pustejovsky (Brandeis University, USA) and
Nikhil Krishnaswamy (Colorado State University, USA)
for the paper entitled
"Multimodal Semantics for Affordances and Actions"
Presented in the context of
HCI International 2022
26 June—1 July 2022
"In this paper, we argue that, as HCI becomes more multimodal with the integration of gesture, gaze, posture, and other nonverbal behavior, it is important to understand the role played by affordances and their associated actions in human-object interactions (HOI), so as to facilitate reasoning in HCI and HRI environments. We outline the requirements and challenges involved in developing a multimodal semantics for human-computer and human-robot interactions. Unlike unimodal interactive agents (e.g., text-based chatbots or voice-based personal digital assistants), multimodal HCI and HRI inherently require a notion of embodiment, or an understanding of the agent’s placement within the environment and that of its interlocutor. We present a dynamic semantics of the language, VoxML, to model human-computer, human-robot, and human-human interactions by creating multimodal simulations of both the communicative content and the agents’ common ground, and show the utility of VoxML information that is reified within the environment on computational understanding of objects for HOI."
The full paper is available through SpringerLink, provided that you have proper access rights.