Larry P. Heck

I am the R. S. Farmer Jr Advanced Computing Concepts Endowed Chair, a Georgia Research Alliance Eminent Scholar, co-Executive Director of the Machine Learning Center (ML@GT), and a Professor at the Georgia Institute of Technology. I have a joint appointment in the College of Engineering (Electrical & Computer Engineering) and the College of Computing (Interactive Computing).

I am interested in next generation Virtual Assistants, Conversational AI, and Natural Language and Speech Processing. My research goal is to create an autonomous digital human that can freely communicate with humans in open, mixed-reality domains.

news

Mar 12, 2023	coming soon…

highlighted publications

AAAI ’23

Commonsense Reasoning for Conversational AI: A Survey of the State of the Art

Richardson, Christopher, and Heck, Larry

In Workshop on Knowledge Augmented Methods for NLP (KnowledgeNLP-AAAI’23) Feb 2023

Abs PDF

Large, transformer-based pretrained language models like BERT, GPT, and T5 have demonstrated a deep understanding of contextual semantics and language syntax. Their success has enabled significant advances in conversational AI, including the development of open-dialogue systems capable of coherent, salient conversations which can answer questions, chat casually, and complete tasks. However, state-of-the-art models still struggle with tasks that involve higher levels of reasoning - including commonsense reasoning that humans find trivial. This paper presents a survey of recent conversational AI research focused on commonsense reasoning. The paper lists relevant training datasets and describes the primary approaches to include commonsense in conversational AI. The paper also discusses benchmarks used for evaluating commonsense in conversational AI problems. Finally, the paper presents preliminary observations of the limited commonsense capabilities of two state-of-the-art open dialogue models, BlenderBot3 and LaMDA, and its negative effect on natural interactions. These observations further motivate research on commonsense reasoning in conversational AI.
ACLConvAI ’22

Multimodal Conversational AI: A Survey of Datasets and Approaches

Sundar, Anirudh, and Heck, Larry

In Proceedings of the 4th Workshop on NLP for Conversational AI May 2022

Abs PDF

As humans, we experience the world with all our senses or modalities (sound, sight, touch, smell, and taste). We use these modalities, particularly sight and touch, to convey and interpret specific meanings. Multimodal expressions are central to conversations; a rich set of modalities amplify and often compensate for each other. A multimodal conversational AI system answers questions, fulfills tasks, and emulates human conversations by understanding and expressing itself via multiple modalities. This paper motivates, defines, and mathematically formulates the multimodal conversational research objective. We provide a taxonomy of research required to solve the objective: multimodal representation, fusion, alignment, translation, and co-learning. We survey state-of-the-art datasets and approaches for each research area and highlight their limiting assumptions. Finally, we identify multimodal co-learning as a promising direction for multimodal conversational AI research.
CHI ’21

Empowering Dyads of Older Adults With Mild Cognitive Impairment And Their Care Partners Using Conversational Agents

Zubatiy, Tamara, Vickers, Kayci L, Mathur, Niharika, and Mynatt, Elizabeth D

In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems May 2021

Abs PDF

Conversational agents (CAs) such as Google Home or Alexa offer empowering opportunities for dyads composed of older adults with mild cognitive impairment (MCI) and their care partners. CAs support coordination and planning between the two, and can amplify the support that the care partner needs to provide. In this study, we observed how ten such dyads interacted with a Google Home over 10 weeks. We logged and analyzed 3,878 total interactions, interviewed the dyads to better understand their experiences, and also surveyed their individual preferences and priorities for automated assistance in the home. We found that CAs empowered both the people who had MCI, and their care partners. We observed that the utility of the CA in the day-to-day lives of users largely depended on how much the care partner scaffolded promising functionality, setting it up and contextualizing it for specific needs and desires.
ACM HCI ’21

Pivoting an MCI Empowerment Program to Online Engagement

Mynatt, Elizabeth D., Vickers, Kayci, LaForce, Salimah, Farmer, Sarah K., Johnson, Jeremy M., Doiron, Matthew, Ramesh, Aparna, Fain, Walter Bradley, Zubatiy, Tamara, and Rodriguez, Amy D.

Proc. ACM Hum.-Comput. Interact. Jan 2022

Abs PDF

In the Spring of 2020, closures and safe distancing orders swept much of the United States due to the COVID-19 pandemic. This paper presents a case study of pivoting an in-person empowerment program focused on lifestyle interventions for people newly diagnosed with Mild Cognitive Impairment (MCI) to an online program. Working as rapidly as possible to sustain participant engagement, our design decisions and subsequent iterations point to initial constraints in telehealth capabilities, as well as learning on the fly as new capabilities and requirements emerged. We present the discovery of emergent practices by family members and healthcare providers to meet the new requirements for successful online engagement. For some participants, the online program led to greater opportunities for empowerment while others were hampered by the lack of in-person program support. Providers experienced a sharp learning curve and likewise missed the benefits of in-person interaction, but also discovered new benefits of online collaboration. This work lends insights and potential new avenues for understanding how lifestyle interventions can empower people with MCI and the role of technology in that process.