Larry P. Heck

Larry P. Heck is a Professor with a joint appointment in the School of Electrical and Computer Engineering and the School of Interactive Computing at the Georgia Institute of Technology. He holds the Rhesa S. Farmer Distinguished Chair of Advanced Computing Concepts and is a Georgia Research Alliance Eminent Scholar. His received the BSEE from Texas Tech University (1986), and MSEE and PhD EE from the Georgia Institute of Technology (1989,1991). He is a Fellow of the IEEE, inducted into the Academy of Distinguished Engineering Alumni at Georgia Tech and received the Distinguished Engineer Award from the Texas Tech University Whitacre College of Engineering. He was a Senior Research Engineer with SRI (1992-98), Vice President of R&D at Nuance (1998-2005), Vice President of Search and Advertising Sciences at Yahoo! (2005-2009), Chief Scientist of the Microsoft Speech products and Distinguished Engineer in Microsoft Research (2009-2014), Principal Scientist with Google Research (2014-2017), and CEO of Viv Labs and Senior Vice President at Samsung (2017-2021). His research interests are on next-generation Virtual Assistants, Conversational AI including Large Language Models (LLMs) and Response Generation, and Natural Language and Speech Processing with the long-term goal of creating an autonomous digital human that can freely communicate with humans in open, mixed-reality domains.

highlighted publications

cTBLS ’23

cTBLS: Augmenting Large Language Models for Conversational Tables

Sundar, Anirudh, and Heck, Larry

arXiv:2303.12024 2023

Abs PDF

An open challenge in multimodal conversational AI requires augmenting large language models with information from textual and non-textual sources for multi-turn dialogue. To address this problem, this paper introduces Conversational Tables (cTBL), a three-step encoder-decoder approach to retrieve tabular information and generate dialogue responses grounded on the retrieved information. cTBL uses Transformer encoder embeddings for Dense Table Retrieval and obtains up to 5% relative improvement in Top-1 and Top-3 accuracy over sparse retrieval on the HyrbiDialogue dataset. Additionally, cTBL performs tabular knowledge retrieval using both encoder and decoder models, resulting in up to 46% relative improvement in ROUGE scores and better human evaluation for response generation on HyrbiDialogue.
AAAI ’23

Commonsense Reasoning for Conversational AI: A Survey of the State of the Art

Richardson, Christopher, and Heck, Larry

In Workshop on Knowledge Augmented Methods for NLP (KnowledgeNLP-AAAI’23) Feb 2023

Abs PDF

Large, transformer-based pretrained language models like BERT, GPT, and T5 have demonstrated a deep understanding of contextual semantics and language syntax. Their success has enabled significant advances in conversational AI, including the development of open-dialogue systems capable of coherent, salient conversations which can answer questions, chat casually, and complete tasks. However, state-of-the-art models still struggle with tasks that involve higher levels of reasoning - including commonsense reasoning that humans find trivial. This paper presents a survey of recent conversational AI research focused on commonsense reasoning. The paper lists relevant training datasets and describes the primary approaches to include commonsense in conversational AI. The paper also discusses benchmarks used for evaluating commonsense in conversational AI problems. Finally, the paper presents preliminary observations of the limited commonsense capabilities of two state-of-the-art open dialogue models, BlenderBot3 and LaMDA, and its negative effect on natural interactions. These observations further motivate research on commonsense reasoning in conversational AI.
ACLConvAI ’22

Multimodal Conversational AI: A Survey of Datasets and Approaches

Sundar, Anirudh, and Heck, Larry

In Proceedings of the 4th Workshop on NLP for Conversational AI May 2022

Abs PDF

As humans, we experience the world with all our senses or modalities (sound, sight, touch, smell, and taste). We use these modalities, particularly sight and touch, to convey and interpret specific meanings. Multimodal expressions are central to conversations; a rich set of modalities amplify and often compensate for each other. A multimodal conversational AI system answers questions, fulfills tasks, and emulates human conversations by understanding and expressing itself via multiple modalities. This paper motivates, defines, and mathematically formulates the multimodal conversational research objective. We provide a taxonomy of research required to solve the objective: multimodal representation, fusion, alignment, translation, and co-learning. We survey state-of-the-art datasets and approaches for each research area and highlight their limiting assumptions. Finally, we identify multimodal co-learning as a promising direction for multimodal conversational AI research.
CHI ’21

Empowering Dyads of Older Adults With Mild Cognitive Impairment And Their Care Partners Using Conversational Agents

Zubatiy, Tamara, Vickers, Kayci L, Mathur, Niharika, and Mynatt, Elizabeth D

In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems May 2021

Abs PDF

Conversational agents (CAs) such as Google Home or Alexa offer empowering opportunities for dyads composed of older adults with mild cognitive impairment (MCI) and their care partners. CAs support coordination and planning between the two, and can amplify the support that the care partner needs to provide. In this study, we observed how ten such dyads interacted with a Google Home over 10 weeks. We logged and analyzed 3,878 total interactions, interviewed the dyads to better understand their experiences, and also surveyed their individual preferences and priorities for automated assistance in the home. We found that CAs empowered both the people who had MCI, and their care partners. We observed that the utility of the CA in the day-to-day lives of users largely depended on how much the care partner scaffolded promising functionality, setting it up and contextualizing it for specific needs and desires.
ACM HCI ’21

Pivoting an MCI Empowerment Program to Online Engagement

Mynatt, Elizabeth D., Vickers, Kayci, LaForce, Salimah, Farmer, Sarah K., Johnson, Jeremy M., Doiron, Matthew, Ramesh, Aparna, Fain, Walter Bradley, Zubatiy, Tamara, and Rodriguez, Amy D.

Proc. ACM Hum.-Comput. Interact. Jan 2022

Abs PDF

In the Spring of 2020, closures and safe distancing orders swept much of the United States due to the COVID-19 pandemic. This paper presents a case study of pivoting an in-person empowerment program focused on lifestyle interventions for people newly diagnosed with Mild Cognitive Impairment (MCI) to an online program. Working as rapidly as possible to sustain participant engagement, our design decisions and subsequent iterations point to initial constraints in telehealth capabilities, as well as learning on the fly as new capabilities and requirements emerged. We present the discovery of emergent practices by family members and healthcare providers to meet the new requirements for successful online engagement. For some participants, the online program led to greater opportunities for empowerment while others were hampered by the lack of in-person program support. Providers experienced a sharp learning curve and likewise missed the benefits of in-person interaction, but also discovered new benefits of online collaboration. This work lends insights and potential new avenues for understanding how lifestyle interventions can empower people with MCI and the role of technology in that process.