David Chan
EECS Department
University of California, Berkeley
Technical Report No. UCB/EECS-2024-15
April 16, 2024
http://www2.eecs.berkeley.edu/Pubs/TechRpts/2024/EECS-2024-15.pdf
If you ask a human to describe an image, they might do so in a thousand different ways. Each of these descriptions depends not only on the image but also on a rich tapestry of contextual hints and clues surrounding the image (up to and including the person doing the describing themselves). Until now, the field of conditional natural language generation has focused almost solely on the perception component of the task: how do we perceive what is in the stimulus -- be it audio, visual, or textual -- and relay it to the user? In this dissertation, we argue that models that focus solely on the stimulus (and not the associated context) suffer significant shortcomings in their ability to generate language that aligns well with human judgments of quality and content while decreasing their overall utility for downstream tasks. This dissertation focuses on three core objectives in the pursuit of building a context-aware conditional natural language generation (CNLG) model: (1) capturing and understanding the information within, among, and between generated conditional texts, (2) developing multimodal models that better integrate contextual information, and (3) designing CNLG evaluation methodologies that better align with human judgment. Through these objectives, we demonstrate the power of context in natural language generation and help to answer the question: "How can we understand, build, and evaluate context-aware models for conditional natural language generation?"
Advisor: John F. Canny
BibTeX citation:
@phdthesis{Chan:EECS-2024-15,
Author = {Chan, David},
Title = {Understanding, Building, and Evaluating Models for Context Aware Conditional Natural Language Generation},
School = {EECS Department, University of California, Berkeley},
Year = {2024},
Month = {Apr},
URL = {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2024/EECS-2024-15.html},
Number = {UCB/EECS-2024-15},
Abstract = {If you ask a human to describe an image, they might do so in a thousand different ways. Each of these descriptions depends not only on the image but also on a rich tapestry of contextual hints and clues surrounding the image (up to and including the person doing the describing themselves). Until now, the field of conditional natural language generation has focused almost solely on the perception component of the task: how do we perceive what is in the stimulus -- be it audio, visual, or textual -- and relay it to the user? In this dissertation, we argue that models that focus solely on the stimulus (and not the associated context) suffer significant shortcomings in their ability to generate language that aligns well with human judgments of quality and content while decreasing their overall utility for downstream tasks. This dissertation focuses on three core objectives in the pursuit of building a context-aware conditional natural language generation (CNLG) model: (1) capturing and understanding the information within, among, and between generated conditional texts, (2) developing multimodal models that better integrate contextual information, and (3) designing CNLG evaluation methodologies that better align with human judgment. Through these objectives, we demonstrate the power of context in natural language generation and help to answer the question: "How can we understand, build, and evaluate context-aware models for conditional natural language generation?"}
}
EndNote citation:
%0 Thesis %A Chan, David %T Understanding, Building, and Evaluating Models for Context Aware Conditional Natural Language Generation %I EECS Department, University of California, Berkeley %D 2024 %8 April 16 %@ UCB/EECS-2024-15 %U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2024/EECS-2024-15.html %F Chan:EECS-2024-15