Page 46 - My FlipBook
P. 46
i cial Intelligence Projects人
工
智 Figure 4 : Visual Story Telling: Integrating visual and text
慧 information for story synthesis.
計
畫 Figure 5 : Integrating photo and text Information for multimodal
news recommendation.
Visual Storytelling
Visual storytelling is a research topic involving
multi-modal integration. Given a set of pictures,
visual storytelling models aim to generate the best
collocated story. For general textual storytelling, it is
already difficult to generate coherent contexts like
those spoken by people. Moreover, visual stories need
to fit the features of related imagery, i.e., grounding,
making this topic even more challenging. We are
tackling this issue by targeting the commonalities
conveyed by both images and texts. First, we extract
key events from the images. Then, we represent these
events using event frames defined in a common
semantic framework, FrameNet, and leverage
knowledge bases to connect them to enrich content.
Finally, we adopt a separate textual story generation
model, trained on a large dataset, to produce the nal
story. We are now working on aspects of coherence,
diversity and length exibility of the stories generated
by this approach. Human evaluators have attested to
the quality of the stories our model generates, and the
results have been published in AAAI 2020.
Multi-View News Recommendation
This project involves integrating news articles and
their accompanying images to generate multi-
modal recommended outputs. Conventional models
use reading histories (session-based), read news
content (content-based), or "hot news" (collaborative
filtering) to recommend news to users. However,
these models overlook that users’ prior knowledge of
the world may greatly impact their current interests
or, more directly, users may simply be attracted by
an image. Our model learns principle concepts from
a large dataset of reading history, representing prior
knowledge, and identi es eye-catching prompts from
the current news article and accompanying images
to recommend the next article. Our experimental
results have confirmed the effectiveness of the
model. Following on from this line of research, we
are studying personalized recommendations and
news descriptions generated from the perspective
of di erent users. Additionally, we are exploring the
influence of recommendation dynamics in terms of
information masking among users.
44
工
智 Figure 4 : Visual Story Telling: Integrating visual and text
慧 information for story synthesis.
計
畫 Figure 5 : Integrating photo and text Information for multimodal
news recommendation.
Visual Storytelling
Visual storytelling is a research topic involving
multi-modal integration. Given a set of pictures,
visual storytelling models aim to generate the best
collocated story. For general textual storytelling, it is
already difficult to generate coherent contexts like
those spoken by people. Moreover, visual stories need
to fit the features of related imagery, i.e., grounding,
making this topic even more challenging. We are
tackling this issue by targeting the commonalities
conveyed by both images and texts. First, we extract
key events from the images. Then, we represent these
events using event frames defined in a common
semantic framework, FrameNet, and leverage
knowledge bases to connect them to enrich content.
Finally, we adopt a separate textual story generation
model, trained on a large dataset, to produce the nal
story. We are now working on aspects of coherence,
diversity and length exibility of the stories generated
by this approach. Human evaluators have attested to
the quality of the stories our model generates, and the
results have been published in AAAI 2020.
Multi-View News Recommendation
This project involves integrating news articles and
their accompanying images to generate multi-
modal recommended outputs. Conventional models
use reading histories (session-based), read news
content (content-based), or "hot news" (collaborative
filtering) to recommend news to users. However,
these models overlook that users’ prior knowledge of
the world may greatly impact their current interests
or, more directly, users may simply be attracted by
an image. Our model learns principle concepts from
a large dataset of reading history, representing prior
knowledge, and identi es eye-catching prompts from
the current news article and accompanying images
to recommend the next article. Our experimental
results have confirmed the effectiveness of the
model. Following on from this line of research, we
are studying personalized recommendations and
news descriptions generated from the perspective
of di erent users. Additionally, we are exploring the
influence of recommendation dynamics in terms of
information masking among users.
44
