The Experts below are selected from a list of 14025 Experts worldwide ranked by ideXlab platform
Laura Dietz - One of the best experts on this subject based on the ideXlab platform.
-
knowledge rich image gist understanding beyond Literal Meaning
arXiv: Information Retrieval, 2019Co-Authors: Lydia Weiland, Ioana Hulpus, Simone Paolo Ponzetto, Wolfgang Effelsberg, Laura DietzAbstract:We investigate the problem of understanding the message (gist) conveyed by images and their captions as found, for instance, on websites or news articles. To this end, we propose a methodology to capture the Meaning of image-caption pairs on the basis of large amounts of machine-readable knowledge that has previously been shown to be highly effective for text understanding. Our method identifies the connotation of objects beyond their denotation: where most approaches to image understanding focus on the denotation of objects, i.e., their Literal Meaning, our work addresses the identification of connotations, i.e., iconic Meanings of objects, to understand the message of images. We view image understanding as the task of representing an image-caption pair on the basis of a wide-coverage vocabulary of concepts such as the one provided by Wikipedia, and cast gist detection as a concept-ranking problem with image-caption pairs as queries. To enable a thorough investigation of the problem of gist understanding, we produce a gold standard of over 300 image-caption pairs and over 8,000 gist annotations covering a wide variety of topics at different levels of abstraction. We use this dataset to experimentally benchmark the contribution of signals from heterogeneous sources, namely image and text. The best result with a Mean Average Precision (MAP) of 0.69 indicate that by combining both dimensions we are able to better understand the Meaning of our image-caption pairs than when using language or vision information alone. We test the robustness of our gist detection approach when receiving automatically generated input, i.e., using automatically generated image tags or generated captions, and prove the feasibility of an end-to-end automated process.
-
knowledge rich image gist understanding beyond Literal Meaning
Data and Knowledge Engineering, 2018Co-Authors: Lydia Weiland, Ioana Hulpus, Simone Paolo Ponzetto, Wolfgang Effelsberg, Laura DietzAbstract:Abstract We investigate the problem of understanding the message (gist) conveyed by images and their captions as found, for instance, on websites or news articles. To this end, we propose a methodology to capture the Meaning of image-caption pairs on the basis of large amounts of machine-readable knowledge that have previously been shown to be highly effective for text understanding. Our method identifies the connotation of objects beyond their denotation: where most approaches to image understanding focus on the denotation of objects, i.e., their Literal Meaning, our work addresses the identification of connotations, i.e., iconic Meanings of objects, to understand the message of images. We view image understanding as the task of representing an image-caption pair on the basis of a wide-coverage vocabulary of concepts such as the one provided by Wikipedia, and cast gist detection as a concept-ranking problem with image-caption pairs as queries. Our proposed algorithm brings together aspects of entity linking and clustering, subgraph selection, semantic relatedness, and learning-to-rank in a novel way. In addition to this novel task and a complete evaluation of our approach, we introduce a novel dataset to foster further research on this problem. To enable a throughout investigation of the problem of gist understanding, we produce a gold standard of over 300 image-caption pairs and over 8000 gist annotations covering a wide variety of topics at different levels of abstraction. We use this dataset to experimentally benchmark the contribution of different kinds of signals from heterogeneous sources, namely image and text. The best result with a Mean Average Precision (MAP) of 0.69 indicate that by combining both dimensions we are able to better understand the Meaning of our image-caption pairs than when using language or vision information alone. Our supervised approach relies on the availability of human-annotated gold standard datasets. Annotating images with, possibly complex, topic labels is arguably a very time-consuming task that must rely on expert human annotators. We accordingly investigate whether parts of this process could be automatized using automatic image annotation and caption generation techniques. Our results indicate the general feasibility of an end-to-end approach to gist detection when replacing one of the two dimensions with automatically generated input, i.e., using automatically generated image tags or generated captions. However, we also show experimentally that state-of-the-art image and text understanding is better at understanding Literal Meanings of image-caption pairs, with non-Literal pairs being instead generally more difficult to detect, thus paving the way for future work on understanding the message of images beyond their Literal content.
Lydia Weiland - One of the best experts on this subject based on the ideXlab platform.
-
knowledge rich image gist understanding beyond Literal Meaning
arXiv: Information Retrieval, 2019Co-Authors: Lydia Weiland, Ioana Hulpus, Simone Paolo Ponzetto, Wolfgang Effelsberg, Laura DietzAbstract:We investigate the problem of understanding the message (gist) conveyed by images and their captions as found, for instance, on websites or news articles. To this end, we propose a methodology to capture the Meaning of image-caption pairs on the basis of large amounts of machine-readable knowledge that has previously been shown to be highly effective for text understanding. Our method identifies the connotation of objects beyond their denotation: where most approaches to image understanding focus on the denotation of objects, i.e., their Literal Meaning, our work addresses the identification of connotations, i.e., iconic Meanings of objects, to understand the message of images. We view image understanding as the task of representing an image-caption pair on the basis of a wide-coverage vocabulary of concepts such as the one provided by Wikipedia, and cast gist detection as a concept-ranking problem with image-caption pairs as queries. To enable a thorough investigation of the problem of gist understanding, we produce a gold standard of over 300 image-caption pairs and over 8,000 gist annotations covering a wide variety of topics at different levels of abstraction. We use this dataset to experimentally benchmark the contribution of signals from heterogeneous sources, namely image and text. The best result with a Mean Average Precision (MAP) of 0.69 indicate that by combining both dimensions we are able to better understand the Meaning of our image-caption pairs than when using language or vision information alone. We test the robustness of our gist detection approach when receiving automatically generated input, i.e., using automatically generated image tags or generated captions, and prove the feasibility of an end-to-end automated process.
-
knowledge rich image gist understanding beyond Literal Meaning
Data and Knowledge Engineering, 2018Co-Authors: Lydia Weiland, Ioana Hulpus, Simone Paolo Ponzetto, Wolfgang Effelsberg, Laura DietzAbstract:Abstract We investigate the problem of understanding the message (gist) conveyed by images and their captions as found, for instance, on websites or news articles. To this end, we propose a methodology to capture the Meaning of image-caption pairs on the basis of large amounts of machine-readable knowledge that have previously been shown to be highly effective for text understanding. Our method identifies the connotation of objects beyond their denotation: where most approaches to image understanding focus on the denotation of objects, i.e., their Literal Meaning, our work addresses the identification of connotations, i.e., iconic Meanings of objects, to understand the message of images. We view image understanding as the task of representing an image-caption pair on the basis of a wide-coverage vocabulary of concepts such as the one provided by Wikipedia, and cast gist detection as a concept-ranking problem with image-caption pairs as queries. Our proposed algorithm brings together aspects of entity linking and clustering, subgraph selection, semantic relatedness, and learning-to-rank in a novel way. In addition to this novel task and a complete evaluation of our approach, we introduce a novel dataset to foster further research on this problem. To enable a throughout investigation of the problem of gist understanding, we produce a gold standard of over 300 image-caption pairs and over 8000 gist annotations covering a wide variety of topics at different levels of abstraction. We use this dataset to experimentally benchmark the contribution of different kinds of signals from heterogeneous sources, namely image and text. The best result with a Mean Average Precision (MAP) of 0.69 indicate that by combining both dimensions we are able to better understand the Meaning of our image-caption pairs than when using language or vision information alone. Our supervised approach relies on the availability of human-annotated gold standard datasets. Annotating images with, possibly complex, topic labels is arguably a very time-consuming task that must rely on expert human annotators. We accordingly investigate whether parts of this process could be automatized using automatic image annotation and caption generation techniques. Our results indicate the general feasibility of an end-to-end approach to gist detection when replacing one of the two dimensions with automatically generated input, i.e., using automatically generated image tags or generated captions. However, we also show experimentally that state-of-the-art image and text understanding is better at understanding Literal Meanings of image-caption pairs, with non-Literal pairs being instead generally more difficult to detect, thus paving the way for future work on understanding the message of images beyond their Literal content.
Tilo Kircher - One of the best experts on this subject based on the ideXlab platform.
-
laterality in metaphor processing lack of evidence from functional magnetic resonance imaging for the right hemisphere theory
Brain and Language, 2007Co-Authors: Alexander Rapp, Dirk T Leube, Michael Erb, Wolfgang Grodd, Tilo KircherAbstract:We investigated processing of metaphoric sentences using event-related functional magnetic resonance imaging (fMRI). Seventeen healthy subjects (6 female, 11 male) read 60 novel short German sentence pairs with either metaphoric or Literal Meaning and performed two different tasks: judging the metaphoric content and judging whether the sentence has a positive or negative connotation. Laterality indices for 8 regions of interest were calculated: Inferior frontal gyrus (opercular part and triangular part), superior, middle, and inferior temporal gyrus, precuneus, temporal pole, and hippocampus. A left lateralised network was activated with no significant differences in laterality between the two tasks. The lowest degree of laterality was found in the temporal pole. Other factors than metaphoricity per se might trigger right hemisphere recruitment. Results are discussed in the context of lesion and hemifield studies.
Dirk T Leube - One of the best experts on this subject based on the ideXlab platform.
-
laterality in metaphor processing lack of evidence from functional magnetic resonance imaging for the right hemisphere theory
Brain and Language, 2007Co-Authors: Alexander Rapp, Dirk T Leube, Michael Erb, Wolfgang Grodd, Tilo KircherAbstract:We investigated processing of metaphoric sentences using event-related functional magnetic resonance imaging (fMRI). Seventeen healthy subjects (6 female, 11 male) read 60 novel short German sentence pairs with either metaphoric or Literal Meaning and performed two different tasks: judging the metaphoric content and judging whether the sentence has a positive or negative connotation. Laterality indices for 8 regions of interest were calculated: Inferior frontal gyrus (opercular part and triangular part), superior, middle, and inferior temporal gyrus, precuneus, temporal pole, and hippocampus. A left lateralised network was activated with no significant differences in laterality between the two tasks. The lowest degree of laterality was found in the temporal pole. Other factors than metaphoricity per se might trigger right hemisphere recruitment. Results are discussed in the context of lesion and hemifield studies.
Jie Yang - One of the best experts on this subject based on the ideXlab platform.
-
the role of the right hemisphere in metaphor comprehension a meta analysis of functional magnetic resonance imaging studies
Human Brain Mapping, 2014Co-Authors: Jie YangAbstract:The role of the right hemisphere (RH) in metaphor comprehension is still controversial. Numerous neuroimaging studies have found that conventionality, sentential context, and task demand can influence the involvement of the RH in metaphor processing. The current meta-analysis used foci from 17 original functional magnetic resonance imaging studies to identify what factors modulate the involvement of the RH in metaphor processing. Activation likelihood estimation was used for quantification. We focused on the contrast of metaphorical Meaning processing versus Literal Meaning processing and calculated the meta-analysis effects when (1) metaphorical Meaning is conventional, (2) metaphorical Meaning is novel, (3) metaphorical and Literal Meaning are presented in words, (4) metaphorical and Literal Meaning are presented in sentential context, (5) task is valence judgment, and (6) task is semantic relatedness judgment. The results indicated that the RH only showed significant effects in metaphor processing when the metaphorical Meaning is novel, when metaphorical Meaning is presented in sentential context, and when the task is semantic relatedness judgment. The effects were located in right fronto-temporal regions, including inferior frontal gyrus, middle frontal gyrus, insula, superior temporal gyrus, and middle temporal gyrus. These results suggest that conventionality, contextual complexity, and task demand can modulate the effect of figurativeness and influence the involvement of RH in metaphor comprehension. The main role of the RH in metaphor processing is related with activating broad semantic fields and integrating concepts that may have distant semantic relations, and hence provide support for the view that the RH is responsible for processing coarse semantic information in language comprehension. Hum Brain Mapp 35:107–122, 2014. © 2012 Wiley Periodicals, Inc.