The Experts below are selected from a list of 164028 Experts worldwide ranked by ideXlab platform
Yilong Yin - One of the best experts on this subject based on the ideXlab platform.
-
Unifying neural learning and symbolic reasoning for spinal medical Report Generation.
Medical image analysis, 2020Co-Authors: Zhongyi Han, Benzheng Wei, Bo Chen, Yilong YinAbstract:Automated medical Report Generation in spine radiology, i.e., given spinal medical images and directly create radiologist-level diagnosis Reports to support clinical decision making, is a novel yet fundamental study in the domain of artificial intelligence in healthcare. However, it is incredibly challenging because it is an extremely complicated task that involves visual perception and high-level reasoning processes. In this paper, we propose the neural-symbolic learning (NSL) framework that performs human-like learning by unifying deep neural learning and symbolic logical reasoning for the spinal medical Report Generation. Generally speaking, the NSL framework firstly employs deep neural learning to imitate human visual perception for detecting abnormalities of target spinal structures. Concretely, we design an adversarial graph network that interpolates a symbolic graph reasoning module into a generative adversarial network through embedding prior domain knowledge, achieving semantic segmentation of spinal structures with high complexity and variability. NSL secondly conducts human-like symbolic logical reasoning that realizes unsupervised causal effect analysis of detected entities of abnormalities through meta-interpretive learning. NSL finally fills these discoveries of target diseases into a unified template, successfully achieving a comprehensive medical Report Generation. When employed in a real-world clinical dataset, a series of empirical studies demonstrate its capacity on spinal medical Report Generation and show that our algorithm remarkably exceeds existing methods in the detection of spinal structures. These indicate its potential as a clinical tool that contributes to computer-aided diagnosis.
-
Unifying Neural Learning and Symbolic Reasoning for Spinal Medical Report Generation.
arXiv: Computer Vision and Pattern Recognition, 2020Co-Authors: Zhongyi Han, Benzheng Wei, Yilong YinAbstract:Automated medical Report Generation in spine radiology, i.e., given spinal medical images and directly create radiologist-level diagnosis Reports to support clinical decision making, is a novel yet fundamental study in the domain of artificial intelligence in healthcare. However, it is incredibly challenging because it is an extremely complicated task that involves visual perception and high-level reasoning processes. In this paper, we propose the neural-symbolic learning (NSL) framework that performs human-like learning by unifying deep neural learning and symbolic logical reasoning for the spinal medical Report Generation. Generally speaking, the NSL framework firstly employs deep neural learning to imitate human visual perception for detecting abnormalities of target spinal structures. Concretely, we design an adversarial graph network that interpolates a symbolic graph reasoning module into a generative adversarial network through embedding prior domain knowledge, achieving semantic segmentation of spinal structures with high complexity and variability. NSL secondly conducts human-like symbolic logical reasoning that realizes unsupervised causal effect analysis of detected entities of abnormalities through meta-interpretive learning. NSL finally fills these discoveries of target diseases into a unified template, successfully achieving a comprehensive medical Report Generation. When it employed in a real-world clinical dataset, a series of empirical studies demonstrate its capacity on spinal medical Report Generation as well as show that our algorithm remarkably exceeds existing methods in the detection of spinal structures. These indicate its potential as a clinical tool that contributes to computer-aided diagnosis.
Eric P. Xing - One of the best experts on this subject based on the ideXlab platform.
-
AAAI - Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation
Proceedings of the AAAI Conference on Artificial Intelligence, 2019Co-Authors: Xiaodan Liang, Eric P. XingAbstract:Generating long and semantic-coherent Reports to describe medical images poses great challenges towards bridging visual and linguistic modalities, incorporating medical domain knowledge, and generating realistic and accurate descriptions. We propose a novel Knowledge-driven Encode, Retrieve, Paraphrase (KERP) approach which reconciles traditional knowledge- and retrieval-based methods with modern learning-based methods for accurate and robust medical Report Generation. Specifically, KERP decomposes medical Report Generation into explicit medical abnormality graph learning and subsequent natural language modeling. KERP first employs an Encode module that transforms visual features into a structured abnormality graph by incorporating prior medical knowledge; then a Retrieve module that retrieves text templates based on the detected abnormalities; and lastly, a Paraphrase module that rewrites the templates according to specific cases. The core of KERP is a proposed generic implementation unit—Graph Transformer (GTR) that dynamically transforms high-level semantics between graph-structured data of multiple domains such as knowledge graphs, images and sequences. Experiments show that the proposed approach generates structured and robust Reports supported with accurate abnormality description and explainable attentive regions, achieving the state-of-the-art results on two medical Report benchmarks, with the best medical abnormality and disease classification accuracy and improved human evaluation performance.
-
Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation
arXiv: Computer Vision and Pattern Recognition, 2019Co-Authors: Xiaodan Liang, Eric P. XingAbstract:Generating long and semantic-coherent Reports to describe medical images poses great challenges towards bridging visual and linguistic modalities, incorporating medical domain knowledge, and generating realistic and accurate descriptions. We propose a novel Knowledge-driven Encode, Retrieve, Paraphrase (KERP) approach which reconciles traditional knowledge- and retrieval-based methods with modern learning-based methods for accurate and robust medical Report Generation. Specifically, KERP decomposes medical Report Generation into explicit medical abnormality graph learning and subsequent natural language modeling. KERP first employs an Encode module that transforms visual features into a structured abnormality graph by incorporating prior medical knowledge; then a Retrieve module that retrieves text templates based on the detected abnormalities; and lastly, a Paraphrase module that rewrites the templates according to specific cases. The core of KERP is a proposed generic implementation unit---Graph Transformer (GTR) that dynamically transforms high-level semantics between graph-structured data of multiple domains such as knowledge graphs, images and sequences. Experiments show that the proposed approach generates structured and robust Reports supported with accurate abnormality description and explainable attentive regions, achieving the state-of-the-art results on two medical Report benchmarks, with the best medical abnormality and disease classification accuracy and improved human evaluation performance.
-
Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
arXiv: Computer Vision and Pattern Recognition, 2018Co-Authors: Xiaodan Liang, Eric P. XingAbstract:Generating long and coherent Reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse Report Generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an off-the-shelf template database, or invoke a low-level Generation module to generate a new sentence. HRGR-Agent is updated via reinforcement learning, guided by sentence-level and word-level rewards. Experiments show that our approach achieves the state-of-the-art results on two medical Report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical Report contents. In addition, our model achieves the highest detection accuracy of medical terminologies, and improved human evaluation performance.
-
hybrid retrieval Generation reinforced agent for medical image Report Generation
Neural Information Processing Systems, 2018Co-Authors: Xiaodan Liang, Eric P. XingAbstract:Generating long and coherent Reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse Report Generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an off-the-shelf template database, or invoke a low-level Generation module to generate a new sentence. HRGR-Agent is updated via reinforcement learning, guided by sentence-level and word-level rewards. Experiments show that our approach achieves the state-of-the-art results on two medical Report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical Report contents. In addition, our model achieves the highest detection precision of medical abnormality terminologies, and improved human evaluation performance.
-
NeurIPS - Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
2018Co-Authors: Xiaodan Liang, Eric P. XingAbstract:Generating long and coherent Reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse Report Generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an off-the-shelf template database, or invoke a low-level Generation module to generate a new sentence. HRGR-Agent is updated via reinforcement learning, guided by sentence-level and word-level rewards. Experiments show that our approach achieves the state-of-the-art results on two medical Report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical Report contents. In addition, our model achieves the highest detection precision of medical abnormality terminologies, and improved human evaluation performance.
Xiaodan Liang - One of the best experts on this subject based on the ideXlab platform.
-
Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation
arXiv: Computer Vision and Pattern Recognition, 2020Co-Authors: Fuyu Wang, Xiaojun Chang, Xiaodan LiangAbstract:Beyond the common difficulties faced in the natural image captioning, medical Report Generation specifically requires the model to describe a medical image with a fine-grained and semantic-coherence paragraph that should satisfy both medical commonsense and logic. Previous works generally extract the global image features and attempt to generate a paragraph that is similar to referenced Reports; however, this approach has two limitations. Firstly, the regions of primary interest to radiologists are usually located in a small area of the global image, meaning that the remainder parts of the image could be considered as irrelevant noise in the training procedure. Secondly, there are many similar sentences used in each medical Report to describe the normal regions of the image, which causes serious data bias. This deviation is likely to teach models to generate these inessential sentences on a regular basis. To address these problems, we propose an Auxiliary Signal-Guided Knowledge Encoder-Decoder (ASGK) to mimic radiologists' working patterns. In more detail, ASGK integrates internal visual feature fusion and external medical linguistic information to guide medical knowledge transfer and learning. The core structure of ASGK consists of a medical graph encoder and a natural language decoder, inspired by advanced Generative Pre-Training (GPT). Experiments on the CX-CHR dataset and our COVID-19 CT Report dataset demonstrate that our proposed ASGK is able to generate a robust and accurate Report, and moreover outperforms state-of-the-art methods on both medical terminology classification and paragraph Generation metrics.
-
AAAI - Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation
Proceedings of the AAAI Conference on Artificial Intelligence, 2019Co-Authors: Xiaodan Liang, Eric P. XingAbstract:Generating long and semantic-coherent Reports to describe medical images poses great challenges towards bridging visual and linguistic modalities, incorporating medical domain knowledge, and generating realistic and accurate descriptions. We propose a novel Knowledge-driven Encode, Retrieve, Paraphrase (KERP) approach which reconciles traditional knowledge- and retrieval-based methods with modern learning-based methods for accurate and robust medical Report Generation. Specifically, KERP decomposes medical Report Generation into explicit medical abnormality graph learning and subsequent natural language modeling. KERP first employs an Encode module that transforms visual features into a structured abnormality graph by incorporating prior medical knowledge; then a Retrieve module that retrieves text templates based on the detected abnormalities; and lastly, a Paraphrase module that rewrites the templates according to specific cases. The core of KERP is a proposed generic implementation unit—Graph Transformer (GTR) that dynamically transforms high-level semantics between graph-structured data of multiple domains such as knowledge graphs, images and sequences. Experiments show that the proposed approach generates structured and robust Reports supported with accurate abnormality description and explainable attentive regions, achieving the state-of-the-art results on two medical Report benchmarks, with the best medical abnormality and disease classification accuracy and improved human evaluation performance.
-
Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation
arXiv: Computer Vision and Pattern Recognition, 2019Co-Authors: Xiaodan Liang, Eric P. XingAbstract:Generating long and semantic-coherent Reports to describe medical images poses great challenges towards bridging visual and linguistic modalities, incorporating medical domain knowledge, and generating realistic and accurate descriptions. We propose a novel Knowledge-driven Encode, Retrieve, Paraphrase (KERP) approach which reconciles traditional knowledge- and retrieval-based methods with modern learning-based methods for accurate and robust medical Report Generation. Specifically, KERP decomposes medical Report Generation into explicit medical abnormality graph learning and subsequent natural language modeling. KERP first employs an Encode module that transforms visual features into a structured abnormality graph by incorporating prior medical knowledge; then a Retrieve module that retrieves text templates based on the detected abnormalities; and lastly, a Paraphrase module that rewrites the templates according to specific cases. The core of KERP is a proposed generic implementation unit---Graph Transformer (GTR) that dynamically transforms high-level semantics between graph-structured data of multiple domains such as knowledge graphs, images and sequences. Experiments show that the proposed approach generates structured and robust Reports supported with accurate abnormality description and explainable attentive regions, achieving the state-of-the-art results on two medical Report benchmarks, with the best medical abnormality and disease classification accuracy and improved human evaluation performance.
-
Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
arXiv: Computer Vision and Pattern Recognition, 2018Co-Authors: Xiaodan Liang, Eric P. XingAbstract:Generating long and coherent Reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse Report Generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an off-the-shelf template database, or invoke a low-level Generation module to generate a new sentence. HRGR-Agent is updated via reinforcement learning, guided by sentence-level and word-level rewards. Experiments show that our approach achieves the state-of-the-art results on two medical Report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical Report contents. In addition, our model achieves the highest detection accuracy of medical terminologies, and improved human evaluation performance.
-
hybrid retrieval Generation reinforced agent for medical image Report Generation
Neural Information Processing Systems, 2018Co-Authors: Xiaodan Liang, Eric P. XingAbstract:Generating long and coherent Reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse Report Generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an off-the-shelf template database, or invoke a low-level Generation module to generate a new sentence. HRGR-Agent is updated via reinforcement learning, guided by sentence-level and word-level rewards. Experiments show that our approach achieves the state-of-the-art results on two medical Report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical Report contents. In addition, our model achieves the highest detection precision of medical abnormality terminologies, and improved human evaluation performance.
Zhongyi Han - One of the best experts on this subject based on the ideXlab platform.
-
Unifying neural learning and symbolic reasoning for spinal medical Report Generation.
Medical image analysis, 2020Co-Authors: Zhongyi Han, Benzheng Wei, Bo Chen, Yilong YinAbstract:Automated medical Report Generation in spine radiology, i.e., given spinal medical images and directly create radiologist-level diagnosis Reports to support clinical decision making, is a novel yet fundamental study in the domain of artificial intelligence in healthcare. However, it is incredibly challenging because it is an extremely complicated task that involves visual perception and high-level reasoning processes. In this paper, we propose the neural-symbolic learning (NSL) framework that performs human-like learning by unifying deep neural learning and symbolic logical reasoning for the spinal medical Report Generation. Generally speaking, the NSL framework firstly employs deep neural learning to imitate human visual perception for detecting abnormalities of target spinal structures. Concretely, we design an adversarial graph network that interpolates a symbolic graph reasoning module into a generative adversarial network through embedding prior domain knowledge, achieving semantic segmentation of spinal structures with high complexity and variability. NSL secondly conducts human-like symbolic logical reasoning that realizes unsupervised causal effect analysis of detected entities of abnormalities through meta-interpretive learning. NSL finally fills these discoveries of target diseases into a unified template, successfully achieving a comprehensive medical Report Generation. When employed in a real-world clinical dataset, a series of empirical studies demonstrate its capacity on spinal medical Report Generation and show that our algorithm remarkably exceeds existing methods in the detection of spinal structures. These indicate its potential as a clinical tool that contributes to computer-aided diagnosis.
-
Unifying Neural Learning and Symbolic Reasoning for Spinal Medical Report Generation.
arXiv: Computer Vision and Pattern Recognition, 2020Co-Authors: Zhongyi Han, Benzheng Wei, Yilong YinAbstract:Automated medical Report Generation in spine radiology, i.e., given spinal medical images and directly create radiologist-level diagnosis Reports to support clinical decision making, is a novel yet fundamental study in the domain of artificial intelligence in healthcare. However, it is incredibly challenging because it is an extremely complicated task that involves visual perception and high-level reasoning processes. In this paper, we propose the neural-symbolic learning (NSL) framework that performs human-like learning by unifying deep neural learning and symbolic logical reasoning for the spinal medical Report Generation. Generally speaking, the NSL framework firstly employs deep neural learning to imitate human visual perception for detecting abnormalities of target spinal structures. Concretely, we design an adversarial graph network that interpolates a symbolic graph reasoning module into a generative adversarial network through embedding prior domain knowledge, achieving semantic segmentation of spinal structures with high complexity and variability. NSL secondly conducts human-like symbolic logical reasoning that realizes unsupervised causal effect analysis of detected entities of abnormalities through meta-interpretive learning. NSL finally fills these discoveries of target diseases into a unified template, successfully achieving a comprehensive medical Report Generation. When it employed in a real-world clinical dataset, a series of empirical studies demonstrate its capacity on spinal medical Report Generation as well as show that our algorithm remarkably exceeds existing methods in the detection of spinal structures. These indicate its potential as a clinical tool that contributes to computer-aided diagnosis.
Xiaolei Huang - One of the best experts on this subject based on the ideXlab platform.
-
IPMI - Improved Disease Classification in Chest X-Rays with Transferred Features from Report Generation
Lecture Notes in Computer Science, 2019Co-Authors: Yuan Xue, Xiaolei HuangAbstract:Radiology includes using medical images for detection and diagnosis of diseases as well as guiding further interventions. Chest X-rays are commonly used radiological examinations to help spot thoracic abnormalities or diseases, especially lung-related diseases. However, the Reporting of chest x-rays requires experienced radiologists who are often in shortage in many regions of the world. In this paper, we first develop an automatic radiology Report Generation system. Due to the lack of large annotated radiology Report datasets and the difficulty of evaluating the generated Reports, the clinical value of such systems is often limited. To this end, we train our Report Generation network on the small IU Chest X-ray dataset then transfer the learned visual features to classification networks trained on the large ChestX-ray14 dataset and use a novel attention guided feature fusion strategy to improve the detection performance of 14 common thoracic diseases. Through learning the correspondences between different types of feature representations, common features learned by both the Report Generation and the classification model are assigned with higher attention weights and the weighted visual features boost the performance of state-of-the-art baseline thoracic disease classification networks without altering any learned features. Our work not only offers a new way to evaluate the effectiveness of the learned radiology Report Generation network, but also proves the possibility of transferring different types of visual representations learned on a small dataset for one task to complement features learned on another large dataset for a different task and improve the model performance.
-
MICCAI (1) - Multimodal recurrent model with attention for automated radiology Report Generation
Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, 2018Co-Authors: Yuan Xue, Zhiyun Xue, Sameer Antani, George R Thoma, L. Rodney Long, Xiaolei HuangAbstract:Radiologists routinely examine medical images such as X-Ray, CT, or MRI and write Reports summarizing their descriptive findings and conclusive impressions. A computer-aided radiology Report Generation system can lighten the workload for radiologists considerably and assist them in decision making. Although the rapid development of deep learning technology makes the Generation of a single conclusive sentence possible, results produced by existing methods are not sufficiently reliable due to the complexity of medical images. Furthermore, generating detailed paragraph descriptions for medical images remains a challenging problem. To tackle this problem, we propose a novel generative model which generates a complete radiology Report automatically. The proposed model incorporates the Convolutional Neural Networks (CNNs) with the Long Short-Term Memory (LSTM) in a recurrent way. It is capable of not only generating high-level conclusive impressions, but also generating detailed descriptive findings sentence by sentence to support the conclusion. Furthermore, our multimodal model combines the encoding of the image and one generated sentence to construct an attention input to guide the Generation of the next sentence, and henceforth maintains coherence among generated sentences. Experimental results on the publicly available Indiana U. Chest X-rays from the Open-i image collection show that our proposed recurrent attention model achieves significant improvements over baseline models according to multiple evaluation metrics.
-
multimodal recurrent model with attention for automated radiology Report Generation
Medical Image Computing and Computer-Assisted Intervention, 2018Co-Authors: Yuan Xue, Rodney L Long, Zhiyun Xue, Sameer Antani, George R Thoma, Xiaolei HuangAbstract:Radiologists routinely examine medical images such as X-Ray, CT, or MRI and write Reports summarizing their descriptive findings and conclusive impressions. A computer-aided radiology Report Generation system can lighten the workload for radiologists considerably and assist them in decision making. Although the rapid development of deep learning technology makes the Generation of a single conclusive sentence possible, results produced by existing methods are not sufficiently reliable due to the complexity of medical images. Furthermore, generating detailed paragraph descriptions for medical images remains a challenging problem. To tackle this problem, we propose a novel generative model which generates a complete radiology Report automatically. The proposed model incorporates the Convolutional Neural Networks (CNNs) with the Long Short-Term Memory (LSTM) in a recurrent way. It is capable of not only generating high-level conclusive impressions, but also generating detailed descriptive findings sentence by sentence to support the conclusion. Furthermore, our multimodal model combines the encoding of the image and one generated sentence to construct an attention input to guide the Generation of the next sentence, and henceforth maintains coherence among generated sentences. Experimental results on the publicly available Indiana U. Chest X-rays from the Open-i image collection show that our proposed recurrent attention model achieves significant improvements over baseline models according to multiple evaluation metrics.