Can Natural Language Processing Improve the Efficiency of Vaccine Adverse Event Report Review

Background: Individual case review of spontaneous Adverse Event (AE) reports remains a cornerstone of medical product safety surveillance for industry and regulators. Previously we developed the Vaccine Adverse Event Text Miner (VaeTM) to offer automated information extraction and potentially accelerate the evaluation of large volumes of unstructured data and facilitate signal detection. Objective: To assess how the information extraction performed by VaeTM impacts the accuracy of a medical expert’s review of the vaccine Adverse Event report. Methods: The “outcome of interest” (diagnosis, cause of death, second level diagnosis), “onset time,” and “alternative explanations” (drug, medical and family history) for the Adverse Event were extracted from 1000 reports from the Vaccine Adverse Event Reporting System (VAERS) using the VaeTM system. We compared the human interpretation, by medical experts, of the VaeTM extracted data with their interpretation of the traditional full text reports for these three variables. Two experienced clinicians alternately reviewed text miner output and full text. A third clinician scored the match rate using a predefined algorithm; the proportion of matches and 95% confidence intervals (CI) were calculated. Review time per report was analyzed. Results: Proportion of matches between the interpretation of the VaeTM extracted data, compared to the interpretation of the full text: 93% for outcome of interest (95% CI: 91– 94%) and 78% for alternative explanation (95% CI: 75 – 81%). Extracted data on the time to onset was used in 14% of cases and was a match in 54% (95% CI: 46 – 63%) of those cases. When supported by structured time data from reports, the match for time to onset was 79% (95% CI: 76 – 81%). The extracted text averaged 136 (74%) fewer words, resulting in a mean reduction in review time of 50 (58%) seconds per report. Conclusion: Despite a 74% reduction in words, the clinical conclusion from VaeTM extracted data agreed with the full text in 93% and 78% of reports for the outcome of interest and alternative explanation, respectively. The limited amount of extracted time interval data indicates the need for further development of this feature. VaeTM may improve review efficiency, but further study is needed to determine if this level of agreement is sufficient for routine use.

Vaccine Adverse Event text mining system for extracting features from vaccine safety reports

OBJECTIVE: To develop and evaluate a text mining system for extracting key clinical features from vaccine Adverse Event reporting system (VAERS) narratives to aid in the automated review of Adverse Event reports.\n\nDESIGN: Based upon clinical significance to VAERS reviewing physicians, we defined the primary (diagnosis and cause of death) and secondary features (eg, symptoms) for extraction. We built a novel vaccine Adverse Event text mining (VaeTM) system based on a semantic text mining strategy. The performance of VaeTM was evaluated using a total of 300 VAERS reports in three sequential evaluations of 100 reports each. Moreover, we evaluated the VaeTM contribution to case classification; an information retrieval-based approach was used for the identification of anaphylaxis cases in a set of reports and was compared with two other methods: a dedicated text classifier and an online tool.\n\nMEASUREMENTS: The performance metrics of VaeTM were text mining metrics: recall, precision and F-measure. We also conducted a qualitative difference analysis and calculated sensitivity and specificity for classification of anaphylaxis cases based on the above three approaches.\n\nRESULTS: VaeTM performed best in extracting diagnosis, second level diagnosis, drug, vaccine, and lot number features (lenient F-measure in the third evaluation: 0.897, 0.817, 0.858, 0.874, and 0.914, respectively). In terms of case classification, high sensitivity was achieved (83.1%); this was equal and better compared to the text classifier (83.1%) and the online tool (40.7%), respectively.\n\nCONCLUSION: Our VaeTM implementation of a semantic text mining strategy shows promise in providing accurate and efficient extraction of key features from VAERS narratives.

text mining for the vaccine Adverse Event reporting system medical text classification using informative feature selection

Objective The US Vaccine Adverse Event Reporting System (VAERS) collects spontaneous reports of Adverse Events following vaccination. Medical officers review the reports and often apply standardized case definitions, such as those developed by the Brighton Collaboration. Our objective was to demonstrate a multi-level text mining approach for automated text classification of VAERS reports that could potentially reduce human workload.

Design We selected 6034 VAERS reports for H1N1 vaccine that were classified by medical officers as potentially positive (Npos=237) or negative for anaphylaxis. We created a categorized corpus of text files that included the class label and the symptom text field of each report. A validation set of 1100 labeled text files was also used. Text mining techniques were applied to extract three feature sets for important keywords, low- and high-level patterns. A rule-based classifier processed the high-level feature representation, while several machine learning classifiers were trained for the remaining two feature representations.

Measurements Classifiers’ performance was evaluated by macro-averaging recall, precision, and F-measure, and Friedman’s test; misclassification error rate analysis was also performed.

Results Rule-based classifier, boosted trees, and weighted support vector machines performed well in terms of macro-recall, however at the expense of a higher mean misclassification error rate. The rule-based classifier performed very well in terms of average sensitivity and specificity (79.05% and 94.80%, respectively).

Conclusion Our validated results showed the possibility of developing effective medical text classifiers for VAERS reports by combining text mining with informative feature selection; this strategy has the potential to reduce reviewer workload considerably.

understanding vaccine safety information from the vaccine Adverse Event reporting system

The Vaccine Adverse Event Reporting System (VAERS) is administered by the Food and Drug Administration and CDC and is a key component of postlicensure vaccine safety surveillance. Its primary function is to detect early warning signals and generate hypotheses about possible new vaccine Adverse Event

Understanding vaccine safety information from the Vaccine Adverse Event Reporting System

The Vaccine Adverse Event Reporting System (VAERS) is administered by the Food and Drug Administration and CDC and is a key component of postlicensure vaccine safety surveillance. Its primary function is to detect early warning signals and generate hypotheses about possible new vaccine Adverse Events or changes in frequency of known ones. VAERS is a passive surveillance system that relies on physicians and others to voluntarily submit reports of illness after vaccination. Manufacturers are required to report all Adverse Events of which they become aware. There are a number of well-described limitations of such reporting systems. These include, for example, variability in report quality, biased reporting, underreporting and the inability to determine whether a vaccine caused the Adverse Event in any individual report. Strengths of VAERS are that it is national in scope and timely. The information in VAERS reports is not necessarily complete nor is it verified systematically. Reports are classified as serious or nonserious based on regulatory criteria. Reports are coded by VAERS in a uniform way with a limited number of terms using a terminology called COSTART. Coding is useful for search purposes but is necessarily imprecise. VAERS is useful in detecting Adverse Events related to vaccines and most recently was used for enhanced reporting of Adverse Events in the national smallpox immunization campaign. VAERS data have always been publicly available. However, it is essential for users of VAERS data to be fully aware of the strengths and weaknesses of the system. VAERS data contain strong biases. Incidence rates and relative risks of specific Adverse Events cannot be calculated. Statistical significance tests and confidence intervals should be used with great caution and not routinely. Signals detected in VAERS should be subjected to further clinical and descriptive epidemiologic analysis. Confirmation in a controlled study is usually required. An understanding of the system’s defined objectives and inherent drawbacks is vital to the effective use of VAERS data in vaccine safety investigations.

an overview of the vaccine Adverse Event reporting system vaers as a surveillance system

We evaluated the Vaccine Adverse Event Reporting System (VAERS), the spontaneous reporting system for vaccine-associated Adverse Events in the United States, as a public health surveillance system, using evaluation guidelines from the Centers for Disease Control and PrEvention. We found that VAERS is simple for reporters to use, flexible by design and its data are available in a timely fashion. The predictive value positive for one severe Event is known to be high, but for most Events is unknown. The acceptability, sensitivity and representativeness of VAERS are unknown. The study of vaccine safety is complicated by underreporting, erroneous reporting, frequent multiple exposures and multiple outcomes.

Adverse Events after anthrax vaccination reported to the vaccine Adverse Event reporting system vaers 1990 2007

Abstract During the period March 1, 1998 to January 14, 2007, approximately 6 million doses of Anthrax vaccine adsorbed (AVA) vaccine were administered. As of January 16, 2007, 4753 reports of Adverse Events following receipt of AVA vaccination had been submitted to the Vaccine Adverse Event Reporting System (VAERS). Taken together, reports to VAERS did not definitively link any serious unexpected risk to this vaccine, and review of death and serious reports did not show a distinctive pattern indicative of a causal relationship to AVA vaccination. Continued monitoring of VAERS and analysis of potential associations between AVA vaccination and rare, serious Events is warranted.

Adverse Event reports following yellow fever vaccination

Abstract Yellow fever (YF) vaccine has been used for prEvention of YF since 1937 with over 500 million doses administered. However, rare reports of severe Adverse Events following vaccination have raised concerns about the vaccine’s safety. We reviewed reports of Adverse Events following YF vaccination reported to the U.S. Vaccine Adverse Event Reporting System (VAERS) from 2000 to 2006. We used estimates of age and sex distribution of administered doses obtained from a 2006 survey of authorized vaccine providers to calculate age- and sex-specific reporting rates of all serious Adverse Events (SAE), anaphylaxis, YF vaccine-associated neurotropic disease, and YF vaccine-associated viscerotropic disease. Reporting rates of SAEs were substantially higher in males and in persons aged ≥60 years. These findings reinforce the generally acceptable safety profile of YF vaccine, but highlight the importance of physician and traveler education regarding the risks and benefits of YF vaccination, particularly for travelers ≥60 years of age. Vaccination should be limited to persons traveling to areas where the risk of YF is expected to exceed the risk of serious Adverse Events after vaccination, or if not medically contraindicated, where national regulations require proof of vaccination to prEvent introduction of YF.

data mining in the us vaccine Adverse Event reporting system vaers early detection of intussusception and other Events after rotavirus vaccination

Abstract The Vaccine Adverse Event Reporting System (VAERS) is the US passive surveillance system monitoring vaccine safety. A major limitation of VAERS is the lack of denominator data (number of doses of administered vaccine), an element necessary for calculating reporting rates. Empirical Bayesian data mining, a data analysis method, utilizes the number of Events reported for each vaccine and statistically screens the database for higher than expected vaccine-Event combinations signaling a potential vaccine-associated Event. This is the first study of data mining in VAERS designed to test the utility of this method to detect retrospectively a known side effect of vaccination–intussusception following rotavirus (RV) vaccine. From October 1998 to December 1999, 112 cases of intussusception were reported. The data mining method was able to detect a signal for RV-intussusception in February 1999 when only four cases were reported. These results demonstrate the utility of data mining to detect significant vaccine-associated Events at early date. Data mining appears to be an efficient and effective computer-based program that may enhance early detection of Adverse Events in passive surveillance systems.