The Experts below are selected from a list of 69969 Experts worldwide ranked by ideXlab platform
Kirk Roberts - One of the best experts on this subject based on the ideXlab platform.
-
an evaluation of two commercial deep learning based Information Retrieval Systems for covid 19 literature
Journal of the American Medical Informatics Association, 2021Co-Authors: Sarvesh K Soni, Kirk RobertsAbstract:The COVID-19 pandemic has resulted in a tremendous need for access to the latest scientific Information, leading to both corpora for COVID-19 literature and search engines to query such data. While most search engine research is performed in academia with rigorous evaluation, major commercial companies dominate the web search market. Thus, it is expected that commercial pandemic-specific search engines will gain much higher traction than academic alternatives, leading to questions about the empirical performance of these tools. This paper seeks to empirically evaluate two commercial search engines for COVID-19 (Google and Amazon) in comparison with academic prototypes evaluated in the TREC-COVID task. We performed several steps to reduce bias in the manual judgments to ensure a fair comparison of all Systems. We find the commercial search engines sizably underperformed those evaluated under TREC-COVID. This has implications for trust in popular health search engines and developing biomedical search engines for future health crises.
Sarvesh K Soni - One of the best experts on this subject based on the ideXlab platform.
-
an evaluation of two commercial deep learning based Information Retrieval Systems for covid 19 literature
Journal of the American Medical Informatics Association, 2021Co-Authors: Sarvesh K Soni, Kirk RobertsAbstract:The COVID-19 pandemic has resulted in a tremendous need for access to the latest scientific Information, leading to both corpora for COVID-19 literature and search engines to query such data. While most search engine research is performed in academia with rigorous evaluation, major commercial companies dominate the web search market. Thus, it is expected that commercial pandemic-specific search engines will gain much higher traction than academic alternatives, leading to questions about the empirical performance of these tools. This paper seeks to empirically evaluate two commercial search engines for COVID-19 (Google and Amazon) in comparison with academic prototypes evaluated in the TREC-COVID task. We performed several steps to reduce bias in the manual judgments to ensure a fair comparison of all Systems. We find the commercial search engines sizably underperformed those evaluated under TREC-COVID. This has implications for trust in popular health search engines and developing biomedical search engines for future health crises.
Stephen J Downie - One of the best experts on this subject based on the ideXlab platform.
-
the scientific evaluation of music Information Retrieval Systems foundations and future
Computer Music Journal, 2004Co-Authors: Stephen J DownieAbstract:Computer Music Journal, 28:2, pp. 12–23, Summer 2004 2004 Massachusetts Institute of Technology. Music Information Retrieval (MIR) is a multidisciplinary research endeavor that strives to develop innovative content-based searching schemes, novel interfaces, and evolving networked delivery mechanisms in an effort to make the world’s vast store of music accessible to all. Some teams are developing ‘‘Query-by-Singing’’ and ‘‘Query-by-Humming’’ Systems that allow users to interact with their respective music search engines via queries that are sung or hummed into a microphone (e.g., Birmingham et al. 2001; Haus and Pollastri 2001). ‘‘Queryby-Note’’ Systems are also being developed wherein searchers construct queries consisting of pitch and/or rhythm Information (e.g., Pickens 2000; Doraisamy and Ruger 2002). Input methods for Queryby-Note Systems include symbolic interfaces as well as both physical (MIDI) and virtual (Javabased) keyboards. Some teams are working on ‘‘Query-by-Example’’ Systems that take prerecorded music in the form of CD or MP3 tracks as their query input (e.g., Haitsma and Kalker 2002; Harb and Chen 2003). The development of comprehensive music recommendation and distribution Systems is a growing research area (e.g., Logan 2002; Pauws and Eggen 2002). The automatic generation of playlists for use in personal music Systems, based on a wide variety of user-defined criteria, is the goal of this branch of MIR research. Other groups are investigating the creation of music analysis Systems to assist those in the musicology and music theory communities (e.g., Barthelemy and Bonardi 2001; Kornstadt 2001). Overviews of MIR’s interdisciplinary research areas can be found in Downie (2003), Byrd and Crawford (2002), and Futrelle and Downie (2002). This article begins with an overview of the current scientific problem facing MIR research. Entitled ‘‘Current Scientific Problem,’’ the opening section also provides a brief explication of the Text Retrieval Conference (TREC) evaluation paradigm that has come to play an important role in the community’s thinking about the testing and evaluation of MIR Systems. The sections which follow, entitled ‘‘Data Collection Method’’ and ‘‘Emergent Themes and Commentary,’’ report upon the findings of the Music Information Retrieval (MIR)/ Music Digital Library (MDL) Evaluation Frameworks Project with issues surrounding the creation of a TREC-like evaluation paradigm for MIR as the central focus. ‘‘Building a TREC-Like Test Collection’’ follows next and highlights the progress being made concerning the establishment of the necessary test collections. The ‘‘Summary and Future Research’’ section concludes this article and highlights some of the key challenges uncovered that require further investigation.
-
the international music Information Retrieval Systems evaluation laboratory governance access and security
International Symposium Conference on Music Information Retrieval, 2004Co-Authors: Stephen J Downie, Joe Futrelle, David TchengAbstract:The IMIRSEL (International Music Information Retrieval Systems Evaluation Laboratory) project provides an unprecedented platform for evaluating Music Information Retrieval (MIR) and Music Digital Library (MDL) techniques, by bringing together large corpora and significant computational resources with the necessary rights management and technical infrastructure to support a variety of MIR/MDL research areas. The standardized research collection being deployed represents a large and diverse corpus of musical examples, which we are hosting in our secure environment for use in evaluating MIR/MDL algorithms. Grid services and NCSA's D2K machine learning environment provide a powerful, highperformance, and secure framework for designing, optimising, and executing complex MIR/MDL evaluation applications. IMIRSEL provides a community resource for researchers who would otherwise not be able to afford the content rights and computational resources to carry out large-scale MIR/MDL evaluations.
-
toward the scientific evaluation of music Information Retrieval Systems
International Symposium Conference on Music Information Retrieval, 2003Co-Authors: Stephen J DownieAbstract:This paper outlines the findings-to-date of a project to assist in the efforts being made to establish a TREC-like evaluation paradigm within the Music Information Retrieval (MIR) research community. The findings and recommendations are based upon expert opinion garnered from members of the Information Retrieval (IR), Music Digital Library (MDL) and MIR communities with regard to the construction and implementation of scientifically valid evaluation frameworks. Proposed recommendations include the creation of data-rich query records that are both grounded in real-world requirements and neutral with respect to Retrieval technique(s) being examined; adoption, and subsequent validation, of a “reasonable person” approach to “relevance” assessment; and, the development of a secure, yet accessible, research environment that allows researchers to remotely access the large-scale testbed collection.
Gilles Hubert - One of the best experts on this subject based on the ideXlab platform.
-
fusing different Information Retrieval Systems according to query topics a study based on correlation in Information Retrieval Systems and trec topics
Information Retrieval, 2011Co-Authors: Anthony Bigot, Claude Chrisment, Taoufiq Dkaki, Gilles Hubert, Josiane MotheAbstract:To evaluate Information Retrieval Systems on their effectiveness, evaluation programs such as TREC offer a rigorous methodology as well as benchmark collections. Whatever the evaluation collection used, effectiveness is generally considered globally, averaging the results over a set of Information needs. As a result, the variability of system performance is hidden as the similarities and differences from one system to another are averaged. Moreover, the topics on which a given system succeeds or fails are left unknown. In this paper we propose an approach based on data analysis methods (correspondence analysis and clustering) to discover correlations between Systems and to find trends in topic/system correlations. We show that it is possible to cluster topics and Systems according to system performance on these topics, some system clusters being better on some topics. Finally, we propose a new method to consider complementary Systems as based on their performances which can be applied for example in the case of repeated queries. We consider the system profile based on the similarity of the set of TREC topics on which Systems achieve similar levels of performance. We show that this method is effective when using the TREC ad hoc collection.
-
on the evaluation of geographic Information Retrieval Systems evaluation framework and case study
International Journal on Digital Libraries, 2010Co-Authors: Damien Palacio, Guillaume Cabanac, Christian Sallaberry, Gilles HubertAbstract:Search engines for Digital Libraries allow users to retrieve documents according to their contents. They process documents without differentiating the manifold aspects of Information. Spatial and temporal dimensions are particularly dismissed. These dimensions are, however, of great interest for users of search engines targeting either the Web or specialized Digital Libraries. Recent studies reported that nearly 20% queries convey spatial and temporal Information in addition to topical Information. These three dimensions were referred to as parts of “geographic Information.” In the literature, search engines handling those dimensions are called “Geographic Information Retrieval (GIR) Systems.” Although several initiatives for evaluating GIR Systems were undertaken, none was concerned with evaluating these three dimensions altogether. In this article, we address this issue by designing an evaluation framework, usefulness of which is highlighted through a case study involving a test collection and a GIR system. This framework allowed the comparison of our GIR system to state-of-the-art topical approaches. We also performed experiments for measuring performance improvement stemming from each dimension or their combination. We show that combining the three dimensions yields improvement in effectiveness (+73.9%) over a common topical baseline. Moreover, rather than conveying redundancy, the three dimensions complement each other.
Ulrich Thiel - One of the best experts on this subject based on the ideXlab platform.
-
cases scripts and Information seeking strategies on the design of interactive Information Retrieval Systems
Expert Systems With Applications, 1995Co-Authors: Nicholas J Belkin, Colleen Cool, Adelheit Stein, Ulrich ThielAbstract:Abstract The support of effective interaction of the user with the other components of the system is a central problem for Information Retrieval. In this paper, we present a theory of such interactions taking place within a space of Information-seeking strategies, and discuss how such a concept can be used to design for effective interaction. In particular, we propose a model of Information Retrieval system design based on the ideas of: a multidimensional space of Information-seeking strategies; dialogue structures for Information seeking; cases of specific Information-seeking dialogues; anti, scripts as distinguished prototypical cases. We demonstrate the use of this model by discussing in some detail the MERIT system, a prototype Information Retrieval system, that incorporates these design principles.