The Experts below are selected from a list of 1108386 Experts worldwide ranked by ideXlab platform
Flavia Costa - One of the best experts on this subject based on the ideXlab platform.
-
Inefficiency in selecting products for submission to national Research Assessment exercises
Scientometrics, 2013Co-Authors: Giovanni Abramo, Ciriaco Andrea D’angelo, Flavia CostaAbstract:One of the critical issues in national Research Assessment exercises concerns the choice of whether to evaluate the entire scientific portfolio of the institutions or a subset composed of the best products. Under the second option, the capacities of the institutions to select the appropriate Researchers and their best products (the UK case) or simply the best products of every Researcher (the Italian case) becomes critical, both for purposes of correct Assessment of the real quality of Research in the institutions evaluated, and for the selective funding that follows. In this work, through case studies of three Italian universities, we analyze the efficiency of the product selection that is intended to maximize the universities' scores in the current national Research Assessment exercise, the results of which will be the basis for assigning an important share of public financing over the coming years.
-
National Research Assessment exercises: a comparison of peer review and bibliometrics rankings
Scientometrics, 2011Co-Authors: Giovanni Abramo, Ciriaco Andrea D’angelo, Flavia CostaAbstract:Development of bibliometric techniques has reached such a level as to suggest their integration or total substitution for classic peer review in the national Research Assessment exercises, as far as the hard sciences are concerned. In this work we compare rankings lists of universities captured by the first Italian evaluation exercise, through peer review, with the results of bibliometric simulations. The comparison shows the great differences between peer review and bibliometric rankings for excellence and productivity.
Giovanni Abramo - One of the best experts on this subject based on the ideXlab platform.
-
Inefficiency in selecting products for submission to national Research Assessment exercises
Scientometrics, 2013Co-Authors: Giovanni Abramo, Ciriaco Andrea D’angelo, Flavia CostaAbstract:One of the critical issues in national Research Assessment exercises concerns the choice of whether to evaluate the entire scientific portfolio of the institutions or a subset composed of the best products. Under the second option, the capacities of the institutions to select the appropriate Researchers and their best products (the UK case) or simply the best products of every Researcher (the Italian case) becomes critical, both for purposes of correct Assessment of the real quality of Research in the institutions evaluated, and for the selective funding that follows. In this work, through case studies of three Italian universities, we analyze the efficiency of the product selection that is intended to maximize the universities' scores in the current national Research Assessment exercise, the results of which will be the basis for assigning an important share of public financing over the coming years.
-
Revisiting the scaling of citations for Research Assessment
Journal of Informetrics, 2012Co-Authors: Giovanni Abramo, Tindaro Cicero, Ciriaco Andrea D’angeloAbstract:Over the past decade, national Research evaluation exercises, traditionally conducted using the peer review method, have begun opening to bibliometric indicators. The citations received by a publication are assumed as proxy for its quality, but they require standardization prior to use in comparative evaluation of organizations or individual scientists: the citation data must be standardized, due to the varying citation behavior across Research fields. The objective of this paper is to compare the effectiveness of the different methods of normalizing citations, in order to provide useful indications to Research Assessment practitioners. Simulating a typical national Research Assessment exercise, he analysis is conducted for all subject categories in the hard sciences and is based on the Thomson Reuters Science Citation Index-Expanded®. Comparisons show that the citations average is the most effective scaling parameter, when the average is based only on the publications actually cited.
-
National Research Assessment exercises: a comparison of peer review and bibliometrics rankings
Scientometrics, 2011Co-Authors: Giovanni Abramo, Ciriaco Andrea D’angelo, Flavia CostaAbstract:Development of bibliometric techniques has reached such a level as to suggest their integration or total substitution for classic peer review in the national Research Assessment exercises, as far as the hard sciences are concerned. In this work we compare rankings lists of universities captured by the first Italian evaluation exercise, through peer review, with the results of bibliometric simulations. The comparison shows the great differences between peer review and bibliometric rankings for excellence and productivity.
Mark Patterson - One of the best experts on this subject based on the ideXlab platform.
-
Reforming Research Assessment.
eLife, 2013Co-Authors: Randy Schekman, Mark PattersonAbstract:It is time for the Research community to rethink how the outputs of scientific Research are evaluated and, as the San Francisco Declaration on Research Assessment makes clear, this should involve replacing the journal impact factor with a broad range of more meaningful approaches.
Marco Schaerf - One of the best experts on this subject based on the ideXlab platform.
-
Are Italian Research Assessment exercises size-biased?
Scientometrics, 2020Co-Authors: Camil Demetrescu, Andrea Ribichini, Marco SchaerfAbstract:Research Assessment exercises have enjoyed ever-increasing popularity in many countries in recent years, both as a method to guide public funds allocation and as a validation tool for adopted Research support policies. Italy’s most recently completed evaluation effort (VQR 2011–14) required each university to submit to the Ministry for Education, University, and Research (MIUR) 2 Research products per author (3 in the case of other Research institutions), chosen in such a way that the same product is not assigned to two authors belonging to the same institution. This constraint suggests that larger institutions, where collaborations among colleagues may be more frequent, could suffer a size-related bias in their evaluation scores. To validate our claim, we investigate the outcome of artificially splitting Sapienza University of Rome, one of the largest universities in Europe, in a number of separate partitions, according to several criteria, noting significant score increases for several partitioning scenarios.
-
On the Shapley value and its application to the Italian VQR Research Assessment exercise
Journal of Informetrics, 2019Co-Authors: Camil Demetrescu, Francesco Lupia, Angelo Mendicelli, Andrea Ribichini, Francesco Scarcello, Marco SchaerfAbstract:Abstract Research Assessment exercises have now become common evaluation tools in a number of countries. These exercises have the goal of guiding merit-based public funds allocation, stimulating improvement of Research productivity through competition and assessing the impact of adopted Research support policies. One case in point is Italy's most recent Research Assessment effort, VQR 2011–2014 (Research Quality Evaluation), which, in addition to Research institutions, also evaluated university departments, and individuals in some cases (i.e., recently hired Research staff and members of PhD committees). However, the way an institution's score was divided, according to VQR rules, between its constituent departments or its staff members does not enjoy many desirable properties well known from coalitional game theory (e.g., budget balance, fairness, marginality). We propose, instead, an alternative score division rule that is based on the notion of Shapley value, a well known solution concept in coalitional game theory, which enjoys the desirable properties mentioned above. For a significant test case (namely, Sapienza University of Rome, the largest university in Italy), we present a detailed comparison of the scores obtained, for substructures and individuals, by applying the official VQR rules, with those resulting from Shapley value computations. We show that there are significant differences in the resulting scores, making room for improvements in the allocation rules used in Research Assessment exercises.
Giuseppe De Nicolao - One of the best experts on this subject based on the ideXlab platform.
-
On the agreement between bibliometrics and peer review: evidence from the Italian Research Assessment exercises
arXiv: Applications, 2018Co-Authors: Alberto Baccini, Lucio Barabesi, Giuseppe De NicolaoAbstract:This paper appraises the concordance between bibliometrics and peer review, by drawing evidence from the data of two experiments realized by the Italian governmental agency for Research evaluation. The experiments were performed for validating the dual system of evaluation, consisting in the interchangeable use of bibliometyrics and peer review, adopted by the agency in the Research Assessment exercises. The two experiments were based on stratified random samples of journal articles. Each article was scored by bibliometrics and by peer review. The degree of concordance between the two evaluations is then computed. The correct setting of the experiments is defined by developing the design-based estimation of the Cohen's kappa coefficient and some testing procedures for assessing the homogeneity of missing proportions between strata. The results of both experiments show that for each Research areas of hard sciences, engineering and life sciences, the degree of agreement between bibliometrics and peer review is -- at most -- weak at an individual article level. Thus, the outcome of the experiments does not validate the use of the dual system of evaluation in the Italian Research Assessments. More in general, the very weak concordance indicates that metrics should not replace peer review at the level of individual article. Hence, the use of the dual system of evaluation for reducing costs might introduce unknown biases in a Research Assessment exercise.
-
do they agree bibliometric evaluation versus informed peer review in the italian Research Assessment exercise
Scientometrics, 2016Co-Authors: Alberto Baccini, Giuseppe De NicolaoAbstract:During the Italian Research Assessment exercise, the national agency ANVUR performed an experiment to assess agreement between grades attributed to journal articles by informed peer review (IR) and by bibliometrics. A sample of articles was evaluated by using both methods and agreement was analyzed by weighted Cohen's kappas. ANVUR presented results as indicating an overall "good" or "more than adequate" agreement. This paper re-examines the experiment results according to the available statistical guidelines for interpreting kappa values, by showing that the degree of agreement (always in the range 0.09---0.42) has to be interpreted, for all Research fields, as unacceptable, poor or, in a few cases, as, at most, fair. The only notable exception, confirmed also by a statistical meta-analysis, was a moderate agreement for economics and statistics (Area 13) and its sub-fields. We show that the experiment protocol adopted in Area 13 was substantially modified with respect to all the other Research fields, to the point that results for economics and statistics have to be considered as fatally flawed. The evidence of a poor agreement supports the conclusion that IR and bibliometrics do not produce similar results, and that the adoption of both methods in the Italian Research Assessment possibly introduced systematic and unknown biases in its final results. The conclusion reached by ANVUR must be reversed: the available evidence does not justify at all the joint use of IR and bibliometrics within the same Research Assessment exercise.