The Experts below are selected from a list of 4737 Experts worldwide ranked by ideXlab platform
Stephen Stark - One of the best experts on this subject based on the ideXlab platform.
-
linking methods for the zinnes griggs Pairwise Preference irt model
Applied Psychological Measurement, 2017Co-Authors: Philseok Lee, Seang-hwane Joo, Stephen StarkAbstract:Forced-choice item response theory (IRT) models are being more widely used as a way of reducing response biases in noncognitive research and operational testing contexts. As applications have incre...
-
Linking Methods for the Zinnes–Griggs Pairwise Preference IRT Model:
Applied Psychological Measurement, 2016Co-Authors: Philseok Lee, Seang-hwane Joo, Stephen StarkAbstract:Forced-choice item response theory (IRT) models are being more widely used as a way of reducing response biases in noncognitive research and operational testing contexts. As applications have incre...
-
adaptive testing with multidimensional Pairwise Preference items improving the efficiency of personality and other noncognitive assessments
Organizational Research Methods, 2012Co-Authors: Stephen Stark, Oleksandr S. Chernyshenko, Fritz Drasgow, Leonard A WhiteAbstract:Assessment of noncognitive constructs in organizational research and practice is challenging because of response biases that can distort test scores. Researchers must also deal with time constraint...
-
can subject matter experts ratings of statement extremity be used to streamline the development of unidimensional Pairwise Preference scales
Organizational Research Methods, 2011Co-Authors: Stephen Stark, Oleksandr S. Chernyshenko, Nigel GuenoleAbstract:Interest in on-demand noncognitive assessment has flourished due to advances in computer technology and studies demonstrating noteworthy predictive validities for organizational outcomes. Computerized adaptive testing (CAT) based on the Zinnes-Griggs (ZG) ideal point item response theory (IRT) model may hold promise for organizational settings, because a large pool of items can be created from a modest number of stimuli, and the items have been shown to be resistant to some types of rater bias. However, sample sizes needed for marginal maximum likelihood (MML) estimation of statement parameters are quite large and could thus limit usefulness in practice. This article addresses that concern and its ramifications for CAT. Specifically, we conducted empirical and simulation studies to examine whether subject matter expert (SME) ratings of statement extremity (location) can be substituted for MML estimates to streamline test development and launch. Results showed that error in SME-based location estimates had little detrimental effect on score accuracy or validity, regardless of whether measures were constructed adaptively or nonadaptively. Implications for research involving small samples and CAT in field settings are discussed.
-
Computerized Adaptive Testing with the Zinnes and Griggs Pairwise Preference Ideal Point Model
International Journal of Testing, 2011Co-Authors: Stephen Stark, Oleksandr S. ChernyshenkoAbstract:This article delves into a relatively unexplored area of measurement by focusing on adaptive testing with unidimensional Pairwise Preference items. The use of such tests is becoming more common in applied non-cognitive assessment because research suggests that this format may help to reduce certain types of rater error and response sets commonly associated with the traditional single stimulus format. Yet there have been no publications evaluating the performance of unidimensional Pairwise Preference adaptive or nonadaptive tests. This article therefore presents the results of a simulation study that examined scoring accuracy for three item selection algorithms (nonadaptive, adaptive-symmetric, and adaptive-asymmetric), two pool sizes (50 and 100 stimuli), two methods for pool composition (even- and over-sampling), and three test lengths (10, 20, and 40 items).
Oleksandr S. Chernyshenko - One of the best experts on this subject based on the ideXlab platform.
-
adaptive testing with multidimensional Pairwise Preference items improving the efficiency of personality and other noncognitive assessments
Organizational Research Methods, 2012Co-Authors: Stephen Stark, Oleksandr S. Chernyshenko, Fritz Drasgow, Leonard A WhiteAbstract:Assessment of noncognitive constructs in organizational research and practice is challenging because of response biases that can distort test scores. Researchers must also deal with time constraint...
-
can subject matter experts ratings of statement extremity be used to streamline the development of unidimensional Pairwise Preference scales
Organizational Research Methods, 2011Co-Authors: Stephen Stark, Oleksandr S. Chernyshenko, Nigel GuenoleAbstract:Interest in on-demand noncognitive assessment has flourished due to advances in computer technology and studies demonstrating noteworthy predictive validities for organizational outcomes. Computerized adaptive testing (CAT) based on the Zinnes-Griggs (ZG) ideal point item response theory (IRT) model may hold promise for organizational settings, because a large pool of items can be created from a modest number of stimuli, and the items have been shown to be resistant to some types of rater bias. However, sample sizes needed for marginal maximum likelihood (MML) estimation of statement parameters are quite large and could thus limit usefulness in practice. This article addresses that concern and its ramifications for CAT. Specifically, we conducted empirical and simulation studies to examine whether subject matter expert (SME) ratings of statement extremity (location) can be substituted for MML estimates to streamline test development and launch. Results showed that error in SME-based location estimates had little detrimental effect on score accuracy or validity, regardless of whether measures were constructed adaptively or nonadaptively. Implications for research involving small samples and CAT in field settings are discussed.
-
Computerized Adaptive Testing with the Zinnes and Griggs Pairwise Preference Ideal Point Model
International Journal of Testing, 2011Co-Authors: Stephen Stark, Oleksandr S. ChernyshenkoAbstract:This article delves into a relatively unexplored area of measurement by focusing on adaptive testing with unidimensional Pairwise Preference items. The use of such tests is becoming more common in applied non-cognitive assessment because research suggests that this format may help to reduce certain types of rater error and response sets commonly associated with the traditional single stimulus format. Yet there have been no publications evaluating the performance of unidimensional Pairwise Preference adaptive or nonadaptive tests. This article therefore presents the results of a simulation study that examined scoring accuracy for three item selection algorithms (nonadaptive, adaptive-symmetric, and adaptive-asymmetric), two pool sizes (50 and 100 stimuli), two methods for pool composition (even- and over-sampling), and three test lengths (10, 20, and 40 items).
-
Normative Scoring of Multidimensional Pairwise Preference Personality Scales Using IRT: Empirical Comparisons With Other Formats
Human Performance, 2009Co-Authors: Oleksandr S. Chernyshenko, Stephen Stark, Matthew S. Prewett, Ashley A. Gray, Frederick R. B. Stilson, Matthew D. TuttleAbstract:In this article, we offer some suggestions as to why tetrads and pentads have become the dominant formats for administering multidimensional forced choice (MFC) items but, in turn, raise questions regarding the underlying psychometric model and means of addressing item quality and scoring accuracy. We then focus our attention on multidimensional Pairwise Preference (MDPP) items and present an item response theory–based approach to constructing and modeling MDPP responses directly, assessing information at the item and scale levels, and a way of computing standard errors for trait scores and estimating scale reliability. To demonstrate the viability of this method for applied use, we show that the correspondence between MDPP scores derived from direct modeling with those obtained using single statement and unidimensional Pairwise Preference measures administered in a laboratory setting. Trait score correlations and criterion related validities are compared across testing formats and rating sources (i.e., s...
-
an irt approach to constructing and scoring Pairwise Preference items involving stimuli on different dimensions the multi unidimensional Pairwise Preference model
Applied Psychological Measurement, 2005Co-Authors: Stephen Stark, Oleksandr S. Chernyshenko, Fritz DrasgowAbstract:This article proposes an item response theory (IRT) approach to constructing and scoring multidimensional Pairwise Preference items. Individual statements are administered and calibrated using a unidimensional single-stimulus model. Tests are created by combining multidimensional items with a small number of unidimensional pairings needed to identify the latent metric. Trait scores are then obtained using a multidimensional Bayes modal estimation procedure based on a mathematical model called MUPP, which is illustrated and tested here using Monte Carlo simulations. Simulation results show that the MUPP approach to test construction and scoring provides accurate parameter recovery in both one- and two-dimensional simulations, even with relatively few (say, 15%) unidimensional pairings. The implications of these results for constructing and scoring fake-resistant personality items are discussed.
Eyke Hullermeier - One of the best experts on this subject based on the ideXlab platform.
-
ECML/PKDD (3) - A Reduction of Label Ranking to Multiclass Classification
Machine Learning and Knowledge Discovery in Databases, 2020Co-Authors: Klaus Brinker, Eyke HullermeierAbstract:Label ranking considers the problem of learning a mapping from instances to strict total orders over a predefined set of labels. In this paper, we present a framework for label ranking using a decomposition into a set of multiclass problems. Conceptually, our approach can be seen as a generalization of Pairwise Preference learning. In contrast to the latter, it allows for controlling the granularity of the decomposition, varying between binary Preferences and complete rankings as extreme cases. It is specifically motivated by limitations of Pairwise learning with regard to the minimization of certain loss functions. We discuss theoretical properties of the proposed method in terms of accuracy, error correction, and computational complexity. Experimental results are promising and indicate that improvements upon the special case of Pairwise Preference decomposition are indeed possible.
-
Multilabel classification via calibrated label ranking
Machine Learning, 2008Co-Authors: Johannes Furnkranz, Eyke Hullermeier, Eneldo Loza mencía, Klaus BrinkerAbstract:Label ranking studies the problem of learning a mapping from instances to rankings over a predefined set of labels. Hitherto existing approaches to label ranking implicitly operate on an underlying (utility) scale which is not calibrated in the sense that it lacks a natural zero point. We propose a suitable extension of label ranking that incorporates the calibrated scenario and substantially extends the expressive power of these approaches. In particular, our extension suggests a conceptually novel technique for extending the common learning by Pairwise comparison approach to the multilabel scenario, a setting previously not being amenable to the Pairwise decomposition technique. The key idea of the approach is to introduce an artificial calibration label that, in each example, separates the relevant from the irrelevant labels. We show that this technique can be viewed as a combination of Pairwise Preference learning and the conventional relevance classification technique, where a separate classifier is trained to predict whether a label is relevant or not. Empirical results in the area of text categorization, image classification and gene analysis underscore the merits of the calibrated model in comparison to state-of-the-art multilabel learning methods.
-
Comparison of Ranking Procedures in Pairwise Preference Learning
2004Co-Authors: Eyke Hullermeier, Johannes FurnkranzAbstract:Computational methods for discovering the Preferences of individuals are useful in many applications. In this paper, we propose a method for learning valued Preference structures, using a natural extension of so-called Pairwise classification. A valued Preference structure can then be used in order to induce a ranking, that is a linear ordering of a given set of alternatives. This step is realized by means of a so-called ranking procedure. In the second part of the paper, we compare the performance of alternative ranking procedures in an experimental way.
-
Pairwise Preference learning and ranking
European conference on Machine Learning, 2003Co-Authors: Johannes Furnkranz, Eyke HullermeierAbstract:We consider supervised learning of a ranking function, which is a mapping from instances to total orders over a set of labels (options). The training information consists of examples with partial (and possibly inconsistent) information about their associated rankings. From these, we induce a ranking function by reducing the original problem to a number of binary classification problems, one for each pair of labels. The main objective of this work is to investigate the trade-off between the quality of the induced ranking function and the computational complexity of the algorithm, both depending on the amount of Preference information given for each example. To this end, we present theoretical results on the complexity of Pairwise Preference learning, and experimentally investigate the predictive performance of our method for different types of Preference information, such as top-ranked labels and complete rankings. The domain of this study is the prediction of a rational agent's ranking of actions in an uncertain environment.
-
ECML - Pairwise Preference learning and ranking
Machine Learning: ECML 2003, 2003Co-Authors: Johannes Furnkranz, Eyke HullermeierAbstract:We consider supervised learning of a ranking function, which is a mapping from instances to total orders over a set of labels (options). The training information consists of examples with partial (and possibly inconsistent) information about their associated rankings. From these, we induce a ranking function by reducing the original problem to a number of binary classification problems, one for each pair of labels. The main objective of this work is to investigate the trade-off between the quality of the induced ranking function and the computational complexity of the algorithm, both depending on the amount of Preference information given for each example. To this end, we present theoretical results on the complexity of Pairwise Preference learning, and experimentally investigate the predictive performance of our method for different types of Preference information, such as top-ranked labels and complete rankings. The domain of this study is the prediction of a rational agent's ranking of actions in an uncertain environment.
Johannes Furnkranz - One of the best experts on this subject based on the ideXlab platform.
-
Multilabel classification via calibrated label ranking
Machine Learning, 2008Co-Authors: Johannes Furnkranz, Eyke Hullermeier, Eneldo Loza mencía, Klaus BrinkerAbstract:Label ranking studies the problem of learning a mapping from instances to rankings over a predefined set of labels. Hitherto existing approaches to label ranking implicitly operate on an underlying (utility) scale which is not calibrated in the sense that it lacks a natural zero point. We propose a suitable extension of label ranking that incorporates the calibrated scenario and substantially extends the expressive power of these approaches. In particular, our extension suggests a conceptually novel technique for extending the common learning by Pairwise comparison approach to the multilabel scenario, a setting previously not being amenable to the Pairwise decomposition technique. The key idea of the approach is to introduce an artificial calibration label that, in each example, separates the relevant from the irrelevant labels. We show that this technique can be viewed as a combination of Pairwise Preference learning and the conventional relevance classification technique, where a separate classifier is trained to predict whether a label is relevant or not. Empirical results in the area of text categorization, image classification and gene analysis underscore the merits of the calibrated model in comparison to state-of-the-art multilabel learning methods.
-
Comparison of Ranking Procedures in Pairwise Preference Learning
2004Co-Authors: Eyke Hullermeier, Johannes FurnkranzAbstract:Computational methods for discovering the Preferences of individuals are useful in many applications. In this paper, we propose a method for learning valued Preference structures, using a natural extension of so-called Pairwise classification. A valued Preference structure can then be used in order to induce a ranking, that is a linear ordering of a given set of alternatives. This step is realized by means of a so-called ranking procedure. In the second part of the paper, we compare the performance of alternative ranking procedures in an experimental way.
-
Pairwise Preference learning and ranking
European conference on Machine Learning, 2003Co-Authors: Johannes Furnkranz, Eyke HullermeierAbstract:We consider supervised learning of a ranking function, which is a mapping from instances to total orders over a set of labels (options). The training information consists of examples with partial (and possibly inconsistent) information about their associated rankings. From these, we induce a ranking function by reducing the original problem to a number of binary classification problems, one for each pair of labels. The main objective of this work is to investigate the trade-off between the quality of the induced ranking function and the computational complexity of the algorithm, both depending on the amount of Preference information given for each example. To this end, we present theoretical results on the complexity of Pairwise Preference learning, and experimentally investigate the predictive performance of our method for different types of Preference information, such as top-ranked labels and complete rankings. The domain of this study is the prediction of a rational agent's ranking of actions in an uncertain environment.
-
ECML - Pairwise Preference learning and ranking
Machine Learning: ECML 2003, 2003Co-Authors: Johannes Furnkranz, Eyke HullermeierAbstract:We consider supervised learning of a ranking function, which is a mapping from instances to total orders over a set of labels (options). The training information consists of examples with partial (and possibly inconsistent) information about their associated rankings. From these, we induce a ranking function by reducing the original problem to a number of binary classification problems, one for each pair of labels. The main objective of this work is to investigate the trade-off between the quality of the induced ranking function and the computational complexity of the algorithm, both depending on the amount of Preference information given for each example. To this end, we present theoretical results on the complexity of Pairwise Preference learning, and experimentally investigate the predictive performance of our method for different types of Preference information, such as top-ranked labels and complete rankings. The domain of this study is the prediction of a rational agent's ranking of actions in an uncertain environment.
Philseok Lee - One of the best experts on this subject based on the ideXlab platform.
-
linking methods for the zinnes griggs Pairwise Preference irt model
Applied Psychological Measurement, 2017Co-Authors: Philseok Lee, Seang-hwane Joo, Stephen StarkAbstract:Forced-choice item response theory (IRT) models are being more widely used as a way of reducing response biases in noncognitive research and operational testing contexts. As applications have incre...
-
Linking Methods for the Zinnes–Griggs Pairwise Preference IRT Model:
Applied Psychological Measurement, 2016Co-Authors: Philseok Lee, Seang-hwane Joo, Stephen StarkAbstract:Forced-choice item response theory (IRT) models are being more widely used as a way of reducing response biases in noncognitive research and operational testing contexts. As applications have incre...
-
Detecting Aberrant Responding on Unidimensional Pairwise Preference Tests: An Application of lz Based on the Zinnes–Griggs Ideal Point IRT Model
Applied Psychological Measurement, 2014Co-Authors: Philseok LeeAbstract:This study investigated the efficacy of the lz person fit statistic for detecting aberrant responding with unidimensional Pairwise Preference (UPP) measures, constructed and scored based on the Zin...
-
detecting aberrant responding on unidimensional Pairwise Preference tests an application of lz based on the zinnes griggs ideal point irt model
Applied Psychological Measurement, 2014Co-Authors: Philseok LeeAbstract:This study investigated the efficacy of the lz person fit statistic for detecting aberrant responding with unidimensional Pairwise Preference (UPP) measures, constructed and scored based on the Zin...
