The Experts below are selected from a list of 4056 Experts worldwide ranked by ideXlab platform
Michel C Desmarais - One of the best experts on this subject based on the ideXlab platform.
-
EDM - The refinement of a Q-Matrix: Assessing methods to validate tasks to skills mapping.
2020Co-Authors: Michel C Desmarais, Behzad Beheshti, Peng XuAbstract:The objective of specifying which skills are reQuired in a given task is fundamental for the accurate assessment of a student’s knowledge and for personalizing tutor interaction towards more relevant and effective assessment and learning. We compare three data driven techniQues for the validation of skills-to-tasks mappings. All methods start from a given mapping, typically obtained from domain experts, and use optimization techniQues to suggest a refined version of the skills-to-task mapping. To validate the different techniQues, we inject perturbations in the Q-Matrix and verify whether the original Q-Matrix can be recovered. Tests are run over both simulated and real data. The analysis of the Q-Matrix refinements of each techniQue over ten data sets shows that, in general, around 1/2 to 2/3 of the perturbations can be restored to their original values, but a number of potentially wrong perturbations are also introduced. The number of correctly restored and falsely switched values vary across the three techniQues and between synthetic and real data. For 1 to 10 perturbations injected, simulated data recovery rate is around 2/3, and invalid alterations introduced vary around 2 to 3. For real data, the two best techniQues generally recover about half the perturbations injected, but introduce between 5 and 7 alterations inconsistent with the original, expert defined Q-Matrix, although some of them may be real improvements.
-
the refinement of a Q Matrix assessing methods to validate tasks to skills mapping
Educational Data Mining, 2014Co-Authors: Michel C Desmarais, Behzad Beheshti, Peng XuAbstract:The objective of specifying which skills are reQuired in a given task is fundamental for the accurate assessment of a student’s knowledge and for personalizing tutor interaction towards more relevant and effective assessment and learning. We compare three data driven techniQues for the validation of skills-to-tasks mappings. All methods start from a given mapping, typically obtained from domain experts, and use optimization techniQues to suggest a refined version of the skills-to-task mapping. To validate the different techniQues, we inject perturbations in the Q-Matrix and verify whether the original Q-Matrix can be recovered. Tests are run over both simulated and real data. The analysis of the Q-Matrix refinements of each techniQue over ten data sets shows that, in general, around 1/2 to 2/3 of the perturbations can be restored to their original values, but a number of potentially wrong perturbations are also introduced. The number of correctly restored and falsely switched values vary across the three techniQues and between synthetic and real data. For 1 to 10 perturbations injected, simulated data recovery rate is around 2/3, and invalid alterations introduced vary around 2 to 3. For real data, the two best techniQues generally recover about half the perturbations injected, but introduce between 5 and 7 alterations inconsistent with the original, expert defined Q-Matrix, although some of them may be real improvements.
-
conditions for effectively deriving a Q Matrix from data with non negative Matrix factorization
Educational Data Mining, 2010Co-Authors: Michel C DesmaraisAbstract:The process of deciding which skills are involved in a given task is tedious and challenging. Means to automate it are highly desirable, even if only partial automation that provides supportive tools can be achieved. A recent techniQue based on Non-negative Matrix Factorization (NMF) was shown to offer valuable results, especially due to the fact that the resulting factorization allows a straightforward interpretation in terms of a Q-Matrix. We investigate the factors and assumptions under which NMF can effectively derive the underlying high level skills behind assessment results. We demonstrate the use of different techniQues to analyse and interpret the output of NMF. We propose a simple model to generate simulated data and to provide lower and upper bounds for Quantifying skill effect. Using the simulated data, we show that, under the assumption of independent skills, the NMF techniQue is highly effective in deriving the Q-Matrix. However, the NMF performance degrades under different ratios of variance between subject performance, item difficulty, and skill mastery. The results corroborates conclusions from previous work in that high level skills, corresponding to general topics like World History and Biology, seem to have no substantial effect on test performance, whereas other topics like Mathematics and French do. The analysis and visualization techniQues of the NMF output, along with the simulation approach presented in this paper, should be useful for future investigations using NMF for Q-Matrix induction from data.
Jimmy De La Torre - One of the best experts on this subject based on the ideXlab platform.
-
Improving Robustness in Q-Matrix Validation Using an Iterative and Dynamic Procedure:
Applied Psychological Measurement, 2020Co-Authors: Pablo Nájera, Jimmy De La Torre, Miguel A. Sorrel, Francisco J. AbadAbstract:In the context of cognitive diagnosis models (CDMs), a Q-Matrix reflects the correspondence between attributes and items. The Q-Matrix construction process is typically subjective in nature, which ...
-
an empirical Q Matrix validation method for the seQuential generalized dina model
British Journal of Mathematical and Statistical Psychology, 2020Co-Authors: Jimmy De La TorreAbstract:: As a core component of most cognitive diagnosis models, the Q-Matrix, or item and attribute association Matrix, is typically developed by domain experts, and tends to be subjective. It is critical to validate the Q-Matrix empirically because a misspecified Q-Matrix could result in erroneous attribute estimation. Most existing Q-Matrix validation procedures are developed for dichotomous responses. However, in this paper, we propose a method to empirically detect and correct the misspecifications in the Q-Matrix for graded response data based on the seQuential generalized deterministic inputs, noisy 'and' gate (G-DINA) model. The proposed Q-Matrix validation procedure is implemented in a stepwise manner based on the Wald test and an effect size measure. The feasibility of the proposed method is examined using simulation studies. Also, a set of data from the Trends in International Mathematics and Science Study (TIMSS) 2011 mathematics assessment is analysed for illustration.
-
An empirical Q‐Matrix validation method for the seQuential generalized DINA model
British Journal of Mathematical and Statistical Psychology, 2019Co-Authors: Wenchao Ma, Jimmy De La TorreAbstract:: As a core component of most cognitive diagnosis models, the Q-Matrix, or item and attribute association Matrix, is typically developed by domain experts, and tends to be subjective. It is critical to validate the Q-Matrix empirically because a misspecified Q-Matrix could result in erroneous attribute estimation. Most existing Q-Matrix validation procedures are developed for dichotomous responses. However, in this paper, we propose a method to empirically detect and correct the misspecifications in the Q-Matrix for graded response data based on the seQuential generalized deterministic inputs, noisy 'and' gate (G-DINA) model. The proposed Q-Matrix validation procedure is implemented in a stepwise manner based on the Wald test and an effect size measure. The feasibility of the proposed method is examined using simulation studies. Also, a set of data from the Trends in International Mathematics and Science Study (TIMSS) 2011 mathematics assessment is analysed for illustration.
-
An Iterative Method for Empirically-Based Q-Matrix Validation.
International Journal of Assessment Tools in Education, 2018Co-Authors: Ragip Terzi, Jimmy De La TorreAbstract:In cognitive diagnosis modeling, the attributes reQuired for each item are specified in the Q-Matrix. The traditional way of constructing a Q-Matrix based on expert opinion is inherently subjective, conseQuently resulting in serious validity concerns. The current study proposes a new validation method under the deterministic inputs, noisy “and” gate (DINA) model to empirically validate attribute specifications in the Q-Matrix. In particular, an iterative procedure with a modified version of the seQuential search algorithm is introduced. Simulation studies are conducted to compare the proposed method with existing parametric and nonparametric methods. Results show that the new method outperforms the other methods across the board. Finally, the method is applied to real data using fraction-subtraction data.
-
On the Consistency of Q-Matrix Estimation: A Rejoinder.
Psychometrika, 2016Co-Authors: Jimmy De La Torre, Chia Yi ChiuAbstract:Abstract This rejoinder responds to the commentary by Liu (Psychometrika, 2015) entitled “On the consistency of Q-Matrix estimation: A commentary” on the paper “A general method of empirical Q-Matrix validation” by de la Torre and Chiu (Psychometrika, 2015). It discusses and addresses three concerns raised in the commentary, namely the estimation accuracy when a provisional Q-Matrix is used, the consistency of the Q-Matrix estimator, and the computational efficiency of the proposed method.
Peng Xu - One of the best experts on this subject based on the ideXlab platform.
-
EDM - The refinement of a Q-Matrix: Assessing methods to validate tasks to skills mapping.
2020Co-Authors: Michel C Desmarais, Behzad Beheshti, Peng XuAbstract:The objective of specifying which skills are reQuired in a given task is fundamental for the accurate assessment of a student’s knowledge and for personalizing tutor interaction towards more relevant and effective assessment and learning. We compare three data driven techniQues for the validation of skills-to-tasks mappings. All methods start from a given mapping, typically obtained from domain experts, and use optimization techniQues to suggest a refined version of the skills-to-task mapping. To validate the different techniQues, we inject perturbations in the Q-Matrix and verify whether the original Q-Matrix can be recovered. Tests are run over both simulated and real data. The analysis of the Q-Matrix refinements of each techniQue over ten data sets shows that, in general, around 1/2 to 2/3 of the perturbations can be restored to their original values, but a number of potentially wrong perturbations are also introduced. The number of correctly restored and falsely switched values vary across the three techniQues and between synthetic and real data. For 1 to 10 perturbations injected, simulated data recovery rate is around 2/3, and invalid alterations introduced vary around 2 to 3. For real data, the two best techniQues generally recover about half the perturbations injected, but introduce between 5 and 7 alterations inconsistent with the original, expert defined Q-Matrix, although some of them may be real improvements.
-
the refinement of a Q Matrix assessing methods to validate tasks to skills mapping
Educational Data Mining, 2014Co-Authors: Michel C Desmarais, Behzad Beheshti, Peng XuAbstract:The objective of specifying which skills are reQuired in a given task is fundamental for the accurate assessment of a student’s knowledge and for personalizing tutor interaction towards more relevant and effective assessment and learning. We compare three data driven techniQues for the validation of skills-to-tasks mappings. All methods start from a given mapping, typically obtained from domain experts, and use optimization techniQues to suggest a refined version of the skills-to-task mapping. To validate the different techniQues, we inject perturbations in the Q-Matrix and verify whether the original Q-Matrix can be recovered. Tests are run over both simulated and real data. The analysis of the Q-Matrix refinements of each techniQue over ten data sets shows that, in general, around 1/2 to 2/3 of the perturbations can be restored to their original values, but a number of potentially wrong perturbations are also introduced. The number of correctly restored and falsely switched values vary across the three techniQues and between synthetic and real data. For 1 to 10 perturbations injected, simulated data recovery rate is around 2/3, and invalid alterations introduced vary around 2 to 3. For real data, the two best techniQues generally recover about half the perturbations injected, but introduce between 5 and 7 alterations inconsistent with the original, expert defined Q-Matrix, although some of them may be real improvements.
Shiwei Ye - One of the best experts on this subject based on the ideXlab platform.
-
EDM - Alternating Recursive Method for Q-Matrix Learning.
2020Co-Authors: Shiwei Ye, Shunya InoueAbstract:The key issue affecting Cognitive Diagnostic Models (CDMs) is how to specify attributes and the Q-Matrix. In this paper, we first attempt to use the Boolean Matrix Factorization (BMF) method to express conjunctive models in CDMs. Because BMF is an NPhard problem [2], we propose a recursive method that updates the attribute Matrix (its rank eQuals to one) in each step. As Boolean algebra is irreversible, it reQuires time to recursively compute and update the Matrix, especially when the number of attributes is large. To speed up computations, we use a Heaviside step function, which allows us to decompose the recursive computing process into normal non-negative matrices and get the results by mapping them back into a Boolean Matrix. Two different algorithms are presented: a deterministic heuristic algorithm and a stochastic algorithm. Simulation results from an actual test show that the proposed method can learn the original Q-Matrix well from item response data.
-
BESC - Q-Matrix learning and DINA model parameter estimation
2016 International Conference on Behavioral Economic and Socio-cultural Computing (BESC), 2016Co-Authors: Shiwei Ye, Guiping SuAbstract:The DINA model is one of the most widely used models in cognitive and skills diagnosis, and several algorithms have been developed for estimating the model parameters. However, since the parameter space is very large and has a mix of binary variables, even medium-sized testing is extremely challenging. To make the model practical, a fast optimization algorithm for parameter estimation is needed. In this study, we converted the deterministic Q-Matrix learning problem into a Boolean Matrix factorization (BMF) problem and developed a recursive algorithm to find an approximate solution while solving the uncertainty parameters analytically using maximum likelihood estimation (MLE). We proved that the MLE is eQuivalent to the minimum information entropy of the DINA model. Simulation results demonstrated that our proposed algorithm converges rapidly to the optimal solution under suitable initial values of skill — item association and is insensitive to the initial values of the uncertainty parameters.
-
Q-Matrix learning and DINA model parameter estimation
2016 International Conference on Behavioral Economic and Socio-cultural Computing (BESC), 2016Co-Authors: Shiwei Ye, Guiping SuAbstract:The DINA model is one of the most widely used models in cognitive and skills diagnosis, and several algorithms have been developed for estimating the model parameters. However, since the parameter space is very large and has a mix of binary variables, even medium-sized testing is extremely challenging. To make the model practical, a fast optimization algorithm for parameter estimation is needed. In this study, we converted the deterministic Q-Matrix learning problem into a Boolean Matrix factorization (BMF) problem and developed a recursive algorithm to find an approximate solution while solving the uncertainty parameters analytically using maximum likelihood estimation (MLE). We proved that the MLE is eQuivalent to the minimum information entropy of the DINA model. Simulation results demonstrated that our proposed algorithm converges rapidly to the optimal solution under suitable initial values of skill - item association and is insensitive to the initial values of the uncertainty parameters.
-
minimum information entropy based Q Matrix learning in dina model
Learning Analytics and Knowledge, 2015Co-Authors: Shiwei Ye, Haobo WangAbstract:Cognitive diagnosis models (CDMs) are of growing interest in test development and measurement of learners' performance. The DINA (deterministic input, noisy, and gate) model is one of the most widely used models in CDM. In this paper, we propose a new method and present an alternating recursive algorithm to learn Q-Matrix and uncertainty variables, slip and guessing parameters, based on Boolean Matrix Factorization (BMF) and Minimized Information Entropy (MIE) respectively for the DINA model. Simulation results show that our algorithm for Q-Matrix learning has fast convergence to the local optimal solutions for Q-Matrix and students' knowledge states A Matrix. This is especially important and applicable when the method is extended to big data.
-
LAK - Minimum information entropy based Q-Matrix learning in DINA model
Proceedings of the Fifth International Conference on Learning Analytics And Knowledge - LAK '15, 2015Co-Authors: Shiwei Ye, Haobo WangAbstract:Cognitive diagnosis models (CDMs) are of growing interest in test development and measurement of learners' performance. The DINA (deterministic input, noisy, and gate) model is one of the most widely used models in CDM. In this paper, we propose a new method and present an alternating recursive algorithm to learn Q-Matrix and uncertainty variables, slip and guessing parameters, based on Boolean Matrix Factorization (BMF) and Minimized Information Entropy (MIE) respectively for the DINA model. Simulation results show that our algorithm for Q-Matrix learning has fast convergence to the local optimal solutions for Q-Matrix and students' knowledge states A Matrix. This is especially important and applicable when the method is extended to big data.
Anders Skrondal - One of the best experts on this subject based on the ideXlab platform.
-
A Constrained Metropolis–Hastings Robbins–Monro Algorithm for Q Matrix Estimation in DINA Models
Psychometrika, 2020Co-Authors: Björn Andersson, Anders SkrondalAbstract:In diagnostic classification models (DCMs), the Q Matrix encodes in which attributes are reQuired for each item. The Q Matrix is usually predetermined by the researcher but may in practice be misspecified which yields incorrect statistical inference. Instead of using a predetermined Q Matrix, it is possible to estimate it simultaneously with the item and structural parameters of the DCM. Unfortunately, current methods are computationally intensive when there are many attributes and items. In addition, the identification constraints necessary for DCMs are not always enforced in the estimation algorithms which can lead to non-identified models being considered. We address these problems by simultaneously estimating the item, structural and Q Matrix parameters of the Deterministic Input Noisy “And” gate model using a constrained Metropolis–Hastings Robbins–Monro algorithm. Simulations show that the new method is computationally efficient and can outperform previously proposed Bayesian Markov chain Monte-Carlo algorithms in terms of Q Matrix recovery, and item and structural parameter estimation. We also illustrate our approach using Tatsuoka’s fraction–subtraction data and Certificate of Proficiency in English data.
-
a constrained metropolis hastings robbins monro algorithm for Q Matrix estimation in dina models
Psychometrika, 2020Co-Authors: Björn Andersson, Anders SkrondalAbstract:In diagnostic classification models (DCMs), the Q Matrix encodes in which attributes are reQuired for each item. The Q Matrix is usually predetermined by the researcher but may in practice be misspecified which yields incorrect statistical inference. Instead of using a predetermined Q Matrix, it is possible to estimate it simultaneously with the item and structural parameters of the DCM. Unfortunately, current methods are computationally intensive when there are many attributes and items. In addition, the identification constraints necessary for DCMs are not always enforced in the estimation algorithms which can lead to non-identified models being considered. We address these problems by simultaneously estimating the item, structural and Q Matrix parameters of the Deterministic Input Noisy “And” gate model using a constrained Metropolis–Hastings Robbins–Monro algorithm. Simulations show that the new method is computationally efficient and can outperform previously proposed Bayesian Markov chain Monte-Carlo algorithms in terms of Q Matrix recovery, and item and structural parameter estimation. We also illustrate our approach using Tatsuoka’s fraction–subtraction data and Certificate of Proficiency in English data.
