The Experts below are selected from a list of 360 Experts worldwide ranked by ideXlab platform
Javier Gardeazabal - One of the best experts on this subject based on the ideXlab platform.
-
do students behave rationally in Multiple Choice Tests evidence from a field experiment
Journal of Economics and Management, 2013Co-Authors: Maria Paz Espinosa, Javier GardeazabalAbstract:A disadvantage of Multiple Choice Tests is that students have incentives to guess. To discourage guessing, it is common to use scoring rules that either penalize wrong answers or reward omissions. In psychometrics, penalty and reward scoring rules are considered equivalent. However, experimental evidence indicates that students behave differently under penalty or reward scoring rules. These differences have been attributed to the different framing (penalty versus reward). In this paper, we model students’ behavior in Multiple Choice Tests as a Choice among lotteries. We show that strategic equivalence among penalty and reward scoring rules holds only under risk neutrality. Therefore, risk aversion could be an alternative explanation to the previously found differences in students’ behavior when confronted with penalty and reward scoring rules. We suggest the use of a modified penalty scoring rule which is equivalent to the reward rule for whatever risk attitudes students might have. To disentangle the effect of framing and risk aversion on students’behavior we design a field experiment with three treatments, each one with a different scoring rule. Two of these scoring rules are equivalent but have different framing, while the third is not equivalent but has the same framing as one of the other two. The experimental results indicate that differences in students’ behavior are due to risk aversion and not due to different framing.
-
optimal correction for guessing in Multiple Choice Tests
Journal of Mathematical Psychology, 2010Co-Authors: Maria Paz Espinosa, Javier GardeazabalAbstract:Abstract Subjects’ decisions in Multiple-Choice Tests are an interesting domain for the analysis of decision making under uncertainty. When the test is graded using a rule that penalizes wrong answers, each item can be viewed as a lottery where a rational examinee would choose whether to omit (sure reward) or answer (take the lottery) depending on risk aversion and level of knowledge. We formalize students as heterogeneous decision makers with different risk attitudes and levels of knowledge. Building on IRT, we compute the optimal penalty given students’ optimal behavior and the trade-off between bias and measurement error. Although MCQ examinations are frequently used, there is no consensus as to whether a penalty for wrong answers should be used or not. For example, examinations for medical licensing in some countries include MCQ sections with penalty while in others there is no penalty for wrong answers. We contribute to this discussion with a formal analysis of the effects of penalties; our simulations indicate that the optimal penalty is relatively high for perfectly rational students but also when they are not fully rational: even though penalty discriminates against risk averse students, this effect is small compared with the measurement error that it prevents.
-
Optimal Correction for Guessing in Multiple-Choice Tests
Research Papers in Economics, 2007Co-Authors: Maria Paz Espinosa, Javier GardeazabalAbstract:Building on Item Response Theory we introduce students٠optimal behavior in Multiple-Choice Tests. Our simulations indicate that the optimal penalty is relatively high, because although correction for guessing discriminates against risk-averse subjects, this effect is small compared with the measurement error that the penalty prevents. This result obtains when knowledge is binary or partial, under different normalizations of the score, when risk aversion is related to knowledge and when there is a pass-fail break point. We also find that the mean degree of difficulty should be close to the mean level of knowledge and that the variance of difficulty should be high.
-
do students behave rationally in Multiple Choice Tests evidence from a field experiment
Research Papers in Economics, 2005Co-Authors: Maria Paz Espinosa, Javier GardeazabalAbstract:A disadvantage of Multiple-Choice Tests is that students have incentives to guess. To discourage guessing, it is common to use scoring rules that either penalize wrong answers or reward omissions. These scoring rules are considered equivalent in psychometrics, although experimental evidence has not always been consistent with this claim. We model students' decisions and show, first, that equivalence holds only under risk neutrality and, second, that the two rules can be modified so that they become equivalent even under risk aversion. This paper presents the results of a filed experiment in which we analyze the decisions of subjects taking Multiple-Choice exams. The evidence suggests that differences between scoring rules are due to risk aversion as theory predicts. We also find that the number of omitted items depends on the scoring rule, knowledge, gender, and other covariates.
Maria Paz Espinosa - One of the best experts on this subject based on the ideXlab platform.
-
do students behave rationally in Multiple Choice Tests evidence from a field experiment
Journal of Economics and Management, 2013Co-Authors: Maria Paz Espinosa, Javier GardeazabalAbstract:A disadvantage of Multiple Choice Tests is that students have incentives to guess. To discourage guessing, it is common to use scoring rules that either penalize wrong answers or reward omissions. In psychometrics, penalty and reward scoring rules are considered equivalent. However, experimental evidence indicates that students behave differently under penalty or reward scoring rules. These differences have been attributed to the different framing (penalty versus reward). In this paper, we model students’ behavior in Multiple Choice Tests as a Choice among lotteries. We show that strategic equivalence among penalty and reward scoring rules holds only under risk neutrality. Therefore, risk aversion could be an alternative explanation to the previously found differences in students’ behavior when confronted with penalty and reward scoring rules. We suggest the use of a modified penalty scoring rule which is equivalent to the reward rule for whatever risk attitudes students might have. To disentangle the effect of framing and risk aversion on students’behavior we design a field experiment with three treatments, each one with a different scoring rule. Two of these scoring rules are equivalent but have different framing, while the third is not equivalent but has the same framing as one of the other two. The experimental results indicate that differences in students’ behavior are due to risk aversion and not due to different framing.
-
optimal correction for guessing in Multiple Choice Tests
Journal of Mathematical Psychology, 2010Co-Authors: Maria Paz Espinosa, Javier GardeazabalAbstract:Abstract Subjects’ decisions in Multiple-Choice Tests are an interesting domain for the analysis of decision making under uncertainty. When the test is graded using a rule that penalizes wrong answers, each item can be viewed as a lottery where a rational examinee would choose whether to omit (sure reward) or answer (take the lottery) depending on risk aversion and level of knowledge. We formalize students as heterogeneous decision makers with different risk attitudes and levels of knowledge. Building on IRT, we compute the optimal penalty given students’ optimal behavior and the trade-off between bias and measurement error. Although MCQ examinations are frequently used, there is no consensus as to whether a penalty for wrong answers should be used or not. For example, examinations for medical licensing in some countries include MCQ sections with penalty while in others there is no penalty for wrong answers. We contribute to this discussion with a formal analysis of the effects of penalties; our simulations indicate that the optimal penalty is relatively high for perfectly rational students but also when they are not fully rational: even though penalty discriminates against risk averse students, this effect is small compared with the measurement error that it prevents.
-
Optimal Correction for Guessing in Multiple-Choice Tests
Research Papers in Economics, 2007Co-Authors: Maria Paz Espinosa, Javier GardeazabalAbstract:Building on Item Response Theory we introduce students٠optimal behavior in Multiple-Choice Tests. Our simulations indicate that the optimal penalty is relatively high, because although correction for guessing discriminates against risk-averse subjects, this effect is small compared with the measurement error that the penalty prevents. This result obtains when knowledge is binary or partial, under different normalizations of the score, when risk aversion is related to knowledge and when there is a pass-fail break point. We also find that the mean degree of difficulty should be close to the mean level of knowledge and that the variance of difficulty should be high.
-
do students behave rationally in Multiple Choice Tests evidence from a field experiment
Research Papers in Economics, 2005Co-Authors: Maria Paz Espinosa, Javier GardeazabalAbstract:A disadvantage of Multiple-Choice Tests is that students have incentives to guess. To discourage guessing, it is common to use scoring rules that either penalize wrong answers or reward omissions. These scoring rules are considered equivalent in psychometrics, although experimental evidence has not always been consistent with this claim. We model students' decisions and show, first, that equivalence holds only under risk neutrality and, second, that the two rules can be modified so that they become equivalent even under risk aversion. This paper presents the results of a filed experiment in which we analyze the decisions of subjects taking Multiple-Choice exams. The evidence suggests that differences between scoring rules are due to risk aversion as theory predicts. We also find that the number of omitted items depends on the scoring rule, knowledge, gender, and other covariates.
Elizabeth Ligon Bjork - One of the best experts on this subject based on the ideXlab platform.
-
Multiple Choice Tests stabilize access to marginal knowledge
Memory & Cognition, 2015Co-Authors: Allison D Cantor, Robert A Bjork, Andrea N Eslick, Elizabeth J Marsh, Elizabeth Ligon BjorkAbstract:Marginal knowledge refers to knowledge that is stored in memory, but is not accessible at a given moment. For example, one might struggle to remember who wrote The Call of the Wild, even if that knowledge is stored in memory. Knowing how best to stabilize access to marginal knowledge is important, given that new learning often requires accessing and building on prior knowledge. While even a single oppor- tunity to restudy marginal knowledge boosts its later accessi- bility (Berger, Hall, & Bahrick, 1999), in many situations explicit relearning opportunities are not available. Our ques- tion is whether Multiple-Choice Tests (which by definition expose the learner to the correct answers) can also serve this function and, if so, how testing compares to restudying given that Tests can be particularly powerful learning devices (Roediger & Karpicke, 2006). In four experiments, we found that Multiple-Choice testing had the power to stabilize access to marginal knowledge, and to do so for at least up to a week. Importantly, such Tests did not need to be paired with feed- back, although testing was no more powerful than studying. Overall,the results support the idea that one's knowledge base is unstable, with individual pieces of information coming in and out of reach. The present findings have implications for a key educational challenge: ensuring that students have con- tinuing access to information they have learned.
-
Optimizing Multiple-Choice Tests as tools for learning
Memory & Cognition, 2015Co-Authors: Jeri L Little, Elizabeth Ligon BjorkAbstract:Answering Multiple-Choice questions with competitive alternatives can enhance performance on a later test, not only on questions about the information previously tested, but also on questions about related information not previously tested—in particular, on questions about information pertaining to the previously incorrect alternatives. In the present research, we assessed a possible explanation for this pattern: When Multiple-Choice questions contain competitive incorrect alternatives, test-takers are led to retrieve previously studied information pertaining to all of the alternatives in order to discriminate among them and select an answer, with such processing strengthening later access to information associated with both the correct and incorrect alternatives. Supporting this hypothesis, we found enhanced performance on a later cued-recall test for previously nontested questions when their answers had previously appeared as competitive incorrect alternatives in the initial Multiple-Choice test, but not when they had previously appeared as noncompetitive alternatives. Importantly, however, competitive alternatives were not more likely than noncompetitive alternatives to be intruded as incorrect responses, indicating that a general increased accessibility for previously presented incorrect alternatives could not be the explanation for these results. The present findings, replicated across two experiments (one in which corrective feedback was provided during the initial Multiple-Choice testing, and one in which it was not), thus strongly suggest that competitive Multiple-Choice questions can trigger beneficial retrieval processes for both tested and related information, and the results have implications for the effective use of Multiple-Choice Tests as tools for learning.
-
Multiple Choice Tests exonerated at least of some charges fostering test induced learning and avoiding test induced forgetting
Psychological Science, 2012Co-Authors: Jeri L Little, Elizabeth Ligon Bjork, Robert A Bjork, Genna AngelloAbstract:Among the criticisms of Multiple-Choice Tests is that—by exposing the correct answer as one of the alternatives—such Tests engage recognition processes rather than the productive retrieval processes known to enhance later recall. We tested whether Multiple-Choice Tests could trigger productive retrieval processes—provided the alternatives were made plausible enough to enable test takers to retrieve both why the correct alternatives were correct and why the incorrect alternatives were incorrect. In two experiments, we found not only that properly constructed Multiple-Choice Tests can indeed trigger productive retrieval processes, but also that they had one potentially important advantage over cued-recall Tests. Both testing formats fostered retention of previously tested information, but Multiple-Choice Tests also facilitated recall of information pertaining to incorrect alternatives, whereas cued-recall Tests did not. Thus, Multiple-Choice Tests can be constructed so that they exercise the very retrieval ...
-
the persisting benefits of using Multiple Choice Tests as learning events
Cognitive Science, 2012Co-Authors: Jeri L Little, Elizabeth Ligon BjorkAbstract:The Persisting Benefits of Using Multiple-Choice Tests as Learning Events Jeri L. Little (jerilittle@wustl.edu) Department of Psychology, Washington University in St. Louis Campus Box 1125, One Brookings Drive, St. Louis, MO 63130-4899 Elizabeth Ligon Bjork (elbjork@psych.ucla.edu) Department of Psychology, University of California, Los Angeles 1285 Franz Hall, Box 951563, Los Angeles, CA 90095-1563 research has indicated that Multiple-Choice testing may have a benefit for later overall performance that is not produced by the taking of prior cued-recall Tests: namely, not only improving the retention of the information tested but also that of non-tested, but related, information—an outcome that would seem to be particularly desirable in educational contexts. When, for example, instructors give quizzes or practice Tests to be followed later in the course by a more comprehensive exam, they are likely to ask questions about related information instead of (or in addition to) questions about the specific information tested earlier. Thus, it seems critical that the use of prior testing in educational contexts should help to improve the retention of both types of information. With respect to the question of how retrieving some information affects the later retention of non-tested related information, previous research has demonstrated that the effects can be negative—that is, answering a cued-recall question can lead to impaired recall of competitive related information on a later test, a finding referred to as retrieval- induced forgetting (Anderson, Bjork, & Bjork, 1994). Even in cases in which retrieval-induced forgetting does not occur, however, the retention of non-tested related information is rarely facilitated as a consequence of a cued- recall test, especially when such non-tested information has a competitive relationship with the tested information. (For exceptions to this finding when tested and non-tested related questions were created to be facilitative, see Chan, McDermott, & Roediger, 2006, and Chan, 2009.) In contrast to such findings for cued-recall Tests, however, Little, Bjork, Bjork, and Angello (in press) recently demonstrated that—in addition to improving retention of previously tested information—taking an initial Multiple- Choice test, as compared to taking an initial cued-recall test, can improve retention of related information when the answers to the competitive related questions have occurred as incorrect alternatives in the initial Multiple-Choice test. Although these findings have clear implications for educational practice, particularly how an instructor might construct a practice exam, the procedure used by Little et al. employed a delay that is considered to lack educational realism (e.g., 5 min). Thus, a primary goal of the present research was to assess whether the demonstrated retention benefits (particularly for competitive related information) occurring as a consequence of taking Multiple-Choice Abstract Taking a test tends to improve the retention of the tested information. Additionally, taking a test often influences the later retention of non-tested information, provided such information is related to the tested information in a specific manner. To illustrate, recent research has demonstrated that Multiple-Choice Tests containing competitive alternatives can improve retention of both tested and non-tested information pertaining to such incorrect alternatives at least over a short delay. The present research investigated whether such improvements in retention would persist with a delay more likely to occur in educational contexts (i.e., 48 hr). Taking an initial Multiple-Choice test improved retention more than a comparable cued-recall test—for both previously tested and related information—and over both short and long delays. Moreover, misinformation effects seen for the Multiple-Choice test at the short delay were reduced. These results thus have important implications for the use of Multiple-Choice Tests as learning opportunities. Keywords: Testing; test effects; Multiple-Choice; retrieval- induced forgetting, retrieval-induced facilitation Introduction Taking a test does more than assess one’s knowledge: It can also improve one’s long-term retention of the tested information. Not all Tests, however, are equally beneficial in this manner. For example, more open-ended Tests (e.g., cued-recall), in general, have been shown to improve long- term retention more than Multiple-Choice Tests (see meta- analyses by Anderson & Biddle, 1975; Hamaker, 1986). Moreover, some research has shown that taking Multiple- Choice Tests can actually harm later performance on open- ended Tests by exposing the test takers to misinformation in the form of incorrect, but plausible, answer Choices, with the consequence of such information sometimes being intruded as incorrect responses to later cued-recall questions— findings referred to as misinformation effects (e.g., Roediger & Marsh, 2005). Thus, Multiple-Choice Tests are often accused of not only being less effective for learning than are more open-ended types of Tests, but also to risk negative misinformation effects, bringing their use as learning tools in educational contexts into question. Although, as indicated, Multiple-Choice Tests can produce misinformation effects on later open-ended Tests, research, in general, has shown that the positive effects of Multiple- Choice testing (as compared to no testing) outweigh any such negative effects they engender. Furthermore, recent
Jeri L Little - One of the best experts on this subject based on the ideXlab platform.
-
Optimizing Multiple-Choice Tests as tools for learning
Memory & Cognition, 2015Co-Authors: Jeri L Little, Elizabeth Ligon BjorkAbstract:Answering Multiple-Choice questions with competitive alternatives can enhance performance on a later test, not only on questions about the information previously tested, but also on questions about related information not previously tested—in particular, on questions about information pertaining to the previously incorrect alternatives. In the present research, we assessed a possible explanation for this pattern: When Multiple-Choice questions contain competitive incorrect alternatives, test-takers are led to retrieve previously studied information pertaining to all of the alternatives in order to discriminate among them and select an answer, with such processing strengthening later access to information associated with both the correct and incorrect alternatives. Supporting this hypothesis, we found enhanced performance on a later cued-recall test for previously nontested questions when their answers had previously appeared as competitive incorrect alternatives in the initial Multiple-Choice test, but not when they had previously appeared as noncompetitive alternatives. Importantly, however, competitive alternatives were not more likely than noncompetitive alternatives to be intruded as incorrect responses, indicating that a general increased accessibility for previously presented incorrect alternatives could not be the explanation for these results. The present findings, replicated across two experiments (one in which corrective feedback was provided during the initial Multiple-Choice testing, and one in which it was not), thus strongly suggest that competitive Multiple-Choice questions can trigger beneficial retrieval processes for both tested and related information, and the results have implications for the effective use of Multiple-Choice Tests as tools for learning.
-
Multiple Choice Tests exonerated at least of some charges fostering test induced learning and avoiding test induced forgetting
Psychological Science, 2012Co-Authors: Jeri L Little, Elizabeth Ligon Bjork, Robert A Bjork, Genna AngelloAbstract:Among the criticisms of Multiple-Choice Tests is that—by exposing the correct answer as one of the alternatives—such Tests engage recognition processes rather than the productive retrieval processes known to enhance later recall. We tested whether Multiple-Choice Tests could trigger productive retrieval processes—provided the alternatives were made plausible enough to enable test takers to retrieve both why the correct alternatives were correct and why the incorrect alternatives were incorrect. In two experiments, we found not only that properly constructed Multiple-Choice Tests can indeed trigger productive retrieval processes, but also that they had one potentially important advantage over cued-recall Tests. Both testing formats fostered retention of previously tested information, but Multiple-Choice Tests also facilitated recall of information pertaining to incorrect alternatives, whereas cued-recall Tests did not. Thus, Multiple-Choice Tests can be constructed so that they exercise the very retrieval ...
-
the persisting benefits of using Multiple Choice Tests as learning events
Cognitive Science, 2012Co-Authors: Jeri L Little, Elizabeth Ligon BjorkAbstract:The Persisting Benefits of Using Multiple-Choice Tests as Learning Events Jeri L. Little (jerilittle@wustl.edu) Department of Psychology, Washington University in St. Louis Campus Box 1125, One Brookings Drive, St. Louis, MO 63130-4899 Elizabeth Ligon Bjork (elbjork@psych.ucla.edu) Department of Psychology, University of California, Los Angeles 1285 Franz Hall, Box 951563, Los Angeles, CA 90095-1563 research has indicated that Multiple-Choice testing may have a benefit for later overall performance that is not produced by the taking of prior cued-recall Tests: namely, not only improving the retention of the information tested but also that of non-tested, but related, information—an outcome that would seem to be particularly desirable in educational contexts. When, for example, instructors give quizzes or practice Tests to be followed later in the course by a more comprehensive exam, they are likely to ask questions about related information instead of (or in addition to) questions about the specific information tested earlier. Thus, it seems critical that the use of prior testing in educational contexts should help to improve the retention of both types of information. With respect to the question of how retrieving some information affects the later retention of non-tested related information, previous research has demonstrated that the effects can be negative—that is, answering a cued-recall question can lead to impaired recall of competitive related information on a later test, a finding referred to as retrieval- induced forgetting (Anderson, Bjork, & Bjork, 1994). Even in cases in which retrieval-induced forgetting does not occur, however, the retention of non-tested related information is rarely facilitated as a consequence of a cued- recall test, especially when such non-tested information has a competitive relationship with the tested information. (For exceptions to this finding when tested and non-tested related questions were created to be facilitative, see Chan, McDermott, & Roediger, 2006, and Chan, 2009.) In contrast to such findings for cued-recall Tests, however, Little, Bjork, Bjork, and Angello (in press) recently demonstrated that—in addition to improving retention of previously tested information—taking an initial Multiple- Choice test, as compared to taking an initial cued-recall test, can improve retention of related information when the answers to the competitive related questions have occurred as incorrect alternatives in the initial Multiple-Choice test. Although these findings have clear implications for educational practice, particularly how an instructor might construct a practice exam, the procedure used by Little et al. employed a delay that is considered to lack educational realism (e.g., 5 min). Thus, a primary goal of the present research was to assess whether the demonstrated retention benefits (particularly for competitive related information) occurring as a consequence of taking Multiple-Choice Abstract Taking a test tends to improve the retention of the tested information. Additionally, taking a test often influences the later retention of non-tested information, provided such information is related to the tested information in a specific manner. To illustrate, recent research has demonstrated that Multiple-Choice Tests containing competitive alternatives can improve retention of both tested and non-tested information pertaining to such incorrect alternatives at least over a short delay. The present research investigated whether such improvements in retention would persist with a delay more likely to occur in educational contexts (i.e., 48 hr). Taking an initial Multiple-Choice test improved retention more than a comparable cued-recall test—for both previously tested and related information—and over both short and long delays. Moreover, misinformation effects seen for the Multiple-Choice test at the short delay were reduced. These results thus have important implications for the use of Multiple-Choice Tests as learning opportunities. Keywords: Testing; test effects; Multiple-Choice; retrieval- induced forgetting, retrieval-induced facilitation Introduction Taking a test does more than assess one’s knowledge: It can also improve one’s long-term retention of the tested information. Not all Tests, however, are equally beneficial in this manner. For example, more open-ended Tests (e.g., cued-recall), in general, have been shown to improve long- term retention more than Multiple-Choice Tests (see meta- analyses by Anderson & Biddle, 1975; Hamaker, 1986). Moreover, some research has shown that taking Multiple- Choice Tests can actually harm later performance on open- ended Tests by exposing the test takers to misinformation in the form of incorrect, but plausible, answer Choices, with the consequence of such information sometimes being intruded as incorrect responses to later cued-recall questions— findings referred to as misinformation effects (e.g., Roediger & Marsh, 2005). Thus, Multiple-Choice Tests are often accused of not only being less effective for learning than are more open-ended types of Tests, but also to risk negative misinformation effects, bringing their use as learning tools in educational contexts into question. Although, as indicated, Multiple-Choice Tests can produce misinformation effects on later open-ended Tests, research, in general, has shown that the positive effects of Multiple- Choice testing (as compared to no testing) outweigh any such negative effects they engender. Furthermore, recent
Kathryn Scantlebury - One of the best experts on this subject based on the ideXlab platform.
-
the role of rasch analysis when conducting science education research utilizing Multiple Choice Tests
Science Education, 2006Co-Authors: William J Boone, Kathryn ScantleburyAbstract:Recent international studies note that countries whose students perform well on international science assessments report the need to change science education. Some countries use assessments for diagnostic purposes to assist teachers in addressing their students' needs. However, in the United States, standards-based reform has focused the national discussion on documenting students' attainment of high educational standards. Students' science achievement is one of those standards, and in many states, “high-stakes” Tests determine the resultant achievement measures. Policymakers and administrators use those Tests to rank school performance, to prevent students' graduation, and to evaluate teachers. With science test measures used in different ways, statistical confidence in the measures' validity and reliability is essential. Using a science achievement test from one state's systemic reform project as an example, this paper discusses the strengths of the Rasch model as a psychometric tool and analysis technique, referring to person item maps, anchoring, differential item functioning, and person item fit. Furthermore, the paper proposes that science educators should carefully inspect the tools they use to measure and document changes in educational systems. © 2005 Wiley Periodicals, Inc. Sci Ed90:253–269, 2006