The Experts below are selected from a list of 6723 Experts worldwide ranked by ideXlab platform

David C Knill - One of the best experts on this subject based on the ideXlab platform.

  • an Ideal Observer analysis of visual working memory
    Psychological Review, 2012
    Co-Authors: Chris R Sims, Robert A Jacobs, David C Knill
    Abstract:

    Limits in visual working memory (VWM) strongly constrain human performance across many tasks. However, the nature of these limits is not well understood. In this article we develop an Ideal Observer analysis of human VWM by deriving the expected behavior of an optimally performing but limitedcapacity memory system. This analysis is framed around rate–distortion theory, a branch of information theory that provides optimal bounds on the accuracy of information transmission subject to a fixed information capacity. The result of the Ideal Observer analysis is a theoretical framework that provides a task-independent and quantitative definition of visual memory capacity and yields novel predictions regarding human performance. These predictions are subsequently evaluated and confirmed in 2 empirical studies. Further, the framework is general enough to allow the specification and testing of alternative models of visual memory (e.g., how capacity is distributed across multiple items). We demonstrate that a simple model developed on the basis of the Ideal Observer analysis—one that allows variability in the number of stored memory representations but does not assume the presence of a fixed item limit—provides an excellent account of the empirical data and further offers a principled reinterpretation of existing models of VWM.

  • an Ideal Observer model of visual short term memory predicts human capacity precision tradeoffs
    Cognitive Science, 2011
    Co-Authors: Chris R Sims, Robert A Jacobs, David C Knill
    Abstract:

    An Ideal Observer Model of Visual Short-Term Memory Predicts Human Capacity–Precision Tradeoffs Chris R. Sims Robert A. Jacobs David C. Knill (csims@cvs.rochester.edu) (robbie@bcs.rochester.edu) (knill@cvs.rochester.edu) Department of Brain and Cognitive Sciences & Center for Visual Science University of Rochester, Rochester, NY Abstract p(x) p(xs | x) We develop an Ideal Observer model of human visual short- term memory. Compared to previous models that have posited constraints on memory performance intended solely to ac- count for observed phenomena, in the present research we de- rive the expected behavior of an optimally performing, but limited-capacity memory system. We develop our model us- ing rate–distortion theory, a branch of information theory that provides optimal bounds on the accuracy of information trans- mission subject to a fixed capacity. The resulting model pro- vides a task-independent and theoretically motivated definition of visual memory capacity and yields novel predictions regard- ing human performance. These predictions are quantitatively evaluated in an empirical study. We also demonstrate that our Ideal Observer model encompasses two existing, but competing accounts of VSTM as special cases. Keywords: Ideal Observer analysis, VSTM, information the- ory, rate–distortion theory Information source Sensory signal xs VSTM Memory encoding / storage p(x ˆ | xm ) Recovered signal xm Optimal encoder & decoder Memory decoding / retrieval Figure 1: A schematic diagram of the Ideal Observer model of VSTM. Introduction but rather only serves to account for observed empirical phe- nomena. Second, the definition of capacity appears largely task-dependent, and it is therefore difficult to form predictions for human performance in different tasks or conditions. In the case of the discrete slot model, there is no strong theoretical justification for determining what visual features or objects can or cannot occupy a single slot, and in the continuous re- source model, the nature of the resource that is being divided is only abstractly specified. Visual short-term memory (VSTM)—defined as the ability to store task-relevant visual information in a rapidly acces- sible and easily manipulated form—is a central component of nearly all human activities. Given its importance, it is per- haps surprising that the capacity of this system is severely lim- ited. Numerous investigations of VSTM performance have revealed that we can only accurately store a surprisingly small number of visual objects or features (for a review, see Luck, In recent years there have been numerous attempts to de- fine and quantify what is meant by VSTM capacity. Until recently, the predominant view has held that capacity is lim- ited to a small, fixed number of visual objects (typically as- sumed to be 4) stored in discrete “slots” (Awh, Barton, & Vo- gel, 2007; Luck & Vogel, 1997; Vogel, Woodman, & Luck, 2001; Cowan & Rouder, 2009). Taking a different approach, Bays and colleagues (Bays, Catalao, & Husain, 2009; Bays & Husain, 2008) explored how the precision of features en- coded in visual working memory may change as a function of the number of features that are concurrently stored. Based on the finding that memory precision appears to decrease even when as few as 2 items are encoded, the authors proposed that VSTM capacity consists of a single, continuous resource that must be divided among items stored in working memory. While both the discrete slot and continuous resource mod- els are able to account for a number of empirical findings, both are ultimately unsatisfactory as complete theories of hu- man VSTM. First, in the case of both models the nature of the capacity limit is somewhat arbitrary: the hypothesized capac- ity limit does not emerge from a principled theoretical basis, The Ideal Observer Model of VSTM In this section we derive an Ideal Observer model of visual short-term memory. We show that from an information- theoretic perspective, our Ideal Observer model is optimally efficient in that it minimizes a particular measure of memory distortion subject to a fixed capacity limit. The resulting model makes several important contributions to the literature on VSTM. First, whereas previous models have postulated abstract or task-dependent definitions of vi- sual memory capacity, the Ideal Observer model provides a quantitative definition of capacity that is task-independent and easily interpreted. This enables results obtained in one ex- periment to generate predictions for performance in another. Second, since our model exhibits provably optimal perfor- mance subject to a fixed capacity, it can be used to obtain an assumption-free estimate of the minimum capacity of hu- man VSTM. Finally, we demonstrate that our Ideal Observer model subsumes two existing models of VSTM: a recent ver- sion of the discrete slot model (Cowan & Rouder, 2009), and the continuous resource model (Bays & Husain, 2008).

  • CogSci - An Ideal Observer Model of Visual Short-Term Memory Predicts Human Capacity–Precision Tradeoffs
    Cognitive Science, 2011
    Co-Authors: Chris R Sims, Robert A Jacobs, David C Knill
    Abstract:

    An Ideal Observer Model of Visual Short-Term Memory Predicts Human Capacity–Precision Tradeoffs Chris R. Sims Robert A. Jacobs David C. Knill (csims@cvs.rochester.edu) (robbie@bcs.rochester.edu) (knill@cvs.rochester.edu) Department of Brain and Cognitive Sciences & Center for Visual Science University of Rochester, Rochester, NY Abstract p(x) p(xs | x) We develop an Ideal Observer model of human visual short- term memory. Compared to previous models that have posited constraints on memory performance intended solely to ac- count for observed phenomena, in the present research we de- rive the expected behavior of an optimally performing, but limited-capacity memory system. We develop our model us- ing rate–distortion theory, a branch of information theory that provides optimal bounds on the accuracy of information trans- mission subject to a fixed capacity. The resulting model pro- vides a task-independent and theoretically motivated definition of visual memory capacity and yields novel predictions regard- ing human performance. These predictions are quantitatively evaluated in an empirical study. We also demonstrate that our Ideal Observer model encompasses two existing, but competing accounts of VSTM as special cases. Keywords: Ideal Observer analysis, VSTM, information the- ory, rate–distortion theory Information source Sensory signal xs VSTM Memory encoding / storage p(x ˆ | xm ) Recovered signal xm Optimal encoder & decoder Memory decoding / retrieval Figure 1: A schematic diagram of the Ideal Observer model of VSTM. Introduction but rather only serves to account for observed empirical phe- nomena. Second, the definition of capacity appears largely task-dependent, and it is therefore difficult to form predictions for human performance in different tasks or conditions. In the case of the discrete slot model, there is no strong theoretical justification for determining what visual features or objects can or cannot occupy a single slot, and in the continuous re- source model, the nature of the resource that is being divided is only abstractly specified. Visual short-term memory (VSTM)—defined as the ability to store task-relevant visual information in a rapidly acces- sible and easily manipulated form—is a central component of nearly all human activities. Given its importance, it is per- haps surprising that the capacity of this system is severely lim- ited. Numerous investigations of VSTM performance have revealed that we can only accurately store a surprisingly small number of visual objects or features (for a review, see Luck, In recent years there have been numerous attempts to de- fine and quantify what is meant by VSTM capacity. Until recently, the predominant view has held that capacity is lim- ited to a small, fixed number of visual objects (typically as- sumed to be 4) stored in discrete “slots” (Awh, Barton, & Vo- gel, 2007; Luck & Vogel, 1997; Vogel, Woodman, & Luck, 2001; Cowan & Rouder, 2009). Taking a different approach, Bays and colleagues (Bays, Catalao, & Husain, 2009; Bays & Husain, 2008) explored how the precision of features en- coded in visual working memory may change as a function of the number of features that are concurrently stored. Based on the finding that memory precision appears to decrease even when as few as 2 items are encoded, the authors proposed that VSTM capacity consists of a single, continuous resource that must be divided among items stored in working memory. While both the discrete slot and continuous resource mod- els are able to account for a number of empirical findings, both are ultimately unsatisfactory as complete theories of hu- man VSTM. First, in the case of both models the nature of the capacity limit is somewhat arbitrary: the hypothesized capac- ity limit does not emerge from a principled theoretical basis, The Ideal Observer Model of VSTM In this section we derive an Ideal Observer model of visual short-term memory. We show that from an information- theoretic perspective, our Ideal Observer model is optimally efficient in that it minimizes a particular measure of memory distortion subject to a fixed capacity limit. The resulting model makes several important contributions to the literature on VSTM. First, whereas previous models have postulated abstract or task-dependent definitions of vi- sual memory capacity, the Ideal Observer model provides a quantitative definition of capacity that is task-independent and easily interpreted. This enables results obtained in one ex- periment to generate predictions for performance in another. Second, since our model exhibits provably optimal perfor- mance subject to a fixed capacity, it can be used to obtain an assumption-free estimate of the minimum capacity of hu- man VSTM. Finally, we demonstrate that our Ideal Observer model subsumes two existing models of VSTM: a recent ver- sion of the discrete slot model (Cowan & Rouder, 2009), and the continuous resource model (Bays & Husain, 2008).

  • Ideal Observer perturbation analysis reveals human strategies for inferring surface orientation from texture
    Vision research, 1998
    Co-Authors: David C Knill
    Abstract:

    Abstract Optical texture patterns contain three quasi-independent cues to planar surface orientation: perspective scaling, projective foreshortening and density. The purpose of this work was to estimate the perceptual weights assigned to these texture cues for discriminating surface orientation and to measure the visual system's reliance on an isotropy assumption in interpreting foreshortening information. A novel analytical technique is introduced which takes advantage of the natural cue perturbations inherent in stochastic texture stimuli to estimate cue weights and measure the influence of an isotropy assumption. Ideal Observers were derived which compute the exact information content of the different texture cues in the stimuli used in the experiments and which either did or did not rely on an assumption of surface texture isotropy. Simulations of the Ideal Observers using the same stimuli shown to subjects in a slant discrimination task provided trial-by-trial estimates of the natural cue perturbations which were inherent in the stimuli. By back-correlating subjects' judgements with the different Ideal Observer estimates, we were able to estimate both the weights given to each cue by subjects and the strength of subjects' prior assumptions of isotropy. In all of the conditions tested, we found that subjects relied primarily on the foreshortening cue. A small, but significant weight was given to scaling information and no significant weight was given to density information. In conditions in which the surface textures deviated from isotropy by random amounts from stimulus to stimulus, subject judgements correlated well with the estimates of an Ideal Observer which incorrectly assumed surface texture isotropy. This correlation was not complete, however, suggesting that a soft form of the isotropy constraint was used. Moreover, the correlation was significantly lower for textures containing higher-order information about surface orientation (skew of rectangular texture elements). The results of the analysis clearly implicate texture foreshortening as a primary cue for perceiving surface slant from texture and suggest that the visual system incorporates a strong, though not complete, bias to interpret surface textures as isotropic in its inference of surface slant from texture. They further suggest that local texture skew, when available in an image, contributes significantly to perceptual estimates of surface orientation.

Chris R Sims - One of the best experts on this subject based on the ideXlab platform.

  • an Ideal Observer analysis of visual working memory
    Psychological Review, 2012
    Co-Authors: Chris R Sims, Robert A Jacobs, David C Knill
    Abstract:

    Limits in visual working memory (VWM) strongly constrain human performance across many tasks. However, the nature of these limits is not well understood. In this article we develop an Ideal Observer analysis of human VWM by deriving the expected behavior of an optimally performing but limitedcapacity memory system. This analysis is framed around rate–distortion theory, a branch of information theory that provides optimal bounds on the accuracy of information transmission subject to a fixed information capacity. The result of the Ideal Observer analysis is a theoretical framework that provides a task-independent and quantitative definition of visual memory capacity and yields novel predictions regarding human performance. These predictions are subsequently evaluated and confirmed in 2 empirical studies. Further, the framework is general enough to allow the specification and testing of alternative models of visual memory (e.g., how capacity is distributed across multiple items). We demonstrate that a simple model developed on the basis of the Ideal Observer analysis—one that allows variability in the number of stored memory representations but does not assume the presence of a fixed item limit—provides an excellent account of the empirical data and further offers a principled reinterpretation of existing models of VWM.

  • an Ideal Observer model of visual short term memory predicts human capacity precision tradeoffs
    Cognitive Science, 2011
    Co-Authors: Chris R Sims, Robert A Jacobs, David C Knill
    Abstract:

    An Ideal Observer Model of Visual Short-Term Memory Predicts Human Capacity–Precision Tradeoffs Chris R. Sims Robert A. Jacobs David C. Knill (csims@cvs.rochester.edu) (robbie@bcs.rochester.edu) (knill@cvs.rochester.edu) Department of Brain and Cognitive Sciences & Center for Visual Science University of Rochester, Rochester, NY Abstract p(x) p(xs | x) We develop an Ideal Observer model of human visual short- term memory. Compared to previous models that have posited constraints on memory performance intended solely to ac- count for observed phenomena, in the present research we de- rive the expected behavior of an optimally performing, but limited-capacity memory system. We develop our model us- ing rate–distortion theory, a branch of information theory that provides optimal bounds on the accuracy of information trans- mission subject to a fixed capacity. The resulting model pro- vides a task-independent and theoretically motivated definition of visual memory capacity and yields novel predictions regard- ing human performance. These predictions are quantitatively evaluated in an empirical study. We also demonstrate that our Ideal Observer model encompasses two existing, but competing accounts of VSTM as special cases. Keywords: Ideal Observer analysis, VSTM, information the- ory, rate–distortion theory Information source Sensory signal xs VSTM Memory encoding / storage p(x ˆ | xm ) Recovered signal xm Optimal encoder & decoder Memory decoding / retrieval Figure 1: A schematic diagram of the Ideal Observer model of VSTM. Introduction but rather only serves to account for observed empirical phe- nomena. Second, the definition of capacity appears largely task-dependent, and it is therefore difficult to form predictions for human performance in different tasks or conditions. In the case of the discrete slot model, there is no strong theoretical justification for determining what visual features or objects can or cannot occupy a single slot, and in the continuous re- source model, the nature of the resource that is being divided is only abstractly specified. Visual short-term memory (VSTM)—defined as the ability to store task-relevant visual information in a rapidly acces- sible and easily manipulated form—is a central component of nearly all human activities. Given its importance, it is per- haps surprising that the capacity of this system is severely lim- ited. Numerous investigations of VSTM performance have revealed that we can only accurately store a surprisingly small number of visual objects or features (for a review, see Luck, In recent years there have been numerous attempts to de- fine and quantify what is meant by VSTM capacity. Until recently, the predominant view has held that capacity is lim- ited to a small, fixed number of visual objects (typically as- sumed to be 4) stored in discrete “slots” (Awh, Barton, & Vo- gel, 2007; Luck & Vogel, 1997; Vogel, Woodman, & Luck, 2001; Cowan & Rouder, 2009). Taking a different approach, Bays and colleagues (Bays, Catalao, & Husain, 2009; Bays & Husain, 2008) explored how the precision of features en- coded in visual working memory may change as a function of the number of features that are concurrently stored. Based on the finding that memory precision appears to decrease even when as few as 2 items are encoded, the authors proposed that VSTM capacity consists of a single, continuous resource that must be divided among items stored in working memory. While both the discrete slot and continuous resource mod- els are able to account for a number of empirical findings, both are ultimately unsatisfactory as complete theories of hu- man VSTM. First, in the case of both models the nature of the capacity limit is somewhat arbitrary: the hypothesized capac- ity limit does not emerge from a principled theoretical basis, The Ideal Observer Model of VSTM In this section we derive an Ideal Observer model of visual short-term memory. We show that from an information- theoretic perspective, our Ideal Observer model is optimally efficient in that it minimizes a particular measure of memory distortion subject to a fixed capacity limit. The resulting model makes several important contributions to the literature on VSTM. First, whereas previous models have postulated abstract or task-dependent definitions of vi- sual memory capacity, the Ideal Observer model provides a quantitative definition of capacity that is task-independent and easily interpreted. This enables results obtained in one ex- periment to generate predictions for performance in another. Second, since our model exhibits provably optimal perfor- mance subject to a fixed capacity, it can be used to obtain an assumption-free estimate of the minimum capacity of hu- man VSTM. Finally, we demonstrate that our Ideal Observer model subsumes two existing models of VSTM: a recent ver- sion of the discrete slot model (Cowan & Rouder, 2009), and the continuous resource model (Bays & Husain, 2008).

  • CogSci - An Ideal Observer Model of Visual Short-Term Memory Predicts Human Capacity–Precision Tradeoffs
    Cognitive Science, 2011
    Co-Authors: Chris R Sims, Robert A Jacobs, David C Knill
    Abstract:

    An Ideal Observer Model of Visual Short-Term Memory Predicts Human Capacity–Precision Tradeoffs Chris R. Sims Robert A. Jacobs David C. Knill (csims@cvs.rochester.edu) (robbie@bcs.rochester.edu) (knill@cvs.rochester.edu) Department of Brain and Cognitive Sciences & Center for Visual Science University of Rochester, Rochester, NY Abstract p(x) p(xs | x) We develop an Ideal Observer model of human visual short- term memory. Compared to previous models that have posited constraints on memory performance intended solely to ac- count for observed phenomena, in the present research we de- rive the expected behavior of an optimally performing, but limited-capacity memory system. We develop our model us- ing rate–distortion theory, a branch of information theory that provides optimal bounds on the accuracy of information trans- mission subject to a fixed capacity. The resulting model pro- vides a task-independent and theoretically motivated definition of visual memory capacity and yields novel predictions regard- ing human performance. These predictions are quantitatively evaluated in an empirical study. We also demonstrate that our Ideal Observer model encompasses two existing, but competing accounts of VSTM as special cases. Keywords: Ideal Observer analysis, VSTM, information the- ory, rate–distortion theory Information source Sensory signal xs VSTM Memory encoding / storage p(x ˆ | xm ) Recovered signal xm Optimal encoder & decoder Memory decoding / retrieval Figure 1: A schematic diagram of the Ideal Observer model of VSTM. Introduction but rather only serves to account for observed empirical phe- nomena. Second, the definition of capacity appears largely task-dependent, and it is therefore difficult to form predictions for human performance in different tasks or conditions. In the case of the discrete slot model, there is no strong theoretical justification for determining what visual features or objects can or cannot occupy a single slot, and in the continuous re- source model, the nature of the resource that is being divided is only abstractly specified. Visual short-term memory (VSTM)—defined as the ability to store task-relevant visual information in a rapidly acces- sible and easily manipulated form—is a central component of nearly all human activities. Given its importance, it is per- haps surprising that the capacity of this system is severely lim- ited. Numerous investigations of VSTM performance have revealed that we can only accurately store a surprisingly small number of visual objects or features (for a review, see Luck, In recent years there have been numerous attempts to de- fine and quantify what is meant by VSTM capacity. Until recently, the predominant view has held that capacity is lim- ited to a small, fixed number of visual objects (typically as- sumed to be 4) stored in discrete “slots” (Awh, Barton, & Vo- gel, 2007; Luck & Vogel, 1997; Vogel, Woodman, & Luck, 2001; Cowan & Rouder, 2009). Taking a different approach, Bays and colleagues (Bays, Catalao, & Husain, 2009; Bays & Husain, 2008) explored how the precision of features en- coded in visual working memory may change as a function of the number of features that are concurrently stored. Based on the finding that memory precision appears to decrease even when as few as 2 items are encoded, the authors proposed that VSTM capacity consists of a single, continuous resource that must be divided among items stored in working memory. While both the discrete slot and continuous resource mod- els are able to account for a number of empirical findings, both are ultimately unsatisfactory as complete theories of hu- man VSTM. First, in the case of both models the nature of the capacity limit is somewhat arbitrary: the hypothesized capac- ity limit does not emerge from a principled theoretical basis, The Ideal Observer Model of VSTM In this section we derive an Ideal Observer model of visual short-term memory. We show that from an information- theoretic perspective, our Ideal Observer model is optimally efficient in that it minimizes a particular measure of memory distortion subject to a fixed capacity limit. The resulting model makes several important contributions to the literature on VSTM. First, whereas previous models have postulated abstract or task-dependent definitions of vi- sual memory capacity, the Ideal Observer model provides a quantitative definition of capacity that is task-independent and easily interpreted. This enables results obtained in one ex- periment to generate predictions for performance in another. Second, since our model exhibits provably optimal perfor- mance subject to a fixed capacity, it can be used to obtain an assumption-free estimate of the minimum capacity of hu- man VSTM. Finally, we demonstrate that our Ideal Observer model subsumes two existing models of VSTM: a recent ver- sion of the discrete slot model (Cowan & Rouder, 2009), and the continuous resource model (Bays & Husain, 2008).

Charles E. Metz - One of the best experts on this subject based on the ideXlab platform.

  • The three-class Ideal Observer for univariate normal data: Decision variable and ROC surface properties.
    Journal of mathematical psychology, 2012
    Co-Authors: Darrin C. Edwards, Charles E. Metz
    Abstract:

    Although a fully general extension of ROC analysis to classification tasks with more than two classes has yet to be developed, the potential benefits to be gained from a practical performance evaluation methodology for classification tasks with three classes have motivated a number of research groups to propose methods based on constrained or simplified Observer or data models. Here we consider an Ideal Observer in a task with underlying data drawn from three univariate normal distributions. We investigate the behavior of the resulting Ideal Observer’s decision variables and ROC surface. In particular, we show that the pair of Ideal Observer decision variables is constrained to a parametric curve in two-dimensional likelihood ratio space, and that the decision boundary line segments used by the Ideal Observer can intersect this curve in at most six places. From this, we further show that the resulting ROC surface has at most four degrees of freedom at any point, and not the five that would be required, in general, for a surface in a six-dimensional space to be non-degenerate. In light of the difficulties we have previously pointed out in generalizing the well-known area under the ROC curve performance metric to tasks with three or more classes, the problem of developing a suitable and fully general performance metric for classification tasks with three or more classes remains unsolved.

  • Behavior of the decision variables of the three-class Ideal Observer for univariate trinormal data
    Medical Imaging 2010: Image Perception Observer Performance and Technology Assessment, 2010
    Co-Authors: Darrin C. Edwards, Charles E. Metz
    Abstract:

    We are attempting to extend receiver operating characteristic (ROC) analysis to tasks with more than two classes. This is difficult because of the rapid increase in complexity, in general, of both Observer behavior and evaluation of its performance, as the number of classes involved increases. Many researchers have proposed addressing this complexity by imposing simplifications on the model; for example, by using univariate data rather than the bivariate data employed by the general three-class Ideal Observer. We have investigated a univariate trinormal model for the underlying data of a three-class Ideal Observer. Although a reasonably complete description of the Ideal Observer's behavior in this case is attainable, this behavior is more complicated than might intuitively be expected.

  • Analysis of proposed three-class classification decision rules in terms of the Ideal Observer decision rule
    Journal of Mathematical Psychology, 2006
    Co-Authors: Darrin C. Edwards, Charles E. Metz
    Abstract:

    Abstract We analyze recently proposed decision rules for three-class classification from the point of view of Ideal Observer decision theory. We consider three-class decision rules proposed by Scurfield, by Chan et al., and by Mossman. Scurfield's decision rule is shown to be a special case of the three-class Ideal Observer decision rule in three different situations. Chan et al. start with an Ideal Observer model and specify its decision-consequence utility structure in a way that causes two of the decision lines used by the Ideal Observer to overlap and the third line to become undefined. Finally, we show that, for a particular and obvious choice of Ideal-Observer-related decision variables, the Mossman decision rule cannot be a special case of the Ideal Observer decision rule. Despite the considerable difficulties presented by the three-class classification task, the three-class Ideal Observer provides a useful framework for analyzing a variety of three-class decision strategies.

  • Evaluating Bayesian ANN estimates of Ideal Observer decision variables by comparison with identity functions
    Medical Imaging 2005: Image Perception Observer Performance and Technology Assessment, 2005
    Co-Authors: Darrin C. Edwards, Charles E. Metz
    Abstract:

    Bayesian artificial neural networks (BANNs) have proven useful in two-class classification tasks, and are claimed to provide good estimates of Ideal-Observer-related decision variables (the a posteriori class membership probabilities). We wish to apply the BANN methodology to three-class classification tasks for computer-aided diagnosis, but we currently lack a fully general extension of two-class receiver operating characteristic (ROC) analysis to objectively evaluate three-class BANN performance. It is well known that "the likelihood ratio of the likelihood ratio is the likelihood ratio." Based on this, we found that the decision variable which is the a posteriori class membership probability of an observational data vector is in fact equal to the a posteriori class membership probability of that decision variable. Under the assumption that a BANN can provide good estimates of these a posteriori probabilities, a second BANN trained on the output of such a BANN should perform very similarly to an identity function. We performed a two-class and a three-class simulation study to test this hypothesis. The mean squared error (deviation from an identity function) of a two-class BANN was found to be 2.5x10E-4. The mean squared error of the first component of the output of a three-class BANN was found to be 2.8x10-4, and that of its second component was found to be 3.8x10-4. Although we currently lack a fully general method to objectively evaluate performance in a three-class classification task, circumstantial evidence suggests that two- and three-class BANNs can provide good estimates of Ideal-Observer-related decision variables.© (2005) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

  • Review of several proposed three-class classification decision rules and their relation to the Ideal Observer decision rule
    Medical Imaging 2005: Image Perception Observer Performance and Technology Assessment, 2005
    Co-Authors: Darrin C. Edwards, Charles E. Metz
    Abstract:

    We analyzed a variety of recently proposed decision rules for three-class classification from the point of view of Ideal Observer decision theory. We considered three-class decision rules which have been proposed recently: one by Scurfield, one by Chan et al., and one by Mossman. Scurfield's decision rule can be shown to be a special case of the three-class Ideal Observer decision rule in two different situations: when the pair of decision variables is the pair of likelihood ratios used by the Ideal Observer, and when the pair of decision variables is the pair of logarithms of the likelihood ratios. Chan et al. start with an Ideal Observer model, where two of the decision lines used by the Ideal Observer overlap, and the third line becomes undefined. Finally, we showed that the Mossman decision rule (in which a single decision line separates one class from the other two, while a second line separates those two classes) cannot be a special case of the Ideal Observer decision rule. Despite the considerable difficulties presented by the three-class classification task compared with two-class classification, we found that the three-class Ideal Observer provides a useful framework for analyzing a wide variety of three-class decision strategies.

Darrin C. Edwards - One of the best experts on this subject based on the ideXlab platform.

  • The three-class Ideal Observer for univariate normal data: Decision variable and ROC surface properties.
    Journal of mathematical psychology, 2012
    Co-Authors: Darrin C. Edwards, Charles E. Metz
    Abstract:

    Although a fully general extension of ROC analysis to classification tasks with more than two classes has yet to be developed, the potential benefits to be gained from a practical performance evaluation methodology for classification tasks with three classes have motivated a number of research groups to propose methods based on constrained or simplified Observer or data models. Here we consider an Ideal Observer in a task with underlying data drawn from three univariate normal distributions. We investigate the behavior of the resulting Ideal Observer’s decision variables and ROC surface. In particular, we show that the pair of Ideal Observer decision variables is constrained to a parametric curve in two-dimensional likelihood ratio space, and that the decision boundary line segments used by the Ideal Observer can intersect this curve in at most six places. From this, we further show that the resulting ROC surface has at most four degrees of freedom at any point, and not the five that would be required, in general, for a surface in a six-dimensional space to be non-degenerate. In light of the difficulties we have previously pointed out in generalizing the well-known area under the ROC curve performance metric to tasks with three or more classes, the problem of developing a suitable and fully general performance metric for classification tasks with three or more classes remains unsolved.

  • Support of the decision variable densities of the three-class Ideal Observer for bivariate trinormal data
    Medical Imaging 2011: Image Perception Observer Performance and Technology Assessment, 2011
    Co-Authors: Darrin C. Edwards
    Abstract:

    Despite theoretical and practical difficulties, we are attempting to extend receiver operating characteristic (ROC) analysis to tasks with more than two classes. Previously we investigated a univariate trinormal model for the underlying data of a three-class Ideal Observer. Although analytically tractable, this is less realistic than a multivariate data model. We have developed expressions for the region of support of the decision variable probability density functions for bivariate trinormal underlying data, given certain constraints on the underlying data covariance matrices. We hope these results will aid in developing computational methods for evaluating Observer performance under such a model.

  • Behavior of the decision variables of the three-class Ideal Observer for univariate trinormal data
    Medical Imaging 2010: Image Perception Observer Performance and Technology Assessment, 2010
    Co-Authors: Darrin C. Edwards, Charles E. Metz
    Abstract:

    We are attempting to extend receiver operating characteristic (ROC) analysis to tasks with more than two classes. This is difficult because of the rapid increase in complexity, in general, of both Observer behavior and evaluation of its performance, as the number of classes involved increases. Many researchers have proposed addressing this complexity by imposing simplifications on the model; for example, by using univariate data rather than the bivariate data employed by the general three-class Ideal Observer. We have investigated a univariate trinormal model for the underlying data of a three-class Ideal Observer. Although a reasonably complete description of the Ideal Observer's behavior in this case is attainable, this behavior is more complicated than might intuitively be expected.

  • Analysis of proposed three-class classification decision rules in terms of the Ideal Observer decision rule
    Journal of Mathematical Psychology, 2006
    Co-Authors: Darrin C. Edwards, Charles E. Metz
    Abstract:

    Abstract We analyze recently proposed decision rules for three-class classification from the point of view of Ideal Observer decision theory. We consider three-class decision rules proposed by Scurfield, by Chan et al., and by Mossman. Scurfield's decision rule is shown to be a special case of the three-class Ideal Observer decision rule in three different situations. Chan et al. start with an Ideal Observer model and specify its decision-consequence utility structure in a way that causes two of the decision lines used by the Ideal Observer to overlap and the third line to become undefined. Finally, we show that, for a particular and obvious choice of Ideal-Observer-related decision variables, the Mossman decision rule cannot be a special case of the Ideal Observer decision rule. Despite the considerable difficulties presented by the three-class classification task, the three-class Ideal Observer provides a useful framework for analyzing a variety of three-class decision strategies.

  • Evaluating Bayesian ANN estimates of Ideal Observer decision variables by comparison with identity functions
    Medical Imaging 2005: Image Perception Observer Performance and Technology Assessment, 2005
    Co-Authors: Darrin C. Edwards, Charles E. Metz
    Abstract:

    Bayesian artificial neural networks (BANNs) have proven useful in two-class classification tasks, and are claimed to provide good estimates of Ideal-Observer-related decision variables (the a posteriori class membership probabilities). We wish to apply the BANN methodology to three-class classification tasks for computer-aided diagnosis, but we currently lack a fully general extension of two-class receiver operating characteristic (ROC) analysis to objectively evaluate three-class BANN performance. It is well known that "the likelihood ratio of the likelihood ratio is the likelihood ratio." Based on this, we found that the decision variable which is the a posteriori class membership probability of an observational data vector is in fact equal to the a posteriori class membership probability of that decision variable. Under the assumption that a BANN can provide good estimates of these a posteriori probabilities, a second BANN trained on the output of such a BANN should perform very similarly to an identity function. We performed a two-class and a three-class simulation study to test this hypothesis. The mean squared error (deviation from an identity function) of a two-class BANN was found to be 2.5x10E-4. The mean squared error of the first component of the output of a three-class BANN was found to be 2.8x10-4, and that of its second component was found to be 3.8x10-4. Although we currently lack a fully general method to objectively evaluate performance in a three-class classification task, circumstantial evidence suggests that two- and three-class BANNs can provide good estimates of Ideal-Observer-related decision variables.© (2005) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

Robert A Jacobs - One of the best experts on this subject based on the ideXlab platform.

  • an Ideal Observer analysis of visual working memory
    Psychological Review, 2012
    Co-Authors: Chris R Sims, Robert A Jacobs, David C Knill
    Abstract:

    Limits in visual working memory (VWM) strongly constrain human performance across many tasks. However, the nature of these limits is not well understood. In this article we develop an Ideal Observer analysis of human VWM by deriving the expected behavior of an optimally performing but limitedcapacity memory system. This analysis is framed around rate–distortion theory, a branch of information theory that provides optimal bounds on the accuracy of information transmission subject to a fixed information capacity. The result of the Ideal Observer analysis is a theoretical framework that provides a task-independent and quantitative definition of visual memory capacity and yields novel predictions regarding human performance. These predictions are subsequently evaluated and confirmed in 2 empirical studies. Further, the framework is general enough to allow the specification and testing of alternative models of visual memory (e.g., how capacity is distributed across multiple items). We demonstrate that a simple model developed on the basis of the Ideal Observer analysis—one that allows variability in the number of stored memory representations but does not assume the presence of a fixed item limit—provides an excellent account of the empirical data and further offers a principled reinterpretation of existing models of VWM.

  • an Ideal Observer model of visual short term memory predicts human capacity precision tradeoffs
    Cognitive Science, 2011
    Co-Authors: Chris R Sims, Robert A Jacobs, David C Knill
    Abstract:

    An Ideal Observer Model of Visual Short-Term Memory Predicts Human Capacity–Precision Tradeoffs Chris R. Sims Robert A. Jacobs David C. Knill (csims@cvs.rochester.edu) (robbie@bcs.rochester.edu) (knill@cvs.rochester.edu) Department of Brain and Cognitive Sciences & Center for Visual Science University of Rochester, Rochester, NY Abstract p(x) p(xs | x) We develop an Ideal Observer model of human visual short- term memory. Compared to previous models that have posited constraints on memory performance intended solely to ac- count for observed phenomena, in the present research we de- rive the expected behavior of an optimally performing, but limited-capacity memory system. We develop our model us- ing rate–distortion theory, a branch of information theory that provides optimal bounds on the accuracy of information trans- mission subject to a fixed capacity. The resulting model pro- vides a task-independent and theoretically motivated definition of visual memory capacity and yields novel predictions regard- ing human performance. These predictions are quantitatively evaluated in an empirical study. We also demonstrate that our Ideal Observer model encompasses two existing, but competing accounts of VSTM as special cases. Keywords: Ideal Observer analysis, VSTM, information the- ory, rate–distortion theory Information source Sensory signal xs VSTM Memory encoding / storage p(x ˆ | xm ) Recovered signal xm Optimal encoder & decoder Memory decoding / retrieval Figure 1: A schematic diagram of the Ideal Observer model of VSTM. Introduction but rather only serves to account for observed empirical phe- nomena. Second, the definition of capacity appears largely task-dependent, and it is therefore difficult to form predictions for human performance in different tasks or conditions. In the case of the discrete slot model, there is no strong theoretical justification for determining what visual features or objects can or cannot occupy a single slot, and in the continuous re- source model, the nature of the resource that is being divided is only abstractly specified. Visual short-term memory (VSTM)—defined as the ability to store task-relevant visual information in a rapidly acces- sible and easily manipulated form—is a central component of nearly all human activities. Given its importance, it is per- haps surprising that the capacity of this system is severely lim- ited. Numerous investigations of VSTM performance have revealed that we can only accurately store a surprisingly small number of visual objects or features (for a review, see Luck, In recent years there have been numerous attempts to de- fine and quantify what is meant by VSTM capacity. Until recently, the predominant view has held that capacity is lim- ited to a small, fixed number of visual objects (typically as- sumed to be 4) stored in discrete “slots” (Awh, Barton, & Vo- gel, 2007; Luck & Vogel, 1997; Vogel, Woodman, & Luck, 2001; Cowan & Rouder, 2009). Taking a different approach, Bays and colleagues (Bays, Catalao, & Husain, 2009; Bays & Husain, 2008) explored how the precision of features en- coded in visual working memory may change as a function of the number of features that are concurrently stored. Based on the finding that memory precision appears to decrease even when as few as 2 items are encoded, the authors proposed that VSTM capacity consists of a single, continuous resource that must be divided among items stored in working memory. While both the discrete slot and continuous resource mod- els are able to account for a number of empirical findings, both are ultimately unsatisfactory as complete theories of hu- man VSTM. First, in the case of both models the nature of the capacity limit is somewhat arbitrary: the hypothesized capac- ity limit does not emerge from a principled theoretical basis, The Ideal Observer Model of VSTM In this section we derive an Ideal Observer model of visual short-term memory. We show that from an information- theoretic perspective, our Ideal Observer model is optimally efficient in that it minimizes a particular measure of memory distortion subject to a fixed capacity limit. The resulting model makes several important contributions to the literature on VSTM. First, whereas previous models have postulated abstract or task-dependent definitions of vi- sual memory capacity, the Ideal Observer model provides a quantitative definition of capacity that is task-independent and easily interpreted. This enables results obtained in one ex- periment to generate predictions for performance in another. Second, since our model exhibits provably optimal perfor- mance subject to a fixed capacity, it can be used to obtain an assumption-free estimate of the minimum capacity of hu- man VSTM. Finally, we demonstrate that our Ideal Observer model subsumes two existing models of VSTM: a recent ver- sion of the discrete slot model (Cowan & Rouder, 2009), and the continuous resource model (Bays & Husain, 2008).

  • CogSci - An Ideal Observer Model of Visual Short-Term Memory Predicts Human Capacity–Precision Tradeoffs
    Cognitive Science, 2011
    Co-Authors: Chris R Sims, Robert A Jacobs, David C Knill
    Abstract:

    An Ideal Observer Model of Visual Short-Term Memory Predicts Human Capacity–Precision Tradeoffs Chris R. Sims Robert A. Jacobs David C. Knill (csims@cvs.rochester.edu) (robbie@bcs.rochester.edu) (knill@cvs.rochester.edu) Department of Brain and Cognitive Sciences & Center for Visual Science University of Rochester, Rochester, NY Abstract p(x) p(xs | x) We develop an Ideal Observer model of human visual short- term memory. Compared to previous models that have posited constraints on memory performance intended solely to ac- count for observed phenomena, in the present research we de- rive the expected behavior of an optimally performing, but limited-capacity memory system. We develop our model us- ing rate–distortion theory, a branch of information theory that provides optimal bounds on the accuracy of information trans- mission subject to a fixed capacity. The resulting model pro- vides a task-independent and theoretically motivated definition of visual memory capacity and yields novel predictions regard- ing human performance. These predictions are quantitatively evaluated in an empirical study. We also demonstrate that our Ideal Observer model encompasses two existing, but competing accounts of VSTM as special cases. Keywords: Ideal Observer analysis, VSTM, information the- ory, rate–distortion theory Information source Sensory signal xs VSTM Memory encoding / storage p(x ˆ | xm ) Recovered signal xm Optimal encoder & decoder Memory decoding / retrieval Figure 1: A schematic diagram of the Ideal Observer model of VSTM. Introduction but rather only serves to account for observed empirical phe- nomena. Second, the definition of capacity appears largely task-dependent, and it is therefore difficult to form predictions for human performance in different tasks or conditions. In the case of the discrete slot model, there is no strong theoretical justification for determining what visual features or objects can or cannot occupy a single slot, and in the continuous re- source model, the nature of the resource that is being divided is only abstractly specified. Visual short-term memory (VSTM)—defined as the ability to store task-relevant visual information in a rapidly acces- sible and easily manipulated form—is a central component of nearly all human activities. Given its importance, it is per- haps surprising that the capacity of this system is severely lim- ited. Numerous investigations of VSTM performance have revealed that we can only accurately store a surprisingly small number of visual objects or features (for a review, see Luck, In recent years there have been numerous attempts to de- fine and quantify what is meant by VSTM capacity. Until recently, the predominant view has held that capacity is lim- ited to a small, fixed number of visual objects (typically as- sumed to be 4) stored in discrete “slots” (Awh, Barton, & Vo- gel, 2007; Luck & Vogel, 1997; Vogel, Woodman, & Luck, 2001; Cowan & Rouder, 2009). Taking a different approach, Bays and colleagues (Bays, Catalao, & Husain, 2009; Bays & Husain, 2008) explored how the precision of features en- coded in visual working memory may change as a function of the number of features that are concurrently stored. Based on the finding that memory precision appears to decrease even when as few as 2 items are encoded, the authors proposed that VSTM capacity consists of a single, continuous resource that must be divided among items stored in working memory. While both the discrete slot and continuous resource mod- els are able to account for a number of empirical findings, both are ultimately unsatisfactory as complete theories of hu- man VSTM. First, in the case of both models the nature of the capacity limit is somewhat arbitrary: the hypothesized capac- ity limit does not emerge from a principled theoretical basis, The Ideal Observer Model of VSTM In this section we derive an Ideal Observer model of visual short-term memory. We show that from an information- theoretic perspective, our Ideal Observer model is optimally efficient in that it minimizes a particular measure of memory distortion subject to a fixed capacity limit. The resulting model makes several important contributions to the literature on VSTM. First, whereas previous models have postulated abstract or task-dependent definitions of vi- sual memory capacity, the Ideal Observer model provides a quantitative definition of capacity that is task-independent and easily interpreted. This enables results obtained in one ex- periment to generate predictions for performance in another. Second, since our model exhibits provably optimal perfor- mance subject to a fixed capacity, it can be used to obtain an assumption-free estimate of the minimum capacity of hu- man VSTM. Finally, we demonstrate that our Ideal Observer model subsumes two existing models of VSTM: a recent ver- sion of the discrete slot model (Cowan & Rouder, 2009), and the continuous resource model (Bays & Husain, 2008).