The Experts below are selected from a list of 57534 Experts worldwide ranked by ideXlab platform
Donald Fucci - One of the best experts on this subject based on the ideXlab platform.
-
Measurement of speech disfluency through Magnitude Estimation and interval scaling.
Perceptual and motor skills, 2006Co-Authors: Douglas Mccoll, Donald FucciAbstract:The purpose was to assess whether equal-appearing interval or Magnitude-Estimation scaling resulted in a data set with a closer correlation to the physical stimuli, made up of speech samples with varying amounts of disfluency. 20 young adults completed two tasks. In Task 1, subjects used a 7-point equal-appearing interval scale to rate the disfluency of 10 speech samples having varying within sentence pause, presented randomly at 65 dB SPL. In Task 2, subjects used Magnitude-Estimation scaling to rate these stimuli, presented in a randomized order. Analysis showed significantly high correlations for both scaling techniques and the speech stimuli (Spearman rho=.90 and 1.00, respectively). It appears that subjects can use either scaling technique to rate accurately varied speech disfluency.
-
Magnitude-Estimation Scaling of Computerized (Digitized) Speech under Different Listening Conditions
Perceptual and Motor Skills, 1999Co-Authors: Ramesh Bettagere, Donald FucciAbstract:The purpose of this study was to evaluate the quality of tape-recorded speech sentences and speech sentences digitized at low, moderate, and high sampling rates by young adults under different listening conditions (quiet vs noise) using Magnitude-Estimation scaling. A single group of 24 young adults participated as subjects. The tape-recorded speech sentences and digitized speech sentences were presented to each subject in quiet and in the presence of background noise at a signal-to-noise ratio of 0 dB. The subjects were instructed to use Magnitude-Estimation scaling to evaluate these sentences by assigning a number that corresponded to the speech quality of each sentence. An analysis of variance with repeated measures was performed to assess the effects of mode of speech and listening condition on the Magnitude-Estimation responses. The analysis showed that the main effects for mode of speech and listening condition were statistically significant. The interactions of mode of speech by listening condition were also statistically significant. Pairwise comparisons showed that the Magnitude-Estimation responses were higher in the quiet condition than in the corresponding noise condition for each level of mode of speech. Based on the results, the implications of the present study and several avenues for later research are discussed.
-
Magnitude Estimation SCALING OF THE LOUDNESS OF A WIDE RANGE OF AUDITORY STIMULI
Perceptual and Motor Skills, 1997Co-Authors: Donald Fucci, Linda Petrosino, Doug Mccoll, Denise Wyatt, Corry WilcoxAbstract:The study of the perception of loudness lends itself well to the psychophysical scaling technique of Magnitude Estimation. This study was designed to extend the range of auditory stimuli used to study the Magnitude Estimation scaling of loudness. The five stimuli chosen were a 1000-Hz pure tone, narrow band noise (700-1300 Hz band width), broad band noise (100-10,000 Hz band width), rock music, and babble speech, i.e., speech in which meaning is not discernible because several individuals are talking at once. Subjects were 30 normal young women (M = 19 yr.). During the auditory Magnitude-Estimation task for each of the five stimuli, a subject was instructed to assign numbers to stimulus presented in a randomly ordered series of nine sensation levels (10, 20, 30, 40, 50, 60, 70, 80, and 90 dB SL). Multivariate analysis of variance for repeated measures indicated there were no significant differences in the numerical responses of the subjects for the five stimuli. A possible explanation for these results is the presence of an underlying stabilizing factor (internal scaling mechanism) that allows adults to scale loudness consistently irrespective of the type of auditory stimulus.
-
Language familiarity in Magnitude-Estimation scaling of loudness by young adults.
Perceptual and motor skills, 1995Co-Authors: Donald Fucci, Ramesh Bettagere, Maria Diana Gonzales, Mary E. Reynolds, Linda PetrosinoAbstract:The purpose was to examine the effect of language familiarity on Magnitude-Estimation scaling of loudness by young adults. Two groups of subjects participated in this study. Group 1 of 20 subjects (M age = 23.95 yr.) were familiar with English and not familiar with Hindi. Group 2 of 20 subjects (M age = 24.30 yr.) were familiar with English as well as Hindi. Two separate Magnitude-Estimation scaling tasks were performed. On the first scaling task, an English sentence was used as the stimulus, and on the second scaling task, a Hindi sentence was used as the stimulus. Statistical analysis showed that there was no significant difference between the two groups in loudness judgments of the English and Hindi sentences. Subjects scaled the loudness of an unfamiliar language in the same manner as they scaled the loudness of a familiar language. The findings suggest that Magnitude-Estimation scaling is an effective measure of loudness whether the language being listened to is familiar to the listener.
-
Hearing threshold shift during auditory Magnitude Estimation.
Perceptual and motor skills, 1992Co-Authors: Daniel Harris, Donald Fucci, Linda Petrosino, Marianne BelchAbstract:The purpose of the present experiment was to determine the extent of hearing threshold shift that may occur during an auditory Magnitude-Estimation task involving stimulus intensities as great as 80 dB sensation level. Also, possible influences of hearing threshold shift on numerical Magnitude-Estimation responses and Magnitude-function slopes were investigated. Results indicated that hearing threshold shift was insignificant (1-2 dB). Consistent small increases in numerical Magnitude responses were observed on a Magnitude-Estimation task where hearing thresholds were retested between stimulus presentations versus a Magnitude-Estimation task where hearing thresholds were not retested. The stability of auditory Magnitude functions across different conditions in the current investigation was in agreement with vibrotactile Magnitude-scaling behavior observed by Fucci, et al. in 1989 and 1991. The over-all results supported the concept of an absolute, internal sensory-scaling mechanism being operable during Magnitude Estimation of auditory stimuli as discussed by Zwislocki and Goodman in 1980.
Eddy Maddalena - One of the best experts on this subject based on the ideXlab platform.
-
the benefits of Magnitude Estimation relevance assessments for information retrieval evaluation
International ACM SIGIR Conference on Research and Development in Information Retrieval, 2015Co-Authors: Andrew Turpin, Falk Scholer, Stefano Mizzaro, Eddy MaddalenaAbstract:Magnitude Estimation is a psychophysical scaling technique for the measurement of sensation, where observers assign numbers to stimuli in response to their perceived intensity. We investigate the use of Magnitude Estimation for judging the relevance of documents in the context of information retrieval evaluation, carrying out a large-scale user study across 18 TREC topics and collecting more than 50,000 Magnitude Estimation judgments. Our analysis shows that on average Magnitude Estimation judgments are rank-aligned with ordinal judgments made by expert relevance assessors. An advantage of Magnitude Estimation is that users can chose their own scale for judgments, allowing deeper investigations of user perceptions than when categorical scales are used. We explore the application of Magnitude Estimation for IR evaluation, calibrating two gain-based effectiveness metrics, nDCG and ERR, directly from user-reported perceptions of relevance. A comparison of TREC system effectiveness rankings based on binary, ordinal, and Magnitude Estimation relevance shows substantial variation; in particular, the top systems ranked using Magnitude Estimation and ordinal judgments differ substantially. Analysis of the Magnitude Estimation scores shows that this effect is due in part to varying perceptions of relevance, in terms of how impactful relative differences in document relevance are perceived to be. We further use Magnitude Estimation to investigate gain profiles, comparing the currently assumed linear and exponential approaches with actual user-reported relevance perceptions. This indicates that the currently used exponential gain profiles in nDCG and ERR are mismatched with an average user, but perhaps more importantly that individual perceptions are highly variable. These results have direct implications for IR evaluation, suggesting that current assumptions about a single view of relevance being sufficient to represent a population of users are unlikely to hold. Finally, we demonstrate that Magnitude Estimation judgments can be reliably collected using crowdsourcing, and are competitive in terms of assessor cost.
-
judging relevance using Magnitude Estimation
European Conference on Information Retrieval, 2015Co-Authors: Eddy Maddalena, Stefano Mizzaro, Falk Scholer, Andrew TurpinAbstract:Magnitude Estimation is a psychophysical scaling technique whereby numbers are assigned to stimuli to reflect the ratios of their perceived intensity. We report on a crowdsourcing experiment aimed at understanding if Magnitude Estimation can be used to gather reliable relevance judgements for documents, as is commonly required for test collection-based evaluation of information retrieval systems. Results on a small dataset show that: (i) Magnitude Estimation can produce relevance rankings that are consistent with more classical ordinal judgements; (ii) both an upper-bounded and an unbounded scale can be used effectively, though with some differences; (iii) the presentation order of the documents being judged has a limited effect, if any; and (iv) only a small number repeat judgements are required to obtain reliable Magnitude Estimation scores.
-
Judging relevance using Magnitude Estimation
2015Co-Authors: Eddy Maddalena, Stefano Mizzaro, Falk Scholer, Andrew TurpinAbstract:© Springer International Publishing Switzerland 2015. Magnitude Estimation is a psychophysical scaling technique whereby numbers are assigned to stimuli to reflect the ratios of their perceived intensity. We report on a crowdsourcing experiment aimed at understanding if Magnitude Estimation can be used to gather reliable relevance judgements for documents, as is commonly required for test collection-based evaluation of information retrieval systems. Results on a small dataset show that: (i) Magnitude Estimation can produce relevance rankings that are consistent with more classical ordinal judgements; (ii) both an upper-bounded and an unbounded scale can be used effectively, though with some differences; (iii) the presentation order of the documents being judged has a limited effect, if any; and (iv) only a small number repeat judgements are required to obtain reliable Magnitude Estimation scores.
-
ECIR - Judging relevance using Magnitude Estimation
Lecture Notes in Computer Science, 2015Co-Authors: Eddy Maddalena, Stefano Mizzaro, Falk Scholer, Andrew TurpinAbstract:Magnitude Estimation is a psychophysical scaling technique whereby numbers are assigned to stimuli to reflect the ratios of their perceived intensity. We report on a crowdsourcing experiment aimed at understanding if Magnitude Estimation can be used to gather reliable relevance judgements for documents, as is commonly required for test collection-based evaluation of information retrieval systems. Results on a small dataset show that: (i) Magnitude Estimation can produce relevance rankings that are consistent with more classical ordinal judgements; (ii) both an upper-bounded and an unbounded scale can be used effectively, though with some differences; (iii) the presentation order of the documents being judged has a limited effect, if any; and (iv) only a small number repeat judgements are required to obtain reliable Magnitude Estimation scores.
-
SIGIR - The Benefits of Magnitude Estimation Relevance Assessments for Information Retrieval Evaluation
Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR '15, 2015Co-Authors: Andrew Turpin, Falk Scholer, Stefano Mizzaro, Eddy MaddalenaAbstract:Magnitude Estimation is a psychophysical scaling technique for the measurement of sensation, where observers assign numbers to stimuli in response to their perceived intensity. We investigate the use of Magnitude Estimation for judging the relevance of documents in the context of information retrieval evaluation, carrying out a large-scale user study across 18 TREC topics and collecting more than 50,000 Magnitude Estimation judgments. Our analysis shows that on average Magnitude Estimation judgments are rank-aligned with ordinal judgments made by expert relevance assessors. An advantage of Magnitude Estimation is that users can chose their own scale for judgments, allowing deeper investigations of user perceptions than when categorical scales are used. We explore the application of Magnitude Estimation for IR evaluation, calibrating two gain-based effectiveness metrics, nDCG and ERR, directly from user-reported perceptions of relevance. A comparison of TREC system effectiveness rankings based on binary, ordinal, and Magnitude Estimation relevance shows substantial variation; in particular, the top systems ranked using Magnitude Estimation and ordinal judgments differ substantially. Analysis of the Magnitude Estimation scores shows that this effect is due in part to varying perceptions of relevance, in terms of how impactful relative differences in document relevance are perceived to be. We further use Magnitude Estimation to investigate gain profiles, comparing the currently assumed linear and exponential approaches with actual user-reported relevance perceptions. This indicates that the currently used exponential gain profiles in nDCG and ERR are mismatched with an average user, but perhaps more importantly that individual perceptions are highly variable. These results have direct implications for IR evaluation, suggesting that current assumptions about a single view of relevance being sufficient to represent a population of users are unlikely to hold. Finally, we demonstrate that Magnitude Estimation judgments can be reliably collected using crowdsourcing, and are competitive in terms of assessor cost.
Andrew Turpin - One of the best experts on this subject based on the ideXlab platform.
-
the benefits of Magnitude Estimation relevance assessments for information retrieval evaluation
International ACM SIGIR Conference on Research and Development in Information Retrieval, 2015Co-Authors: Andrew Turpin, Falk Scholer, Stefano Mizzaro, Eddy MaddalenaAbstract:Magnitude Estimation is a psychophysical scaling technique for the measurement of sensation, where observers assign numbers to stimuli in response to their perceived intensity. We investigate the use of Magnitude Estimation for judging the relevance of documents in the context of information retrieval evaluation, carrying out a large-scale user study across 18 TREC topics and collecting more than 50,000 Magnitude Estimation judgments. Our analysis shows that on average Magnitude Estimation judgments are rank-aligned with ordinal judgments made by expert relevance assessors. An advantage of Magnitude Estimation is that users can chose their own scale for judgments, allowing deeper investigations of user perceptions than when categorical scales are used. We explore the application of Magnitude Estimation for IR evaluation, calibrating two gain-based effectiveness metrics, nDCG and ERR, directly from user-reported perceptions of relevance. A comparison of TREC system effectiveness rankings based on binary, ordinal, and Magnitude Estimation relevance shows substantial variation; in particular, the top systems ranked using Magnitude Estimation and ordinal judgments differ substantially. Analysis of the Magnitude Estimation scores shows that this effect is due in part to varying perceptions of relevance, in terms of how impactful relative differences in document relevance are perceived to be. We further use Magnitude Estimation to investigate gain profiles, comparing the currently assumed linear and exponential approaches with actual user-reported relevance perceptions. This indicates that the currently used exponential gain profiles in nDCG and ERR are mismatched with an average user, but perhaps more importantly that individual perceptions are highly variable. These results have direct implications for IR evaluation, suggesting that current assumptions about a single view of relevance being sufficient to represent a population of users are unlikely to hold. Finally, we demonstrate that Magnitude Estimation judgments can be reliably collected using crowdsourcing, and are competitive in terms of assessor cost.
-
judging relevance using Magnitude Estimation
European Conference on Information Retrieval, 2015Co-Authors: Eddy Maddalena, Stefano Mizzaro, Falk Scholer, Andrew TurpinAbstract:Magnitude Estimation is a psychophysical scaling technique whereby numbers are assigned to stimuli to reflect the ratios of their perceived intensity. We report on a crowdsourcing experiment aimed at understanding if Magnitude Estimation can be used to gather reliable relevance judgements for documents, as is commonly required for test collection-based evaluation of information retrieval systems. Results on a small dataset show that: (i) Magnitude Estimation can produce relevance rankings that are consistent with more classical ordinal judgements; (ii) both an upper-bounded and an unbounded scale can be used effectively, though with some differences; (iii) the presentation order of the documents being judged has a limited effect, if any; and (iv) only a small number repeat judgements are required to obtain reliable Magnitude Estimation scores.
-
Judging relevance using Magnitude Estimation
2015Co-Authors: Eddy Maddalena, Stefano Mizzaro, Falk Scholer, Andrew TurpinAbstract:© Springer International Publishing Switzerland 2015. Magnitude Estimation is a psychophysical scaling technique whereby numbers are assigned to stimuli to reflect the ratios of their perceived intensity. We report on a crowdsourcing experiment aimed at understanding if Magnitude Estimation can be used to gather reliable relevance judgements for documents, as is commonly required for test collection-based evaluation of information retrieval systems. Results on a small dataset show that: (i) Magnitude Estimation can produce relevance rankings that are consistent with more classical ordinal judgements; (ii) both an upper-bounded and an unbounded scale can be used effectively, though with some differences; (iii) the presentation order of the documents being judged has a limited effect, if any; and (iv) only a small number repeat judgements are required to obtain reliable Magnitude Estimation scores.
-
ECIR - Judging relevance using Magnitude Estimation
Lecture Notes in Computer Science, 2015Co-Authors: Eddy Maddalena, Stefano Mizzaro, Falk Scholer, Andrew TurpinAbstract:Magnitude Estimation is a psychophysical scaling technique whereby numbers are assigned to stimuli to reflect the ratios of their perceived intensity. We report on a crowdsourcing experiment aimed at understanding if Magnitude Estimation can be used to gather reliable relevance judgements for documents, as is commonly required for test collection-based evaluation of information retrieval systems. Results on a small dataset show that: (i) Magnitude Estimation can produce relevance rankings that are consistent with more classical ordinal judgements; (ii) both an upper-bounded and an unbounded scale can be used effectively, though with some differences; (iii) the presentation order of the documents being judged has a limited effect, if any; and (iv) only a small number repeat judgements are required to obtain reliable Magnitude Estimation scores.
-
SIGIR - The Benefits of Magnitude Estimation Relevance Assessments for Information Retrieval Evaluation
Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR '15, 2015Co-Authors: Andrew Turpin, Falk Scholer, Stefano Mizzaro, Eddy MaddalenaAbstract:Magnitude Estimation is a psychophysical scaling technique for the measurement of sensation, where observers assign numbers to stimuli in response to their perceived intensity. We investigate the use of Magnitude Estimation for judging the relevance of documents in the context of information retrieval evaluation, carrying out a large-scale user study across 18 TREC topics and collecting more than 50,000 Magnitude Estimation judgments. Our analysis shows that on average Magnitude Estimation judgments are rank-aligned with ordinal judgments made by expert relevance assessors. An advantage of Magnitude Estimation is that users can chose their own scale for judgments, allowing deeper investigations of user perceptions than when categorical scales are used. We explore the application of Magnitude Estimation for IR evaluation, calibrating two gain-based effectiveness metrics, nDCG and ERR, directly from user-reported perceptions of relevance. A comparison of TREC system effectiveness rankings based on binary, ordinal, and Magnitude Estimation relevance shows substantial variation; in particular, the top systems ranked using Magnitude Estimation and ordinal judgments differ substantially. Analysis of the Magnitude Estimation scores shows that this effect is due in part to varying perceptions of relevance, in terms of how impactful relative differences in document relevance are perceived to be. We further use Magnitude Estimation to investigate gain profiles, comparing the currently assumed linear and exponential approaches with actual user-reported relevance perceptions. This indicates that the currently used exponential gain profiles in nDCG and ERR are mismatched with an average user, but perhaps more importantly that individual perceptions are highly variable. These results have direct implications for IR evaluation, suggesting that current assumptions about a single view of relevance being sufficient to represent a population of users are unlikely to hold. Finally, we demonstrate that Magnitude Estimation judgments can be reliably collected using crowdsourcing, and are competitive in terms of assessor cost.
Falk Scholer - One of the best experts on this subject based on the ideXlab platform.
-
the benefits of Magnitude Estimation relevance assessments for information retrieval evaluation
International ACM SIGIR Conference on Research and Development in Information Retrieval, 2015Co-Authors: Andrew Turpin, Falk Scholer, Stefano Mizzaro, Eddy MaddalenaAbstract:Magnitude Estimation is a psychophysical scaling technique for the measurement of sensation, where observers assign numbers to stimuli in response to their perceived intensity. We investigate the use of Magnitude Estimation for judging the relevance of documents in the context of information retrieval evaluation, carrying out a large-scale user study across 18 TREC topics and collecting more than 50,000 Magnitude Estimation judgments. Our analysis shows that on average Magnitude Estimation judgments are rank-aligned with ordinal judgments made by expert relevance assessors. An advantage of Magnitude Estimation is that users can chose their own scale for judgments, allowing deeper investigations of user perceptions than when categorical scales are used. We explore the application of Magnitude Estimation for IR evaluation, calibrating two gain-based effectiveness metrics, nDCG and ERR, directly from user-reported perceptions of relevance. A comparison of TREC system effectiveness rankings based on binary, ordinal, and Magnitude Estimation relevance shows substantial variation; in particular, the top systems ranked using Magnitude Estimation and ordinal judgments differ substantially. Analysis of the Magnitude Estimation scores shows that this effect is due in part to varying perceptions of relevance, in terms of how impactful relative differences in document relevance are perceived to be. We further use Magnitude Estimation to investigate gain profiles, comparing the currently assumed linear and exponential approaches with actual user-reported relevance perceptions. This indicates that the currently used exponential gain profiles in nDCG and ERR are mismatched with an average user, but perhaps more importantly that individual perceptions are highly variable. These results have direct implications for IR evaluation, suggesting that current assumptions about a single view of relevance being sufficient to represent a population of users are unlikely to hold. Finally, we demonstrate that Magnitude Estimation judgments can be reliably collected using crowdsourcing, and are competitive in terms of assessor cost.
-
judging relevance using Magnitude Estimation
European Conference on Information Retrieval, 2015Co-Authors: Eddy Maddalena, Stefano Mizzaro, Falk Scholer, Andrew TurpinAbstract:Magnitude Estimation is a psychophysical scaling technique whereby numbers are assigned to stimuli to reflect the ratios of their perceived intensity. We report on a crowdsourcing experiment aimed at understanding if Magnitude Estimation can be used to gather reliable relevance judgements for documents, as is commonly required for test collection-based evaluation of information retrieval systems. Results on a small dataset show that: (i) Magnitude Estimation can produce relevance rankings that are consistent with more classical ordinal judgements; (ii) both an upper-bounded and an unbounded scale can be used effectively, though with some differences; (iii) the presentation order of the documents being judged has a limited effect, if any; and (iv) only a small number repeat judgements are required to obtain reliable Magnitude Estimation scores.
-
Judging relevance using Magnitude Estimation
2015Co-Authors: Eddy Maddalena, Stefano Mizzaro, Falk Scholer, Andrew TurpinAbstract:© Springer International Publishing Switzerland 2015. Magnitude Estimation is a psychophysical scaling technique whereby numbers are assigned to stimuli to reflect the ratios of their perceived intensity. We report on a crowdsourcing experiment aimed at understanding if Magnitude Estimation can be used to gather reliable relevance judgements for documents, as is commonly required for test collection-based evaluation of information retrieval systems. Results on a small dataset show that: (i) Magnitude Estimation can produce relevance rankings that are consistent with more classical ordinal judgements; (ii) both an upper-bounded and an unbounded scale can be used effectively, though with some differences; (iii) the presentation order of the documents being judged has a limited effect, if any; and (iv) only a small number repeat judgements are required to obtain reliable Magnitude Estimation scores.
-
ECIR - Judging relevance using Magnitude Estimation
Lecture Notes in Computer Science, 2015Co-Authors: Eddy Maddalena, Stefano Mizzaro, Falk Scholer, Andrew TurpinAbstract:Magnitude Estimation is a psychophysical scaling technique whereby numbers are assigned to stimuli to reflect the ratios of their perceived intensity. We report on a crowdsourcing experiment aimed at understanding if Magnitude Estimation can be used to gather reliable relevance judgements for documents, as is commonly required for test collection-based evaluation of information retrieval systems. Results on a small dataset show that: (i) Magnitude Estimation can produce relevance rankings that are consistent with more classical ordinal judgements; (ii) both an upper-bounded and an unbounded scale can be used effectively, though with some differences; (iii) the presentation order of the documents being judged has a limited effect, if any; and (iv) only a small number repeat judgements are required to obtain reliable Magnitude Estimation scores.
-
SIGIR - The Benefits of Magnitude Estimation Relevance Assessments for Information Retrieval Evaluation
Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR '15, 2015Co-Authors: Andrew Turpin, Falk Scholer, Stefano Mizzaro, Eddy MaddalenaAbstract:Magnitude Estimation is a psychophysical scaling technique for the measurement of sensation, where observers assign numbers to stimuli in response to their perceived intensity. We investigate the use of Magnitude Estimation for judging the relevance of documents in the context of information retrieval evaluation, carrying out a large-scale user study across 18 TREC topics and collecting more than 50,000 Magnitude Estimation judgments. Our analysis shows that on average Magnitude Estimation judgments are rank-aligned with ordinal judgments made by expert relevance assessors. An advantage of Magnitude Estimation is that users can chose their own scale for judgments, allowing deeper investigations of user perceptions than when categorical scales are used. We explore the application of Magnitude Estimation for IR evaluation, calibrating two gain-based effectiveness metrics, nDCG and ERR, directly from user-reported perceptions of relevance. A comparison of TREC system effectiveness rankings based on binary, ordinal, and Magnitude Estimation relevance shows substantial variation; in particular, the top systems ranked using Magnitude Estimation and ordinal judgments differ substantially. Analysis of the Magnitude Estimation scores shows that this effect is due in part to varying perceptions of relevance, in terms of how impactful relative differences in document relevance are perceived to be. We further use Magnitude Estimation to investigate gain profiles, comparing the currently assumed linear and exponential approaches with actual user-reported relevance perceptions. This indicates that the currently used exponential gain profiles in nDCG and ERR are mismatched with an average user, but perhaps more importantly that individual perceptions are highly variable. These results have direct implications for IR evaluation, suggesting that current assumptions about a single view of relevance being sufficient to represent a population of users are unlikely to hold. Finally, we demonstrate that Magnitude Estimation judgments can be reliably collected using crowdsourcing, and are competitive in terms of assessor cost.
Marc J Buehner - One of the best experts on this subject based on the ideXlab platform.
-
Magnitude Estimation reveals temporal binding at super second intervals
Journal of Experimental Psychology: Human Perception and Performance, 2009Co-Authors: Gruffydd Rhys Humphreys, Marc J BuehnerAbstract:Several recent studies (e.g., Haggard, Aschersleben, Gehrke, & Prinz, 2002; Haggard & Clark, 2003; Haggard, Clark, & Kalogeras, 2002) have demonstrated a "Temporal Binding" effect in which the interval between an intentional action and its consequent outcome is subjectively shorter compared to equivalent intervals that do not involve intentional action. The bulk of the literature has relied on the "Libet Clock" (Libet, Gleason, Wright, & Pearl, 1983; but see also Engbert & Wohlschlager, 2007; Engbert, Wohlschlager, Thomas, & Haggard, 2007; Engbert, Wohlschlager, & Haggard, 2008). Here we demonstrate that Temporal Binding is a robust finding that can also be reliably achieved with a Magnitude Estimation procedure, and that occurs over intervals far greater than those previously explored. Implications for the underlying mechanisms are discussed.
