The Experts below are selected from a list of 6954 Experts worldwide ranked by ideXlab platform
John C. Callender - One of the best experts on this subject based on the ideXlab platform.
-
THE EFFECTS OF UNREPRESENTED STUDIES ON THE ROBUSTNESS OF Validity Generalization RESULTS
Personnel Psychology, 2006Co-Authors: Steven D. Ashworth, H. G. Osburn, John C. Callender, Kristin A. BoyleAbstract:Researchers conducting meta-analyses such as Validity Generalization can never be certain that their review contains all studies relevant to the research domain. Indeed, several authors in the past have noted ways in which research reviews may be systematically biased. A few techniques have emerged for addressing the issue of “missing studies” including the use of Rosenthal's (1979) file-drawer equation. Noting that Rosenthal's technique is inappropriate when applied to Validity Generalization findings, this paper develops a new method for assessing the vulnerability of Validity Generalization results to unrepresented or missing studies. The results of this new procedure are compared to the results of file-drawer analyses for 103 findings from Validity Generalization studies. We illustrate that this procedure more appropriately estimates the robustness of Validity Generalization results.
-
A note on the sampling variance of the mean uncorrected correlation in meta-analysis and Validity Generalization.
Journal of Applied Psychology, 1992Co-Authors: H. G. Osburn, John C. CallenderAbstract:This article compares the accuracy of several formulas for the standard error of the mean uncorrected correlation in meta-analytic and Validity Generalization studies. The effect of computing the mean correlation by weighting the correlation in each study by its sample size is also studied. On the basis of formal analysis and simulation studies, it is concluded that the common formula for the sampling variance of the mean correlation, V r =V/ r /K where K is the number of studies in the meta-analysis, gives reasonably accurate results. This formula gives accurate results even when sample sizes and ps are unequal and regardless of whether or not the statistical artifacts vary from study to study. It is also shown that using sample-size weighting may result in underestimation of the standard error of the mean uncorrected correlation when there are outlier sample sizes
Frank L. Schmidt - One of the best experts on this subject based on the ideXlab platform.
-
A Failed Challenge to Validity Generalization: Addressing a Fundamental Misunderstanding of the Nature of VG
Industrial and Organizational Psychology, 2017Co-Authors: Frank L. Schmidt, Chockalingam Viswesvaran, Deniz S. OnesAbstract:The lengthy and complex focal article by Tett, Hundley, and Christiansen (2017) is based on a fundamental misunderstanding of the nature of Validity Generalization (VG): It is based on the assumption that what is generalized in VG is the estimated value of mean rho ($\bar{\rho}$). This erroneous assumption is stated repeatedly throughout the article. A conclusion of Validity Generalization does not imply that $\bar{\rho}$ is identical across all situations. If VG is present, most, if not all, validities in the Validity distribution are positive and useful even if there is some variation in that distribution. What is generalized is the entire distribution of rho ($\bar{\rho}$), not just the estimated $\bar{\rho}$ or any other specific value of Validity included in the distribution. This distribution is described by its mean ($\bar{\rho}$) and standard deviation (SDρ). A helpful concept based on these parameters (assuming ρ is normally distributed) is the credibility interval, which reflects the range where most of the values of ρ can be found. The lower end of the 80% credibility interval (the 90% credibility value, CV = $\bar{\rho}$ – 1.28 × SDρ) is used to facilitate understanding of this distribution by indicating the statistical “worst case” for Validity, for practitioners using VG. Validity has an estimated 90% chance of lying above this value. This concept has long been recognized in the literature (see Hunter & Hunter, 1984, for an example; see also Schmidt, Law, Hunter, Rothstein, Pearlman, & McDaniel, 1993, and hundreds of VG articles that have appeared in the literature over the past 40 years since the invention of psychometric meta-analysis as a means of examining VG [Schmidt & Hunter, 1977]). The $\bar{\rho}$ is the value in the distribution with the highest likelihood of occurring (although often by only a small amount), but it is the whole distribution that is generalized. Tett et al. (2017) state that some meta-analysis articles claim that they are generalizing only $\bar{\rho}$. If true, this is inappropriate. Because $\bar{\rho}$ has the highest likelihood in the ρ distribution, discussion often focuses on that value as a matter of convenience, but $\bar{\rho}$ is not what is generalized in VG. What is generalized is the conclusion that there is Validity throughout the credibility interval. The false assumption that it is $\bar{\rho}$ and not the ρ distribution as a whole that is generalized in VG is the basis for the Tett et al. article and is its Achilles heel. In this commentary, we examine the target article's basic arguments and point out errors and omissions that led Tett et al. to falsely conclude that VG is a “myth.”
-
measurement error obfuscates scientific knowledge path to cumulative knowledge requires corrections for unreliability and psychometric meta analyses
Industrial and Organizational Psychology, 2014Co-Authors: Chockalingam Viswesvaran, Frank L. Schmidt, Deniz S. Ones, Huy Le, Insue OhAbstract:All measurements must contend with unreliability. No measure is free of measurement error. More attention must be paid to measurement error in all psychological research. The problem of reliability is more severe when rating scales are involved. Many of the constructs in industrial-organizational (I-O) psychology and organizational behavior/human resource management research are assessed using ratings. Most notably the organizationally central construct of job performance is often assessed using ratings (Austin & Villanova, 1992; Borman & Brush, 1993; Campbell, Gasser, & Oswald, 1996; Viswesvaran, Ones, & Schmidt, 1996; Viswesvaran, Schmidt, & Ones, 2005). The reliability of its assessment is a critical issue with consequences for (a) validation and (b) decision making. For over a century now, it has been known that measurement error obfuscates relationships among variables that scientists assess. Again for over a century, it has been known that statistical corrections for unreliability can help reveal the true magnitudes of relationships being examined. However, until mid-1970s, corrections for attenuation were hampered by the fact that the effect of sampling error is magnified in corrected correlations (Schmidt & Hunter, 1977). Only with the advent of psychometric meta-analysis was, it possible to fully reap the benefits of corrections for attenuation because the problem of sampling error was diminished by averaging across many samples and thereby increasing sample sizes. Since the advent of psychometric meta-analysis 38 years ago, scientific knowledge in the field of I-O psychology has greatly increased. Hundreds of meta-analyses have established basic scientific principles and tested theories.Against this backdrop, LeBreton, Scherer, and James (2014) have written a focal article that distrusts corrections for unreliability in psychometric meta-analyses. They question the appropriateness of using interrater reliabilities of job performance ratings for corrections for attenuation in Validity Generalization studies. Because of length limitations on comments in this journal, we will address only major errors, not all errors in LeBreton et al.'s strident article. The focal article is unfortunately more emotional than rational in tone and conceptually and statistically confused. In our comment, we address only the two latter problems.We have organized our comment in five major sections: (a) purpose of validation and logic of correction for attenuation, (b) reliability of overall job performance ratings, (c) Validity estimation versus administrative decision use of criteria, (d) accurate Validity estimates for predictors used in employee selection, and (e) correct modeling of job performance determinants.
-
Refinements in Validity Generalization methods: Implications for the situational specificity hypothesis.
Journal of Applied Psychology, 1993Co-Authors: Frank L. Schmidt, Kenneth Pearlman, John E. Hunter, Kenneth S. Law, Hannah R. Rothstein, Michael A. McdanielAbstract:Using a large database, this study examined three refinements of Validity Generalization procedures: (a) a more accurate procedure for correcting the residual SD for range restriction to estimate SDP, (b) use of f instead of study-observed rs in the formula for sampling error variance, and (c) removal of non-Pearson rs. The first procedure does not affect the amount of variance accounted for by artifacts. The addition of the second and third procedures increased the mean percentage of Validity variance accounted for by artifacts from 70% to 82%, a 17% increase. The cumulative addition of all three procedures decreased the mean SDf estimate from .150 to .106, a 29% decrease. Six additional variance-producing artifacts were identified that could not be corrected for. In light of these, we concluded that the obtained estimates of mean SDP and mean Validity variance accounted for were consistent with the hypothesis that the true mean SDP value is close to zero. These findings provide further evidence against the situational specificity hypothesis. The first published Validity Generalization research study (Schmidt & Hunter, 1977) hypothesized that if all sources of artifactual variance in cognitive test validities could be controlled methodologically through study design (e.g., construct Validity of tests and criterion measures, computational errors) or corrected for (e.g., sampling error, measurement error), there might be no remaining variance in validities across settings. That is, not only would Validity be generalizable based on 90% credibility values in the estimated true Validity distributions, but all observed variance in validities would be shown to be artifactual and the situational specificity hypothesis would be shown to be false even in its limited form. However, subsequent Validity Generalization research (e.g., Pearlman, Schmidt, & Hunter, 1980; Schmidt, Gast-Rosenberg, & Hunter, 1980; Schmidt, Hunter, Pearlman, & Shane, 1979) was based on data drawn from the general published and unpublished research literature, and therefore it was not possible to control or correct for the sources of artifactual variance that can generally be controlled for only through study design and execution (e.g., computational and typographical errors, study differences in criterion contamination). Not unexpectedly, many of these meta-analyses accounted for less than 100% of observed Validity variance, and the average across studies was also less than 100% (e.g., see Pearlman et al., 1980; Schmidt et al., 1979). The conclusion that the Validity of cognitive abilities tests in employment is generalizable is now widely accepted (e.g., see
-
Meta-Analysis of Integrity Tests: A Critical Examination of Validity Generalization and Moderator Variables
1992Co-Authors: Frank L. Schmidt, Deniz S. Ones, Chockalingam ViswesvaranAbstract:Abstract : A comprehensive meta-analysis was conducted to investigate whether integrity test validities are generalizable and to estimate differences in Validity due to potential moderating influences. The database included 665 Validity coefficients across 576,464 data points. Results indicate that integrity test validities are positive and in many cases substantial for predicting both job performance and counterproductive behaviors on the job such as theft, disciplinary problems, and absenteeism. Validities were found to be generalizable. The estimated mean operational predictive Validity of integrity tests for supervisory ratings of job performance is .41. For the criterion of counterproductive behaviors, results indicate that use of concurrent validation study designs may overestimate the predictive criterion-related Validity applicable in selection situations. Our results based on external criterion measures (i.e., excluding self reports) and predictive Validity studies using applicants indicate that integrity tests predict the broad criterion of organizationally disruptive behaviors better than they predict the narrower criterion of employee theft alone. Our results also indicated substantial evidence for the construct Validity of integrity tests. Perhaps the most important conclusion of this research is that despite the influence of moderators, integrity test validities are positive across situations and settings.
Martin T. Wells - One of the best experts on this subject based on the ideXlab platform.
-
using random rather than fixed effects models in meta analysis implications for situational specificity and Validity Generalization
Personnel Psychology, 1996Co-Authors: Amir Erez, Matt Bloom, Martin T. WellsAbstract:Combining statistical information across studies (i.e., meta-analysis) is a standard research tool in applied psychology. The most common meta-analytic approach in applied psychology, the fixed effects approach, assumes that individual studies are homogeneous and are sampled from the same population. This model assumes that sampling error alone explains the majority of observed differences in study effect sizes and its use has lead some to challenge the notion of situational specificity in favor of Validity Generalization. We critique the fixed effects methodology and propose an advancement–the random effects model (RE) which provides estimates of how between-study differences influence the relationships under study. RE models assume that studies are heterogeneous since they are often conducted by different investigators under different settings. Parameter estimates of both models are compared and evidence in favor of the random effects approach is presented. We argue against use of the fixed effects model because it may lead to misleading conclusions about situational specificity.
-
USING RANDOM RATHER THAN FIXED EFFECTS MODELS IN META‐ANALYSIS: IMPLICATIONS FOR SITUATIONAL SPECIFICITY AND Validity Generalization
Personnel Psychology, 1996Co-Authors: Amir Erez, Matt Bloom, Martin T. WellsAbstract:Combining statistical information across studies (i.e., meta-analysis) is a standard research tool in applied psychology. The most common meta-analytic approach in applied psychology, the fixed effects approach, assumes that individual studies are homogeneous and are sampled from the same population. This model assumes that sampling error alone explains the majority of observed differences in study effect sizes and its use has lead some to challenge the notion of situational specificity in favor of Validity Generalization. We critique the fixed effects methodology and propose an advancement–the random effects model (RE) which provides estimates of how between-study differences influence the relationships under study. RE models assume that studies are heterogeneous since they are often conducted by different investigators under different settings. Parameter estimates of both models are compared and evidence in favor of the random effects approach is presented. We argue against use of the fixed effects model because it may lead to misleading conclusions about situational specificity.
H. G. Osburn - One of the best experts on this subject based on the ideXlab platform.
-
THE EFFECTS OF UNREPRESENTED STUDIES ON THE ROBUSTNESS OF Validity Generalization RESULTS
Personnel Psychology, 2006Co-Authors: Steven D. Ashworth, H. G. Osburn, John C. Callender, Kristin A. BoyleAbstract:Researchers conducting meta-analyses such as Validity Generalization can never be certain that their review contains all studies relevant to the research domain. Indeed, several authors in the past have noted ways in which research reviews may be systematically biased. A few techniques have emerged for addressing the issue of “missing studies” including the use of Rosenthal's (1979) file-drawer equation. Noting that Rosenthal's technique is inappropriate when applied to Validity Generalization findings, this paper develops a new method for assessing the vulnerability of Validity Generalization results to unrepresented or missing studies. The results of this new procedure are compared to the results of file-drawer analyses for 103 findings from Validity Generalization studies. We illustrate that this procedure more appropriately estimates the robustness of Validity Generalization results.
-
A note on the sampling variance of the mean uncorrected correlation in meta-analysis and Validity Generalization.
Journal of Applied Psychology, 1992Co-Authors: H. G. Osburn, John C. CallenderAbstract:This article compares the accuracy of several formulas for the standard error of the mean uncorrected correlation in meta-analytic and Validity Generalization studies. The effect of computing the mean correlation by weighting the correlation in each study by its sample size is also studied. On the basis of formal analysis and simulation studies, it is concluded that the common formula for the sampling variance of the mean correlation, V r =V/ r /K where K is the number of studies in the meta-analysis, gives reasonably accurate results. This formula gives accurate results even when sample sizes and ps are unequal and regardless of whether or not the statistical artifacts vary from study to study. It is also shown that using sample-size weighting may result in underestimation of the standard error of the mean uncorrected correlation when there are outlier sample sizes
Adrian Thomas - One of the best experts on this subject based on the ideXlab platform.
-
THE EFFECT OF CORRELATED ARTIFACTS AND TRUE Validity ON THE ACCURACY OF PARAMETER ESTIMATION IN Validity Generalization
Personnel Psychology, 1998Co-Authors: Nambury S. Raju, Tobin V. Anselmi, Jodi S. Goodman, Adrian ThomasAbstract:The effect of non-zero intercorrelations among the three artifacts (criterion reliability, predictor reliability, and range restriction on the predictor) and true Validity on the accuracy of estimation in two Validity Generalization models (Model 1and Model 2) was investigated. Only the Thylor Series Approximation 1(TSA1) procedure from Model 1 and the procedure with complete sample-based artifact data from Model 2 were included in this study. Six intercorrelation conditions (zero, low positive, medium positive, low negative, low negative and positive, and medium negative and positive) and two distributions of artifacts were used in this investigation. The two major conclusions from this study are: (a) Both models yielded reasonably accurate estimates of the mean of true validities (Mρ) and (b) only Model 2 produced accurate estimates of the variance of true validities (Vρ). The need for additional research for accurately estimating Vρ when the three artifacts and true Validity are correlated is recommended.
-
THE EFFECT OF CORRELATED ARTIFACTS AND TRUE Validity ON THE ACCURACY OF PARAMETER ESTIMATION IN Validity Generalization.
Academy of Management Proceedings, 1995Co-Authors: Nambury S. Raju, Tobin V. Anselmi, Jodi S. Goodman, Adrian ThomasAbstract:The effect of non-zero intercorrelations among three artifacts and true Validity on the accuracy of Validity Generalization (VG) estimates was investigated, resulting in the following conclusion: T...
