– In the public sector, Training and Experience (T&E) exams assess prior experience and are one of the most often used methods for selecting job applicants. This study uses a KSA approach, where raters judge the quality of job relevant prior experience, not its duration or quantity. It was hypothesized that an additional rater and a consensus meeting between raters would increase reliability and validity.
– T&E and supervisory ratings were obtained over a 12-year period for 166 candidates seeking promotion to a budget analyst position. Validity was measured by the correlation between T&E scores and supervisory ratings. Consensus was required only for T&E scores differing by a specific amount (hybrid consensus).
– Intraclass reliability was 0.73, 0.84, and 0.95 in the one-rater, two-rater, and hybrid consensus conditions with each coefficient greater than the next (p<0.05) showing the benefit of multiple raters and consensus for reliability. Validity was significant at 0.21, 0.26, and 0.251 for each rating condition, respectively (two-tail test; p<0.01). Validity was greater in the two-rater condition than in the one-rater condition (one-tail test; p<0.05). Consensus did not improve validity beyond that of two raters. For consensus T&Es (n=76), two raters improved validity (one-tail test; p<0.05), moving from 0.112 to 0.231 but not reliability; consensus improved reliability (two-tail test; p<0.05) but not validity.
– There has been a vacuum in T&E research for close to 20 years. Validity data are difficult to obtain but critical for meta-analysis. T&Es showed validity. Use of two raters improved validity but consensus did not increase the gain.
