Article · 5 minute read
How Can You Know if an Assessment Actually Works?
5th May 2023
The latest effectiveness research on assessment methods from the 2023 SIOP conference – the leading conference for the professional association for the science and practice of organizational psychology in the USA.
I attended the recent annual SIOP (Society for Industrial Organizational Psychology) conference in Boston which was a great opportunity to hear some of the latest research on the effectiveness of assessment methods and learn from expert academics and practitioners in occupational psychology.
As always in our field, validity of assessments (i.e how we know an assessment is actually working) was one of the key themes across much of the research presented, with some great advice on what to look for and delve into to know if an assessment actually works.
What are some of the key issues when evaluating whether an assessment is valid?
- What are you trying to predict with the assessment? Most of the time, if we are using an assessment for selection, we are trying to predict overall job performance, but we may also want to predict things such as how well someone will perform on a training program, or other narrower facets of performance e.g. leadership potential. Defining specifically what you want to predict can be really useful.
- What does ‘performance’ mean in your organization? How is it measured? What ‘performance’ means is changing and in order to line up your assessment process to predict it, it’s useful to know how it is defined in your organization. The growing importance of technology and evolving organizational priorities mean that how we measure individual and team performance is likely to be changing too but knowing what you are going to be measuring people on can give you a helpful lens through which to view an assessment’s effectiveness.
- Is there an opportunity to do the research to establish evidence that your assessment method can predict your measure of performance? There are different ways to do this. Either by researching the link between the assessment and current performance, or future performance in your organization; this can be done in partnership with your assessment provider to establish that evidence base for your assessment use.
What does the latest research say are the most effective assessment methods for predicting performance?
The latest research has re-visited some research design assumptions which has given us new effectiveness data. It shows that job-specific measures are the most predictive; this means using assessments that measure key competencies most relevant to the job work better than those that are more generalized.
This has a few implications:
- Structured interviews based on key job-specific competencies were found to be highly effective – making sure the interviews are standardized, based on job-specific competencies and have consistent scoring methodologies is key. Our range of easy-to-use Interview Guides provide a strong structured basis for job-specific structured interviews which can be easily picked up by hiring managers.
- When using behavioral/personality assessments, ensure that these are deep diving into job-specific behaviors only (rather than the whole profile/report) to maximize effectiveness.
- Work samples which mimic the specific job requirements are also highly effective (as opposed to an assessment center exercise that is more general / group-based). This reflects the Saville Assessment approach to assessment centers – moving away from lengthy exercises assessing competencies in multiple ways, to one or two highly-relevant work sample exercises – maximizing validity and minimizing undue burden on candidates and assessors.
The evaluation of certain research design assumptions means the validity of cognitive ability assessments has been re-calculated and is not quite as high as previously suggested. This, along with potential challenges from a D, E & I perspective, has the potential to put some people off using these tools. However, their validity is still solid and, in terms of ‘off-the-shelf’ predictors, they remain one of the most valid predictors of overall job performance, training performance and narrower facets of overall performance.
There was also discussion on how multi-faceted processes (e.g. combining aptitude assessments scores with scores from behavioral assessments) can improve D, E & I outcomes, compared to using cognitive measures alone. This is something we at Saville Assessment have long advocated for and our pioneering mix’n’match methodology drives powerful results.
To maximize validity, it is best to combine assessment methods to benefit from the incremental gains of using multiple assessments together – for example, a job-specific situational judgment test, cognitive ability test and structured interview based on job-specific behaviors/competencies is likely to be more powerful than one of those assessments alone.
This is the approach we advised that Jaguar Land Rover use when experiencing high volumes of applicants for their graduate scheme. This culminated in a much more efficient and streamlined process with a higher quality of applicant making it through to the final stages. You can read the full case study.
Find Out More
Our Consultancy team will be happy to discuss the latest research on validity and to help you create an assessment process that’s effective, job-specific and practical to implement.