Article · 5 minute read
The truth about self-reports and what we can learn from them
21st March 2024
Traditional criticism of self-reports is often:
“They’ll just tell you what you think you want to hear.”
“It’s just what they think of themselves.”
“Others might say differently.”
“AI has answered for them.”
Sound familiar? These are common views associated with ‘self-report’, but actually they reflect the myths we will tackle in this short article.
At Saville Assessment, we are unapologetically scientific. Our extensive research program developing Wave®, the most scientifically proven solution for identifying potential and developing performance, gave us great insight into the value of self-reports. This understanding not only challenges many traditional misconceptions associated with self-report, but also provides practical tips for ensuring the data gleaned is optimized for maximum effectiveness and offers a fresh perspective against the new backdrop of generative AI in hiring.
Myth - We don’t use any ‘self-report’ in our process
Truth: Self-reports are everywhere: interviews are self-reports, CVs are self-reports, applications are self-reports; the user is choosing how best to present themselves. All of these tools are used globally to help recruiters make hiring decisions and the chances are that you are using one, if not all of them, yourselves.
The use of bio-data in hiring has not been without scrutiny, particularly through the lens of potential bias and DE&I. The generative AI spotlight is now shining brightly on these methods. Applicants are using ChatGPT to write cover letters, construct CVs and prepare for interview questions. HR experts have shared their first-hand experiences of issues arising from this; ”It’s like smoke and mirrors. I get so many cover letters where the applicant’s personality just doesn’t come through. You can completely tell it has been AI generated.”
To understand how a person behaves and how good a match they are for the role, your organization’s culture and values, you need a personality assessment. These can also help facilitate highly-structured interviews which probe strength and challenge areas and really drill down on the key areas important to you and your organization. We would also recommend introducing these as early in the process as possible. Whilst this is best practice, it can help prevent the other consequences of people using generative AI as part of their initial application. Daniel Wolken, a Human Resources Expert at DailyRemote shared that including rigorous assessments early on in the process helped to ensure time wasn’t wasted on candidates who can ‘talk the talk’, but can’t ‘walk the walk’.
Myth - Self-reports can be ‘faked’
Truth: One of the common concerns around self-reports is that the user can ‘second guess’ them, trying to manipulate them to give the employer what they think they want to hear, or even using generative AI to inform their responses.
Self-report assessments underpinned by strong scientific validity mitigate against this.
Our Wave Personality Questionnaires are the most scientifically proven measures of performance and potential in the market. Built in to the Wave Personality questionnaires are smart scientific algorithms, or checks and balances if you will, to test for consistency. Its unique ‘rate-and-rank’ format make it difficult for an individual to second-guess the questionnaire. ‘Red-flags’ are highlighted in the report outputs via the unique ‘deep dives’ which indicate for areas of inconsistency in responses, how positive or negative they have been about themselves, potential over-exaggeration of capabilities and the alignment between what they say they enjoy doing and what they say they are good at doing (motive vs talent).
Crucially, ChatGPT does not know how to create a personality profile that is appropriate for a particular job, or reflects the candidates personality. You can read more about ChatGPT and the world of assessment here.
Myth - Assessment centres are more valid
Truth: In a world where advancements in technology have contributed to a rise in mistrust, observing people in assessment centers must surely be the most reliable way to get an objective perspective on a person, right? Wrong…!
When done correctly, interviews do a very good job of predicting workplace performance. The same can be said for personality questionnaires. In fact, evidence suggests that both of these methods are better at forecasting potential job performance than the type of observational techniques used at assessment centers.
One of the problems is that the exercises at these centers usually aren’t probing areas that are actually relevant to the role. They generally don’t measure competencies, so they don’t tell you what an individual is good or bad at, only how good they are at performing the specific task set, which will only be predictive of workplace performance if that task forms a major part of the job – generally they are not!
Another potential issue is that assessment centers usually have multiple observers covering a number of candidates; even with the best training and calibration in the world, there will still be variability in observers’ notetaking, scoring and final assessment, which makes them somewhat unreliable, compared to other methods.
The below graph shows the effectiveness of all the assessment methods generally deemed acceptable for use in hiring across different occupations:
In summary, when it comes to self-report in terms of interviews or personality questionnaires, the following will optimize effectiveness:
- Directly target the specific behaviors that matter to you
- Evidence suggests that skills-based interviewing is much better at this compared to strengths-based interviewing.
- Not all personality questionnaires are equal in this regard; some have badly-written, ineffective questions, so make sure yours doesn’t!
- Standardize responses onto a scale that allows for comparison to others.
- It’s just as critical to have structured scoring and calibration processes as it is to have a properly structured interview.
- You can go one step further with personality questionnaires and benchmark individuals against large external comparison groups.
- As mentioned earlier, ensure a framework that allows for the detection and mitigation of response bias; for example, either very positive or very critical responders
- Personality assessments tend to do much better than interviews in this area. You can, however, ensure you are doing this at interview using probing questions such as “What did you learn?” and “What mistakes did you make?” (the candidate who replies “None!” to this second question may be presenting themselves is an overly positive manner…you may not be getting at their ‘authentic’ self).
- Again, not all personality questionnaires are equal and some do this far better than others, so make sure you understand how yours works.
Tap into the Science of Human Potential
We are often asked how Wave® also has such strong validity. The answer is actually very simple; it is all in the questions. One of the main lessons we took from developing Wave is that exact wording of questions is critical and small differences really do matter; the slightest change can impact on a question’s ability to predict performance.
For example, getting people to agree or disagree to the statement ‘I rarely make mistakes’ would be a poor quality question for a personality assessment. This is one you could disagree with because you believe you never make mistakes, when in disagreeing to it, the questionnaire assumes you always make mistakes. This is bringing in complexity and adding to the cognitive load of the assessment, which means you end up measuring cognitive skills rather than behavior.
There are 216 questions in the Wave Professional Styles questionnaire. We started with over 4,000 questions, carefully crafted by a team of four highly-skilled psychometricians, who between them had over 100 years’ experience. We selected the very best of those questions based on how well they targeted the key behavior and then used data to identify those most effective at predicting performance.
Want to know more?
To discuss how the most scientifically-proven solutions for realizing untapped potential could transform your hiring and development activity, get in touch using the button below.