Get my free whitepaper "10 Ways AI Will Transform Talent Acquisition" – Download Now
Transform Talent Acquisition – Find out how my digital course will help you shape the future of talent acquisition in your organization – Click Here
Assessment is the one area of talent acquisition practice where we have proper peer-reviewed science about what is and isn't effective. Unfortunately, the use of science in assessment has tended to only happen in larger organizations and then not necessarily consistently.
AI has the potential to change everything, making assessment technology faster and more accessible to employers of all sizes. However, there are huge dangers here. Not only does AI bring with it the potential for more bias in the process, things are moving so quickly that it may well be that some tech solutions coming into the market are built without due consideration of the fundamental principles of science and psychology that should be applied to assessment.
So, what steps can TA Leaders take to ensure their technology is fit for purpose and embed a more strategic approach to assessment?
My guest this week is Dr Charles Handler. Charles is the President & Founder of Rocket-Hire, an organization that designs, implements, and validates employee selection solutions. He is also a Futurist, working extensively to research and understand the impact of AI on the future of assessment.
In the interview, we discuss:
- How assessment has evolved over the last five years
- Making the science more accessible and assessments more user-friendly
- The importance of properly evaluating assessment technology.
- The opportunities and threats of AI
- Finding a positive way to respond to the growing use of AI by jobseekers
- How to think more strategically about assessment
- What does the future look like
Follow this podcast on Apple Podcasts.
"Hi, this is Matt. Just before we start the show, I want to tell you about a free white paper that I've just published on AI and talent acquisition. We all know that AI is going to dramatically change recruiting, but what will that really look like?" For example, imagine a future where AI can predict your company's future talent needs, build dynamic external and internal talent pools, craft, personalised candidate experiences, and intelligently automate recruitment marketing. The new white paper, Ten Ways AI Will Transform Talent Acquisition, doesn't claim to have all the answers, but it does explore the most likely scenarios on how AI will impact recruiting. So, get a head start on planning and influencing the future of your talent acquisition strategy. You can download your copy of the white paper at mattalder.me/transform. That's mattalder.me/transform. "There's been more of scientific discovery, more of technical advancement and material progress in your lifetime than mine, at all the ages of history." Hi there, welcome to episode 631 of Recruiting Future with me, Matt Alder. Assessment is the one area of talent acquisition practice where we have proper, peer-reviewed science about what is and isn't effective. Unfortunately, the use of science and assessment has tended to only happen in larger organisations, and then not necessarily consistently. AI has the potential to change everything, making assessment technology faster and more accessible to employers of all sizes. However, there are huge dangers here. Not only does AI bring with it the potential for more bias in the process, things are moving so quickly that it may well be that some tech solutions that come into the market are built without due consideration of the fundamental principles of science and psychology that should be applied to assessment. So what steps can TA leaders take to ensure their technology is fit for purpose and embed a more strategic approach to assessment? My guess this week is Dr Charles Handler. Charles is the founder of Rocket Hire, an organisation that designs, implements and validates employee selection solutions. He's also a futurist, making extensively to research and understand the impact of AI on the future of assessment. Hi, Charles, and welcome to the podcast. Hey, thanks so much for having me. This is one I listen to a lot, and it's always amazing to have the opportunity to be a guest on podcast that you listen to. Well, the pleasure is all mine, so please could you introduce yourself and tell everyone what you do? Yeah, so I'm an organisational psychologist. I've been doing that for 30 some years, and I'm a specialist in talent assessments from a lot of different angles from the compliance angle. I've built assessments. I've filled assessments for global enterprise companies. I'm an analyst for the industry, so I kind of track the vendors and what's going on. I'm totally immersed in the AI aspect of things, the ethics and the efficiency, et cetera, of all that stuff. I'm the host of the Psych Tech at Podcast on which you've been a distinguished guest, and just an all around fan of good hiring and good people practices. Excellent stuff. Unsurprisingly, I want to talk about AI in a bit more depth, but before we do that, I suppose by way of context and background, let's talk about just about how assessment has kind of evolved over the last five years or so. What's been changing, what's been going on, what's the general direction of travel then? Yeah, well, compared to a lot of other industries and even HR tech, it unfortunately hasn't evolved at the level that other things are. Part of that is because what we use, we know it works, but we have to be so careful. I mean, when you're evaluating people for important high stakes things like jobs and promotions, you absolutely have to make sure that what you're using is relevant and fair, et cetera. I think as well, the providers in the space, the long-term providers in the space, they have a lot of business. They don't have a lot of incentive to change rapidly, nor can they really do that from a technology standpoint. We do see a lot of exciting startups. We see things that people are doing with AI and pretty much everything new now has AI built into it. The things that I see happening more are definitely a level of user friendliness for candidates. I think in the past, you might have been hit with a 200-item assessment with questions that don't really seem job-relevant at all. We can't get away with that. We're seeing shorter, we're seeing better UIs, we're seeing the opportunity to use mostly natural language processing to listen to what people say or type and be able to pull some elements of predictive ability out of that. I think that the biggest change in the industry is not necessarily coming from the industry itself. It's coming from providers who are offering the ability to go out and find source candidates and bring candidates into the funnel using AI and the type of technologies I talked about. They don't have people like myself at the table a lot of time. What they're saying is we can predict who's going to be successful in a job, but they're not measuring things about people in the way that we know how to do it in my world. That's an interesting point. Let's follow up to that. Is the science becoming more accessible? As we see more assessment platforms opening up, different price points, is it more accessible for employers to use sound science in their selection processes? Yes, absolutely. It's getting built into some of the platforms, as you mentioned. I've tracked the market and providers for 20-some years. Every certain amount of years depends on what's going on. I'll publish here's a list categorized of all the folks that are doing predictive hiring. My database is now up to 400 plus. I just added two today. I added a couple a few days earlier. They're popping up everywhere, which to me says it has value. What we're doing has value, which is a good thing. It is becoming more accessible because of all those various options. It's coming down market a little bit. What you have to watch out there for is, from a science standpoint, the more you can really look at a job and study the job and build a set of predictors relative custom to that job, the better you're going to be. Oftentimes, it's really hard to do that at scale. To your question of accessibility to science, a lot of times we and I personally have stood there and said, "Hey, we know how to do this right. Give us the data. Give us the subjects. Let us get in there." It's very difficult sometimes. The commitment to really doing it right on the client side is often the biggest limitation, which I think to me, and in some ways, I've only really started to properly think about this in the last couple of years, to me is kind of crazy. This is the only part of talent acquisition where there is proper peer-reviewed science about how to do something, but it seems that it's kind of ignored and not adopted in the way that it should be. Yeah. What you see is, if you have people like myself running an assessment program in a global enterprise company, which, by the way, is happening more and more and more, and I speak from the global enterprise perspective because that's where I've lived and worked for 20, 30 years, but those folks typically have a lot of buy-in. They have mature programs or programs that are evolving, and the business commits to giving them what they want. What I've found is an external person coming in as the voice or representation of what I do, it is often a lot harder. I deal with a lot of high-volume jobs. Taking an employee off the floor, off the seat, whatever it is, getting them sometimes, they're not allowed to use their own personal time most of the time to answer, because I'll back up. The paradigm for making it successful is typically you're going to study the job, and you're going to talk to people who do the job and be able to understand, "Okay, what is the job requirements that we want to evaluate?" Otherwise, you're completely wasting what you're doing. That takes some effort, and to do that right in a compliant way needs to be a nice-sized engagement. Once you get an assessment in place, typically we do what's called a criterion validation, we give the assessment to a healthy sample of incumbents, and then we do some correlational model building work to see what things predict what things that we're excited about predicting, that we need. That takes pulling people off the floor, often supervisors rating their employees. There's a lot of surveying, there's a lot of time. It doesn't have to be a massive commitment. Contact centers, retail, get people off the floor to participate. It's a heavy lift sometimes. It really has to be emphasized from the very top. I think that's what I'm talking about. We obviously mentioned AI right at the beginning, and you mentioned some of the tools that are coming in that might not necessarily have the proper basis behind them. Is AI being used to select people at the moment? As we move forward, what time acquisition professionals need to look out for when it comes to the use of AI in this kind of way? Yeah, that's a great question. I just mentioned a massive set of friction points. Companies are recognizing through their actions that those friction points can be overwhelming. Hiring has a lot of friction anyway. If you look at AI, what do we want to do? We want to use it to reduce the friction maybe to understand what a job requires without doing it hands-on, maybe to understand what a possible employee or a job applicant brings to the table. We can use AI in ways to do that. Building those matching algorithms, AI can do that. To my point earlier, there's a lot of people who are going all in on that, who may not have our background who are really empirically driven. That's one segment of what's going on. If you move closer into the assessment industry, what you see is, and I'm studying this, I'm interviewing people and organizations that do what I do about how their companies use AI. I'm consulting, I'm watching. We're using it in the industry to build stuff, right? In other words, the risk of saying, "Oh, man, I've got to build all these structured interview questions for the job of software engineer." AI, chat GTP, whatever, can get you 80 percent of the way there really fast. The drudgery, just like in every other area, we're using AI too. That's not facing candidates. That's not making evaluations. The next step, where are we doing that? It's mostly natural language processing. It's where you're looking at words or speech and pulling features that are predictive out. We're seeing a good deal of that. What I would say too, and there's more, but I don't want to ramble on. What I would say to organizations considering looking at this stuff is you got to evaluate any AI tool the same way you would evaluate any predictive hiring tool. Is it predicting differentially for applicants in one bucket or another? That's a really important thing. Is it applicant-friendly? If it's completely stealth, are you letting candidates know that you're doing that? I built a model called the ethics hire model where here's several things that it's initials. Each one of these things is going to be something that you need to critically look at when you're evaluating a vendor. What has changed is the complexity of what you look at. If you're looking at models and it's using a large language model or it's using a deep learning model, it's pretty hard to unpack those and say, "Oh, this is what's happening here." Whereas traditionally in our field, it's just algebra. The models are algebra. You could take a look at that. You could look at the data that was used to build those. There's a lot of clarity there. It's getting more complicated. The last thing I'll say on that is if your organization has an AI governance model, it needs to extend to everything. There's some general high-level things there that you need to be looking at. You must apply that to looking at a hiring tool. It has to be relevant to the job. You could have the best tool in the world. If you're not getting in there and understanding what the job requires and you're missing that calibration, that connection, it'll never be able to predict what you're looking for. I suppose on the flip side of this, we're seeing a big impact now from job seekers and candidates and applicants using AI themselves in the process. What are the implications of that, do you think? The implications are we better get with it and figure out how to deal with it because it's not going away. That's the first thing. There's some good research that I've consumed. That's pretty available out there that says job seekers expect to be able to use generative AI on the job, especially younger demographic, graduates, early career. They're using this stuff every day. People expect to use generative AI on the job, whether they're supposed to or not, especially younger demographics. There's been a lot of research that shows that. They don't want to work for a company that doesn't allow them to do that. There's a couple of things, several things that need to happen. We need to understand it's here to stay and that it's going to be something that folks do. To preempt that, you can either fight it or you can go get with it and say, "How can we understand that?" and turn lemons into lemonade. That would be, "What's the policy?" Helping job seekers understand what your policy is. Potentially even giving them some coaching and tips and things like that about, "This is how we would like to see it done." Even more so than just changing some of the methods. ChatGTP, this is not even 4.omni4.0, which is even more powerful if you use this stuff. It can complete a personality test if you tell it what job. It can complete a cognitive test really well, just like in other disciplines. There's types of tests, more interactive, experiential, like simulations and things that make it much more difficult for this to happen. You need to start in the industry, needs to start looking to those types of things. There's, "Hey, do you want to go the old-fashioned way and bring someone in in person and sit there and watch?" There is a level or a thing I call AI Darwinism. One of the funniest stories I like to tell is I was sitting at a conference with people whose job it was to do test security. Multiple people told me they've busted people using ChatGTP in an interview, a recorded interview, because they could see it in their glasses. They could see the reflection of it. At the same time, you can use ChatGTP to help you prepare for an interview, to role play and do mock interviews, so that's valuable. I've come up with a concept I call AI-enabled work ethic. What that means is that's a competency you need to actually evaluate in applicants. Are they able to use this stuff in a creative and positive way on the job? That's something that you want to happen. We've got to get with it. It's not going with any security thing. It's going to be hard to keep up. Let's look at it a different way. No, I 100% agree with you. I think that's the only way forward with this and a really positive way forward as well. Almost by way of summary of some of the things that you've already said, how should TA leaders, employers, be thinking more strategically about these test assessment tools and technologies? What would your advice be to people? My advice would be, this probably goes without saying, look to where there's a problem, right? But also, look at the bigger picture. I think a lot of times assessment is used tactically to fight fires. We have horrible turnover in this call center, et cetera. In the UK, and I've had the pleasure of working with a bunch of UK-based companies, for graduate recruitment, it's a lot more common to do more of an assessment center or role playtime. These are things that I feel like have more strategic value because you're often looking at the overall competencies and things that people need to bring to the table. I think it's really valuable to think about the skills-based hiring paradigm, where you're looking past academic credentials, et cetera. Assessment is critical for that. If you're just using inferential methods, some kind of AI technology to try and tell you this person has these skills. For hard skills, that works OK. For soft skills, we're not really there yet. So don't rely on that. Don't believe the hype, I would say. The paradigm that we use works really well. I would also say, evaluate your vendors very carefully. I do a lot of vendor evaluation and companies, the first part of our exploration is, what in the heck are you looking to do here? Who are you? What are the parameters that will make an assessment program successful? OK, now let's go look. Don't just say, oh, I got, you know, I watched a webinar from this provider. It looks like they're doing amazing stuff. So-and-so uses this provider, although that is valuable feedback, you know, who your peers are using. You've got to understand before you go shopping, you've got to make that shopping list. And then you've got to be very thorough and critical about it. Take the assessment. Ask for the-if they can't provide you with a technical manual that shows how it was built and, you know, the actual science that goes into it with a good sample size, et cetera, you've got to run screaming because the minute you get audited by someone, you're in trouble, but even more so, that shows that a level of responsibility, I think that's really important. And the last thing I'll say, I can't overemphasize this. I get into this battle of name that tune sometimes in some sense. Like, you need to hire this person in five minutes or less, you know, and I'll say, well, give me 10 minutes. Maybe I can do it in 10 minutes. You don't get something for nothing. And the other thing that's important is people always say, well, applicants aren't going to sit for an assessment. Well, that may be some of them, but I've also seen great research, you know, from the candies of candidate experience folks there. They are the talent board. They have done some great research that show-they interviewed our-excuse me-survey job applicants, thousands of them. And the preponderance is we want a chance to demonstrate our skills. We want a chance to show what we can do. And so the more job-related, you know, another mantra I have, candidates never leave a candidate asking, why are they asking you this? Or wondering, excuse me, why are they asking you this? It should all be very clear. So, you know, a 20-minute assessment is not bad. There's good research that shows the inflection point is around 20 or 25 minutes. And in fact, candidates view very short assessments as less credible, right? So job-related, in a good timing, they're engaging if you can. I think those are all really, really important things to think about. Don't limit yourself by saying, we don't have enough applicants, you know, that's going to drive applicants away. Do you really want to hire somebody that's not going to give you 20 minutes of their time? I know I wouldn't, you know? So those are all things to think about. There's barriers that get thrown up. Don't believe that hype either. And so as a final question to you, where is all this going? What does the future look like? How are things going to develop? What's going to happen over the next two or three years? That's a great question. It's the one everybody's asking, and I don't claim clairvoyance. But what I can say is I've dedicated my whole career to thinking about this. We're not going to have tests anymore. You're not going to take a Likert scale personality test, you know, in the future, absolutely not. There's going to be two ways that it goes. Way number one is complete stealth. You're going to be evaluated and probably eventually pretty accurately through the preponderance of digital exhaust that you have through some other application standards like interviews and materials that you send in. So there'll be the stealth aspect, and then there's going to be the hands-on aspect. We're going to be able to simulate jobs and interactions and work product on jobs at scale much more easily than we ever have before. So and think about it. You know, that's been a long-standing truth. You want to see how someone will do on the job, give them a piece of the job to do, for goodness' sakes, right? So that has been difficult to do at scale in a very precise way. As I mentioned earlier, that precision is just really important. So I believe that we're going to see no assessment and we're going to see highly job-related assessment. Those two can fill different purposes as well. Maybe the no assessment is at the top of the funnel. We're already starting that, but again, the inferential stuff is typically not being driven by science. It's being driven by data science, empiricism, and not a blend of rationality and empiricism. And that's a problem. And the simulation world has been limited. So, you know, and that's the kind of stuff I'm working on. Like, I'm betting my future on simulations, right? Large language models can role play. Large language models, when they text a video is accessible. You're going to be able to simulate you and I have in a podcast and the dialogue and everything that's happening, but we won't be there, you know? And so that is going to change the game. That's going to change the game. Absolutely. We live in interesting times. Charles, thank you very much for talking to me. Yeah, thanks so much for having me on. It's a real honor and I appreciate it. My thanks to Charles. You can follow this podcast on Apple Podcasts, on Spotify, or via your podcasting app of choice. Please also subscribe to our YouTube channel by going to metal.tv. You can search all the past episodes at recruitingfuture.com. On that site, you can also subscribe to our newsletter, Recreating Future Feast, and get the inside track about everything that's coming up on the show. Thanks very much for listening. I'll be back next time and I hope you'll join me. (upbeat music) [BLANK_AUDIO]
