Avoid the "Bad Data Landfill" Using Responsible AI in Hiring Assessments With Mike Campion

What is AI?

We begin our discussion by level setting on what exactly AI is- With Mike’s annotated definition being:

“It's commonly interpreted as a computerized assistance with sensing data, compiling data, doing analyses for the purpose of decision making or a prediction. Now we've been doing that for a long time, but this takes it to a different level. Probably one of the biggest things is that it has broken down the silos, if there ever were any, between statistics that we know and the computer science analyses of data, which are more software engineering.”

While the applications of AI are almost endless, our conversation focuses on the analysis of qualitative data to reliably, fairly, and accurately predict outcomes. With this definition in hand, most of our conversation focuses on how AI is able to:

“create variables from data probably in a slightly superior way”.

Specifically, Mike shares his experience in creating programs that extract variables from text or speech and using them to score predictive hiring tools with the same level of accuracy as humans while stripping away bias. Which is important because, according to Mike:

“That's probably the biggest breakthrough because qualitative data, meaning text data and narrative data spoken or written has been generally ignored in favor of quantified data.”

Mike has approached this by conducting long term applied research programs that use tried and true methodologies to develop predictive tools based on qualitative data. We discuss two such programs: scored application blanks and structured interview ratings.

AI in the Application Process

Mike’s success in scoring job applications via AI involves: 

“The applications that people submit may include a resume, be completely unstructured in that way, or they will have an application that's a combination of education, work experience, some things maybe you can quantify. And then textual information like past jobs, past job duties, special skills and things. So the value of artificial intelligence and the most common application currently is scoring the entire application to get a score that allows the pre-selection of candidates from the large candidate pool. And historically that has been done by a recruiter usually poorly because a recruiter is kind of skimming through resumes looking for keywords.” 

While the automated scoring of structured interviews involves: 

“You might have an interview question that you extract concepts from it or terms they go by different labels, you might call 'em features, you might call 'em concepts, you might call 'em categories Anyway, you might extract several hundred and then find out that some subset of that are predictive and then that becomes your predictive model. And then candidates who use those phrases get higher scores. And so it isn't like the machine is thinking, it scores everything, and it scores everything in a very thorough and objective way. And so even though the computer doesn't understand what it's doing, it can predict human scores as well as humans can predict human scores.”

Balancing Fairness with Validity

When asked about the most exciting things that Mike is working on with these programs of research, he suggests that it is the use of machines to find the optimal balancing point between fairness and validity in a way that is more effective and fair than traditional methods.

“I think use of artificial intelligence to try to solve the validity adverse impact dilemma. I think that's a big one. So I think we'll find that we can score information in a way that draws out skills and abilities based on text data that will likely reduce the amount of subgroup, race and gender differences and at the same time improve our prediction job performance. So I think that's a big one. I think using statistical manipulations to try to make the adverse impact go away mathematically are going to be a dead end.”

The big takeaway from our conversation is that when AI is created using tried and true models, we can build new tools that are more accurate and actually introduce less bias.

“The truth is the way the computer goes at it, which is a thousand little teeny hammers seems to work even though it doesn't really understand. But that's not a problem because it works just as well and it doesn't make mistakes and it doesn't have bias, it can't discriminate.”

Using AI Responsibly 

in order to use AI responsibly it is important to understand that it has to be built on a great deal of careful work that provides a foundation of quality data on which to build.

These models are only as good as the data they're trained upon. Yes. You said garbage in, garbage out before. Well in machine learning they like to call it landfill in and landfill out cause they deal with so much more data. But it's the same thing here. So if your organization has historically done a poor job of hiring of a pre-screening candidates and you train your model against that, then it really limits you.

At the end of the day- when it comes to the usefulness of AI tools for hiring – there is a fork in the road. The point of divergence is in approaches that frontload the prediction of outcomes based on poor quality data vs. those that put quality data collected using tried and true methodologies first.

People in This Episode

Catch Mike Campion on LinkedIn 

Read the Transcript

Announcer:

Welcome to Science 4-Hire with your host, Dr. Charles Hander. Hiring is hard. Pre-hire talent assessments can help you ease the pain. Whether you don't know where to start or you just want to stay on top of the trends, Science 4-Hire provides 30 minutes of enlightenment on best practices and news from the front lines of the employment testing universe. So get ready to learn as Dr. Charles Handler and his all-star guests blend old-school knowledge with new wave technology to educate and inform you about all things talent assessment.

Dr. Charles Handler:

Hello everybody and welcome to the latest edition of Science 4-Hire. I am your host, Dr. Charles Handler, and today I'm extremely honored to have our guest today who is Michael Campion, who's a distinguished professor of management and perennial IO psychologist at Purdue University. So hey Michael, how are you today?

Mike Campion:

I'm doing just great. Thanks for that introduction.

Dr. Charles Handler:

Yeah, no worries. So just a little quick introduction of yourself for our listeners. I always say who knows you better than you. So tell our audience a little bit about yourself and then we'll get into our topic today, which is AI and machine learning and science and hiring and all that good stuff.

Mike Campion:

Okay. Well I'm an industrial psychologist by training. I teach in the business school at Purdue University, but I'm very involved throughout my career in publishing. I publish a lot of articles compared to most anybody else you might have as a guest. And I'm also, I'm just very active consultant. I think I've worked for 200 organizations about a 1400 projects and on every kind of topic, but probably the latter half of my career it's been dominated by selection more than anything else and legal issues as well. And I was one of the early adopters to get into artificial intelligence and machine learning for personnel selection. Saw an application in 2013, which is well ahead of the wave and did a big study and published one, what I think was the first scientific article in an industrial psychology journal on it in 2016. And have become a real advocate because I see it as probably the biggest trend in the profession of personnel selection. Probably the biggest influence since the Equal Rights Act in terms of its overwhelming impact that it will have on the field. So I changed all of my research pretty much to focus on artificial intelligence these days.

Dr. Charles Handler:

Very cool. I can't wait to get into a little of that. And I do, one of the reasons I really wanted to make sure we could get you on the show is because I really like the way you approach it. A lot of people and me, myself sometimes disparage and say, this isn't any good. It's all a bunch of hype and it's really important to have a balanced point of view and there is responsible use of these things. So I can't wait to talk about that, but I gotta tell a story first just to <laugh> that relates to you, and you probably don't know this story at all, but I signed up for at siop, our industrial psychology conference. I remember probably 20 14, 20 15, I signed up for the mentoring, speed mentoring like event <laugh>. And I thought I was signing up as a mentor. I'm like, well I've been doing this 20 years. There's probably a lot of younger people who might wanna know about consulting and how to have your own small company and stuff. And so I sat down at the table, Nope, you were mentoring me. I was like, actually I'm not a mentor, I'm here to be mentored. Well good. That's humbling for me cause

Mike Campion:

I don't remember.

Dr. Charles Handler:

Well it was really funny. So anyway, and you're going to mentor me again today a little bit in this topic because I haven't done a lot of research. I read stuff, I listen to stuff, I see how it's applied quite a bit cuz I work with a lot of applications in industry of various selection systems. So let's talk a little bit today for, I think that it would be great for you to just lead off with, let's define ai. You and I were talking about this in the pre-call, right? You referred to it as machine learning and obviously machine learning is the flavor of it or under the bigger umbrella. But as you go into this work, how are you defining things? What's the way you see the world in terms of the terminology, which we know is fluid and

Mike Campion:

I'm probably less concerned about drawing borders around things than actually using the methodologies. But it seems to me like it's commonly interpreted as a computerized assistance with sensing data, compiling data, doing analyses for the purpose of decision making or a prediction. Now we've been doing that for a long time, but this takes it to a different level. Probably one of the biggest things is a broken down the silos, if there ever were any between statistics that we know and the mach and the computer science analyses of data, which are more software engineering and there's different assumptions and other things, but they basically are able to create variables from data probably in a slightly superior way. That's part of it. But the big news is they have methodologies for analyzing qualitative data. And I think that's probably the biggest breakthrough because qualitative data, meaning text data and narrative data spoken or written has been generally ignored in favor of quantified data.

Whether that be test scores or ratings provided by interviewers or whatnot. And it provides a real potential opportunity to make some major strides for illustration. I think we can and have some research to show that we can improve the prediction of job performance beyond our normal measures and we might be able to reduce adverse impact meaning statistical differences in passing rates between race, racial and gender subgroups. And as you know, that's been a hard nut to crack cuz usually the more valid section systems are, the more they measure everything and they also pick up group differences. So if there are any group differences, they measure those more as well. And so you got this dilemma between wanting to do the best prediction but wanting not to have too much adverse impact. And so I think this offers a first time when we've had a real breakthrough. Yeah,

Dr. Charles Handler:

It's a trade off, it's all trade-offs, right? And so you gotta trade that fairness for the accuracy. But boy, like you said, if we could find something that's able to do both with fewer compromises, that would be really good. The other area, and I'd love to dig into what tools are behind what you're just talking about, but I wanna surface kind of my thought about where I would like to see this stuff really having the most impact. To me, the killer app as they used to say, is the ability for an AI to be able to rate complex behaviors, be it a simulation or an assessment center or something like that without just to equal human judgment or maybe even exceed human judgment in complex things that have to synthesize a bunch of information to give a score or an evaluation. And there's definitely companies working on that.

I've seen some really interesting things. So that's the other side of the fence. One side of the fence we see a lot of AI is let's look at a job description, let's look at some information and say these are the skills you need for a job. This is how well someone matches a job. The other side of the fence is, well could we use AI to evaluate selection inputs or stimuli, whatever you'd call 'em. Predictors evaluate complicated predictors. So it'll be interesting to hear if you've done anything with that. I'd like to hear from you when you say, you know, found some breakthroughs and some really good results with predicting performance. What are the tools, what are the assessment tools that are being used in that paradigm?

Mike Campion:

Well it seems to me like most of the action these days are thinking, well what's the best way to classify this? But they're in several categories. One is scoring of the entire application. The applications that people submit may include a resume, be completely unstructured in that way, or they will have an application that's a combination of education, work experience, some things maybe you can quantify. And then textual information like past jobs, past job duties, special skills and things. So the value of artificial intelligence and the most common application currently is scoring the entire application to get a score that allows the pre-selection of candidates from oh wow, the large candidate pool. And historically that has been done by a recruiter usually poorly cuz a recruiter is kind of skimming through resumes looking for keywords. So that's an obvious application and one with very high volumes, fairly refined procedures in my experience that is the most common current application. The next application, we've not seen too much research yet, but it's obvious. And that's going to be scoring interview answers in other, what we call constructed responses where you would ask candidates questions, maybe a simulation of some kind even online. And they respond narratively you score that narrative information. So the interview and some kind of a exercise that collects unstructured responses like verbal responses, that's probably the next big application. There are some other things Bill, lemme stop there.

Dr. Charles Handler:

Yeah, well there's a lot of companies that are doing both of those things. I don't know how well they're doing 'em or how backed they are with any research. So I'm curious about this. So many great questions here in the deference to the idea of garbage in, garbage out. So when you're scoring those things, you have to have some kind of lexicon or some kind of understanding and what the various words that all might be pointing to the same construct or the same piece of information. So there's many labels we can use for a skill or a competency or whatever. So how are you training or just making sure that whatever is interpreting the resume and the application process truly understands how to sort that information so that it's clustering and around real things that are happening. Does that make...

Mike Campion:

Sense? Yeah. And the process appears to be in most situations is kind of a two-step process. First, the software allows you to extract text variables. Now the text variables may be single words like leadership or they can make multiple words like team leadership or leadership in school. And so the first step is to extract all these various terms and combinations of terms. And as you can imagine with the concept of leadership, you can have leadership show up in many contexts, leadership and vol. And thing about this software is, it can check words that are sort of close by. So it doesn't have to be connected words, but leadership in school, leadership, volunteer organizations, leadership in the military, senior leadership, you know name it. There can be potentially hundreds of ways of describing leadership that a candidate might exhibit. And then you have all these variables and you usually need what we call a labeled data.

Basically what you and I would call a criterion against which to select the variables that predict. So that's called supervise machine learning because you are influencing the machine learning model by how well it predicts other things. So just an illustration, you might have an interview question that you extract concepts from it or terms they go by different labels, you might call 'em features, you might call 'em concepts, you might call 'em categories Anyway, you might extract several hundred and then find out that some subset of that are predictive and then that becomes your predictive model. And then candidates who use those phrases get higher scores. And so it isn't like the machine is thinking it's what computers do well, it scores everything and it scores everything in a very thorough and objective way. And so even though the computer doesn't understand what it's doing, it can predict human scores as well as humans can predict human scores.

So that correlation between the computer scores and human ratings is the same as the correlation between human ratings. And I think it's really the computer doesn't understand the nuances but the computer doesn't miss anything. Right? And so unlike people, the computer doesn't miss anything and so it scores everything. And I'm just wondering when you're listening to an interview answer in scoring leadership, if you hear the candidate talk a bunch about leadership, you give them a higher rating and that's kind of what the computer is doing now. It's not somewhere in the middle that the person says, well I'm not a good leader. And you know would hear that if you were in person and the computer won't hear that, right? But that never happens. So there you can think of extreme examples but it just never happens. The truth is the way the computer goes at it, which is a thousand little teeny hammers seems to work even though it doesn't really understand. But that's not a problem because it works just as well and it doesn't make mistakes and it doesn't have bias, it can't discriminate. So

Dr. Charles Handler:

Interesting. So I just thought of <laugh> when you said that I remember it, it's not so much anymore, but I remember even 20 years ago when ATSs were apple contracting systems first coming out, people would say what you do is you know, take a white font and you write all these skills in the background as a waterfall.

Mike Campion:

We all did that.

Dr. Charles Handler:

Yeah. So I picture somebody in a job interview going leadership, leadership, leadership, leadership, leadership, right? So that if you're saying the computers picking up on how many times people say those words, but it doesn't necessarily like you could use the words but use 'em in a way that's not correct or not really showing indicative performance. I guess that's where the criterion and the comparing it to the criterion would probably help you make those decisions or the model to show that just cuz you're tracking everything doesn't mean that everything is related to performance. So

Mike Campion:

It seems as though when something is new and people don't understand it, they can conceive of some wild idea where the computer could misre something. But it just simply doesn't happen because first of all, when you fill out an application, if you put a bunch of idiotic things like list leadership 50 times in your application, they're going to see that and you know will not be brought in. Or if you're brought in you'll be not hired. Second of all is efforts to try to see if you can fake good luck to you. I bet. I bet you, I bet if you show somebody the algorithm and say go ahead and try to fake it, they don't do it very effectively. So we can imagine for some odd circumstance where something can go awry but it simply doesn't happen because there are self-correcting forces as I mentioned, plus people. It isn't that sensitive to that sort of manipulation. So I think it's a complete false alarm, but when you don't understand something it's easy to think that the world's flat look out the window. The world is obviously flat. Are you an idiot? Yeah. So anyway, I feel sorry for people but that's the way human nature is and maybe if we hadn't, we'd have been eaten by the animals years ago. So it's good that we're suspicious about everything

Dr. Charles Handler:

For oh for sure. Caution is important, especially in today's world then again fortune, the bowl. So you gotta balance those things out. So from a bias perspective though, I'm starting to think are there any cultural differences in the words people use and how do you make when they're say writing out a resume or filling out an application or even in an interview situation, how, cuz people would say, well and you hear it all the time, machine learning, ai, it's bias. It's bias, it's bias. There are places where it introduces bias. I could talk about that all day long, but it's not a universal slinger of bias, stamping bias on everything. But in the paradigm you're talking about, you mentioned it doesn't have bias. So how do you control for differences in how people might describe or interpret things if they're from another culture say

Mike Campion:

Yeah, that has not been an issue with me and my clients. I have probably implemented a dozen systems myself, but it has with the companies I advise. So a big company, well known everyday kind of company that sometimes even has gotten some negative press around this topic. I do advisement for them. So their legal staff asks me to review their models as an outside reviewer and they are role models, not the bad guys, they're the bad guys cuz they're big and people wanna go after 'em to earn some money by exploiting the legal system. But they do lyes of it. And that the best analyses I have seen focus on the translation of voice response to text. Cause he got turn it into text for the computer to read it and they found or not found, but their study was looking at that topic and they did not find much by African American or Hispanic.

But they did find some for Asian candidates of which this organization has a lot cuz this tool they were using was used internationally. And so the study was all about how much of a influence that had in terms of creating mistranslations where the text of voice to text what was not, how many errors it created. So sort of an error rate per a hundred words or per a thousand. And that's like the best I've seen. And those particular candidates in this context scored quite well cause of content reasons. So the fact that they had a greater number of language related problems was not viewed as a major issue because it did not create any statistical disparities to speak of. And quite honestly, English skills were a job requirement. Yeah,

Dr. Charles Handler:

There you go. So well let's talk a little bit about, when we talk about building these models, you're looking empirically at a set of predictors and some criterion and you're finding kind of the best fitting line that predicts. But what about what we call generalizability of those models. So if you build one of those models off of one sample and you may even cross validate, which is always good, you hold out sample and you see if the model holds. But in these big corporations they may be using this model in different geographies, they may be using it for three years and things change over time. So how are you controlling for the stability of the model? So it's not kind of a one snapshot in time that may not be as dynamic as you need to be able to count for randomness and change and other factors.

Mike Campion:

So a couple of answers two in particular come to mind. One is this hasn't been researched. We just finished a special issue in personnel psychology on the topic of machine learning, artificial intelligence applications to personnel selection. And that'll come out next year. But we had large number of submissions. So one of the articles we accepted is on what the authors called algorithmic construct generalizability, which between you and I, that'd be like alternate foreigns tests. So if you use different prompts for the open-ended question, tell me again about your leadership skills versus five different versions of that question. Can the same computer model predict across those versions? And they found quite good prediction to novel prompts, especially when the assessments were the same type of assessment, like past behavior interview questions compared to other past behavior interview questions. And when the data were seated, a little bit seated basically means when you created the alternative questions, you ensured some of the words from the primary question are still included in that.

Which by the way, that's a whole nother world. You can have the computer create questions for you that psychometrically similar to each other. Yeah. So what answer is these models will generalize across similar questions. Just like a rating scale in a interview might generalize across a range of questions on a given topic. The second is will these models degrade over time? That has not been something that has been studied widely. This computer models that we developed, we developed 'em in 13 and implemented 'em in 14. We run 'em three times a year. So whatever that's been three times to the number. So we've run 'em mid twenties now and we're just rebuilding 'em now. And that's for a major client of mine. And we rebuilt it once, we didn't extract the concepts or variables, we just up the revise the regression equation that put it all together. We started out with 0.6 correlations over about a dozen years. It degraded to like the mid five correlations and then when we rebuilt it, it went back to the six. Geez. And then now we're rebuilding it again just to see if we can do better. And I don't have anything new to report on that other than it looks promising. Well

Dr. Charles Handler:

That's super.

Mike Campion:

So I think they're pretty stable is the answer to the question.

Dr. Charles Handler:

That's superhero stuff, pulling correlations of 0.6 and criterion validity pretty darn good? No,

Mike Campion:

No. What we're doing is we're predicting the human ratings that's not talking.

Dr. Charles Handler:

Oh, gotcha. I was going to say I thought you're predicting performance with the 0.6.

Mike Campion:

No, no, no. But the goal is usually to predict as well as a human. And then a separate question is does it predict job performance?

Dr. Charles Handler:

Ah, right. Gotcha. Interesting. And so are you doing this for, are these models for one particular job or are they for more of a general kind of thing? So I guess, and this is important because people who are listening out here too, there's so many applications of the tools and techniques we're talking about here that I immediately go to, oh you're predicting performance with this stuff. And that may be true, but you're also as predicting again, how well does this match what a human would do? So that's another dimension of this stuff that we're talking about today. And I think it's important to understand that it gets infinitely more complex by all the different applications that you can use with these tools. And each one of those takes a lot of research and a lot of planning and coding and all that stuff. So I guess back to my question, is what you're doing related to one particular job or is it a job family or more general?

Mike Campion:

Yeah, that particular application I was talking about was five occupational groups in one big organization. So was, if you can imagine this organization has five major areas of management and this was hiring management level personnel for these five major areas of their business and which is all of them in their business, but they hire centrally. So all the hiring goes through the same

Dr. Charles Handler:

Right

Mike Campion:

Process. And so that's why we were able to apply to all of them. But it does, as I say, we're predicting the scores of human judges, but we're not predicting actual job performance.

Dr. Charles Handler:

Right, right. So the human judges are using say structured interview guides. Are they competency based kind of things obviously.

Mike Campion:

Yeah. And that's a really important point. These models are only as good as the data they're trained upon. Yes. You said garbage in, garbage out before. Well in machine learning they like to call it landfill in and landfill out cause they <laugh>. Oh nice. They deal with so much more data. But it's the same thing here. So if your organization has historically done a poor job of hiring of a pre-screening candidates and you train your model against that, then it really limits you. So this particular organization was a model application cuz they really pay attention to it for every candidate they evaluate by three assessors who are full-time and they have about 40 or 50 full-time assessors who evaluate thousands of candidates. And they have a highly refined scoring rubric like anchored rating scales and they make a bunch of judgements. Each one of 'em spends about 20 minutes per applicant. So we have three assessors for each applicant spending a total of one hour 20 minutes each. And so their data are highly reliable. They correlated at about a 0.6 in their ag aggregate reliability in the eights.

Dr. Charles Handler:

So yeah, that's really interesting. If you think about the roi, you just said what 40 people working, I mean if you can have a machine do that and have those people hopefully repurpose to do other <laugh>, other things, right? Then boy, think about the the roi, the savings to the organization of that. That's incredible. That's millions and millions of dollars.

Mike Campion:

That'd be a good example. We only use it to select out the poorest scoring candidates. So we save about 200, a little over 200,000 a year. Yeah,

Dr. Charles Handler:

Gotcha.

Mike Campion:

We spend 40,000 in developing the model and we've been running the model for 15, 17 years now. Oh wow. So it's really, I've been a big payoff and well worth it. But I work with this organization as their inside guy. So I don't know if you were to hire a consulting firm to do it for you and they charge you on a per candidate basis whether it would be quite as favorable, but I don't know that it wouldn't either. Right. I just don't know. Sure.

Dr. Charles Handler:

Well the big thing too, you're coming at this from a very consulting focused mindset, which I have had that myself being in consulting for 20 some years. But think about it from a product mindset. There's a lot of companies out there, tons of them IO type companies that are actually doing this responsibly. Some companies that just say they're using AI and God knows what they're doing. But think about it from a productization standpoint, you can't always go in and do the wonderful methodologically solid work that someone like yourself does. A lot of times you just try to automate this thing and say we have this product and we can turn this product on say every resume that you get and then this is going to give you a fit score and going to say this person fits with this job better than these other people.

That's the use case that people in talent acquisition and you go to HR technology show and that that's the kind of stuff that people are trying to do or saying they're doing and are actually doing. And some of that involves IO psychologists, but some of it's just pure data science with no real deference to the fact that we're measuring individual differences here. And you have to understand those. So to what extent have you worked with something that's a little bit more like what I'm talking about here? Maybe it's a predictive model that scans resumes. I mean interview once you convert something to text and that text is all categorized and built, trained and all that kind of stuff. I'm probably butchering the words the vocabulary, but at that point it doesn't matter if it's a resume or an interview or what have you, right? It's taking that voice, turning it to text, taking that text and interpreting it. But have you had experience with predictive tools that just look at this text and can then say, hire this person instead of that person?

Mike Campion:

I'd be real suspicious of those. There are some around that are generically scoring capabilities. They work to some extent they work as well as a lot of the procedures we use in our science now. Hope, I'm not stepping on anybody's toes, but a lot of the personality tests and they work a lot better than personality tests, but they work as well as they could work if they were developed well, using your own data and having a good criterion against which to create a model for your organization. No, but I do actually think that they can improve over your current situation pretty much. For sure. I would emphasize the point you just mentioned, companies in old due deference, they're like 50% entrepreneur, they're 40% it in their 10% content knowledge. So they know the least about personnel selection. They're mostly IT entrepreneurs who are trying to make a buck, they don't really understand the nuances. So for example, they will, if you say I don't want any adverse impact, they will just simply fix it. And that's illegal as you know, can't just adjust the scores. No, but they don't know that and bless your hardware, they don't know our professional, we don't know theirs very well. So really if you pick a vendor, make sure their vendors are well advised by having people on their staff who understand selection and the legal issues

Dr. Charles Handler:

For sure. And just that if you can do a local job analysis, local validation, you can use that to generate some automated tools. We do that all the time. And talking about application blanks, I mean and weighted application blanks are always been a really good predictor when you're able to get in there and calibrate them to what combination is most predictive. But you also have to have some rationality. If your shoe size or your favorite condiment choice is predictive, well you don't really wanna be paying attention to that because it's either an artifact or it's, it's not job related so you don't want to use it anyway I guess. So there's a lot of complexities to it, but at the end of the day, getting into the local place and that's why I always encourage people who are putting selection systems in place or just nurturing and using a selection system over time to take that extra step and do a local validation study where you can really see what's happening and make adjustments to what's happening. And I'd say in my experience, most large enterprise companies who are using assessment at scale typically do that kind of stuff. They typically don't just buy something off the shelf and stuff it in there. And so that's a good thing because when we have a chance to do that, whether it's using an AI tool, cuz you can do a validation study with an AI-based tool just as much as you could with anything else. So it it's not precluding anything.

Mike Campion:

The best companies do exactly that. They try to evaluate things. I'd love HR as a profession, I love HR people but anyone else they'd just to buy a solution and just use it and just make the decision, move on to the next problem. It's kind of a sell to encourage them that really you need to do some research here. In fact, in that way I'm almost afraid to say it, but the legal staff has helped us encourage the HR staff to take those extra steps because hundred

Dr. Charles Handler:

Percent.

Mike Campion:

Cause they already see the liabilities and even though attorneys often slow progress down, sometimes they can be helpful. And this is one where it seems to me that well we need, I wanna tell a story if I can, just a really brief one. You are not old enough to remember when lie detector tests were just invented. Initially everybody was IED

Dr. Charles Handler :

<laugh> not invented, but I did police hiring even in the nineties, know

Mike Campion:

All about

Dr. Charles Handler:

It. We had to use a polygraph in the mid nineties. I was just going, oh my god, what are we

Mike Campion:

Doing? Yeah, well anyway, sorry interrupt. Society took society didn't trust it and they took away that tool. We'll never know if it'll work if it works or not cuz it's illegal to use them. And so the same could happen here,

Dr. Charles Handler:

But for jobs that sensitive law enforcement, you could use it because the Polygraph Protection Act was way before the mid nineties anyway.

Mike Campion:

Yep, yep. Yeah you could. And that's because Congress, they knew what they didn't know and so they allowed it. But the point I'm trying to drive is public sentiment raised up too quickly and took a tool away and now we'll never know. The same could happen with artificial intelligence. People get all worried about it before they understand it. And that has really bothered me. It's like we don't seek to understand first and that is not, scientists are supposed to be right. And so the reception publicly is maybe understandable, but the reception by the profession where they're doubting first seems to me to be. Yeah. The second thing they should do, they should first make sure they understand what they're talking about, which almost none of them do, honestly. They don't look into it in deeply enough and they can get it. This stuff is not that complicated.

Dr. Charles Handler:

I mean the concepts for sure. So that's a really good segue too cuz as we we're running up on our time here, what is the thing that you're most excited about in the future in this realm? What do you think is really going to be something to behold over time?

Mike Campion:

Well I, I've already kind of showed you my hand. I think use of artificial intelligence to try to solve the validity adverse impact dilemma. I think that's a big one. So I think we'll find that we can score information in a way that draws out skills and abilities based on text data that will likely reduce the amount of subgroup, race and gender differences and at the same time improve our prediction job performance. So I think that's a big one. I think using statistical manipulations to try to make the adverse impact go a away mathematically are going to be a dead end. But there are a number of things that have been tried. This special issue, we have a bunch of papers there and basically anything you do to reduce the adverse impact in introduces bias itself, it must statistically introduce bias. And that actually turns out hurting the high performing minorities completely.

You would never expect it, right? So expecting machine learning just to make it go away is not going to work. But there are lots of other applications where we're going to see this. You can have, let's say personality, it's fundamentally flawed by self-report. So people are not going to be honest with you when the data are used for hiring. But if there was some way to get at their personality where they cannot fake it, then there might be hope. And so using the way people talk and the expressions they use and the terms they use reflects some amount of personality like personal IT traits like persistence and enthusiasm and empathy and actually conscientiousness and agreeableness. And you can pick up on those things based on the words people use. So actually I think there will be uses there. So your application and other information, your interview answers will be scored in part by the personality traits that you present through the choice of your words. Yeah, I think that's kinda interesting. Yeah,

Dr. Charles Handler:

That's happening already. It's, that's being productized quite a bit. Again, it's such a range. Just cuz you're doing it doesn't mean you're doing it. Right. Who knows? But there you go. Well, great, thank you so much. And I always ask our guests as we close out to let everyone know how they can follow and find you. I will just say, all you gotta do is Google this gentleman's name and you'll probably find 400 publications, textbook, all kinds of stuff. So you're not hard to find, there's a lot of good wisdom from you. But on a basis, and especially we have a lot good mixture of listeners who are IO psychologists and not, so if someone's maybe not as steeped in the depths of what we're doing as a profession, are there things that lay people come, lay people can access where they can keep in touch with the things you're talking about?

Mike Campion:

So many things. You can't rely on the news. The news is entertainment and it basically publishes extreme things. So you gotta be a wise consumer. So don't think that just Googling artificial intelligence is going to help you much that most likely will mislead you. Yeah. Yeah. So this stuff is, I know it's it is really not that complicated if you would just take the time to learn about it and there's many accessible sources, but they're not as fun as doing a click on your phone and get in 30 seconds. Yeah. Wanna understand this stuff. It's going to take something deeper than that. And I'm sorry, I mean that's the same with everything else. Your health, finances, the home improvement, pick, anything else. They all take a little bit of investment and we shouldn't feel like all knowledge is available in 30 seconds on our phone. So Good.

Dr. Charles Handler:

Good point. I like it. We're talking about high tech, low tech and everything, so thank you so much. I really appreciate it. It's been real fun and look forward to seeing you around the block as they say.

Mike Campion:

All right, P, thanks for having me. We'll talk to you soon.

Dr. Charles Handler:

Yeah. Science 4-Hire is brought to you by Sova Assessment Recruiting Software powered by Science . Sova's Unified Talent Assessment Platform combines leading-edge science with smart, flexible technology to make hiring smarter and easier.
Visit sovaassessment.com to find out how your organization can provide an exceptional candidate experience while building a diverse, high performing and future ready workforce.