Katryn Wright & James Elfer - ‘But is it actually working?’

[00:00:00] Sarah Abramson: Speak to the Human is a podcast that explores how we build connections with people in their professional work. It's about the human experience at work, and about how to foster that connection and belonging to support people and their organisations to flourish. I'm your host, Sarah Abramson, and I'm looking forward to you joining me in hearing from our brilliant guests.

[00:00:25] For this episode, I'm joined by James Elfer and Katryn Wright from Behavioural Science Practice More Than Now. We explore what it means to take an experimental approach in an organisation. So when it comes to any business challenge, whether that's around leadership, change, recruitment, ethics, inclusion, or anything else, how do you know how to tackle it?

[00:00:46] What behaviors should you target? And how do you know if it's working? How might you measure and evaluate? For example, are leadership retreats really about instilling lasting changes in how leaders engage with their teams, or are they more about making a splash and doing [00:01:00] something impressive but which doesn't really change the behaviours of leaders?

[00:01:04] Or how often are simple interventions being overlooked, even if they're more effective than a bigger ticket approach? And, is anyone asking the questions like this and taking a really rigorous approach to measure and analyse impact? James and Katryn bring expert insight and a wealth of examples to this topic, and we dig into some interesting ideas such as how to measure the absence of a problem, and the fresh start effect for picking a good time to prompt change.

[00:01:32] I hope you enjoy our conversation. As always, please do drop me a line with your thoughts, feedback, and suggestions for future guests.

[00:01:45] Hi, I'm joined today by James Elfer and Katryn Wright from More Than Now. It's great to have you both with us. Hi.

[00:01:52] Katryn Wright & James Elfer: Great to be here. Thank you.

[00:01:53] Sarah Abramson: So to get us started, it would be really good to hear from both of you a bit about your background and what got you into [00:02:00] behavioral science, which is the area that you're working in.

[00:02:02] Katryn Wright: Yeah, sure. I'm happy to go first. I, my name is Katryn and I spent 10 years working in the business and human rights world, which is sort of at the intersection of corporate responsibility, human rights, labor rights, corporate power. Um, and I was really interested in the impact of organisations on people's lives, whether that was workers in supply chains, communities around mine site, um, consumers and users of products and tech products from a privacy perspective.

[00:02:31] And And I worked with really large organisations like, um, Coca Cola, like Microsoft, to understand how do we translate big UN standards into practice. And through that, I became obsessed with two aspects of the work. One was really behavior. Like what is it that people need to really practically do differently within those organisations to ensure that we have those, um, positive impacts on people and [00:03:00] we avoid negative impacts on people.

[00:03:02] But also how do we measure it? So a common question in the business human rights space is always, how do you measure the absence of something? How do you measure the absence of the human rights or a labor rights impact not happening? And obviously you can't do that, right? But through a focus on behavior and the data that we can collect on all behaviors changing that are relevant to those outcomes, um, we can understand that more, so I became really interested in that, um, retrained as a behavioral Now, working with more than now with organisations, I lead our social impact practice means that my work focuses on human rights.

[00:03:42] Also diversity and inclusion and health equity increasingly. Um, and I'm sure we'll go a little bit more into detail about what that means we do from a research perspective. So that's me, James, how did you get into this?

[00:03:53] James Elfer: Uh, I also had 10 years before getting into behavioral science. So I, I started out in this sort of HR space.[00:04:00]

[00:04:00] Um, so doing a few different bits and bobs, um, I got into technology organisations. So. I had short stints in HR at Google, short stint in HR at Sapient as well, and then kind of stumbled more into the consulting side of things. Um, and at the end of that period, I just started to become increasingly disillusioned with the type of work that I was doing.

[00:04:22] So the project that I always quote is One huge company had bought another enormous company, and we had been commissioned to do a big leadership event in Paris, and we'd kind of booked out this glamorous glass dome in the Parc Floral, and there were peacocks outside, and we'd hired a NASA astronaut to do the keynote speech.

[00:04:42] You know, he was saying how precious the Earth looked, and that's why. The big organisational purpose is so important and what, what I was really struck by because I was sort of getting into behavioral science and getting into some of the books at this stage, was everyone was so pleased with how things were going, everyone was so pleased with the event, and I couldn't even ask the most basic [00:05:00] questions, like why were we doing this?

[00:05:01] What was the point of this event? How are we measuring its effectiveness? Was anything going to happen afterward? And I kind of thought, oh, I can't. I can't keep going in this industry in this way. So, um, yeah, similar to Katrin, I also went back to school, um, to retrain in behavioral science and the thing that captivated me particularly while I was doing that was this idea of evaluation and measurement.

[00:05:26] And, and that's what took me into, uh, Founding more than now and, uh, and focusing on that for the last now, eight years, so since 2000.

[00:05:34] Sarah Abramson: Interesting. So what was your vision for more than now? How did you get it started? You founded it yourself. So yeah,

[00:05:41] James Elfer: just experiments really. So like I, I was really struck by, and maybe it was because I've had this first 10 years that when I was doing that course and again, things have moved on now, but it was sort of 2016.

[00:05:53] The thing that everyone was talking about in the outside world was, you know, lists of biases and heuristics. Um, thinking about [00:06:00] call nudges that worked in some instances. But the thing that really captured my attention was this idea of causality and how you could get to it with randomized controlled experimentation.

[00:06:10] And it was really the first time I'd ever heard of those concepts at all, especially applied to human behavior. So I immediately thought, you know, wow, you. There is actually the potential to evaluate the things we're doing and seeing if they work. Um, and the other thing that really excited me about that is I'd never really seen it in the space of HR and the workplace and culture change.

[00:06:32] So I wondered, could, could that happen, right? And that was what, that was what more than now, came from just the idea of whether we could measure stuff properly.

[00:06:42] Sarah Abramson: So tell us a bit about the types of projects that you do at More Than Now to bring that to life.

[00:06:47] Katryn Wright: Yeah, sure. So organisations come to us with very real business challenges.

[00:06:52] So in the world, um, that I need at More Than Now, that might be around diversity and inclusion or health equity and [00:07:00] human rights. Um, and then we really try to do two things. One is to understand what do we need people to do to tackle the big challenges. And we do that by breaking it down into really sort of small concrete behaviors that need to be changed or colorless and build up to address the larger challenge.

[00:07:19] And then the second piece around measurement. So in the diversity and inclusion space, we will do research around, um, aspects such as how do we influence hiring decisions? How do we de bias performance ratings? In the human rights space, we're doing some interesting work around how do we get suppliers to be more honest in their risk disclosure to, um, to their customers?

[00:07:43] How do we understand what motivates workers to raise grievances and to what extent does anonymity really matter in that space? Um, in the health equity space at the moment, we're doing some work with big pharma companies around how do we get underrepresented population to participate [00:08:00] in genomic research or clinical trials.

[00:08:03] Um, and then, yeah, we really do two things. The first is a, a real sort of hyper focused on behavior. Um, particularly because we know that people say they do versus what people do, do in reality can often be different, right? And then the second is that piece around experimentation and measurement. So, as James said, we, we run our work like clinical trials, you know, we know that in this space, there are challenges with before, after measurement, right?

[00:08:33] So. I'd roll out an intervention if I do a pre post. Is it because my intervention has caused that, or is it because the underlying structure has changed in the economic environment, or people's home lives, or people's situations? Um, so the thing that we always hear is organisations saying retention is up or accident for down compared to last year, or would that have just happened anyway?

[00:08:59] Is it [00:09:00] really because of what you've done or not? Um, so James said, like, we're really interested in building evidence around if our treatment better than our control, better than our business as usual, better than what we've been doing before. And then the second really interesting thing is that change experts at NEVD, whether you're a behavioral scientist or not.

[00:09:18] You're always making judgment calls, right? Should I do X or Y? Is it better if I talk about my intervention in this way or that way? Is X intervention better than Y intervention? And by doing randomized control trial experiments, we get to test those judgment calls. Um, so some of the work that we've been doing recently has been looking at when organisations go out with their employee value proposition and why organisations should, why individuals should apply to work organisations.

[00:09:48] There are obviously lots of different ways that organisations can disrupt benefits of working there to a potential prospective employee. Um, they may or may not perform better certain messages over [00:10:00] others, and they may or may not attract certain demographics over different demographics. So we've been exploring those sorts of questions through randomized control trials to help organisations to address those sorts of challenges.

[00:10:11] James, what would you add to that?

[00:10:12] James Elfer: I don't know. That was a pretty exhaustive list. I think, I think that, uh, that's good for now. Maybe we can get into some of those examples.

[00:10:19] Sarah Abramson: Yeah, well, I was just going to ask first, just to unpack the bit about identifying specific behaviors of our first part of the approach that you talked about, um, which I think can be, can be tricky, particularly in terms of how people think, um, you know, leaders within organisations, whether they think in terms of specific behaviors, how do you get to what those behaviors should be?

[00:10:41] James Elfer: That's probably the hardest bit of our work, I would say. Um, because you're right, like intuitively, we don't really think in those precise terms, especially, and you brought up the example of leadership. So let's, let's go down that road. When organisations come to us, they'll say, we want our leaders to be more inclusive, or we want our leaders to be [00:11:00] better communicated.

[00:11:01] And the sort of intuitive assumption is that by telling them that we want you to be more inclusive or by taking them away on a three day course and teaching them how to be more inclusive, they will become more inclusive. And actually the evidence, uh, from a lot of behavioral science research, maybe, um, would suggest that that might not be the case or certainly it might be really easy for those leaders to have the best of intentions within those courses.

[00:11:26] But when they go back to that day job, just fall back into the same habit. And actually it might be more, um, effective if we can really pin down exactly what we mean by that. That can be quite hard because leaders want to think big, right? They want to have these like grand ideas and actually to be able to say, Well, maybe inclusive leadership just means running your next one to one meeting in a different way.

[00:11:50] Or maybe it means starting a project review in a slightly different way rather than kind of giving your opinion first, for instance. And obviously there might be tens, [00:12:00] maybe even hundreds of those small behaviors that we might want to think about, but we really do have to pin them all down. So when we start talking in that way, You tend to split people into two groups.

[00:12:09] Either the ones that kind of realize that the approach that we're talking about might be more effective, but it also sounds like a lot of work and kind of back away. Or organisations who say we actually really get that, let's focus on one, two, three meaningful behaviours, see what we learn and then move on after that.

[00:12:28] Um, but it is relatively easy to pin those behaviours down, it's just that we don't normally think about doing it in that way and it challenges. The kind of the big leadership workshops, training programs, inspiration pipe star. That at the moment is sort of 95 percent of culture and change. Yeah,

[00:12:44] Sarah Abramson: definitely.

[00:12:45] And that's kind of how organisations I think are often used to approaching and thinking about those things and commissioning them, frankly. Um, do you use any sort of frameworks for getting at what those behaviors should be or, um, for helping to roll them out? So I'm [00:13:00] thinking, you know, there are frameworks like from the behavioral insights team of the East framework.

[00:13:05] Um, Where it's about easy, what are they? Easy, accessible, social and timely, that kind of thing. Do you use that, that, anything like that to identify and roll out behaviors?

[00:13:16] James Elfer: Yeah, we do. I mean, we, we always have those sorts of frameworks in mind. I don't think we do it in as formal a way, probably as, um, as other consultancies.

[00:13:24] And part of that is because I have a personal opinion that so much comes from the people that we're working with. So one of the golden rules, of course, of behavioral science is this idea of context, and that's it. That different interventions might work differently, depending on the organisation you're working for, the situation you're in, the leadership group that you're focusing on, and there is so much expertise that exists within HR teams.

[00:13:48] So in a way, I want, uh, us at More Than Now to kind of rock up with a few frameworks in our mind, and of course, whatever literature review that we've done on that particular problem, but we want to be fairly humble about [00:14:00] that and interact that with the expertise of the HR team and the leaders that are involved.

[00:14:04] Because, of course. They are expert in their context, in their culture, and those things come together in a bit of a co design. The idea that we can use frameworks to make very accurate predictions and then not test the hypotheses, I think, is one that exists. In a lot of corporate behavioral science and is one that, that we don't subscribe to.

[00:14:27] Sarah Abramson: Hmm. Interesting. Yeah. And, and Catherine, how does, um, the, the spaces that you've worked in sort of in human rights in particular and those, um, diversity and inclusion spaces, are there elements of that kind of cultural context that really come to the fore when you're working in those spaces?

[00:14:47] Katryn Wright: Yeah, definitely.

[00:14:48] And I think that one of the big things that we see, as James was alluding to, was the idea that sort of big problems require really big solutions. And so, [00:15:00] when we're thinking really small, we're also thinking about timely moments within organisations. And so, within those cultural contacts, or just those very process oriented contacts, We see specific moments where if we intervene, we can really change things from a diversity and inclusion perspective or a human rights perspective.

[00:15:22] So in the diversity space, one of our, um, quite successful pieces of research for the last couple of years have been around, um, raising women's leadership aspirations. So we were working with a large telecom company and Um, there was this challenge with women not, uh, not, not sort of aspiring to leadership management positions at the same, uh, rate that they would have hoped.

[00:15:48] And so, you know, again, right, a big, a big solution to that problem would be women's leadership retreat, development program, away days, a really small [00:16:00] focused, um, behaviorally informed intervention would be around In performance review, uh, stations, when women are asked, do you want to, do you want to lead shift position?

[00:16:14] Do you want to be in a management position over the next 12 to 18 months? Just to say, yeah. Right. And that, that then sparks a conversation with your line manager around actually, how do we get you there? And so that was what we did. It was really simple. We intervened in that process. We told women about the company's commitment to diversity.

[00:16:35] And that was, that was one of our treatments. We called it the sort of diversity target treatment. And then our second intervention, cause we also, you know, we like testing multiple different, if you say, judgment calls around what are the best ways to, um, change that behavior. We also tested the target plus expressing an incentive around it.

[00:16:53] So the idea that, um, the, the board and the senior leadership were financially incentivized [00:17:00] to do so by having the achievement of this target linked to their, uh, remuneration packages. And yeah, fascinating results. We found that it worked. We found that in particular, the in front of condition was for women that had been in the organisation for a long time, which we think is interesting and interesting indication that women need to know that the company is putting its money where its mouth is right.

[00:17:23] And that there's some real skin in the game to achieve those targets. So that'd be an example where we really looked at compact, really focused on a small behavior. And we really focus on a timely moment where we can get things in a meaningful way.

[00:17:37] Sarah Abramson: I think that speaks to something I'm really interested in, which is how people can make the business case.

[00:17:43] So somebody that wants to roll something out within their organisation, how do they make the case for doing it, particularly if it's doing it differently? And I wonder if, Does it feel a little bit like, you know, sometimes organisations need to do something because it's more visible. So maybe having [00:18:00] that three day retreat looks a bit better, or they can tell a bigger story about it.

[00:18:06] Um, but trying to help them shift to, you know, okay, there is this. It feels small intervention that you're doing in, in people's review, but it's more effective. Is there something around that about helping them to surface the business case? And I guess that gets to your evidence based approach too. Yeah.

[00:18:24] James Elfer: I mean, we always have to be careful that the goal that someone comes to us with, they actually care about, which is not certainly not always the case.

[00:18:33] And I think that's kind of what you're alluding to. So sometimes Let's take the example of well being. I'm not a cynic. Like, I'm sure that lots of people really care about well being. They really want their employees to be, um, uh, have better mental health, for instance. But it might be that, as a head of well being, what you're really, um, expected of you and what you are kind of remunerated on in terms of a bonus is a whole sequence of calendar events.

[00:18:59] And it's [00:19:00] making the organisation look really great. And that's what people expect, and that's how you're going to be judged. So when we come along and say, okay, but none of those things might be effective. It might be that we're just not the partners for you. Experimentation certainly isn't for you because what you really need.

[00:19:15] Is a comms agency or an events agency and they will come and do that work for you and then they will measure it and I think this is really telling by asking people whether they liked it or not, like it's a kind of a, it's a PR mechanism and in those instances, we're never going to be right. There might be other people who say, actually, we really have done that type of work before or we do have some measures that say there are legitimate mental health concerns within our organisation and we really do want to get after that.

[00:19:42] We're just not sure how to go about it. That is the moment when someone's willing to go into a sort of exploratory process, more of a scientific process, where we might have to come in. Um, but I think you're, it's a great question because the experiment that Katrin just talked about kind of articulate, articulates it [00:20:00] perfectly.

[00:20:00] Because the answer is boring, right? We want to get more women into leadership. It's amazing to have these, uh, leadership development events. Like it feels really good to do large mentoring programs. It's uncomfortable to ask about the effectiveness of those things. And when someone comes and says, well, what should we do instead?

[00:20:18] And we say, Well, when you actually ask people what their career aspirations are during the performance review, why don't you just do something then and there, and actually just telling people, you know, what, we're really supportive of more gender diversity in our leadership team. And guess what? Our board are remunerated for the app causes this big spike in how many women in your organisation actually do that.

[00:20:41] But in a, in a sense, I think it too easy, right? It feels, feels like. It's not the answer to this big problem. I mean, in actual fact, in that instance, I would think that that would be a completely wrong way to look at it, because of course, we haven't solved the problem. All we've done is raised aspirations.

[00:20:59] Then, [00:21:00] there are all these other small moments, right? What happens next? Do women actually apply when leadership roles are put out there? Do they experience bias within the process? How are leaders evaluated within the organisation? How are leaders then on boarded? What is their expectation of them? There are sort of lots of follow up experiments to that first trial that we need to do to get to our actual outcome, which is of course gender equality within leadership teams, both in terms of getting there and staying there and performing well.

[00:21:28] So I think it actually is a really long process, but it's very different and almost For completely different purposes than the event, noisy type stuff that organisations kind of do at the moment. Yeah,

[00:21:42] Sarah Abramson: there's a sort of time aspect to that as well, isn't there? You've got to kind of be in for the long term, for the, you know, for the hard work over a longer period, rather than a kind of big bang.

[00:21:52] This is exciting. This is going to be something that. Makes a splash, um, but the kind of boring grunt work over a long period of [00:22:00] time. And do you, as more than now, do you manage to retain that long term relationship so you can see what's happening sort of three years down the line?

[00:22:06] James Elfer: For sure. Like you, you, you have to have that philosophy.

[00:22:09] So I talk about this in terms of performance management all the time, where at the moment, there's a tendency to go through what organisations would call transformation cycle. Like really quickly, maybe only every two or three years where they throw maybe out with a bath water and they get in a new performance management system.

[00:22:25] And they go round and round and round around. It was actually the point I would make about performance management is you're never going to complete it. Like if you think about your organisation in 10 years, 20 years, 30 years time, you're still going to be thinking about how we can improve performance management.

[00:22:41] So you have to think about what you're doing today in those timescales, right? What, what are we going to learn now that's going to be useful to us or our successors? Within three years or five years time, you want to build a base of evidence. You don't want to go three years into the future, say, we're not sure whether what we just [00:23:00] did worked or not, so we're going to tear it up and start again from scratch.

[00:23:03] So to come on that journey with us, you really do need to understand that and understand that what we're doing, and by the way, this is the whole point of the name Waller Now, is to create evidence today that we can look back on in five years or ten years and say, We learned something, right, and we built on it and we're improving and progressing.

[00:23:22] Oh, brilliant.

[00:23:22] Sarah Abramson: I love that. That's great. Great to understand the reason for the name better. Um, sorry, Katryn, I think you were about to say something. Yeah,

[00:23:30] Katryn Wright: I was just going to say the other thing that we see building long term relationships with people is that people really pick up the experimentation bug.

[00:23:40] When you learn that there are really simple ways to test anything that you're rolling out, but then also different versions of it, it really sparks people's curiosity. And so people really say, okay, you know, should I talk about targets or incentives? Or should I talk about diverse teams? Or should I talk about effective team?

[00:23:59] [00:24:00] Or could I talk about my organisation this way or that way? Um, should we be thinking about, you know, countability or objectivity? Like they're all a bit different. Judgment that I was as professional expert make every day, right? And the majority of the time we're sort of relying on our intuition and best guesses, and that is how the world has functioned for a long time.

[00:24:28] Right. But the, the trend now is I think regulators and civil society organisations and investors are interested in how we demonstrate the effectiveness of what we're doing. But people also pick up on that, right? One of our favorite things to do when we launch any experiment is to place bets on what's going to work.

[00:24:51] And we have a team of six or seven people, everybody bet wildly different things. We did this last week with an experiment we were [00:25:00] launching. You know, you ask six different people, you get six different responses. Um, we also ask chatGPT, what does chatGPT think is going to be the most effective intervention?

[00:25:09] And we're always surprised, you know, it's never the same person that's right every time. Um, I'm always consistently wrong. And, you know, that's not anything about my judgment in particular, but it's, it's the limits of expertise, right? And it's the, the benefit of going out and collecting data and understanding our effects on behavior change.

[00:25:29] So, um, I think people get quite excited about that and begin to see this. world of possibility. And so quite quickly you realize one experiment isn't going to change anything, three experiments, you know, we're going to get somewhere. We're going to be able to do some meaningful work and hopefully change things, but really it's when you, when you build this into your habits and way of working and you realize that we can use it around any sort of question and organisation, but it gets really exciting.

[00:25:55] Sarah Abramson: I love that idea. Yeah.

[00:25:58] James Elfer: Yeah, it's sort of, it's interesting because [00:26:00] when you first start talking about it to people, it sounds unfamiliar and it sounds really innovative. But actually, across a lot of questions in the workplace, it's wild that we're not doing this already, like just completely wild. So one simple example, let's say you're a multi billion dollar organisation.

[00:26:16] You've got a hundred thousand people, you're going out, you know, in a two week period to go and ask them to set goals for the year, right? You're going to have a whole load of processes around that. You're going to have a framework that you ask them. You're going to have a technology system that you put it in.

[00:26:29] All those things are influencing how they set goals. Thanks. And how you do that might encourage them or discourage them to set goals in different ways. Like maybe the goals will be more precise. Maybe the goals will be more or less aspirational, for instance. Maybe there'll be more personal, like you could make it all about the business, or you can encourage it to be much more about me and my career and integrate that with the business plan as well.

[00:26:52] And organisations have so little idea about how the process and the frameworks and the communication that they've constructed are [00:27:00] influencing those goals. But this is a hugely influential process and because it is a process and it is happening at such incredible scale, and there are incredible data points that you can get from HR systems, and it's very easy to intervene because it's easy to change the columns and easy to change the frameworks.

[00:27:17] It's just the perfect placement to run experimental research and find out can we optimize this process? Can we encourage people to be more aspirational with their goals, more precise? Is the way that we do that mean they're going to adhere to those goals over a six month or nine month process? These are empirical questions that we can test.

[00:27:36] And if we can do that for a multi billion pound organisation, then we can improve things for them and hopefully improve things for the individuals involved. And at the moment we are on almost ground zero, like not many people do that type of work at all. And as Katrin says, that's just one example of the hundreds.

[00:27:53] Of interesting questions that are out there in organisations just waiting to be, to be answered.

[00:27:57] Sarah Abramson: It's quite fun as well, isn't it? I really like what [00:28:00] he was saying, Catherine, about people kind of get hooked on it and like the idea of experimentation. And we can see kind of our organisations as a bit of a laboratory to test things out.

[00:28:09] And that's actually really, you know, it piques curiosity. It's really, it's, it's, it's potentially fun, interesting, but leads you to a more impactful outcome. Um, can you. Talk a little bit about randomised control trials, explain what you mean by that, maybe give a couple of examples of work that you've done involving those.

[00:28:31] Katryn Wright: Yeah, sure. Um, so randomised control trials, we mean the way that randomised control trial or RCT mean in a clinical or medical setting. So we take our population that might be a group of managers in an organisation. It might be a collection of suppliers. It might be a collection of people working for the organisation that becomes our sample and we randomly allocate them to control and treatment group.

[00:28:59] Control [00:29:00] group goes through the business as usual process, goes through the normal process, and then you can have treatment group or treatment groups, multiple treatment groups. When you give different interventions or different versions of your interventions to, um, by looking at what we would call behavioral measures or outcome measures at the end of that, which might be particular behavior that you expect to change or, um, or sometimes sentiment within a group.

[00:29:26] If, if the outcomes on average are different between the treatment group and the control group, then your treatment works, your treatment is effective, um, to a certain extent. Right. And it can tell you what that extent is. Um, And you can see which of your treatments might be more or less effective than others.

[00:29:44] What's really interesting about that is that generally I think we assume that doing anything is going to be proven doing nothing, right? We keep hearing this bias for action at the moment. Um, we often don't see that that's the case. So sometimes we see that treatment can be Um, just as [00:30:00] effective as, as business as usual.

[00:30:02] And I can give an example of that in a minute. Sometimes treatment can make things worse. And often, and obviously what we're always aiming for is that treatment can make things better. But it can be more of a complex picture than just, oh, obviously any intervention or any treatment going to work. Um, so one of the ways that we, that we've deployed that recently, there was a financial services organisation that we were working with.

[00:30:26] Um, that was interested in understanding and evaluating the effectiveness of an inclusive leadership program. Um, so a training that sort of a thousand or so managers would go through that would teach them how to be more inclusive leaders. And we didn't design the program, we were just brought in as evaluators.

[00:30:44] Um, but what was fascinating was that as all, as leaders went through it, when we compared the leaders that had gone through the training with those that hadn't, we didn't see any difference between the outcomes for those in the treatment group versus those in the control group. Which showed us that the training [00:31:00] wasn't leading to the type of behavior change that the organisation had hoped.

[00:31:04] Um, that's not to say, to your earlier point, Sarah, that there weren't valuable reasons for the organisation to do it, right? That it doesn't send a valuable signal that the organisation is committed to inclusive leadership. That it doesn't make people, um, have less of a good time, right? So, in terms of the The qualitative evaluation, the sort of standard post workshop feedback forms.

[00:31:25] Everybody loved it, right? Managers really enjoyed going on the training. They said that they were different people. They said they had learned a lot. We didn't see that translate into the behavior change. Um, but, uh, you know, that, that sort of shows that the randomized control trial methodology gives us that sort of evidence and data around do things work and to what extent, and then it becomes a management that won't crawl around.

[00:31:49] What are some of those other benefits of doing so beyond the whole of the change?

[00:31:53] Sarah Abramson: Interesting. Yeah. James, have you got examples of other projects where you've, um, used that kind of experimental [00:32:00] approach?

[00:32:00] James Elfer: Yeah, I can keep talking about goal setting as that was the one I brought up a moment ago. So that works really well.

[00:32:06] I should say that not all questions within the workplace are right for randomized control trials. Like there's a load of different research methods. They all have their own, um, pros and cons. Um, and RCTs work particularly well when things are really measurable and they happen at scale, when there's sort of precision in scale, I would always say as well, you can deploy this methodology particularly good.

[00:32:27] So goal setting is the example I gave before. And in the experiment we've most recently done, we focused on this idea that you might've heard of from Professor Katie Milkman of Wharton called the Fresh Start Effect, which is the idea that she identified, and it sounds fairly intuitive, that at certain points in our lives, for instance, January, or a birthday, or a move to the new house, a move to a new house, we are particularly open to behavior change.

[00:32:52] Like those are times where we can think about things that we want to do differently, and then we're more likely to be successful in that endeavor. And [00:33:00] we are really interested in kind of exploring that particular phenomenon in the workplace, you know, after transformations, for instance, or at points of time in new strategies.

[00:33:09] And of course, goal setting, particularly when it happens earlier in the year. So a, um, uh, experiment we did recently at the goal setting period. We asked, um, a treatment group of people. We, we, what we call primed the, um, fresh start effect pretty much just by asking a leader to do a video for them that talked about how she uses kind of the moment within January to reflect on what's been achieved in the previous year.

[00:33:35] Yeah. but also to think about what she would like to achieve in the year ahead, what she would like to start, what she would like to start, and how she integrates sort of personal career objectives into what she knows she needs to achieve the organisation. So there was a bit of role modeling in there, but ultimately it was encouraging people at the goal setting period to draw a bit of a line in the sand, take some achievements and lessons from the previous period, and [00:34:00] take those into the following year.

[00:34:01] Um, we found that that was really effective as well across multiple grounds. So Pete Poole Wrote more detailed goals, they wrote more goals. And what's really interesting is from a sentiment perspective, they also felt closer to their executive leadership team. You saw that in sort of engagement surveys.

[00:34:19] That was a particularly interesting insight for me, because we have always found that a really difficult sentiment to measure. We found it quite easy to change team dynamic because it's quite easy to change relationships between teams, they're close and direct. How do you change relationships between people and executive leaders that they might never speak to and only see, you know, doing speeches or feel the consequences of their decisions?

[00:34:43] Actually, it seemed like when an executive leader showed up in a sort of peer to peer way to give you some advice at a point in time that maybe you needed it or you found it helpful, that made you view, not just that leader, but the whole executive leadership team in a fairly different way. And, and that [00:35:00] has given me all sorts of interesting hypotheses about maybe how we think about leadership and how we can change the relationship between executives and tend to tens of thousands of people beneath them, not in.

[00:35:11] Big inspirational person on a stage in lights type way, but in a actually digitally I can show up in relatively day to day moment, give you a bit of advice and if I do so in a peer to peer way, you might see me as a human that you could trust and you take some value of and that might change how you feel about me.

[00:35:29] Not just your manager and people around you, but actually the organisation as a whole.

[00:35:33] Sarah Abramson: Yeah. So interesting. I'm just going back to that, um, January effect. Um, I think there's something that we can all just relate to on a really personal level because we, you know, most of us every year go through that kind of, Oh, what's my new year's resolution this year?

[00:35:46] And then how do you, um, overcome though the February effect? Well, it was a bit hard.

[00:35:55] James Elfer: You got any tips for that,

[00:35:56] Katryn Wright: Katryn? Overcoming the February effect. [00:36:00] I mean, having, I'm going to cop out by saying I didn't work on the goal setting piece, I'm not sure. Um, I mean, I think like longevity is hard, right? And even with this methodology, you know, we're looking at moments in time, obviously we're interested in longitudinal data and what happened those times, um, But it's hard, you know, we know that sort of in the behavioral science literature as a whole, sustaining behavioral change is challenging.

[00:36:29] Um, one of the things that sort of drives me crazy around, I'm not sure if you saw over the weekend, there was a flurry of, it's a Wall Street Journal article and a Guardian article about the limits of nudging and the limits of behavioral science. And like, there are real limits to this work. But the one that drives me mad is when people say, It doesn't, it doesn't work, right?

[00:36:51] So there was a meta analysis that came out that showed more effect sizes than we'd anticipated. And what drives me mad about that is like, we're [00:37:00] collecting data on it. So we know whether it works. So we don't have people going around saying, you know, here's this silver bullet that I've got, here's this, you know, the solution that's going to solve all problems.

[00:37:11] Like we collect data and that means that we know the effect of our short term, long term over time, unlike. That's incredible, and for all of its limitations, this idea that we know whether things work or not, may be infused in more sort of social change spaces and more organisational change spaces. Um, so that's one of the things that,

[00:37:35] Sarah Abramson: interesting, makes me madder.

[00:37:37] Yes, yeah, I completely see that. Is there, is there something? to get at around habit forming, sustaining behavior change. And do you, do you see that? Do you get involved in any of that kind of thing? Yeah,

[00:37:51] James Elfer: sure. I mean, I think when we're talking about habits, we're, we're always thinking about one specific habit that we're trying to change in tens of [00:38:00] thousands of people, but if you're just talking about an individual, right, who has a specific goal that they want to, um, they want to go after, like, I would always just give two bits of advice.

[00:38:11] So the first is be precise. And this is super obvious, it just means smarter goals, right? So if you turn up in January, like, I want to get fitter this year, or I want to run a marathon. Actually, I haven't done any running in the previous year. And then you set no goals for yourself, nothing else. You just sort of go out into the world hoping that that's going to happen.

[00:38:30] But of course, if February comes around, like you're probably not going to have got anywhere. So the first stage is just take the broad aspiration and put that down into some shorter term, tangible, specific, measurable goals, all that type of stuff. Now that is hardly groundbreaking, but hopefully you can get to that stage.

[00:38:50] Now what that stage allows you to do is then the accountability bit, which is to say, Actually, I need to find a way to hold myself [00:39:00] accountable for achieving those short goals. And we can be as creative as we like about that. And actually, I think people are intuitively quite good at that. So, how can I hold myself accountable?

[00:39:10] Well, maybe I team up with a mate, right, who wants to do a similar goal. If we both want to run a marathon, then we're going to meet on Sundays, right, and do our first 5K. Well, actually, it's going to be way harder for me. to, uh, stay in bed if I know my mate is waiting for me in the rain, right? And we know that type of social contract work.

[00:39:29] And there might be loads of other types of ways that we hold ourselves accountable. And I think that's stage two. So if you can do those two things, maybe you get over the February effect. You don't need to hold yourself accountable forever. Just enough for that habit to form and it to feel fairly natural.

[00:39:44] And before you know it, you know, you're running all the time. Now, these days. Sometimes I think behavioural scientists talk about them like they're magic. They're not guarantees, right? They're just going to help you. They're just going to help you improve a little bit, make it more likely that maybe you hold on to that goal.

[00:39:58] But they're worth a shot, right? And [00:40:00] if they can help one individual do more, and they can certainly at scale help tens of thousands of people, um, hold a goal, maybe 5%, 10 percent more effectively, then they might have delivered a colossal amount of value, especially if that goal has, you know,

[00:40:14] Sarah Abramson: Brilliant. I'm going to resist the temptation to dive into the marathon example too far because being a marathon coach and runner, I just, yeah, I could talk endlessly about it and I don't want to bore our listeners with that, but I absolutely agree that habit forming, having a goal that you're focused on, everything that you just said is absolutely crucial.

[00:40:35] So, I'm just kind of aware of time. I'm really grateful to you both for fantastic, insightful examples. It's so interesting to hear about all your work. As a last question to each of you, um, sort of speaking to you as a human, uh, if you could hear about like what you're enjoying at the moment, what you're looking forward to over the next few months, um, in terms of projects, ambitions, anything that [00:41:00] you're, you know, that you're working on and what you're motivated in.

[00:41:02] Start

[00:41:04] Katryn Wright: with Katrin. Yeah, I mean, I'm sure it really shows. I think, I won't speak for James, but I know that this is true, that we're both really, really motivated and obsessed with, as you can tell, data and evidence and what works. And when we get interesting questions, when organisations ask us interesting questions to solve experimentally, that motivates us, that excites us.

[00:41:30] That's why we, you know, get out of bed in the morning. Um, and so, yeah, I think some of the things that we're looking, that I'm looking forward to over the next few months, we're doing some really interesting, deep biasing project. Um, particularly in, yeah, really key organisational processes where we have data that shows, um, that gives indications that there is bias that exists in those process.

[00:41:53] That's really exciting. I'm really personally interested in, in how we grow and expand the human rights [00:42:00] work. So. In particular, we're doing some exploration at the moment around complaints and grievance mechanisms. Um, you know, who are the best people who are the most trusted individuals to sort of go out and and reassure workers around the value of raising issues and raising complaints through these mechanisms.

[00:42:19] Um, what are some of the biggest predictors of trust for organisations in that? Is it response time? Satisfaction rates? Is it, um, any number of sort of variables that we could be looking at there? And then, you know, through the conversation that's come up through your interest there in marathons and running, I was also just sort of starting to think about, how should I be applying this to my personal life?

[00:42:45] And, you know, we did an event, we did an education program, I guess, about six months ago, where we were challenging people to sort of, yes, learn about behavioral science, but also, yes, learn about how they could do it. be applying it in their personal lives. And for that, I said, you know, [00:43:00] I really want to, um, to, to make sure that I'm dedicating time in my week to drum practice.

[00:43:05] So I started learning drums about a year ago and I love it. Um, but I, I, you know, as we all do, we struggle. I struggled at home to do that. And so. I am definitely motivated by the idea of being a better drummer six months from now. So

[00:43:18] Sarah Abramson: cool.

[00:43:19] Katryn Wright: But I don't give enough focus on a daily or weekly basis. So how am I going to get myself there?

[00:43:25] Um, what is my, what are my habits that I'm putting in place together? So as always, I know that I need to use more of my professional expertise to change my personal life. But as we know, knowledge isn't enough, intention isn't enough. We have to focus on the behavior change, which is what we need to. So, um, that would be, that would be my goal and motivator.

[00:43:44] Sarah Abramson: Great, and James?

[00:43:45] Katryn Wright: Um,

[00:43:46] James Elfer: I never thought I'd say this, but I'm, I'm getting quite interested in the leadership space. I've mentioned it a couple of times in the chat, and that's probably why it's the sort of thing on my, on my mind at the moment, and the reason why I say I've always avoided it in the past, that it just seems like [00:44:00] such a chaotic area of HR, particularly.

[00:44:04] I think that's. It's mainly because you have this powerful set of people and the stuff that we do for them, we have done historically, we've measured it by how much they like it, right? And, and that has turned into what you would expect, which is lots of amazing experiences for very powerful people that they enjoy, right?

[00:44:22] Who wouldn't want to go off on a three day retreat at a lovely country house and be wined and dined and, you know, listened to? I don't know, uh, their, their latest elite sports stars coach, right? To tell them how they're so elite and, you know, how they can, um, they can start to inspire the people beneath them.

[00:44:40] But whether it does anything, I have no idea. And I've always just thought that was untouchable, right? It was such a strong dynamic that we could never get involved. Whereas actually, more recently, we've done a couple of experiments, like the goal setting one, where we've put leaders in a place where they do seem to have been really helpful to people.

[00:44:57] And also talking to the [00:45:00] leaders, they've really valued a little bit of help in moments that are quite hard or interesting. So like little stuff once a year, all your leaders are going to go out and give people bonuses, for instance. And as we all know within organisations, Sometimes that moment can do more harm than good, right?

[00:45:19] Cause you give, you pay out tens of millions of pounds of bonuses and all you manage to do is to make most of your workforce feel disgruntled or that those bonuses have been given unfairly. Actually, that could be a moment that's really nice, right? It could be a moment where. It's a, it's a bit of an interaction between a leader and their team where you get a moment to say thank you.

[00:45:38] And of course, maybe use that as a bit of a, a platform for the following year and what you can do differently. And that might improve the relationship and it might make people feel better and it might mean that they're looking forward to their work ahead a little bit more. And actually, I'm so much more interested in those moments of precision.

[00:45:57] Rather than helping someone be, you know, [00:46:00] a great leader overall, whatever their perception of that is in their mind. So I'm going to try and get. Back into that. I think it's got lots of challenges and I think it needs more mixed method. We will obviously be doing our RTT things. There are loads of other research methods that will add loads of value there.

[00:46:15] And that's the sort of stuff that I love. You know, you get an idea or a theme or a concept in your mind and you think, I don't know what we're going to learn over the next 12 months, but I'm going to It seems like a space worthy of exploration. So that's what I'm looking forward to.

[00:46:28] Sarah Abramson: Definitely. That's so cool.

[00:46:30] I love that you've both sort of landed on points that feel very human there. It's sort of, you know, how do we relate the things that we think about in our work to our. Personal habits, drumming, whatever it is that just that's really cool. But also those relationships, those moments that you're talking about, James, where, um, interactions really matter.

[00:46:50] So thank you both so much. That's been such an insightful conversation. I've really enjoyed it. I hope that our listeners do too, um, really appreciate your time. Thank you [00:47:00] both.