The Innovation Show
Annie Duke - Thinking in Bets
Today's guest teaches us how to get comfortable with uncertainty and make better decisions as a result. In Super Bowl 49, Seahawks coach Pete Carroll made one of the most controversial calls in football history. With 26 seconds remaining and trailing by four at the Patriots' one-year line, he called for a pass instead of handing off to his star running back. The pass was intercepted and the Seahawks lost. Seahawks called it the dumbest play in history, but was the call really that bad, or did Carroll actually make a great move that was ruined by bad luck? Even the best decision doesn't yield the best outcome every time. There's always an element of luck that you cannot control and there is always information that is hidden from view. So key to long-term success and avoiding worrying yourself to that is to think in bets. How sure am I? What are the possible ways that could turn out? What decision has the highest odds of success? Did I land in the unlucky 10% on the strategy that works 90% of the time or is my success attributable to dumb love rather than great decision-making? By shifting our thinking from a need for certainty to a goal of accurately assessing what you know and what you don't, you'll be less vulnerable to reactive emotions, knee-jerk biases and destructive habits in your decision-making. You'll become more confident, calm, compassionate and successful in the long run. Our guest today is a world poker champion, a cognitive psychologist, a co-founder of the Alliance for Decision Education Nonprofit, a member of the National Board of After School All-Stars and the Board of Directors of the Franklin Institute and she is the author of Quiz, How to Decide and today's book that she's kindly went back into the library for Thinking in Bets, making smarter decisions when you don't have all the facts. Annie Duke, welcome back to the show. Thank you for having me, I'm excited to be back. It's great to have you back. I had the greatest pleasure of having you on the show before to cover Quiz. We got so many positive feedback from that show. Annie, even when I mentioned in the newsletter yesterday that you were forthcoming guest on the show, got so many emails. You have lots of people out there probably don't reach out to you to tell you, but you made them with bad marriages, jobs, and they're so grateful for that. So just to let you know that that's going on out there. I have to say, I'm so, so happy when people do reach out to me and say, "Hey, you know, you got me to clip my job." And the thing is, as predicted, they're always happier because if you're thinking that hard about like, you're probably miserable in what you're doing. And so just like venture into the great unknown and it'll probably be better. And it did for me as well as Celia that's looking back on that episode in preparation as well as reading the book. It was great to do both because stuff I had missed or stuff that's happened in my life since we recorded a couple of years ago. It just made more sense now as well because I've more experienced more water under the bridge. So really, really helpful. But I told we'd start with some context for this book because in our episode on Chris, I'll put a link in the shout outs by the way, we shared how the last day you got ill and your career break turned into a 20 year career as a professional pumpkin player. So let's give a bit of context that will get stuck into the content. Yeah. And this also relates to quit, by the way. So I was an academic. I was at the University of Pennsylvania doing my doctoral work at a National Science Foundation Fellowship working with an amazing woman, Lila Gleitman and her husband, Henry Gleitman also, amazing. And you know, if you had asked me then, I would have told you, yeah, I'm going to be an academic, I'm going to go get a professorship on a tenure track. And that's what we're going to do with my life. And in fact, after five years there, so, so just to kind of set the stage so people understand like, five years in, you've already done your qualifying exams that other people call them major area exams. That's like a really big deal. You know, I'm already teaching at that point, not, not just TAing and I've done all my dissertation work. And I'm actually on the job market, so I have a bunch of job, job talk, set up my first one at NYU Duke, Cornell, UT Austin, like, I had like eight job talks set up. And so that's what we're going to do. I'm going to go get a tenure track job. I had been struggling over probably the last year that I was in graduate school with like a, it was sort of a chronic stomach illness, like, it's pretty bad and it became really acute. And I ended up in the hospital for two weeks because it made us so that I couldn't keep any food or liquid down, but so obviously that was bad. So, so I ended up in the hospital for two weeks. It happened to be, first of all, it's just kind of like when my job talks were, and second of all, it's very clear, like, I just needed to address this health problem because I'd been kind of thinking, okay, well, I'm kind of sick, but like, I'm dealing with it because obviously I was still doing my research and things like that. And I'll deal with this once I'm through the job process, you know, and my body was like, no, you will, you need to deal with it now. So at that point, I canceled my job talks and, you know, I was just planning to take some time off to address my health, health, and then come back, finish my PhD work, and then go back out on the job market the next year. So that was, that was my plan. So during the time that I took off, I didn't have my fellowship, which meant I didn't have any money, and I needed some money. And so my brother, Howard, actually, he had been playing poker already for about 10 years professionally. And so he suggested to me that I might want to just try poker out because I watched him play a bunch. There were a few times during graduate school where he had, like, blown me out to Las Vegas and I got to play some low stakes poker, and he was like, maybe you can make some money doing that. So I was like, sure, and he sent me 2,400 bucks, I remember. And I went off to play, I was in, I was recuperating in Montana, which is where my husband's family lived, which was obviously my husband at the time. And so I was recuperating out there, and there was legal poker there. And so I started playing, it was actually kind of ideal because I didn't feel well every day, so like, let me sort of decide when I was going to play and when I wasn't, and I didn't want to reboot a new career and all those things. So I started playing and I loved it. But just like, just to set the stage, because I think that people don't realize this, this was in the 90s. So poker wasn't on television, there was no internet poker. So if you were playing poker, people kind of thought you were gambling at it, which isn't true now, but that was what it was like that. But I loved it, and I actually did really well. And in fact, I loved it so much that I didn't go back to academia. So for the next eight years, I played poker exclusively. And then in 2002, and this was really the beginnings of thinking of that. I got asked by a hedge fund to come speak to them how poker might inform their thinking about risk. And I didn't quite talk about that. I mean, I did. I talked about risk attitudes, which is a little bit different. But what I was really talking about was the kind of intersection between poker and cognitive science. So I've been studying cognitive science, particularly learning and decision making under uncertainty in graduate school. And I'm sure that those things were informing each other in some implicit way during those first eight years that I was playing. But all of a sudden I kind of realized, you know, when you're thinking about decision making under uncertainty, poker is kind of it. It's really terribly matted. And gosh, you know, this really has a lot to say, like it can really inform the cognitive science, I think, in a really interesting way. So I gave that first talk about how whether you've been winning or losing recently changes your attitudes toward risk. Really using kind of poker as an example, but that really speaks to original work by Daniel Kahneman and Amos Tversky, which ended up kind of informing prospect theory. And that was kind of the moment where I started thinking about this idea of, you know, what is the interplay between poker and cognitive science and how can we think about decision making and improving decision making under those conditions of instrument uncertainty, not just thinking about the science, but also thinking about how do you deal with this at poker, right? How are you dealing with this problem at poker? So so I started giving talks in that space. And it was kind of 2000, I think it was right around 2009 or 2010. So 2002 was when I gave that first talk. Then I sort of developed a suite of talks, which were all about this issue. And then I think it was 2009. I started saying, Oh, you know, this would be an interesting book, but funnily enough, I was like, but there's also an interesting book because I was awful teaching poker at the time doing seminars. And I thought, well, actually the cognitive science really informs poker as well, like it works in both direction. So when I was teaching poker, I was teaching a little bit differently than other people, a sort of through like a cognitive science lens. And I was like, that would be an interesting book too. And I don't know. I just had the thought of like, well, I'm a poker player. So I should write that book first, which I did, and that got published in 2012. And then, and then I actually wrote another little poker book after that and a few, I don't know, I have a few poker book. And then I eventually got around to the, you know, well, I'm going to write the reverse now. I'm going to write about how poker can really inform your thinking about decision making under uncertainty. And that's, that's really what ended up becoming thinking about, which is just a great killer title as well. I did not come up title, by the way, my editor did. I pitched a different title. I'm not even going to tell you what it is because I'm embarrassed. I'm terrible at title, by the way, although I did title quit. And when I went in and pitched my publisher, my, my editor was in the room and she said, what do you think about calling it, thinking in back? And I was like, Yes, well, it speaks to exactly what you're talking about in the book, which is listening to people, diverse opinions, et cetera, et cetera. I would tell you before we get one there, a great friend of the show, Charles Kahn wrote this book, The Imperfectionist, and it's strategic mindsets for uncertain times. And he mentions you several times. He told me he's a big fan of your work. And what that is essentially about is mindsets like lenses through which to look at unfolding events. And I wondered beyond the book that as you were going through your career before you started to write Thinking Embats, did you unconsciously use the frameworks you had learned from behavioral economics, behavioral sciences to make decisions? Because you know the way when something's implicit, you don't even know when you're using it. So I wondered, was it that moment of the hedge fund talk or were you doing this unconsciously the whole time? Well, I mean, I'm sure that I was doing it subconsciously the whole time, but you know, the hedge fund is when I really started thinking patiently about it. And in particular, just I really started thinking deeply about this like problem of when you're winning, you know, how does that change your risk attitudes when you're losing? How does it change? How does that change your risk attitude? How do you actually close feedback loops in a game that has so much uncertainty, right? So I was super obsessed with trying to figure out why I lost a hand or why I want to hand. What was the contribution of luck and skill to either of those outcomes? And I think that people look at poker and they say, well, you find out really fast, because a couple things. One is they say, you see your opponent's card. That's when I get a lot like, well, life is different because you don't always see your opponent's cards. You don't quote unquote, right? And I'm like, okay, but like, first of all, at the top levels of the game, you get to what we would call showdown, which means both people show their cards 11% of the time. So by the way, you mostly don't see your opponent car, just to be clear. But even when you do, you know, it's that thing of like, you're never, you know, I people see the rounders movie and when Matt Damon is calling out, he's watching a game and he calls out everybody's exact hand, that's not really how poker work. There's been a couple of I remember one time where I was like, that guy has the ace jacket club. And he did. But it was like, that's very unusual for the way they're playing the hand to stall narrow it down that I can hold the cards, right? Mostly what you're doing is you're coming up with some range of court. So even when they show you the hand, it's not like you were like, I think they have king queen and look, they actually have two seven. It's those things were kind of either in the range of hands that I was thinking about or outside of the range of hands that I was thinking about. So is it helpful for that? Yes, I, you know, but you, you know, you're setting a range and hopefully as you get better, you can set the range more narrowly. So yeah, that's helpful feedback, but you just don't get it very often. What you're mostly getting is I won the hand or I lost the hand. I fold it. Was that right? Right? My opponent folded, did I, did they fold for the reason that I thought they were folded? Right. Like there's all these questions and you don't really know. And then there's this separate issue of even if you win a hand, you don't know if you maximize. Today, did you actually play a line of play that extracted the most money from your opponent? Right? And if you lose, you don't know, did you lose too much? Sometimes did you lose too little, actually, sometimes, sometimes that's the case. If you lose, was there a different way to play the hand that could have won it? Right? If you win, did you just get lucky? Like these are all actually really hard questions to ask when mostly hands end in a cloud of uncertainty. And so I think that I was just right, but really intensely focused on that problem. I'm sure because of my academic background and then very explicitly so once I started giving talks and I started trying to figure out, well, in a system that has so much uncertainty where there is so much short-term luck involved and there's so much hidden information because I can't see my opponent's cards, how do you actually start to close these feedback loops? Because the thing that I noticed, like, supremely that I really wanted to avoid was a problem called self-serving bias. So there's kind of two problems that we can think about in terms of the way that we close feedback loop. One which I talk about a lot in the book is resulting. And resulting is that you kind of fit the outcome to the line of play. So if you lose a hand, I'm going to sort of, I'm going to interpret the way you played it poorly because I want them to match. And if you win a hand, I'm going to interpret the way you played the hand well. And in fact, if all I know is that you won or lost, I'm going to say you played poorly when you lost and you played well when you won. So that's just for like a stranger and we do things like that all the time, right? So this is a really big problem in decision making and that's where I talk about P. Carroll, which we can get into. But there's an interesting thing that happens when it's you. So I'm deciding about me. And this is something like Daniel Kahneman really says is that we're really all trying to have a positive narrative of our life. And so the question is, if I lose a hand and I say, oh, I played that hand really poorly, how is that advancing my positive narrative, right? Well, I can give you a good story about why it is and that's sort of the mind trick that you have to do. But like our natural tendency is that that's not going to advance our positive narrative. So what do we do? It's something about self-serving bias, which is if I win, it's because I played great and if I lose, it's because I got unlucky. So this is like what a thing that I was so focused on because in a system that does have so much luck, luck becomes a scapegoat, right? It's like it's the way that I can get out of having to confront that I might not have played well. And that particular bias, and you can hear it all the time, by the way, if you're walking along the halls of a casino, all you hear is players talking about how this donkey got lucky, beat me in a hand and they play or they outplayed everybody and they're so great. And obviously it's always a combination of luck and skill, right? And I was terrified of that because I realized if you want to make sure that you're not learning then interpret the world that way. Because when I win, I may have played the hand well, I may have played a poorly, I don't know. And when I lose, I may have played it well and I may have played it poorly. I don't know. When I win, I might have gotten lucky, you know, or I might have earned it. But I lose, I might have gotten unlucky or I might have earned it. And how am I going to become a better player in the long run? If I can't overcome that buy it, right? So that was the thing that really kept me up at night as a poker player and a lot of the things that I did when I was playing poker and in life, by the way, are just trying to avoid that particular problem. And what was very clear to me was your constant growth mindset as well, where you put together an accountability partner, a team, a pod to look at you, to be this mirror to you, this black mirror sometimes held up against you. One of the stories you tell is when one of your grace mentors, Eric's, Sidel, you stuffed to me like gone, oh, you started to moan and go, oh, I had such bad luck. And he's like, wait a second, sister, they gave it to you. And I just, before you tell the story, I just want to ask your audience, has anyone ever given you feedback? And then you go, what a jackass, how dare they speak to me like that? But where are they actually giving you really positive feedback that could have sent you in a much better direction, because you embrace this feedback? I think that's a great question to ask because a lot of times our first reaction is to be defensive. And you know, and we blame the person who's delivering the message. But you know what, that's how you grow. So you should try to think about it from their perspective, hopefully, and listen to the feedback and try to incorporate it and see if you can make a change. You know, the thing is that this tendency really is very strong. And this was, by the way, I mean, this was really early in my career. It was before, I think it was before I actually even sort of declared being a professional poker player. Oh, I didn't get that context from the book. God, that's really interesting. Yeah. Like, I mean, I was a professional poker player at the time, but I hadn't sort of declared it. And so I was, you know, I was still at the moment where I was still like, I'm going to go back to academics. So anyway, I made the final table of a tournament and there were six people left and I had a pot where I could have gotten a chip lead. And I raised with two Jacks and this guy Gus from Costa Rica. I still remember. This is a very painful hand. It was a long, long time ago, trying to think out long ago. You always remember the painful one, not the, you're not so much the positive ones. Right. I'm going to say this was 1994. So if I still remember was Gus from Costa Rica. So there's six people left. I raised with two Jacks. He moves all in. Which is not an easy call. I really think for a long time and I call him and he asked two nine. So just to be clear, I'm 18, I'm 81 and a half percent to win this. So this is round up to 82. He's got an 18% chance. And this is for the chip lead. Right. So this is like a really big deal. Like I've made a really tough hero call and this is for the chip lead. And I'm 82% and he made a straight. So I was pretty upset. It's that I saw your side down and I was like, like, I'm like, can you hear this? And he, he literally looked at me and goes, is there a point to this topic? I feel like you can start a point in this story. Gus from Costa Rica got so lucky and I had the chip lead and I, you know, whatever. There was a point to this and he was like, he basically was just like, did you do anything wrong? Because like he was like, if you want to talk about, could you have played the hand differently here? I'm all for it. But if all you're trying to do is offload your emotional distress at me about luck and intervening in a way you didn't like, I have to deal with that all day myself because I played poker all day too and people get lucky on me too. And I don't need yours to crap on top of that, right? And I was like, oh, okay. And I was, I was mad that I was like, oh, you know, got kind of a point here. So I wouldn't say that I connected it at that moment to self serving bias, but what I did realize is like probably not very good to focus on the luck element, except in as much as sometimes, sometimes you're saying luck intervened, right? But it's interesting because where the people, where people get the most upset and poker is in those moments where they're a favorite to win and they get unlucky, weirdly enough, they don't get upset when they're on the other hand of the equation. And so what he was kind of saying to me is if you want to focus on the skill elements of the game, then I will talk to you all day about it. And that's the right way to approach the game, right, is to kind of treat it in that way like chess. I mean, the thing about chess is that very hard to get unlucky in chess, right? So you're kind of forced to focus on what were the moves? What were my opponents possible moves? Did I miss something? Did I open up a line that something, you know, you kind of have to focus on that because you can't just sort of say, like, I got so unlucky and the world is against me and I'm so sad I'm going to go cry to my friend. And that's the only thing I'm going to think about the he. So that's all he was saying, right, was if you have a question about how you played the hen, I'm happy to discuss it, but I'm not happy to discuss your bad luck. Boy, was I lucky that he said that to me. That was a big moment of luck for me, very, very early career because I then started thinking about how could I actually do that, right, like how could you actually implement that? And if you do that, you're just much more likely to not fall into either the resulting problem or the self-serving bias problem, because what you're really getting down into is analyzing the moves in the hand. And ultimately that's what's going to cause you to win in the long run, right? Because poker is kind of weird, right? In the long run, it's all skill. In the short run, it's quite a bit of luck. So what you're really trying to do is what we call grind the net, right? Make sure that you're playing a game where you're expected to win. And you can get in enough hands with a positive expectation to realize that expected win. So to kind of ground that, let me try to sort of ground that. So let's imagine that we were flipping coin and we had a coin that we knew was fair. Okay. So we know it's a fair coin and it's going to land heads 50% of the time, tails 50% of the time. You say that I offer you a bet at two to one. So I'm going to pay you $2 if you call the coin correctly and you're going to pay me a dollar if you don't. So we know that you're 50/50 to pay the coin correctly and I'm offering you $2 when you call it right and $1 when you call it wrong. What that means is that on every flip, your expectation, your positive expectation is 50 cents because 50% of the time you're going to win $2, 50% of $2 is $1 and 50% of the time you're going to lose a dollar, 50% of a dollar is 50 cents. So $1 which is your positive expectation minus 50 cents which is the downside is 50 cents. So that puts you at 50 cents positive expectation. Okay. Now here's where I hope people can understand the luck skill element. If we do that, there is no flip on which I will give you 50 cents. Right. So this idea of this positive expectation is theoretically what are you earning on each dollar that you bet? Because I'm either going to give you $2 or you're going to give me $1. So this 50 cent exchange is not going to occur. What this means is that that's what the positive, the upside of that bet is for you. But on one time, you might lose. So if you flip that coin and you call tails, invite land heads and you might have to give me a dollar, that's going to happen half the time. In fact, 25% of the time you lose two times in a row, 12 and a half percent of the time, you'll lose three times in a row, six and a quarter percent of the time, you'll lose four times in a row, right? A little over three percent of the time, you'll lose four times in a row, right? We can keep going for five times in a row, right? We can keep going like that, right? So we know that we can get these, it's actually going to be a pretty common event that you'll lose two in a row. And actually a pretty common event that you're going to lose three in a row, right? Okay. So that's where you can see that short term luck happening. But what we can say is, but what if we flip that coin a thousand? Are you ever going to come out a loser a thousand time? And the answer is no, right? Because obviously, over a thousand times you're flipping it enough that your expectation is that the coin is going to lay in heads 50% of the time, tails 50% of the time. So we can expect you to get that right 50% of the time, which means that over a thousand times, you should be earning $500. So in the long run, it's going to be all skill, right? Did you take a good bet or not? So now the skill element for you is not in the calling of the coin because that's just luck. It's in what bet are you taking, right? Am I laying you two to one? Are you taking that bet, right? Or are you taking a bet where it's a dollar 10 to one, which would only be a five cent earn for you, right? So that's a choice. Both of those are positive expectancy, but you've made a very poor choice to choose the dollar 10 to one if the two to one is available. Where are you actually winning 80 cents, but losing a dollar? Now you would be negative expectations. So how we handle that coin, the skill element, is going to be in what bet are you willing to take given that the coin is 5050, okay? And then that's going to turn, you know, but here's the, now, this is where we can see the problem. Let's say you're like, okay, I'm going to pay you 80 cents and you're going to pay me a dollar. And I'm going to do that because I think I'm really good at flipping coin. And you win on the first one. So I pay you the 80 cent that this is, this is what we do in life. You say, see, I told you I was really good at it. That's the problem, right? So this is the poker problem, right? So if I start moaning about my bad luck, I'm not focused on the thing that I need to be, which is am I actually making good decisions that in the long run are going to end up earning me money and we can substitute money for happiness, well-being, fulfillment, health, anything that we value. So we can think about this in the exact same way about like, what are the foods that we choose to eat? How does it make me feel short run long run, you know, short run versus long run, right? So there's all sorts of stuff that I might do today that's bad for me in the long run. Am I thinking about that and what the effect is going to be because I don't get immediate. I don't necessarily get immediate feedback on that, right? How am I thinking about decisions about what I'm driving? Who I marry with job I take, right? These are all in the same category and the thing that I love about poker is that it really distills it down into something very understandable that makes it so clear that you have to focus on the skill element and somehow move yourself past those luck elements. What I say is like, don't pretend it doesn't exist, which I think is what a lot of people do. They want to pretend the left doesn't exist and instead embrace it and just say of course there's luck. All I'm trying to do is reduce the probability of a batter. I want to really emphasize, thank you for bringing this through that, that it's not about poker. This book is not about poker, it's about improving your decision making and dealing with when the look doesn't go your way. I was telling you before I had a career in professional rugby and my son now is really addressing to become a professional soccer player and he's really interested and he's only 11 and he asked me the other day so innocently he's like how did you become a professional player and I said in this order discipline, tiny bit of talent and look and all you can control is the discipline, like really like you have talent but all you can control is the discipline and I saw so many, I'm sure you did with poker, so many much, much more talented players than me, way more talented, never go anywhere and then they'd blame the world, they'd blame bad luck or blame, oh you were lucky, you spoke French or something, you know, some crazy thing and my whole thing was like walk away, if you're going to walk away or it doesn't work out, at least you'll have no regrets because you control the controllables of it. And I'm saying that to get us to a key point that I really want our audience to hear. We had a Ellen Langer on the show a few weeks ago, Ellen has this beautiful say which ties beautifully together with something you said, she said instead of becoming paralyzed over making the right decision, make the decision and then make the decision right and it echoed what you say and this is one of the core things I talked from this book is that there's a huge difference between outcome quality and decision quality and you need to know the difference with them because it helps you deal with no regrets in life. Yeah, I mean, obviously a central table of the book is that there is a huge decision between outcome quality and decision quality. So you know, I opened the book with this story about Pete Carroll in the 2015 Seattle Seahawks and they're in the Super Bowl against Patriot, the New England Patriot and there's 26 seconds left in the game and the Seahawks are the Patriots one yard line and there's just a so there's a cloth management problem, which is that the Seahawks only have one time out. So 26 seconds left in the game, it's actually there's a constraint in terms of how many plays you're going to be able to get off. It's second down so they have three downs to try to get into the end zone. Now, the expectation of the crowd is that Pete Carroll's going to call for a run place specifically he's going to hand it off to a guy named Marcia Lynch who is one of the greatest running backs of all time. And I guess what people think is that Marcia Lynch is obviously just going to go through the Patriots defense like a hot night through butter and score. Pete Carroll actually does something really different. He calls for a pass play into the front corner of the end zone. That ball is intercepted and the game ends with the Patriots winning because the Seahawks at that moment, which I forgot to say, we're down by four. Okay, so down by four, the Seahawks have to score a touchdown. They can't score a field goal, so they needed to get the sit. So obviously when that ball is intercepted, Patriots are up by four, they can just run the clock out. That's the end of the game. The headlines were just brutal. The fans were just brutal. I still play this video when I give talks and people in the audience, like there's always a reaction. Some people laugh when they see this still that I put up on the screen. Some people go like I'll often get like should I hand it off to Marcia on lit? They'll get yelled in the audience. And people just have very strong opinions about this play to this very dead. So the first thing that I say to them is, and I'll ask you this, what if it'd been caught for the game winning touchdown? What would the headlines have been? Master stroke by Carol. Right. Nobody saw it coming. This is why it's going to be in the Hall of Fame. So this right here should tell you that there's something wrong because the play is the play. You know, this is the problem. Like this is that poker hand that I went and cried to Eric Seidel about 82% of the time ish. I'm going to win 18% of the time I'm going to lose. If I if I lose the 18%, you shouldn't say I played the hand poorly. And if I win the 82%, you shouldn't say I played it well. It depends on whether I made a good call or not in that situation. That's really what matters, right? In that particular situation, I think I played the hand quite well, but there's been lots of situations that played it poorly, right? And this is the Pete Carroll issue, right? Is that if that is the game winning touchdown, everybody's talking about how brilliant it is. If the pass is incomplete, people don't make a peep about it. It's only under these circumstances where the ball is intercepted that people say it was an idiotic play. But the play is the play. The outcome on a single try doesn't tell you anything about whether the play was good or bad, right? In the same way that if you call a coin heads and it lands heads, it doesn't tell me you're a good coin caller. It doesn't tell me anything. So what we have to do is get back to, well, let's look at what the possible outcomes were and what the probability of those outcomes were to start to try to understand whether that decision was good or bad. And this is like a primary idea in the book is that any decision you make is associated with some stat of possible outcomes. The world is not deterministic. If I make a decision, there isn't only one outcome that could occur. There's a set of possible outcomes that can occur. And each of those that isn't going to occur at some probability. So my job as a decision maker is trying to say, given the probability of any of those outcomes and the pay offs associated with those outcomes, am I making a decision that causes me to advance toward my goals on average? Because I cannot control the outcome that I actually observed. So let's think about the Pete Carroll thing. All right, so 26 seconds left needs to score a touchdown, three possible downs, but only one time out. So this is a problem. So I'm going to ask you this question. Let's imagine that he hands it off to Marshall Lynch, just like everybody wants him to. What happens to the clock if Marshall Lynch doesn't score? It keeps running. So what does Pete Carroll have to do? He has to use his one time out, right? Now if he hands it off to Marshall Lynch again, the clock runs if Marshall Lynch doesn't score and that's the end of the game. So he's only going to get two plays. If he hands it off to Marshall Lynch, twice in a row, on second and third down. Okay, but what if, on either second or third down, I don't really care which one, he calls a pass play. Now it becomes interesting because what happens when a pass is incomplete to the clock? When a pass is incomplete, the clock stops all on its own. So you don't need to use a time out. So we can think about three outcomes that are associated with passing. One is a touchdown. We're obviously very happy with that. One is incomplete. Interestingly enough, we're also fine with that because it stops the clock. And it stops it in about six seconds, doesn't take very long. And then the other one is an interception. So let's imagine that we really want to hand the ball off to Marshall Lynch twice. If we have a pass play somewhere in there, either on second down or third down, we can hand it off to Marshall Lynch twice, Ian, also try to pass the ball into the end zone. So I assume, having played rugby, that you would like three chances to score rather than two. Right? Absolutely. So now what we can say is, okay, but like, first of all, what's the chances that Marshall Lynch is going to score? Obviously, if he's like 90% to score, he'd probably just want to go that way, right? Well, it turns out he's about 50% to score in that situation. So going back to the coin flip problem, what that means is that 25% of the time, when you hand it off to Marshall Lynch twice, he will not score. Okay. So let's remember that he will not score 25% of the time. So I think that tells you, okay, let's now look at the other possibility. Close to half the time that ball will be caught for attention. All right, that's pretty good because we know that 50%, that's about how often Marshall Lynch is going to score on the one track. Close to half the time, the ball is going to be incomplete. So what that means is that the probability of an interception is actually super duper low because what you're paying to get that third try to score is the interception rate. That's the new outcome that can occur. And it turns out it depends on who you ask, but it's one to two percent. That's it. So we can go back to that poker thing there. I lost a hand where he was 18 and a half percent to win. And I was like super duper sad, right? Think about how much more that is than one to two percent to win. So what people didn't understand is that they were just observing an incredibly low probability event and actually the play was brilliant because it gave you three accounts instead of two and it wildly increased the chances that you would win the game because you're basically giving yourself 350-50 shot instead of two, which reduces the chances that you're going to lose the game from 25% to 12 and a half percent cuts it in half. So you know, it's like it's clearly just a very good play. The problem is that we can't see that because look at what I just had to do to show you that it was a really good play, right? I had to go through all this stuff and I just say all this meh and whatnot. So this is the problem that we have as decision makers is that getting to decision quality and understanding decision quality is actually quite difficult. It's quite hard to know what the quality of the decision was. And so we use these proxies and the easiest proxy because this is not complicated at all is was it a good outcome or was it a bad outcome? Oh, it was an interception. It must have been a crappy call. Oh, it was a touchdown. It must have been the most brilliant call ever and we use these proxies all the time. And what it does is it really messes up our decision making because it makes people not throw past plays in that situation, right? Or you know, it's sort of like the equivalent of like you go through a red light, you get in an accident and you're like, I'm never going through a red light again. Now obviously in that particular case, it's like settled. We know that those are the rules of the world, but it's very rare that we actually know for sure whether that was a good decision or not. And we're changing our decisions all the time. So I'll give you like a super simple example. You hire someone. They work out. You say, I'm really good at hiring clearly the process that I use to hire that person is great, which may have just been like a friend recommended them. I had lunch with them and I really died, you know, which it was a very bad hiring process, by the way, but like it can work out or you hire someone. And they don't work out. And you say, well, obviously I make a made of a stake. I shouldn't hire someone with, you know, that fit that profile ever again. And both of those are huge mistakes, but we make those all the time in hiring because we don't realize like how noisy hiring is. I mean, when you hire someone, it's about 50, 50 to work out. If you've got a really good hiring process, you can get it to about 60 for eight. And that's it. And then we're trying to sort of like, we change the way that we make decisions going forward based on that good or bad outcome on the one time, on the one hire. That's not. But that's really the problem that decision making under uncertainty is causing. Then there's this other problem that occurs, which made me think about the Ellen Langer, the Ellen Langer quote that made me think about it, which is that for a lot of people, we still don't want to make decisions under uncertainty because we're so uncomfortable with it. The idea that somehow we can't determine what that outcome is going to be, which is just how life is, that we will go in one of two directions. I think there's sort of two profile. One is this sort of, you know, analysis by paralysis, right, that trying to continually gather more evidence to try to get to a hundred percent certainty before you're willing to make a decision. That obviously is very bad. It really slows you down. It's incredibly bad for innovation. How can you innovate under circumstances where you need to be a hundred percent sure? That's actually the opposite of innovation. But this is what happens, and it particularly happens in settings that do a lot of resulting, which is an enterprise setting. And the reason why I know an enterprise setting does a lot of resulting is because they have something called a post-mortem, and a post-mortem is literally resulting. It's like, "Yeah, we had a bad outcome, I'm now going to put you in a room and quiz you about this." Right? Like, that's what it is. And when you have a good outcome, there's no quid. So post-mortems are literally institutionalized, resulting. You want to know why your enterprise isn't innovating, who on earth is going to be willing to innovate, because they're going to expose themselves to the probability of a good outcome that's just way too high, and they're never going to do it. Right? So that's one thing. So now they're going to get consensus, and they're going to gather all this evidence, and they're going to come in with a briefing book that they could flap on the table and say, "What could all the work that I did? You can't be mad at me for a try." Terrible. The other thing that happens is, "I'm just going to go with my gut." So it's almost like a giving up. It's like, "Well, I can't possibly know, so therefore, I'm just going to go with my gut." And both of those are huge mistakes. What you actually want to do is live in the middle where you say, "I'm going to embrace the uncertainty, but I'm also going to figure out what I know," and I'm going to do both of those things at the same time. And I think that that's actually a better place to live. And it gets to the core of the title of the book, which is Thinking in Bet, because I think that when you really embrace this idea of thinking about that gets you into this focus on both knowing what you don't know and knowing what you know. And I'll tell you what. So if you have a gut decision maker, somebody's just really into their gut. And they'll declare something like, "We should hire this person. I know they're going to be great," or, "We should launch this product." And I know, "I guarantee you, it's going to be amazing because my gut says so." If I say, "Oh, really? How much do you want to bet on them?" What happens to them every single time? They're like, "Wait, no." They back off, right? And they're like, "I didn't mean," well, no, but you just said you did because you're like, "I just know my gut tells me I have a nose for value. I know what's going to work." And it's like, as soon as you say, "But do you want to bet that they get into this more reasonable place of being like, "No, what I really mean is like 60% of the time it's going to work out." Right? Because that's really what the essence of betting is, is what's the probability and the payoff, right, associated with the two things. And if I put my resources in here, like how often do I think this is actually going to go in a direction that I like? And that's an amazing place. That's an amazing place to live. And that's where I want to get people as a way from the need for 100% certainty, which is completely unrealistic, which you can get to in two ways, because both of them are expressed by a need for 100% certainty. One is, "My gut is great, so I'm just going with my gut." And the other is, "I'm going to do so much work that I never actually do any." And thank you for linking it to innovation, because one of the things that I want our audience to see, our audience or CEOs, decision makers, startup founders, and that this is so valuable, this type of thinking. Even getting stopped by Eric Sidel and told, "Look in the mirror here, achieve." That happens to innovators all the time, and it's usually the biggest critic. And we walk away from them, because they're not confirming the biases we have. And actually embracing them is absolutely core. And I wanted to just say, because there's a quote by John Maynard Keynes, I think it is that, wordily wisdom teaches you it's better to fail, then succeed on conventionally, and it speaks to the Pete Carroll call. But one of the things you taught me through this book, and I think it's so important, is that what you are literally doing with innovation is betting to learn, to get feedback. The way you would, I want to pull the next card, I want you to play the next, so I can build on that if that's the right terminology. But you say that it is often the case that our best choice doesn't even have to have a particularly high likelihood of succeeding. So even when we play the right call, it doesn't sometimes work. And that happens in innovation, it happens in entrepreneurship all the time, and we blame ourselves. Instead of kind of going, "Sometimes, yeah, the conditions are wrong, the market conditions are wrong, it was the wrong time, it was too early, it was a recession." Like you say, the hot hand fallacy of so many people who were like great at buying and selling property until '08 hits, and then they got proved all wrong. There's a lot in there, but I'd love you to riff on that. So there's different ways when we're thinking about the coin flip that you might take, that you can have a positive expectation. Okay, so one is that you could have a coin that is going to land heads a lot of the time, and I'm making the right bet for you, right? So let's imagine a situation where the coin is going to land heads 90% of the time, okay? And I'm going to pay you 20 cents if you get it right, and you're going to pay me a dollar if you get it wrong. So this is one kind of mistake that people make, where they're like, "Oh, I can't take that because I'm only going to win 20 cents when I'm right." But if we do the math, right, 90% of 20 cents is 18 cents. And 10% of a dollar is 10 cents. So you're actually winning 8 cents on that bet. Did you follow? Was that clear? So basically, you're just taking the probability that it lands heads because I assume you're going to call heads. It's a 90% point. The probability that it lands heads is 90%, I'm going to give you 20 cents when that happens. So 90% of 20 cents, which is just 9 times 2, is 18 cents. When you lose when you call heads and that 10% hits where it lands tails, you're going to lose a dollar. So 10% of the time you're going to lose a dollar, which is 10 cents. So now we can take your winnings, which is 18 cents, subtract your losing 10 cents from that, and it's 8 cents every dollar that you bet. So that's an 8% return. And that's going to be a pretty sure thing because it's going to be very rare that that coin is actually going to land a dollar. So let's call that indexing the market. So that's sort of what that is. The market is very likely to go up. There's not a lot of risk in this bet, and the return is reflected because you're getting 8% back, which obviously indexing the market would be pretty good, but it's a lower return. But you have more certainty about what the outcome is going to look like. In the short run, you have more certainty about what the outcome is going to look like. It can be a mistake though, to go that route. It depends on what your goals are, right? Like if you're saving for retirement, that's a good idea, but not, for example, if you're trying to innovate, right? And this is what a lot of practices and businesses push people to do. They push people more for the sure thing. They don't have a big return. But now there's another way that you can make a bunch of down, and let's change it to where you're going to lose 90% of the time. So basically what I'm saying to you is you have to, you have no choice but to call tails. So it's going to land heads 90% of the time, 10% of the time, I'm forcing you to call tail. But what I'm going to do is I'm going to pay you $20 when it lands tails. And you're going to pay me a dollar when it lands head. So now we can do the same math, right? So we can say, well, 90% of the time, you're going to lose a dollar. So that's 90 set, 10% of the time, you're going to win $20. So 10% times 20 is $2. So we can take $2, which is now going to be your upside, and subtract that 90 cents, which is your downside. And you can see you're making a dollar 10. So for every dollar that you bet, you're getting more than double your money back. You're getting a dollar 10. Obviously, that's a good bet. That's really what innovation is, right? So you're going into something that has a low likelihood of succeeding, but a really high payout, if it does. Now, there's more risk in that bet. So notice in one case, you're making eight cents and the other case, you're making a dollar 10, right? So you're getting more than double your money back and the other case, you're making 8%. There's less risk in the 8% bet in the sense that you're going to realize you're being in a much shorter period of time. There's much less swing going around. In the sort of innovative, in the innovative example, there's a lot of risk involved, meaning that it's going to be a lot of failure. But when you hit that 10%, you're going to make a ton of money. How do we handle those two situations? Well, that has to do with this idea of what are you kind of willing to put at risk. So you should be willing to bet a lot more of your net worth on this situation where you're going to win 90% of the time, not all of it, but a lot more than in the situation where you're going to win 10% of the time. So when we think about innovators, basically what we want to do is say, okay, so how can we sort of think about de-risking that for someone? There's a variety of ways you can do that. One is you could be very young, where if you fail, you still have lots and lots of time to do other things and build well. So just being young is a way to be able to innovate. Another thing is, and this is really what venture capital does, is to get venture funding. Because what that does is it de-risk it for somebody who can't afford to lose. And then basically the way that's being paid out is the venture fund is taking the monetary risk. And so they're getting paid for that, right? So they're getting equity in exchange for that. And you're putting in obviously sweat equity and you're getting paid for that equity, and hopefully everybody comes out of that happy. So that's another way that you could do it. Another thing is you could be rich, it's another thing you could do. You'll see that for example with multi-time founders, right? Or somebody was employee number eight at Uber, obviously they have a lot of money and then they start throughout the day. That's a way to de-risk. One of the things that I think is really tragic is that enterprises are de-risk. They have so much, they have so many resources, and yet they don't, they aren't really willing to take these back, which really goes to the innovator's dilemma, right? And that I think is a real shame because they're the most de-risk of all. And yet you really don't see a lot of innovation coming out of enterprise situations. I think, again, because of a lot of some of the things that we've already talked about. But I think it's really important to understand, and then we can take it a step further. And we can say, okay, if it's the case that innovation by definition is going to be decision making under the highest degree of uncertainty. Because you're doing things for which you're cobbling together past experiences to try to figure out what might work in the future, but clearly if you're innovating, people haven't done it before, right? So this isn't going toward Greek and light. This is the opposite of going toward Greek and light, right? Many people have gone toward Greek and light. We know a lot about it, but this is going into the unknown. So if we know that we're going to be, that's going to be the situation under which we have the greatest uncertainty. What we want to do is minimize the number of decisions that we're making under conditions of the worst case of uncertainty and start to stage our decisions out in order to start to gather information so that we can improve decision quality as we go along. And the way to do that is really that's really what minimum viable product is doing. It's basically say, I'm going to do a little thing. So in the smallest way it would be don't build anything, go talk to customers first, potential customers first. So notice I'm not committing a lot of resources to go build something. If I haven't talked to a customer yet about whether they would be interested in me solving a problem, is this a problem for you? Is the solution that I'm thinking of something that you would want? So what that's doing it to be risking you because it lowering the resources that you're putting into the decision before you take the next step, right? And then you could start to think about like minimum viable product and that's what that doing for you. Now, again, a lot of time people do the opposite where they build the whole thing. And then they're sad that it doesn't work and it's okay, but you put yourself under the stress of the most uncertainty possible when you could have actually pulled that back for yourself a little. That speaks so much to the idea of going all in on a hand. And I thought about the concept of betting the farm. We saw this with I don't know if you ever saw the case study of CNN plus and they did it in a vacuum and didn't right to learn but I thought it was a nice segue for a huge section of the book. There's actually two chapters really that are dedicated to this perspective taking. How do I guess the devil's advocates involved? How do I engage diverse opinions, neuro diversity, et cetera. I told it was important for us to say a word and not because it really is something you build towards and this was Eric Seidel talking to you there saying stuff that is uncomfortable to hear. And as you say, do I take the red pill or the blue pill and actually go, I'm going to see the world as it is because that's actually where the learning is. Yeah, and I want to say I just about that, I want to say there's a battle that we have within ourselves between what we want for ourselves in the future and what we want for ourselves now. So I want I want to eat the cupcake now, but I also want to live to be a hundred, right? And there's tension there. Do I make myself happy today or do I make myself happy in the long run? And that's really what I was confronted with and we're all confronted with when we get feedback like what Eric Seidel gave me because I could just say, okay, he's an asshole. And the reason I could say that is because he wasn't making me feel good, but what he was really trying to tell me is do you care about being a good poker player in the long run? Because if you care about being a good poker player in the long run, you're going to have to recognize that sometimes you play hands like rap. Most of the time, actually, sadly, because it's decision-making on Earth's Earth Day. So you're generally the default is that you're making mistake. Okay. So the first thing that you have to do is you have to make a decision that I want to see the world as objectively as possible because that is good for my long-achieving my long-term goals. It may feel bad in the moment, but I have to be okay with that because this will make me happier in the long run. And I really need to take care of future me because there's a lot more versions of future me than there is me now, there's infinity of those, basically, and there's just me. But that's actually a really hard decision to make. So I really encourage people to try to make that decision that it's okay if you make a mistake now because one of the things that I think one of the best ways to get yourself into the situation where you're willing to make that decision is to, I guess, no pun intended, or pun intended, have your cupcake and eat it too. So the way that you do that is say, well, and this is kind of what I did, remember I told you that you'd be walking the halls of a poker room and people would be like, oh, that don't get so bad and I got so unlucky or, oh, I plead so great and did it. It was always the same thing when they were winning, it was because they were freaking brilliant. And they were losing, it was also because they were brilliant. And the other people were just lucky and they were unlucky. And that kind of horrified me. So I said, oh, well, I don't want to be like that. And one of the ways that we really derive a lot of feeling of self-worth is through distinctiveness. We like to be distinct from other people. This is really a lot of how politics works. It's not just about belongingness to a party, it's about being distinct from the other party. So distinctiveness is actually a big driver of sort of identity and feeling of self-value and really moving along that positive self-narrative that Kahneman talked about. So you can instead of just saying, oh, I feel bad because someone said I mean thing to me or they gave me negative feedback, if you say, you know what, I'm going to be really open-minded to this and we're exploring with, think about it, you can then say to yourself, I'm doing something that other people aren't willing to do and that makes me special. So that's having your cake and eating it too. So that's really where I came to in that. Was this idea of, okay, I still get to feel good about myself because I'm doing something that these people who I'm listening in the poker halls and the halls of the casino aren't doing. And that makes me feel good about myself that I'm obsessively trying to figure out what my errors are. So I would really highly encourage that. Now, there's a problem though, and there's two problems. Problem number one is it's hard to actually get other people's opinion. And the reason is that other people have two issues. One is they're generally going to try to agree with you, particularly if they're your friend. So when you're trying to get their advice, they're often just going to give you what they think you want to hear, which is going to be affirmation of whatever choice that you wanted, or they made, what I said before, try to conform the outcome to the decision quality. In other words, oh, you must have made a bad decision because, or you must have made a good decision because. So you want to avoid both of those problems. So the solution that I came up with, which is one that I advise businesses to do, it's actually how I set up their decision processes, really comes down to this really important idea, which is if you want to know what someone thinks, don't tell them what you think and certainly don't tell them the outcome. So the way you see this in business all the time, so I'll give you the poker example and then the business is in poker, what you'll see is, oh, I had a hand and someone raised in front of me and I had a screen and I re-rate them and I call and they, they called and then the word came in eight, nine, three and they chapped and I bet and they raised me and I, I called and then the next card they did and all the, and then it turned out they had a skin. Right. So like that, I've literally told you everything that happened to the hand. Not only that, I've told you how it turned out and I've told you all of my own opinion because if I raise with a screen, it means my opinion was I should raise with the a squeeze. So I've told you everything, right. So now if I ask your opinion, well, you know, I lost the hand first of all. So you're probably going to look for error and you're probably not going to want to make me feel bad. Like, I mean, now I've got all these problems where I'm not actually going to know what you think. So the way that I started describing hands was this, I'd be like, someone raised in front of me. I had ace cream. Here's how many chips we had. They'd been raising a lot before that. I, I came across at that point. I was pretty tight because like I had, I was on a bad street cards, you know, well, I'll give you some details, right? And then I'll say, what do you think I should have done? Now notice I already did it. I know how they interned out. I know what I did, but I'm not telling you. The reason I'm not telling you is that I want your opinion. So this is the best way to get somebody to tell you you're wrong because they don't know that they're telling me that I'm wrong. All right. So, so this happens all the time in meetings. How many times have you been at the meeting where someone gets up and they say, well, I'm thinking about this go to market motion. And here's the strategy that I'm considering. And here's all the reasons why. What do you invest in my company? Yeah. Right. Look here, you're not, there's no possibility that that team is going to give you the feedback that you're actually looking for, right? So, so a lot of what I do in my consulting work is try to make it so that people aren't sharing their opinions with each other before they get everybody else's opinion. So I create like these online forums where people have to go and fill them out before they come into a room and meet with each other in order to sort of recreate that thing that was happening in poker. And the science is actually headably strong on it that you need to be eliciting people's opinions independently because that's the way that you actually capture the diversity of the group, the diversity of opinions, the diversity of thought in the group is to allow them to express those independently because otherwise there's too much crossed contamination and group fate and striving for alignment, which I think is the stupidest word ever and all that stuff. Okay. So that, that's kind of like the first problem that we have that we really want to address by just keeping your mouth shut. It's the best way to get people to tell you you're wrong and you ought to be really happy with that happen. Like you're eager to get people to tell you when you're wrong, so you have to set up systems to let them do that. They won't naturally do it. The second problem is that we're actually really bad in the moment of making a decision in taking into account the future. So this is something that, again, like Daniel Kahneman talks about, he uses the word in it. How Hirschfeld field, who has a great book called Future You, that that's also, he really talks a lot about this problem too, that we're just not very good at it. You know, where we get caught up in the moment, we can't sort of get to that 10,000 foot view. So one of the ways to actually get really good perspective. So we want to think about getting the perspectives of other people, but we also want to get the perspective of us in the future, right? So let's imagine that I'm playing poker and I'm losing and I'm thinking about going and rebying. And one thing that I know is that what I'm losing, I'm going to be actually a pretty poor judge of why I'm losing, because I know that I'm going to have this tendency to blame it on luck. So if I go and I decide that I'm going to go buy in, I could stop myself and say, when I wake up tomorrow morning or next week when I look back on this, do I think I'm going to be happy with this decision? So I'm sort of recruiting my future self to now sort of give me some advice here, right? And the answer is to be no, sure, can you think you are an idiot? And so maybe that's going to stop me from buying in. So that's one thing you can do is kind of have this interrupt of like, how do I feel about this? Like when I look back on this in a week, am I going to be happy I made this decision? You know, this comes up in like arguments with your spouse a lot. When you're about to say something, I'll be good to say like, I'm going to regret this tomorrow. Oh, I love, I love how you go this, because I miss a lot of people. It's when Marty McFly reads Marty McFly in the future. Right. And it's like, don't do that. Right. Back to that. Right. Don't kiss your mother, don't do it. So that that's one thing that you can do is that interrupt. Another thing you can do is set something up in advance. So what I can do is I can say there's this class of decisions that I think are going to be pretty bad at. And so I'm going to set up what's called a pre commitment contractor, a Ulysses contract that's going to actually improve the chances that I stick to it. Katie Milkman, who has a wonderful book called How to Change has done quite a bit of work on this. David Eagleman is writing a book on this topic as well. And a pre commitment contract is basically, so we can think about the strongest form of a pre commitment contract. Let's imagine that I don't want to eat ultra processed food. But I know that when I'm hungry and it's late at night and the convenience and whatever, I'm going to eat that. I can throw them all out of my house. Right. So I'm not in the moment of the decision, right? Like I don't want any, I'm not hungry at the moment. And I'm recognizing that when I get hungry, cause ultra processed foods are quite addictive. I'm probably going to want some. And so what I'm going to do for my future, I'm going to protect my future self. And I'm going to throw all this stuff out of the house. Piece of that, that's one way to sort of take time traveling and bring it to your advantage. Now you can't always do something and drastic is that. So what I can do is just actually make a commitment and say, I'm not going to eat ultra processed foods. And then it's really good if I bring you into it and I say, Hey, can you help me? I'm making commitment, not to make ultra processed food. So now you become like an accountability buddy for me. And that's going to really reduce the chances that that I do that. So so that's another form of perspective taking, right? So there's getting the perspective of other people and how do we actually do that with high fidelity? But there's also getting the perspective of the future version of you, which is also then for your decision making. I just want to remind our audience, that's in quiz, the idea of kill criteria. And as you said, your accountability partners had you go, okay, if you're full below, if you lose $600, you're right, no matter what, because as you said, in that state, you're probably in fight or flight mode and your decision making is flawed. That was, I thought that was so, so important to talk about. And quiz, I'll link to that as well that episode. The book is brilliant as well as thinking in bet. I love this quote and I, I thought we really needed to share this because the book is ultimately about you having a better life. Yes, it's brilliant for business people. Yes, it's brilliant for decision making and any context, but one of the things I loved was a quote you shared by artist and writer, John Copto. And he said, we must believe and look for how else can we explain the success of those we don't like. And it reminded me, I had on the show, Carol Tavareson, a few weeks back, read, read, book mistakes were made, but not by me. And she talked about how, say if you're having a disagreement over a decision, she said in one case, psychologists did their study where they changed the author of the paper and then people all thought it was a great idea because it was their own. And you say the key is that in explicitly recognizing that we feel the night come as a best, we consider a greater number of alternative causes more seriously than otherwise we would have had. And that's truth seeking. So this concept would inside an organization. And here you share Robert Merkin's work where the kudos concept, and we won't have time to go into all that. But I just thought the importance of setting up a like the government has a line. What's the word you said, you know, sorry, a dissenting line. So people have the right to dissent. In fact, you're seeking out that dissent. Like you said, when the customer goes, yeah, not interested in that, I would never buy it. It hurts a lot, but thanks for saving me a lot of hassle. I think what you just said is kind of interesting because I think one of the reasons why people don't do that is because they don't want to share that people don't want it, right? Like, you know, it's, it's that thing of like, there's, I can't remember who did the study, but people want it, you know, it's why do people avoid like medical tests? And it's like, well, because somehow if you don't actually get the test, you don't have it because they don't want to get the bad news, but that's actually a real thing. No mirrors. No mirrors anywhere. Right. Exactly. So I do think I do think that that's, that's, I think there's something very deep in what you just said, right? Like, well, why aren't people just going with a wireframe or something? I didn't go talk to customer, you know, so, yeah, I mean, anyway, whatever, I did, that just made me think of that. And it happens in innovation all the time, we fall in love with our idea, you know, the idea effect where you're kind of going, oh, I put somewhat jeffered into this thing. And in innovation, we do that all the time where, and then to the point of you have an erics side L who's a gift to you, who goes, look, I work in regulation, this thing's never going to fly. We're never going to get past regulation. You're going to go, la, la, la, la, I don't want to hear that type of thing. So it's, it's this idea of setting up a, a panel or some way for the organization to hear the truth is so, so key. And I thought we'd, we'd end on your advice for that. Maybe work that you do with your consulting for people to set up the dissent line in an organization. Yeah. So basically the way that I do it in organizations is I set up the decision-making process so you don't need a dissent line. So we can think about solutions that are after the fact solution. So a dissent line would be an after-the-fact solution where people can dissent with a particular policy. In this particular case, maybe they weren't decision makers in the policy and you would need a dissent line. The groups that I work with tend to be pretty small. So we don't really have that problem. So we wouldn't have a reason for a dissent line. But that what that would be like in, in the CIA, there's some group of decision makers who do something and then the rest of the, I mean, obviously you're not going to include every CIA agent or state department person, so then you have a dissent line for that. So that's kind of a, the decision has been made now and people are dissenting. And I think that that's healthy. It's not just not something that I have to set up. So that's an after-the-fact solution. The other after-the-fact solution would be you make the decision, you get the outcome and then you go back and do what I call knowledge tracking, which is, because I think part of the big problem that we have is we think we knew things at the time of the decision that we didn't know. Right? So when we think about like, why are people so critical about Pete Carroll's decision is sort of like, they feel like he should have known it was going to be intercepted, right? That's really kind of what the problem is. Like, how could you not know that? And so what's happening is that you're sort of bleeding your state of knowledge today into an evaluation of the state knowledge at the time. So what I ask people to do is actually write down what did you know at the time, write down what you knew at the time, write down what you learned afterward, and then figure out, could I have known that beforehand? So sometimes the answer will be yes, I could have known it beforehand, then you have to ask what the cost was, because you know, you need to make a decision in a week and it's going to take you a year to know it. It doesn't matter if you could have known it beforehand before you made the decision because you didn't have time. If you could have known it within the time frame that you needed to make the decision, then that's fine. You just added to the decision process going forward. But mostly the answer is going to be no, I couldn't have known it beforehand because mostly what you learned is the help go, which you can't know before. So those are kind of after the fact solution. What I do with the organizations that I work with is I really try to come up with before the fact solution. So the before the fact solution would be, I'll give you an example from first round capital. So I'm a special partner at first round capital. And what we do there is we have a point of view, which is expressed in a decision rubric about what are the things that you should be judging that would tell you whether you should invest in something. So you can think about broadly market founder team product. Okay. So let's say about broadly those four categories. And then within each of those categories, we have what we call mediating judgment, which are things that you would have to judge in order to be able to judge the founder well, or things that you would have to judge in order to be able to judge the market well. So you could imagine rating something about Tam. And that would tell you something about the market, right, about the market size is it growing? Is it shrinking? Those kinds of questions. So we have this rubric for each thing. They rate the favorability of those things on a scale of one to seven. So notice we're sort of quantifying qualitative opinions here. And then when they make a rating of say the market as a whole, they give a rationale for why they're giving that rating. All right. So here's how we sort of avoid a dissent channel. A founder comes in and pitches that looks normal. It's a normal pitch. The founder leaves none of the partners say anything to each other. Remember that thing about keep your mouth shut, right? Though none of them say anything to each other. They go to an online forum, which is this rubric where they fill out all these judgments on a scale of one to seven, and then they give some rationales and their forecasts and so on and so forth. That all gets dumped into a spreadsheet. So now we know where they disagree with each other because it's already been recorded. So we have a free recording now of what the point of view of everybody was, right, at the time. And now they have a discussion about those opinions. So we're not actually soliciting anybody's opinion in the group discussion because we already know them beforehand. This is really important. So then you lose that cross contamination and all of that stuff. And now what's driving the discussion is the places where people disagree. So the discussion will kind of go like this. Everybody thinks the market is pretty good, but there's a lot of disagreement about whether this is the right founder. And that's what you talk about. So literally it's an acknowledgment that people are pretty much in agreement about the quality of the market. They like the market that that person is playing in, but they have a lot of disagreements about whether this is the right founder to invest in. And that's where we're like, why do you rate the pie? Why do you rate them low? Why are you sort of met? You know, and we're sort of driving discussion around where the dispersion of opinion it. And then they make a decision. Okay, so notice now the dissent is already built in, right? Like what we've done is we've created a decision process that gets the dissent to show up. And this now makes it a lot easier to do a look back of the decision, because I know we've all been in situations where someone was like, well, I knew it was going to turn out this way. And I totally disagreed at the time and you're like, I was in that meeting. No, you didn't. And this is like this memory creep that occurs with hindsight bias. But now we can just go look and we can say, okay, we invested in this company. It did really poorly. What was our point of view at the time? Sometimes you can say we invested in this company. It actually ended up raising a really strong series A or a really strong series B. What was our point of view at the time? Did we miss them, right? And now we can actually go look and it just, it makes the decision process so much better. It makes it so much easier to close feedback loops. And it's a thing that I could never do in poker because in poker, it's this real time. I can't go off and be like, hey, I'd like to get every piece of previous, could you write them all down before I decide what my move is going to be? So it actually solves a really big problem that like poker players have. And the science is very, very strong. This is a place where the science is really good, that this way of making decisions really improves decision quality. And then you don't need a dissent channel. Brilliant. One of the things I just wanted to really nail was that John Cocteau quotes what I share that was, if I'm a company, like your company that you work with there, and I look at somebody else's, if I'm not trained in your way of thinking, I'll kind of go, those idiots can't believe they made those mistakes. And what you're trying to say is, put yourself in their shoes. So have empathy. Actually, make the decision, have an assumption, capture that assumption, and then check against what actually happened in that as well, which is such an important thing in all aspects of life. Because we do it in like that fundamental attribution bias where we go, those people made such a terrible decision, but when it happens, you, it was local the time and you're trying to inoculate that. Right. Yeah. And I think the other thing, I think the other really important thing to understand is that, let's say that I'm playing in someone who's really bad. It doesn't mean that they're worth it everything then. So I ought to be paying attention to that and try to figure out what it is that they do well. Because I need to know that, first of all, I need to know that in order to play well against them. But I also need to know that because I might be able to incorporate that one thing into my game. So when you're looking at another company and you're like, what a bunch of idiot there, we're on, you should be like, but what do they do it better than us? Like, you should actually be super obsessed with that. And I think that people are the opposite of obsessed with that. I think that it's so, you know, it's so nice to eat the cupcake and just be like, I'm better than them, whoa, right? And boy, that this is really bad for improving and learning, right? And that outcome by, as you talked about, when most of us in business school, we learned by case studies, but we know that we know the outcome. It's crazy. Yeah. And, you know, you're, you're hypothetical. What would you do if these were your chips? This was the time that these are the cards. What would you do is a much better question or very hard to bring into the education system at this, at this late stage? Last one for you. This, this dawned on me, right? So I recently, I was thinking about, there's a guy I know I had on the show. He's retiring from a major consultancy. And I thought, I wonder what would happen if that consultancy interviewed him and created a digital twin of all his knowledge. And then do they own that IP, does he own the IP? Can they use it again? Do they hire other people, et cetera? And I thought of the bad view. I thought myself, I wonder is there going to be an attitude boss, right? Which should be the decision making, thinking embeds decision making applied to business. What would you do? Because, and then I thought to your point about chess, it would be easy to do for chess, which is why we've seen the AI's credit that beat Lisa at all in the game of Go. We've seen deep mind, chess, scary cast, robots, et cetera, et cetera. And I wonder, can, can AI play poker, given all the different permutations that happen in poker versus chess being a very mathematical game? Yeah. So AI has gotten pretty good at poker, actually. And a lot of it has to do with poker can sort of be boiled down to game theory optimal. So this goes back to John Nash and a Nash equilibrium table. So there is some art to it because you have to have some sense of what the range is of hands that your opponent is playing. But basically, you can generate a table for any situation that tells you what you should do with this range of hands, this range of hands, this range of hands in any situation. So clearly, you have to be smart enough to memorize that. But I would argue that an LLM is going to be better to memorize in the stables than I am. But it took a long time, right? So deep blue was in the '90s. They started getting pretty good at poker, kind of the mid, like 2015, maybe, somewhere in there, where they started getting, you know, being able to beat people heads up. And now I think they're probably pretty good in at least low-state poker games. So there's some, you know, but again, there's a game theory optimal solution to any situation that you're playing in poker. You know, I retired in 2012, so if I tried to go back and play now, I would get slaughtered. And it was willing to do the work of really memorizing those game theory optimal tables. People hadn't really generated them back then. We understood that idea, right? Like, we had some knowledge of it, but you couldn't do the kind of data work that you can do now. And now it's kind of insane what people have available to them in terms of these solutions, that what we call poker solver, and because I retired in 2012, I haven't studied the poker solvers. And so I'm sure I could beat you at poker, but I would not try to beat a professional at poker anymore. That would not be good. In terms of, you know, what I do, you know, I'm sure at some point there'll be something that can do what I do. That's fine with me. You created, though. That's the F I guess. Your point about the Pete Carroll, Pete Carroll, probably, they probably use real-time AI to give permutations of the next play. Well, that's the thing is that they have, they actually, so I think I'm not sure that they're allowed to do that, but because the NFL makes you use paper. But I know that that's sort of what those binders are, right? It's like they have these binders and there's someone sitting there sort of looking up the situation that they're in. Pete Carroll knew that he was making a good decision there. He's defended it. By the way, other people have defended, Bill Belichick defended it. I think generally within the, you know, sort of like the analytics people on poker, the people who are more analytics minded, they all understand that he made a good decision. I mean, it's not like he got fired. The fans were mad. The pundits were mad. But the people who really knew were like, yeah, I mean, I made a good decision. But you know, this is also a bind, right? And just sort of speaking to this because of innovation. One of the things that I think is really interesting about the Pete Carroll situation is that if he had handed the ball to Marsha on Lynch twice, which would have been a very bad decision, and that's all he'd done any, it lost the game, nobody would have cared. And the reason that it was, it was expected. And so there's this interesting thing that happens, which is if you do sort of the expected thing, the not innovative thing, the thing that's like going through a green light, nobody gets mad at you when you fail, nobody, nobody, because they're like, well, what could you do? You went through a green light. And it goes back to that postmortem problem, right, which is, okay, so now you're going to make me gather way more information than I need in order to decide because I want to defend myself in the room where I'm going to make sure that I consensus across the whole company or I'm going to make sure it's just SOP, right, because that's going to get me out of it. It was the innovation. It was that it was unexpected that made it so that that outcome got such a big response. So when you do something that's SOP and you win to it, people are like, good job. And when you lose to it, people are like bad luck. When you do something that's innovative and you win to it, they call you a genius, which is what would have happened had that play worked out. But they also call you like an idiot and a moron if you lose to it. So this becomes a real dilemma, right? So in a startup, the expectation is failure. So you can innovate all you want, right, because there isn't, there's no SOP, right? Everything is innovation. It's just the expectation. But once you get into an enterprise setting, that's not the case anymore. And now all of a sudden, it's like, it's very hard for people to do new things because the critique that comes from failing in those things is so huge. So yeah, like for Pete Carroll, the insiders know it was a good decision, but look how much flaky tape. Ten years later, people are still like, what an idiot. Great point then. One will finish on. There was one line I wanted to share as a finale and it was you talking about yourself, but I think we could all derive a lesson from it. You said, you rewired your reward systems and habits to get your dopamine fix by trying to be the best credit giver, the best mistake admitter and the best finder of mistakes and good outcomes. And I thought that was what you could do in businesses, what you can do in sports, what you can do, what you can teach to your children to go. That's all you could do. Actually, if you could, if you do those things, you'll make progress and you'll inch forward and you'll win the towing coin flips over a period of 10 years as they come towards you. It's always a massive pleasure happening on the show. Annie, for people who do want to find out, I know you have a sub-stack channel. You have courses now and thinking in bets. Where's the best place for people to find you? Yeah. So I've run cohorts on maven.com as people want to take a class for me. The other thing you could do is show up at Wharton's executive ad, which I do twice a year, a whole host of other amazing professors like Murray Schweitzer, Joe Simmons, Adi Weiner, Kate Massey. I mean, it's really like, it's an amazing issue. He wanted to go take an exact end cast word, you know, check that out. I have a sub-stack which is called thinking in bets and then, you know, the easiest place to get in touch with me is through AnnieTube.com. Also, of thinking in bets, quit and how to decide AnnieTube, thank you for joining us. Thank you so much for having me.