(upbeat music) - Hi, I'm Eric Dots. - And I'm John Wessel. - Welcome to "The Data Stack Show." - "The Data Stack Show" is a podcast where we talk about the technical, business, and human challenges involved in data work. - Join our casual conversations with innovators and data professionals to learn about new data technologies and how data teams are run at top companies. (upbeat music) - Welcome back to the show. We are here today with Evan Wimpy from Elder Research. Evan, welcome to "The Data Stack Show." We are super excited to chat with you. - Awesome, yeah, thanks guys, happy to be on. - Okay, give us a quick background. - Sure, so yeah, you mentioned Elder Research. It's where I'm a director of analytics. I've been there about five years doing analytics consulting work as a data scientist and now tell another data scientist what to do. And whenever I get a chance, I'm trying to make them laugh as well, do a little data comedy. Sometimes they're able to research and sometimes for anybody in the data space who could use a little bit of laugh. - That's awesome, Evan. So one of the topics we spoke on before we started the show here was synthetic controls, kind of a data sciencey topic, so I'm excited to dig in more on that. Any other, anything else you want to cover on the show? - Yeah, I won't be able to start myself from telling a couple of jokes, but yes, synthetic controls are great and would dive into some of the technical stuff. - Great, well, let's do it. - Yeah, looking forward to it. - Okay, Evan, I think you are the only professional data comedian that I've met and so, which is a special thing for the show here. We're very excited, but I'm going to give you the option. Do you want to tell a couple of jokes now or do you want to make people wait on the edge of their seat to the end of the show? - I think we want to build their hopes up all the way until the end. - Perfect, yeah, let them down. - Let them down in one giant crash there. - Okay, great, yeah, okay, perfect. So let's start earlier in sort of your data history. So you have two master's degrees, is this correct? - It is, yeah, you got to make a decision at some point. - Yeah, well, have you decided, have you... - No, I'm going for another one sometimes. - Okay. - Yeah, triple crown. - The audience will learn quickly how little I know and they'll think, wow, you can get two master's degrees and I don't know anything. - What's wrong with our education system? - What are the degrees in? - Yeah, so first master's is in Applied Economics, which I think is pretty close to the data and analytics space. So it was from East Carolina. I got that one a dozen years or so ago, just before it got into the Marine Corps. So got in the Marine Corps was a finance officer, so don't picture like kicking down doors, but more like sitting behind a computer. - Kicking down spreadsheets. - Yeah, exactly. - Yeah, I did not have an M16. I was issued Excel spreadsheet, that's it. (laughing) - Basically the same thing. - Yeah, just assemble and clean your weapon, please. - Yeah, data cleaning is what we do. That was all my boot camp. So yeah, I would say that I still use that degree, some, you know, it's econometrics, studying. You learn a lot about linear regression and all the assumptions behind it. You learn a lot about how people make decisions. What I got on the Marine Corps was working in finance for about 10 minutes. And the place that when I was working in finance, I saw opportunities for using analytics. My wife was actually at the time in a master's program in analytics. And so I thought, wow, we could be doing this at the bank. We could be doing this at the bank. And there wasn't a whole lot of appetite for it. I said, you know what? I wanna go somewhere where there is appetite for it, but I need to learn a little more, as I couldn't stand my wife knowing so much more than me. So I actually followed her to the same master's in analytics program at NC State. And so that was the second master's. - And how does you and your wife having, both having a master's in analytics affect your decision-making process together? - That's the big question I have here. - That's a great question. When you think about like different weights that you give to different input variables, her weight is a one, mine is a zero, so. - Okay. So it's actually pretty easy. He's from a modeling. - Pretty five-air, yeah. - A very simple model. - Yeah, very simple model. - That's great. - Percemonius is what we are. - That's great. Okay, so did you go to work for elder research after that degree? - I did, yeah. So elder research actually recruits pretty heavily out of that program at NC State. My boss is a graduate. The CEO of the company is a graduate. And so when I was there, those are clarifier real quick. Elder research is like an analytics consulting firm. They don't do elder research. They're just founded by a guy named John Elder. You haven't heard John Elder. As soon as this episode's over, go and read John Elder and read his books and listen to his talks. He's great. He really motivated me when he came and spoke to our graduating class. So yeah, I was really excited to get a foot in the door with Elder. And I've been there for a little over five years since then. - And what is, so analytics is super broad. What does elder, do you have a specialties at elder or? - Yeah, this turns into a long answer. Analytics is super broad. Elder research is also very broad. So in the, when I talk to the consulting friends, like management consulting, I say we're very specific, we're very focused. But when I talk to data and analytics people, we're really generalist within data and analytics. So if there's a specialty, we have a pretty big footprint with US government, department of defense. So that turns into a lot of, if you can believe it, fraud detection, threat detection, network analysis. So I think that's where we probably got the biggest start and the biggest growth, 25-ish or so years ago. I don't work with government clients. They have higher standards. So I work with-- - Is that quality or security risk in the area? - That's security, look, that was a long time ago, okay? Yeah, so I work with commercial clients and there it's very broad on the types of problems we tackle. So you think of any big corporation, sometimes small startup, more often bigger fortune thousand companies that have marketing teams that need to know who to market to and how to market to them. They have operations and logistics that are building things, moving them around. So they have a lot of data and they need to make a lot of decisions. And so we sort of help with that. - Cool. Well, John, you were really, I think we should just go for the jugular and bite into the juiciest topic. And I cannot wait for this topic, but I think you're more excited than me, so-- - The data jokes? - No, well, we know we're waiting to the end, but of course that's-- - No, yeah, I wanna dig into the synthetic controls. We talked about this a little bit before the show, but I was just sharing with Evan, it was on vacation recently, got off on an exit in the middle of nowhere, there's a gas station under construction, and then there's one that looks like it's been there for 20 years. - Right next to each other? - Yeah, right next to each other, like same exit. - You're making a right turn to get it to you. - Yeah, yeah, right. Same exits, they're on the same side of the road, and let's say there's less than a quarter mile between them, so pretty much next to each other. So you go, and then the gas station, it's doing great, like the cars are full, the pumps are full, people are going in and out, like it's doing great, but you walk into it, like it's kinda dirty, you know, you walk in, it's kinda got that, this has been around for 20 years, like not really cupped up, the restrooms are gross, like that type of thing, right? And then my driving back out, pay more attention to the new one, so it's a quick trip, QT, which is popular in this area, and then they kinda have this business model, it's like really, always really fast, like keep it clean, they've got like so clean. - Yeah, unreal. - They've got like a kitchen, you know, like full food service, like a whole deal, just to kind of a different experience. So I'm thinking in my mind from a data background, that general manager, or whoever's owns that particular location, like knows that it's coming, they know that A, the other place has this strategic location, it's just slightly closer, right, to the interstate exit. But bigger, they know like that got feeling of like, man, things are gonna get bad for us, but they have no way of quantifying that, and let's say that like that general manager's like, man, I wanted $250,000, $500,000 worth of improvements to try to compete, or maybe we should go ahead and shut this place down instead of letting it, you know, bleed out for a year or two. But there's no like quantity, there's no way to quantify that, I highly doubt either like anybody there is trying to quantify it. So there's like the background, we were talking about synthetic controls, and I'd love to hear past it to you, like how would you handle that situation, or a similar situation? - Yeah, I think that's great. And I, John, I sort of share that with you, just think about things in terms of data and analytics, and what it would mean for data and analytics out from the real world. So I loved it, that's what came to your mind in this dingy old side. - We put like the bars over the window. - Yeah, you're taking your kid in here, like my wife is not gonna like, if I told her how dirty this thing is, it's like where I'm taking my kid to the bathroom, and you're like, how could you quantify the potential failure here? - Like you walk in, and you walk in here, and you're like, don't touch anything, totally. - Yeah, the young kids, don't touch that, don't touch that. - Yeah. - We got hands and a thousand in the car, don't use that. - Exactly. - Yeah, right. - Yeah, so I'll even take it one step forward before I set up synthetic controls, but let's say the quick trip opens, and then you've got six months worth of data in that new gas station, and even then it's really hard to quantify what happened, because you know, almost certainly there's going to be an impact of the quick trip opening, in this case there's like, this sounds like a very extreme scenario where, okay, sales are going to plummet, because everybody's going to go to the nice lane quick trip, but what if there were already five gas stations there, and now they're opening it up? - Sure. - It's really hard to measure, even after the fact what happened. And so this is where synthetic controls, we've used this with a few clients that older research, and it's been around for several years, but I've only heard of it for about a year ago. I'm pretty slow, but man, it's been super powerful, and we've blocked it. So I'll try to sort of set it up where in the context of this gas station, where if it's just a single gas station, there's not a lot you can do, but let's say it's a franchise that has a few hundred locations. Well, ideally when you want to measure the impact of something, you've got a test and a control, but you can't do that in this case, you can't have a test scenario where you say, okay, open the quick trip, now go back in time and don't open the quick trip, and then we can measure what the impact was. But what you can do is try to synthesize that control. You're gonna have the test scenario, and if you've got a hundred other, maybe not even a hundred, if you've got 10,000 other gas stations, then you can find some combination of those gas stations that look similar, have similar demographic, have similar sales numbers, same location or whatever, similar location, and you can say, you know, whatever, these stores, A, B, C, and D, they map, maybe with different weights, maybe with nonlinearities, but they can predict pretty well this store E. So they can tell us this is what, if we can predict store E based on these other stores, and these other stores are not gonna have a quick trip open right next door at the same time. So you're synthesizing a new theoretical store from a combination of other stores that would closely map this one that's about to be impacted. Exactly, spot on, and then as long as you've got some historical data, you can go and validate those stores to say, hey, does it actually work, you know, train it, and then validate it on out of sample before the test event, so you can say, hey, it does a pretty good job, it does a very close job of mapping what this gas station and question is. One question quickly, so how intensive is the process of building the control group, actually synthesizing that, right? Especially, so you mentioned, you know, if it's a hundred stores, and, you know, we'll go back to the kicking down spreadsheet saying, if it's a hundred stores, and I have them listed, and there's, you know, maybe 50 columns of data in an Excel sheet, right? You can sit down in an afternoon and like do a human synthesis of like, okay, I can sort of filter down to the list, I think that's good. If you have 10,000 stores, what kind of process are you using to try to build the synthesis control? Yeah, that's a great question. And I think the biggest risk in this is, if you've got 10,000 stores, you almost certainly can fit training data perfect on that 10,000 in first stores. You can find some combination that make it perfect. And so you've got to be careful. And this is where I think there's some technical aspects to this, where you've got a hold out set that's still in the past, but there's some human element to this as well, where you're not gonna choose a store in New York for to map to the store, to this gas station in South Carolina, where the roads are very different, the competitors are very different. Even if the sales numbers are perfect and you can map the sales exactly and that's ultimately what you care about, there's some qualitative sort of feature engineering that's gonna go into, okay, the market conditions are gonna be so differently in this location or this type of store than it is for this store that we care about. But ultimately, it turns into basically a feature engineering where you're trying to model this new store and you've got 10,000 features that are all your old stores or all of the other stores. - And as you go through this process, sorry, I wanna keep going, but this is just so interesting to me like the source election. How much do you time do you spend studying the industry, studying those sorts of contextual aspects, like you said, right? - Sure. - Okay, well, if we just, you can write a model to go find the perfect fit, but it's gonna lack the context that would actually make the synthetic control that would imbue it with its deepest levels of meaning, right? So are you spending a lot of time studying, 'cause you haven't, I'm assuming you haven't, you know, managed many gas stations? - I've been to a long time. (laughing) - So is there a lot like, and just from an elder perspective, like you're digging in, studying the industry, like all that stuff? - Yeah, this is a great question. And I think it's at the crux of sort of what differentiates a consulting as a service firm, like elder versus a SaaS or a product firm that is gonna come in and say, hey, here's a solution that you can plug in and do this, versus we're gonna come in and we're gonna charge you money, but we're gonna charge you time too, because we're gonna ask a lot of questions. We're gonna ask what makes this store this. We're gonna try to talk to store operators. We're certainly gonna talk to the people who are forecasting the sales or the impact or the cases. So, you know, a lot of that is, a lot of that you can do almost like in an academic setting where you're doing research to find what you can about managing a gas station, but much more of that is, John's your point, if you talked to the manager of that gas station, he or she knows that the quick trip is coming and they know what is about to happen and they talk to their customers all the time and the context that they have is what you want for that qualitative context that you need when you're building it to. - And depending on the company and industry, they may even talk to other operators and have a good idea of like, oh, like, yeah, I typically trend with like this other store, this, you know? - Yeah, some of them even have an idea with stuff like that. - Yeah, exactly. Yeah, very spot on. - Okay, so ID reeled us for a minute there. - Okay, so we have 10,000 stores. We've done, we've dug in from a qualitative standpoint. We have a model that takes that context. We have a good fit. And so now we have our synthetic control. So where do we go from there? - Yeah, so now we're able. Now, it doesn't necessarily answer the question of what happens when a quick trip opens right next door. If we've never observed that in the past, then it's gonna be hard, but it lets us measure that in the future. There's almost, you know, this gas stations aren't a new industry. New ones have opened up next to old grungy ones all the time. So, probably we've got something close to that has happened in the past. And previously, there's no counterfactual. So we can't measure what the impact was. We can say, oh, look, sales kept going up after that. Yeah, but maybe they would have gone up way more if the quick trip that you've been. So now we've got this in the past. We can say, when a gas station that meets some sort of object, you know, some quantitative criteria, like new build in the last six months is, you know, top 10 franchise for gas stations. And, you know, whatever the criteria is, here's what we have seen as an impact in the past. And that gives you, you know, hopefully with some uncertainty bounds, which this method is pretty good because as long as your model has some prediction interval or confidence window, then, you know, you can be pretty confident. Hey, it's going to impact sales directionally. And here's some sort of measured interval of what we think the impact will be on sales or number of customers or whatever it is. Yeah. I don't want to get too specific on the gas station example as so others got a pretty big footprint in like consumer packaged goods and the hospitality space where there are a lot of franchises, there are a lot of products that have competing products. And so that makes a great use case for synthetic controls because out of all the tens of thousands of products that some manufacturer makes and puts on a shelf at Walmart, you can see what happens when a Walmart shuts or when they expand a parking lot or when, you know, they put a feature promotion at the end of an aisle. So there's all these events that take place and synthetic controls gives you a way to sort of retroactively measure what the impact of those events were. Yeah. When you start one of these projects, do your clients generally have a really good idea of the question that they want to answer or is it more general like, is it more general along the lines of we want to understand the risk factors for this product line? Is it more general or is it more specific? Yeah, it sort of spans the gamut. I'm trying to debate if I have a favorite and maybe I like the mix and that's why I like being in the consulting industry. Yeah, we've got some clients that have good data and they've got a prioritized backlog of analytics projects and they run agile and they work with their IT and they deploy great stuff and they know, hey, we want to measure the impact of new store openings on our stores and maybe they've never heard of synthetic controls but they know exactly what they're trying to measure and they're asking us to do it. Nice. That's more rare than-- I would suspect that, that's what I was gonna say, like who and how often does that happen? That's a very small sample size. So, you know, not meaning that this generalize every, we work with some pretty analytically mature clients, which would surprise me a little bit because you'd think they're analytically mature. They've got their own data scientists and engineers to do that. But having an outside perspective come in can often be helpful. But oftentimes it's more, hey, the general manager at this gas station has been complaining that this QT is about to put them out of business and we don't know what to do, we don't know what's gonna happen. Can you help us figure out what's gonna happen? And then that usually doesn't start with us coming in and says, yes, we'll build synthetic controls for you and we'll quantify with uncertain estimates. So no, okay, well, what are you trying to do? What are your goals? Very sure. It's trying to measure. Yeah, and it becomes a much more consultative problem than a technical problem. Right, right. I was actually gonna ask a question to both of you as consultants. And this is like a, yeah, this is, I'm genuinely interested. So let's take a franchise for example. We could, we'll just keep it the gas station thing, right? But let's say it's a big, you know, national franchise of gas stations. And so of course they have a lot of analytical horsepower at the company because they need to understand the real estate market. They need to understand, you know, fluctuations in the cost of oil, food, operating expenses, all, I mean, there's huge, you know, the firepower is huge, right? And so you would think, you know, okay, wow, we actually do have a ton of analytical horsepower, right? But there's, in every company, there's a point at which an internal resource specializing in something really specific, it means that you're not working on the actual product anymore, right? So now your analytics team is, you know, sort of embarking on something that is helpful to the business, but is, you know, sort of divergent from the core work that's actually, you know, sort of driving it day-to-day necessarily, right, to answer your questions. So how do you think about when to make that decision? And as you think about your clients, right? Like what should you keep in house and what are the situations that you need to outsource? - So I think the most interesting thing with large clients, say I've got 500 horsepower and that could be 500 people or some other unit of whatever. - Yeah, 500 horsepower. And it's all allocated some of it specifically to like, like we've talked before with somebody that was doing forecasting for a call center for benefits for, I think it's for Amazon. - Oh, right. - Very, very specific, like in Amazon's case. So, but it's all deployed, some of it's deployed specifically to marketing or finance or maybe even a subset of HR or whatever. And then there's some that may be under more generic corporate roles that do, like maybe they do data engineering or maybe they do more generic, they do reporting to roll up to investors or the board, like more generic things. - Yeah, exactly to be. - Yeah, right. But if you had a project that kind of spans between those groups, A, those can be gaps in knowledge between the groups, even if they have the right skill sets. And B, as far as redeploying horsepower, like most people do not have flexible resources even if they have the right resources. So that's what it's like, well, it's a big company. Like, why can't they just do it? It's like, well, this person's practically like, A, they're just focused on this. B, they need to keep doing their job that they were hired to do. And C, they don't have the broader context of all these moving pieces. So I think even if you technically have the worst power, redeploying it is really challenging. So I don't know if you've had that same experience, Evan. - I think that's a great point. I was gonna mention something else, and I will, but I'll touch like, I think that's a great point and it brings an example to mine, which happens all the time, but was really amplified during COVID. So we work with a bunch of like consumer goods, manufacturing groups, and they've got marketing analytics folks, and they've got operations analytics folks. And if you went shopping during the pandemic, you might have noticed the shelves weren't always full of everything. And so marketing analytics folks running specials and promotions on goods that you can't even keep on the shelf is an absolute waste. It's just a marketing waste. And so you've got your operations folks that are trying to supply the right stuff at the right time. And you've got your marketing folks that are trying to push the right products to the right people. And there's not sort of that cohesion in between them that don't market the stuff that we can't keep on the shelves right now. Or start to market the stuff that is taking up too much inventory that was not moving fast enough. And so like that's a pretty specific example, but where these analytics folks and these analytics folks are doing great, but the outside perspective, it becomes easier to connect those dots. - Yeah, 'cause they're each optimizing for their thing. Like I'm trying to sell more in my category. Like that's how you gold me. Like that's where my incentives are. Of course, I'm gonna keep trying that when in actuality, like you said, you maybe you have not enough inventory in that category and too much to another. And yeah, so that makes a lot of sense. - I would say one, the first thing that I was gonna mention that's not as good an answer is I think oftentimes, you know, this comes with, yeah, I work, John Elder founded this company about 30 years ago and he's a well-regarded name in the space and a lot of times just like the change management aspect. So driving some project can be really helpful to bring in an outside, you know, an analytics team that is peer-to-peer with a sales team that can't get traction and can't get a sales director, manager to listen and do this. But hey, we brought in these outside experts that specialize in this thing and they can do this. And now a sales director, not always, but maybe more inclined to work with that or listen or institute some change. - Yeah, yeah, that makes sense. - Yeah, makes a lot of sense. Yeah, it's always an interesting question, you know, build versus buy. Changing gears a little bit. One of the other topics that I'm really interested in that we discussed briefly before the show was understanding model performance, right? So, you know, it's easy for people to think about, like, ooh, there's a data scientist, right? So they're just gonna throw this model, if you don't know a lot about the space, right? Like you can throw this model at something and wow, it does all this crazy stuff and like we get something at the end, but there's this huge element of understanding whether what the machine is doing is actually working the way that you want it to. And I mean, we don't even need to get into the question of AI, right? Because it sort of highlights the insanity of what's going on, but how do you deal with that at Elder? And how do you think about that? - Yeah, I would say that that, you asked earlier what our specialty is and I sort of punted and said, well, we're more generalist. I would say we've got, that's where we've built most of our credentials is on validating models, validating findings. And, you know, if anything, it's tough to talk to clients or prospective clients because the flashy stuff sells the, you know, we can forecast this, we can tell you exactly what this is gonna be. Where what we, what we at Elder, what Elder Research likes to do is come in and say, well, this is how well we can forecast. And this is how much error you should expect. And these are the places where it's gonna generalize well and where it's not going to. And so a lot of the work that we end up doing is actually model validation for work that people have inherent, teams that have inherited or have built internally. And, you know, sometimes that starts out of, we just wanna stamp of approval. Somebody just sanity check this work, make sure it's good, but often that turns into, you know, you're reporting a general performance, but, you know, it's performing well in certain subsets and poorly in certain subsets. Or you've not validated it well out of sample and it's not going to generalize because you've got a leak from the future or, you know, conditions have changed at this point. And, you know, as of this time period, it's not going to generalize well. So I think that is becoming a more important thing is there are more and more tools and vendors that are out there that you can plug in modeling and some of it's super powerful. But it's really tempting to use a lot of really powerful tools and then not know how good they're actually performing. And so we're trying to keep folks grounded in being able to measure how good a model actually performs. - So I think this is a huge issue in the customer data marketing tech space. Huge issue because almost all of the customer data tool, like marketing focus customer data tooling, they're all introducing predictive models like on churn, on predicted next order, like things like that. And I have yet to see one that provides any sort of metrics around like with what level of confidence do we think this thing, you know, it'll just give it. It just spits it out. It's now the persona is marketing people. So from-- (laughing) - So no, I have to-- - I have this respect-- - From the former marketers. - Not very because-- (laughing) - Yes, because we are easy prey. I mean, that's the-- - No, I'm just saying that like the persona is marketing people and they're like, they wouldn't know what to do with like persona. - Yeah. - But the numbers, so like in a sense, like I understand what-- - Yeah, benchmarks. - Yeah, or like model like part. - Yeah, like our values, like, that's like like, what are they gonna do with that? So from a products perspective, I understand why that information would not be shown. But as more data teams like get involved in the space where you get more and more complex and you end up having a warehouse more at the center of your data stack and then you're tying in all these marketing tools, I think it's becoming more relevant. And the data teams are asking, oh, so you're sending all these emails or messages around like churn, like, where did that come from? Oh, it came from this marketing tool. Okay, great, like how do we know that's even close to being right? And the answer is like, we don't. - Yeah. - And now, I mean, you wanna trust your vendors, right? You wanna trust that somebody at one of those marketing companies validated the thing, but you don't really know. And there's at least a few of the tools where some people in the data science community have kind of come out and be like, hey, we've done this research and these numbers are not very good. - Did you implement any of those in a past life where you like turned something on that was predictive but you had no idea what was going on? - Yeah, oh, for sure. So we, I mean, especially on the, like email was the easiest one, right? Like my theory of email is it's about touch points. Like you can get really into A/B testing everything. And a lot of email is like the more touch points, like the more it like converts. So it doesn't really matter what the email says. So as a way of like, yeah, like let's do a campaign around like prediction next purchase. So we did that and probably gained some like incremental benefit from it. So from that standpoint, it's like, great, like who really cares? - Yeah. - But once you like, you know, get past that, like initial stages and you're trying to refine things and we don't want, and then we see like, maybe we're seeing higher churn because, you know, it's 2020 and your box is even more full of email marketing than it's ever been in your life, right? - Right. - Like, okay, it's time to like, we're seeing this high churn. We need to like cut back on these things. Like, how do we like make this a lot more precise? And then it's like, well, what does this model do? Like, is it accurate? Like, I don't know. - Yeah. - Yeah, yeah. - I'm gonna, I'm gonna mention one thing, a little series, one thing, a little silly here, but like, even a model that does really well that makes point predictions is so much more limited in what it can do than a model that gives some type of prediction interval or window. And we've done, I've done forecasting work where they've already got a model that's in place that doesn't natively have any type of prediction intervals. And if you think like, if you're in charge of a restaurant and making sure there are enough french fries at the restaurant and your restaurant predicts, you're gonna sell 100 cases this week and the other restaurant predicts 100, wait, you treat those the same. But if one says, you know, the 90% confidence interval is between 90 and 110 versus it's between zero and 1,000. Well, that really changes how you want to manage your inventory there. But I think having measuring uncertainty and reporting on uncertainty is not a language. It's like when you start talking about distributional estimates or distributional predictions, like people's eyes glaze over, but if you can frame frame it the right way, it's like, you need to understand that the models doesn't know what it's talking about here, but it knows what it's talking about here. But I think that's really useful though, because especially in that use case, if I told you in like common English, hey, you need between 95 and 102 cases of fries, like that's super easy. And I can make a decision, right? Whereas like it's not as clear, like you need 100 cases of fries with a 89% confidence, right? Like it'd be much better, like we can get to like, we're almost 100% confident it is in this range. And I think even that like phrasing is way more helpful than people. - I also what we're talking about if we're bashing marketers, this is. - Yeah, we love bashing marketers. - We do it, we do it too. - And I actually like it even more now that I'm not on the marketing theme anymore. - I've moved from marketing to product. - Yeah, recently moved to products. - No, it even more fun. - Good promotion there. He can wash his hands. - Yeah, this is, I'm hesitant to tell it 'cause it's not even my story. I'm just borrowing it from one of my colleagues at all the research, but working with a client who had segmented their, this has been several years ago, but had segmented their customers into, they called them cubes, but it's like, hey, this, you know, 18 to 24 upper middle class suburban really likes this product. And this, you know, retired 65 plus widower, really liked this type of product. And we're trying to convey like, you've not tested that at all. That could just be by chance by, you know, you break it up small enough, you're gonna find random things. And they argued it and they didn't appreciate this. And no, look, the data shows that this is what it is. And they had, this was recovid, so it was in person. And their, their floor is this black and white, like tiled checkered floor. And so the presenter is very senior. So he could get away with this, picked up the candy bowl that was on the table and just threw them all on the floor. - Okay. - And went and looked and said, oh, look, the Snickers all land in the black tiles. And oh, look, the Skittles always land in the white tiles. So we should sell Skittles to the white one. - That is unreal. - I mean, it hits home pretty hard. It's like, well, they, they, that's just by chance. They say, yes, exactly. It's just by chance. - Yeah. - Wow. That is an amazing story. - Yeah, that's cool, yeah. - It got the point of, we, we, we, we were not invited back to that client anymore, but I think they're doing better administrative clients now. - But you, you talk, somebody learned something about that. - Yeah. - So here's a question for both of you. So one of the things that's really interesting about this is setting our expectations around, I mean, technology generally, right? But if we think about sort of data science as a discipline and this modeling where you're getting into a space that's predictive, right? Which is way different than, you know, sort of historical look back, you know, analytics, right? This is my time series and I'm just sort of understanding what happens, right? I mean, actually amazing how often that is done poorly. - All the time. All the time. - Right. And I think that even reinforces my point, which is, how do you think about setting expectations around technology, right? And I love that, you said, okay, at Elder, we say, this is how accurate we can be, right? Which is setting a really different expectation for data science than here, like, I'm gonna give you a, like, tidy packaged answer, right? - Yeah. - I mean, that, that's a constant struggle for us. And I think probably for everybody in the field, especially in the post chat GPT world where there is so much noise out there of look what you can do with this new AI tool. And demos look great. And in very specific circumstances, tools can be super powerful. And we, to come in as the voice of skeptics, even you've been in the space for 30 years or been working in the space for 10 years, it, you come in as a skeptic and it's, okay, well, this guy just doesn't believe, he doesn't think that this is gonna work and we're gonna jump to it and it's gonna work. And so, yeah, I think it's tough to try to balance that excitement about new capabilities with, like, grounded in realistic expectations. And it's hard, and as a consultant, when you're trying to sell services out there, the flashy stuff sells. So I don't know, maybe our strategy is to wait for the flashy stuff to break. And then say, see, we told you so, you want us to fix it? - Yeah, I mean, actually a great strategy, honestly. - Yeah, I mean, honestly, like, there's a consultant we worked with that worked with a very specialty kind of niche software and like his whole, and he'd been doing it like 30 plus here years. And his whole like thing was like, I'm the cleanup guy. Like I, like it was a very small operation. And like the big operations would come in and like, and would kind of know the software, but not really. And he would just like basically be the cleanup guy, come behind each one of them, fix what they screwed up. Like, like and did it. I mean, it was like a 30 year business model. He did quite well, you know, and it's, yeah. - Yeah, I mean, I mentioned model validation earlier. Like that, broadly in IT, but in analytics specific, like there's absolutely a niche there for model validation. - For sure. - I like it, the cleanup guy. - Yeah. - Yeah. - Yeah, that's super, inside of a company as well, I'm just thinking about some of the stakeholders that you had, John, even at the executive level. And the, it's, it seems like a really similar dynamic, right? I mean, you're selling services as a consultant, right? So, but inside of a company, you have to sell as well. - Right. - And I don't know, would you say that's like easier or harder? - I think it depends, it can be easier in, in some sense and harder in other, easier in the sense that like, you, like if you've been there a little while, you have that precursor, like track record, right? Of like if you've been there several years, and that matters. It can be harder in the sense where if you're getting into something more niche, then maybe you haven't like, like we got into some very like specific types of forecasting, like I'm not an expert in this type of forecasting, right? So like that's where it can be, that's where it can be harder. But I think either way, it's a sales job, you know? Like especially, I mean, I think data, I think data people are maybe skewed more of this way than a lot of disciplines, like they do not want to sell. They want to present the data, right? This is the data. - Yeah. - Evan's nodding. - Yep. - For sure. - And I mean, that's how I feel too. Like I, like I just want to present the data. But yeah. And the sale becomes clarity and communication and simplification often, like without losing the core of it. And that's, I think the hard part. - Yeah. - Yeah, spot on. - Yeah, I would agree. - Okay. Enough of the serious talk. - Yeah. It's time. - Oh gosh. How did you get into data comedy, Evan? This is fascinating to me. - I majored in it in college. (laughing) - You got a four-year degree in data college. - No, it's that sort of the byproduct of two master's degrees. (laughing) - Yeah, sorry, dad, if you're listening. I didn't. I majored in, well, business administration. So probably equally useful. (laughing) Yeah, so I, this is shortly into the pandemic. At Elder Research, we're a Slack company. I don't get paid by them to say this, but if you're on Teams, maybe just go on Slack for a little bit and just see how nice it is. I'm Slack-straight for all the internal communication and had, we were pretty involved academically. We've got people that do research and write books and somebody was going to a conference on undergraduate statistics education. So basically a bunch of stats professors and they had a call for papers. And I don't know any stats papers to my name, but they, in the call for papers, they said, "Hey, we're also looking for fun stuff." So if you've written a poem, if you've written a song, performed music. - Well, talent show ab ad joke. - Yeah. - Yeah. - So they solicited that. - Yeah. And so I don't tell my boss, but you know, I spent like 40 billable hours just thinking up statistics. (laughing) - Of course. - Came up with one, submitted, actually I came up with three, submitted one of them one, one first place. So I was invited to this conference virtual, but I was invited to this virtual conference, got an award, read the joke. I could like hear people's eye roll on the virtual conference, but I got a $50 prize. So that made me a professional statistics. - That you have been paid in award winning. - Yes. - In award winning professional. - Yeah, exactly. And so I absolutely wore that out. I told that joke to my colleagues, hundreds of times over, my poor wife has heard it so many times and just start to tell it another, start to think of some more jokes and you become the guy with the stats jokes. And so we did several teaching engagements with clients, say, hey, before we teach this forecasting class, why don't you open with a few jokes? And so I tell a few jokes. - Yeah. - And started getting a pretty warm reception, started thinking of more jokes. Eventually wrote, I did a little market research. If you wanna write a joke book, you have to have a hundred or a thousand in one, but there's no way I was getting to a thousand in one. So had a hundred one jokes, published a joke book. - That's a lot of jokes. - Yeah, that is a lot. Yeah, the quality drops down a lot there at the end. - Right. - It's all flow. - Well, statistics is like a bell curve. - Yeah, yeah, yeah. - Exactly. - Yes, yeah. I would, I made this as a very far right skewed or I guess left skewed. - Yeah, then, so you have a book and then some conference organizers get a hold, say, you wanna tell jokes at this conference. And some university programs want you to come along. So now, I still try to tell some jokes for all the research clients, but now I'm not quitting my day job, but every chance I get, you know, if somebody wants me to come tell jokes for their data or analytics or tech team, then I've been doing stand up for folks. - Okay, so I wanna hear the jokes, but there's a big difference between writing a funny joke. - Yes. - And delivering that in front of people, especially live. Can you talk about that a little bit? 'Cause like, that's a big transition. - Yeah, that is a really great point. And I think it mirrors to sort of our earlier conversation where you've got data science people that may be able to build these great technical tools, but they can't sell it, they can't communicate it in a simple way. And so, you know, I think you could almost get technical on joke writing, but the joke delivery is very much a human endeavor. It is very much how do you inflect, how do you stay engaged with an audience? - I've done a few virtual and it is infinitely harder virtual than it is in person when you can make eye contact and engage with folks. So when I do stand up comedy, I don't tell a single joke out of the book unless somebody requests it. They're just, the jokes from the book are standalone jokes. The stand up comedy is much more storytelling around data and analytics and finding the humor in it. So yeah, it's very different disciplines. - Yeah, man, that is amazing. I could keep going down that because that's fascinating to me. - Super awesome. - Well, we could do a whole 'nother episode. - Yeah, yeah, data comedy episode. - Stand up comedy episode. - Yeah, okay, we're close to the buzzer here. So- - All right, both of us in this long, good enough. - We have made you wait this long. That was a great call making people wait to the end. - Yeah, I'm sorry for all the angry emails you guys are gonna get. - Yeah. - We'll take it. - So the joke that I've told more than any other is the one that made me an award-winning professional. - Yes, I was gonna ask if you weren't gonna share it. I was gonna ask her that one specifically. - Yeah, actually I'll tell it first, but I got a funny story about it too. Did you hear about the Bayesian who built a model to tell her when she needs to go to the dentist? Well, it said she didn't need to go for six months, but that's probably because she just had a week prior. (laughing) - Yeah, that's so- - That's great, that is great. - My boss is way funnier than me. I'm glad he has a successful consulting firm. So he doesn't get in the data comedy, but every time he hears me tell that joke, which has been way too many times, he counters with Evan today, ever tell you if anybody else submitted a joke to that contest? (laughing) Thanks, John. - Wow, that actually makes it even more funny. That is great. - Yeah, yeah. - Oh man. - This one, this one, this is, I got the book. Yeah, I'm looking at the back of the book cover. This one is, do you hear about the 12th grade student that failed his machine learning exam so badly that he had to go back to 11th grade? It was a classic case of scholastic grade descent. (laughing) One of the best things about telling very nerdy jokes is like the built-in defense mechanism. If people don't laugh, then I just assume they just don't get, they just don't- - Right, right, yeah. - You can feel good about yourself because like, oh, they just have a process for writing these or do they like, are they more lightning strikes? - Yeah, this is a great question and it's tough. So there are a few in the book that end up being sort of formulaic almost and I don't like those as much, but most of the time it's, I had a commute, I've got about a 30 minute commute on days when I go into the office and when I, I don't do it so much anymore, but when I was writing the joke book and when I'm trying to make new standup material, I just ride in silence, just ride with the no radio on, like windows up and it feels so bizarre in today's world to not have, you know- - Some sort of input, yeah, yeah. - The tech stack podcast playing in the background. - Yeah, yeah. - But then just freeing mind and then realize basically that no time in my life where there's no input and so I started taking walks, driving with no radio on it, just doing things where there's sort of the freedom to be bored and let your mind wander around things and there's been a ton of dead end chases, but then I come up with something that I think, oh, okay, that could stick, that could make it. - Yeah. - Yeah. - Yeah. - I love it. - Awesome. - All right, well, do you have one more for us before we close it out today? - Yeah, I will tell another one. This is even, this is objectively worse. It got, it didn't win the prize, but it was submitted to the same joke competition there. So you've got a guy recently graduated, studied data scientists, studied data science, looking for a job as a data scientist. Can't find anything, job market's tough. So he applies for a logistic analyst job. He doesn't know anything about logistics, but hey, it's got analysts in the title. He goes in, he's interviewing with the guy. He's got a big map behind him, the boss does. Says, "Let's say you had to put a new distribution center. "Here's our network right now. "Where do you think you'd put a new distribution center?" They thought about it for a Senate. And he went and put a pen right in the middle of a divided highway and the boss just laughed. So yeah, you're gonna put our distribution center in the middle of a highway that's not gonna work. He said, "Sir, a normal distribution center "is always on the median." - Yeah. (laughing) - I don't know, I don't know a lot. - It's hilarious. - It's a background. - That's cool, wow. - All right. - Yeah, I love it. - I love it. - I love it. - It's at best the second best joke that I get. - It's amazing. All right, well, where can listeners get the book? - Predictablejokes.com. - Predictablejokes.com. - It's on Amazon, you can find it, but I'm getting buried by all the people who know how to do good SEO and keywords. But predictablejokes.com, you got to stand up on there, you got a few clips and you can get the book. The book is called Predictable Jokes. Go to Evan's site. You know, we'd rather give him the money than. - Yeah, exactly. - Yes. - Exactly. - Thank you. Thank you. I'm coming for you, Jeff. (laughing) - He started with a bookstore, you're starting with a book. - That's what I did, exactly. - Exactly. - Yeah. - If we were doing synthetic controls, like he does his trajectory on Mathwell to mine. - Yeah, I was gonna say, I don't know if you're gonna actually be able to produce a good synthetic control there, but all righty, well Evan, so great having you on the show. This is a great time, excited to check out the book. Thanks for joining us. - Absolutely. Great conversation. Appreciate y'all having me on. - The Datasack Show is brought to you by Rudderstack, the warehouse native customer data platform. Rudderstack has purpose built to help data teams turn customer data into competitive advantage. Learn more at Rudderstack.com. (upbeat music) (upbeat music) (upbeat music) [MUSIC PLAYING]
