A conversation with Convertro's Nathan Janos about methodologies used to help advertisers understand the affect each of their marketing efforts (print, SEM, display, skywriting, etc.) contributes to their overall return.
Data Skeptic
Advertising Attribution with Nathan Janos
(upbeat music) - Welcome to the Data Skeptics Podcast. I'm here today with my guest, Nathan Janos. - Hi. - Actually it's Janos, right, or is it Janos? - Well, you know, our family goes by Janos but I also go by Janos, or whatever you want. That's fine, it's Hungarian for John. - Oh really? - Janos. - So you and I know each other for a while, professionally, and as friends since. We started working together at a place where we were doing a lot of search engine marketing stuff. And I believe you're still in the advertising area. - That's right, kind of followed that lead from there and just took it to the next step wherever that was. - So Doug, if you would mind, give us a little bit of your background, education, interests, things like that. - Well, I have an undergraduate degree from MIT in computer science. I did a focus on artificial intelligence there. I also worked on a couple of projects at the MIT Media Lab, one in interactive cinema. Multi-linear narrative type exploration. That's where if you're watching a movie, it would read your vital signs. Actually design this little egg you would hold and the movie would morph depending on how you were reacting to it. - Oh that's awesome, I didn't know that actually. - Yeah, I worked on another project there, it was pretty cool too, where we created this, basically this device that was based on neural networks and genetic algorithms that would allow anyone to improvisationally express themselves in a musical way. So the guy whose project it was, Paul Nemirovsky, he's basically a musical prodigy and he wanted anybody to be able to pick up something and be able to express himself musically without having to learn the instrument itself and spend all the time doing that. So that was an interesting project also. I always liked math and computers and exploring data and modeling really. Really for me computers are a kind of tool, they're a kind of microscope into computational complexity. So whereas a lens might let you see the moons around Jupiter, by dilating the universe and exposing something new to us, I think that's where computers are, they're like a microscope or a telescope into computation. - Yeah, I love that analogy. - So after school, I didn't want to go down to academic route and started a company with the buddy of mine where we made turntables that would let you scratch and remix any type of audio-visual content. That was a company called EJ Enterprises. And I learned a lot about business, learned a lot about talking to people, learned a lot about firmware, video codecs. And then I fell into, I did some other stuff, I fell into this world of, I guess when we were starting and they called algorithms and specifically applied to online advertising. And that's where I met Kyle. His company was acquired by a company and acquired our company. So we were like two cousin companies coming together. - I didn't know it though when it was acquired. It sounded like I was a rich man, but now? - Oh yeah, but Kyle was a big part of the other company. And Kyle to this day is probably one of the finest engineers and data scientists I've ever met. And he taught me a lot of stuff. - Now you need to get out more. - Yeah, I wish I could work with him again sometime. - Yeah, one of these days. - I guess we're working together now, I'll let it be. - So we're gonna talk a little bit about what data science is doing for the advertising space and how we can measure it and some of the claims that are being made. And I hope the podcast can be for a general audience, not just a skis. So maybe we should give sort of a definition of what advertising is, especially when we say digital ads. - Sure, so there's different kinds of online advertising. They usually call these channels, different types of touch points. And right now I'm not a super expert in the whole ecosystem there. I'm more on the modeling side, but there's display advertising, there's pay-per-click advertising. You could think of organic search as a not a type of advertising, we go to the type of navigational source. There's affiliate advertising, there's email advertising. And there's different types of social marketing and advertising and linking, so. There's a whole spectrum in what people are exploring this new frontier. And they wanna find out this stuff is effective. And people are gonna be advertising across this whole spectrum of channels, including offline channels as well, which I also deal with now. And people are spending money on all this and they know they're increasing the sales most of the time, but they don't know how to attribute it. They don't know how to attribute the value in a scientific way back to everything. So that's what where the industry's really been going. And they've been doing this, really the new ecosystem is user-level data. We track every single touch point in interaction. - So I went on this site, Zenith Optimedia, and you can take early their statistics for what they were, but they report, 161 billion was spent in advertising in 2012. I think some numbers are bigger, some are smaller, but certainly we can agree a huge amount of money spent than advertising. 40% of that, they said was TV. And this part surprised me a little bit. 18.7% in newspapers and slightly less, 18.3% on internet, which I didn't realize newspapers were still around, I guess. And the internet was such a small slice, but certainly a growing slice. So with so much being spent, anybody who's spending that money has the challenge of deciding, where do I allocate my money and what's working for me? And that's kind of this attribution problem that many people are working on and certainly has some value, given the amount of money being spent there. - Right, and that's really where the industry is going. There's some new terminology out there, we call it multi-touch attribution. And the idea is to get the real value of all your advertising cross-channel. So that means how do all your channels interact with each other? And how do they share the effort for that final conversion, that final sale? And so that's the main problem right now, that's where we've evolved as an industry from this sort of last click, as they call it, type of attribution, where all the credit was generally given to your Google PPC type advertisement. - Yeah, Google plays such an interesting role in the whole discussion, not only because they're giant, and of course there's also the Bing Yahoo product ad center. So I mean to be a vendor agnostic podcast, but apologies if I just default to Google and say AdWords more than I say anything else. When they kind of came on the scene, they offered a lot more transparency and data than I think anyone ever saw before. When you were doing TV ads, I mean there was Nielsen, and I think there was some equivalent for radio, but you were pretty blind to measure most anything, except for maybe sweeping effects. Like if you create a vanity phone number that you'd never used before, and you put that in a radio ad, of course you can say, numbers people started calling it within 15 minutes, and we never used it again, and by two weeks later it was dead, and we could count the number of calls. But that's one thing Google did, and for anyone who's not familiar, I'd say it's worth the time and spending 20, 30 bucks out of your pocket to sign up for an AdWords account, and drive traffic to, if you don't have something you wanna promote, your Etsy page or something, drive it to a friend's website, or a restaurant you really like, 'cause going through and seeing the processes I think gives a lot of people who don't have any some perspective on what internet advertising is, in particular the statistics they'll give you. So you bid on the specific search terms you want, which is way different from traditional advertising where you're essentially doing an all or nothing approach. You can't just put an ad in male subscribers to a magazine's version versus females. It's, you're getting everyone. But on the online world I can say, no only people who search for blue cheese, do I want them to see my ad, and then Google will report to you what they call impressions, which are how many times they displayed that ad. The clicks you got, which is how many people who clicked on your ad, and then if you'll participate in their conversion tracking system, which is really, really easy, you can, let's say you have a website that sells blue cheese, on your thank you page after the checkout's complete, you put a little piece of JavaScript that calls back to Google and records that a transaction has happened. So you can go look and say, when I advertised for best in the world blue cheese, this many impressions, this many clicks, and turned into this many sales. So it really feels like, wow, I can calculate my ROI very precisely. If that sale was worth $100 profit to me, and I had to buy 50 clicks, well, then my break even point is $2 a click. And because it's so precise, and the data's so readily available, and I can do something with that programmatically, it really feels like, hey, this is where it's gotta go, all the money needs to go into this channel because I can measure it. And not only that, but I'm going to give all the credit there. But kind of, as you were pointing out, the fact that I even went to search for world's finest blue cheese, probably just didn't start out of thin air. Something had to give me that idea that I'd like to get some blue cheese. - Right, we call that the funnel. And that's where the TV people keep insisting that the TV works in this whole process. And that's really, that's one of the main drivers for this cross-channel attribution. And my company, we have a lot of clients come and they say, hey, we know TV works, but we're having a hard time showing the marketing department works. So easy for the guys doing the Google advertising to show that their marketing works. And in fact, they're probably overstating their effectiveness because when you're looking through Google's dashboard, it only knows about Google. So it assumes that all the advertising effort for that conversion and hence that conversion attribution to their click came from them. But we know that's not true when you have other marketing channels that are active in the ecosystem. So typically the advertisers classify different channels as being introducers. You know, there might be the brand awareness type thing. There's influencers and there's closers. There's probably different terms for those, but they're moving the customer through the chain, making them aware of the product, getting a connection maybe, and then pushing them to do that final sale. And so the idea is to give the proper credit throughout that chain. And so that you can give the proper return investment for all those different parts of the chain. - So one of the challenges I see in going down that path as an advertiser is you've got all these signals, you know, you've got that Google dashboard we talked about and you've got hopefully some means of measuring TV, but everything is like any scientific measurement, a weak measurement. A lot of online advertising is based on cookies for tracking. Yet if you trust ComScore, and maybe you do, maybe you don't, but they report that a study they did found, 20% of cookies get deleted by different privacy, or 20% of people have their first person cookies deleted, meaning like a website stores a cookie for that website, and 30 to 40% of third party cookies get deleted. So if I'm using cookies to measure something, I've got to, and we could argue about the numbers, but certainly there's some margin of error that there's a dark figure I'm not gonna be able to, I'm gonna have a lot of false negatives in this case. - Yeah, so that is a problem. At Convertro, we actually solve for that problem by relying on a type of cookie matching, where we tune it to minimize those false positive associations. And we do this, and we establish an ID on our end, given characteristics about your browser, not including any personally identifiable information. That way we can actually bridge the gap when people do delete their cookies. And we also always use a first party tracking with our clients, because those have a lower deletion rate. - I was doing so, and without, you know, I'd be so mad, I'm not gonna ask you for trade secrets or anything, so stop me if I'm walking up to a point. - It's patented technology. - Well, there you go. So you're saying that even without the cookie, you might say, well, the guy, we know the precise version like 5.67093 of Firefox, he's running, plus his language, plus two or three other things. And that's fairly unique enough, ish, that we might say this guy is the same guy with some reliability. - Yes. - Interesting. - That's what we're doing. And yeah, there's perhaps Bloom filters involved in there. - Yeah. - That's the final Bloom filter, 'cause they're a cool data structure that I think a lot of people won't know. - Why don't you define it for me? You're better at these. (laughs) - I didn't look it up before yet. From the best of my memory, Bloom filters are a data structure that's attractive because they're compact. So you can store many things in there without, with a low likelihood of collision, but also, so it's a hash, essentially. So you can do a quick look up, and it's, because it has some loss on insert, because of collision. - Collision, yeah. - You might have a false positive, but you'll never have a false negative, I think, are the features that make Bloom filter. - I think that's right. And then, like you said, it's fast, which is important. So we mentioned, or you mentioned Convertro, but we never mentioned what that is if you want to give a background. - Convertro's a company that was founded out of the e-commerce world and tracking all the online advertising touch points. You know, they've extended themselves into the offline world now, and the idea of Convertro's to measure the real value of all the different marketing channels, cross-channel and cross-device, and to give a return on investment. And to do this as an agnostic third party, as a service to our clients, we do that through collecting their marketing exposure data through tags and, you know, offline feeds. And associating that with, through this cookie list tracking system, associating that with any of the eventual conversion events. And then we run some math models on it, which I think is another discussion that we could talk about. - Absolutely, let's get into those. - Right, so again, going back to this, what is data science? What is big data? What are, you know, what's going on with all this? Why do we have these terms? Are they sprouting out of our vernacular? The type of modeling we're talking about is a type of modeling which is done on big data sets. And because of that, you know, we have this user-level data that we're talking about. You're gonna find it in industries where there's just a lot of user-level data readily available, and one of those, it's just kind of out there and open, is marketing and advertising. So, you know, this, a lot of the supplies to any kind of industry and industry, and kind of a problem that we might have commercially or even maybe with organizing society, there's lots of sort of big data problems that are probably really interesting out there. But as far as this problem, we have millions or hundreds of millions of users that may have seen tens or hundreds of thousands of touchpoints, so you can just think of this matrix. It's hundreds of millions of rows and tens or hundreds of thousands of columns. You've got these big matrices that you gotta perform operations on, and so your basic sort of ordinary least squares analysis is gonna break down for a lot of reasons. And so I think one of the interesting things going on is that there are econometricians and there are statisticians, and our computer scientists working together now to create new software packages to let us run different types of regression, different types of machine learning on these large datasets. One of the people doing this is this guy Hasty at Stanford, and he has a good free book online, just called machine learning. And it's all about big data and how to look at these problems. But he's also developed a lot of the like cutting edge packages out there right now, like the GLMNet package, which is a way of attacking these problems. So I was gonna describe that a little bit 'cause I think a lot of people are using this technique to look at these big problems. So what happens when you have a lot of predictors? So let's go back and define this problem. So really what we're talking about in this case is a type of supervised learning. We know the answer, we're looking at data, we know when people convert it and we know what they're exposed to. So we kinda wanna learn that relationship. And one of the ways to do that is through a regression, a regression analysis. And one of the problems you're gonna have if you have tens of thousands of predictors is some of them are gonna be what they call collinear. They might be moving at the same time together. And so how do you decide which one is actually explaining the conversion behavior in this case? Another is that you're just gonna have a selection problem and there are gonna be different classes of variables as well. You can imagine there's a whole bunch of display classes. And at Convertor, we model all the way down to the creative level. So when possible, if there's data, we're modeling at that individual display, at that individual Google PPC app. We use a lot of people out there, but I just wanna talk about the GLM that, it's one of the things we use. It's a package that attacks this problem through something called a regularized regression. It's called a plastic net, regularized regression. And it uses a different type of penalty function. So what it lets you do is tune between two extremes. One where you select very few variables to explain the output, and one where you select a little bit of every variable. And so it lets you tune between those. And really that's it's lasso and ridge regression are some other words you hear for that. Or L1 and L2 norm penalization. And they're just a different way of giving penalty. There's a sort of a squared way, and a linear way of giving, an absolute value of giving penalizing, not fitting the data. And so at the heart of a lot of these, you'll talk about this penalization function, the heart of a lot of these, any kind of models, you need to define a fitness function or conversely a penalization function. It tells how well it's doing. So you know if you should select that solution or not. So in this case, we're just using a combination of penalties. And it lets us tune the behavior so that we can control for inter-class correlation and the selection of variables. Previously, there are other techniques, like one is called stepwise regression, where you would either introduce or remove a variable from your regression to find out which ones mattered. But the problem is you could imagine that's a large search space when you have 10,000 variables, which ones do you remove at each step? And where do you start by adding them or removing them? And you're the kind of randomly wandering around inside of this space. And stepwise regressions proved to be unstable and they're not very robust. That's one of the benefits you get with your elastic net type of regression, this robust. You could remove one variable and the whole solution's not gonna change. And there are other packages, so sometimes just getting this data in memory is a big deal. So the other package is that we'll let you distribute these types of regression analyses across multiple machines. And that's where we're intersecting with big data right there. - So in the problem, is it something that can be parallelized? In other words, I can write a map and a reduce function and do something like that do that? - I think that was, the first algorithms from that are probably from like the 70s. And a lot of the singular value decomposition kind of matrix multiplication stuff can, does fit into the map reduce framework. But there's these other new techniques coming out like these sort of Monte Carlo techniques where okay, I have a thousand nodes and I have this problem with millions of pieces of data. Well, let me just send a little bit of data to each one of them, have them do their job. You know, and they're gonna be a little bit inaccurate 'cause we're not working on all the data. But they're gonna come up with some answer. Then I'm gonna collect all the answers and kind of average it out. That's sort of like a, I guess that's also a boosting type technique, what they might call. - 'Cause that like a particle filter approach or? - I don't know. I don't know the particle filter approach is what is that? - It sounds like what you're describing. It's where you have a complex stochastic problem where it's difficult to maintain your state estimate. So you have a pool of discrete assumptions about the state. So if you're in some state or you're uncertain your state, you look at where you'll go next based on the actions that took place. And rather than trying to have a posterior probability distribution over that state space instead, you just say, well, I'm very likely to be in this state. So I'll assume I'm there. But then obviously that's a pretty gross assumption. So I'll make 20 similar assumptions proportionate with my expectation. And then I'll explore each of those. So there's a chance I skip the true state, in which case I'll be very confused about where I am. But if my pool's big enough and I sampled well, one of my solutions was a correct guess sort of. - Yep, yep. There's generally I find that the fancier the wording for things, there's actually if you could kind of explain it on a whiteboard, it's not that complicated usually. - Yeah, I have the same complaint about a lot of stuff. I remember looking at some of the first game theory literature I looked at, I was like, wow, this stuff is so complex, man. The guys that dream this up must be real geniuses. And weeks later, when I finally penetrated it, I was like, no, these concepts are actually pretty simple. You just have a lot of symbols in here. - Yep, yep. Yeah, people like to be very precise about how they specify things. - Yeah. - Which has its merits. You need to lay down the groundwork to the foundation to make sure these techniques are actually founded and mathematically founded in truth, yeah. - So going back to your, I like the way you formulated the problem of saying I've got all the user data and matrix crossed with all these observations I have about those users. I would guess that in an attribution model, some of it you want to have time series consideration. So Google, you know, a click, and if it's, if you're going to attribute it to just Google AdWords, that click and conversion almost certainly need to come on the same day. But something like a billboard ad or a bus stop ad, these are things that use a better industry term than I have, but I would call them like they're planting the seed for a later purchase. So if I, if you think I saw a billboard on Tuesday and maybe I buy the product on Wednesday or maybe Thursday, there's something interesting about how many times I have to see it and how long it takes until that treatment has an effect. Yet you can't, it would be hard I would think to explore, well, what if it's one day? What if it's two days up to end days? How do you deal with time delay and decay modeling and that sort of thing? - Yeah, so the, you know, I kind of class these all into the temporal effect. And there are just a ton of ways that people are trying to get at it. Traditionally people got at it with this concept called ad stock, which is really just a, you pick a decay value and it's the decay of the awareness of say a television advertisement over the next three weeks. And you can actually look at people, you can pick the amount of time it decays and the decay factor for your particular business, but really the ad stock factor traditionally through the 60s and 70s was modeled on the order of two, three, four weeks. - So there's other ways to get at it. You can look at a sort of source decay. So after I saw that display ad, you know, how long did it affect me? At Convertero we have data-driven models. Well, we'll look at all the converting paths that have ever had display on it. And we can look at the, you know, this is sort of a hazard curve or a hazard function type of analysis that they might use for analyzing the failure of hard drives after time and manufacture. And you can see that, hey, look at after three months, 99% of the people who will ever convert after seeing that display ad have already converted. So I know that perhaps I shouldn't include a display ad when I'm, you know, determining the source signatures to put into my matrix for regression if it's older than three months. Hey, let's not include it because we know it's decayed. There are other decay notions in attribution modeling that again, these are sort of heuristic models and they're kind of basic. Then there's, you know, basic time series modeling. So since the '60s and '70s, people have been doing this mixed media modeling, which is an econometric time series type modeling usually either on the weekly or daily level of all the amount of, of all the money that was spent in these different channels and also the sales. And so you can build in leg dependent variables in there. You can build in leg in different types of ways. You can say, hey, today's sales were also a function of yesterday's sales. That's called a leg dependent variable. You could say today's sales were a function of yesterday's advertising and a day before that, advertising the day before that, advertising. And so that's another way of including time. You can also say if you're analyzing chains of events, say where there was an event where someone put something in their cart online and then they purchased or maybe they did a lead form and then they subscribed and then they resubscribed, you can break down each of those sub paths in between the conversion events and iteratively regress on those and pass the value back through the chain. So that's another way of getting at it. So those are a bunch of techniques. Another one is a dummy variables, also known as fixed effects in some contexts. And you could add a dummy variable into these regression models for each. Typically, you could do one for each month. You could do them for holidays. You could do them for day of week. You can do them for weekends or weekdays and they allow you to establish sort of a, what I described is sort of a secondary baseline or secondary intercept in the function for that particular time. And those dummy variables will soak up any of the other effects that we're not modeling for during those time periods such as weather and seasonality and maybe just the difference between how people buy sunglasses and a Los Angeles compared to how they buy them in Seattle. - So what's the value for the dummy variable going into the regression? - The value, well, so the value, you'll have to look at it in terms of a design matrix. And the value's gonna be a one just like it was a constant. So in that design matrix, you're gonna have a one save for your baseline constant. Then you're gonna have another one, but it's only gonna occur sometimes. Say, for instance, we have a dummy variable for if you're on the eastern or western side of the United States. Say, we'll look at your IP address and for that particular path, if you're on the western, then we'll give you a one for the western side. If you're on the eastern, maybe one for the eastern side. It makes sense. So it's dependent on the metadata, whether or not you include an additional constant offset for that time period. - Yeah. So then, does that help you? One of the challenges you'd have in coming out of the regression saying, did I account for every variable I should have accounted for? So if there's some crazy news story, drove a lot of interest towards or away from some particular product, if you don't have a variable capturing a new story equal true at that time. Well, I guess first question is, how are your models, which it would be fair for them to fail in that situation? 'Cause you haven't informed them with enough data. In what way will they fail? And can you tell that they failed? - Yeah, so generally, you're gonna tell if your models are failing by looking at some kind of fitness factor or fit statistic that comes out of running your model. That might be in some cases in R squared type fitness statistic. It might be deviance. One I really like is just looking at the, just the residual error. Look at the residual error on both the data that you just fit and also in a cross-validation set, a hold out set of data. Run it on that and see how well, what the residuals look like on that. You know, residuals in this context for definition are just the difference between your model and what the real data was. And you can actually quantify that as a percentage. You can say, hey, there was a 15% mean absolute percent error. Okay, now I can compare, I'll tweak the model. Now there's 14%. Okay, maybe it's a little better. And this is generally what you're doing. So now as far as adding variables, so this really smart guy I worked with explained linear regression this way to me. I thought it was pretty good. He said, all we're doing is we're trying to explain the variance and the output. So you have some output in this case, some kind of sale. Either a sale happened or not. No, that's maybe a logistic regression, which we've kind of been talking about. Or maybe there's an amount of a sale or how many sales per day. So that's your dependent variable. That's your outcome that you're trying to understand. Now things are gonna be going up and down, you can imagine. You're trying to explain why does it go up and down? Well, you can put in factors. And let's just say, okay, I'll just put in a factor. How much I spend on TV every day. And you can have a really simple model. My sales are related to y equals mx plus b here. In this case, b is gonna be that base. It's gonna be that constant. It's gonna be how many people are buying stuff without any exposure to advertising. And then the m in this case is gonna be how well that advertising works. Now, maybe you were also doing other types of advertising, but you didn't put it into this model. So what's gonna happen there? The other two factors, other two parameters that this model can move within, which is the m and the b in this case, are gonna absorb that. They're gonna absorb that effect. And generally the idea is that that base that constant's gonna absorb that effect because it becomes part of the, not knowing about the other advertising becomes part of the inherent conversions that are happening without you going. There's just kind of noise in a way. Yeah. And also, technically there is an error distribution that's associated with all these linear regressions as well. So when you look at the residuals approach, one way I might characterize it is to say that that tells you the power your model has in explaining the variables you're interested in. That's right. Given the inputs you gave it. So if you perhaps created a model that took into account a person's age and eye color, it would come up with some regression, some fit, but likely have a very poor fit to the data. You'd know that that way. Age is a nice indicator, eye color probably is irrelevant, but you're missing something and you can tell that there. As you add more and more variables and knowing the deluge of data we have available to us, at some point it's plausible that you come up with a regression that has a strong fit to the data but isn't necessarily meaningful. Exactly. And this is something that we all have to watch out for and that it's sort of a pitfall that you might see in some types of statistical models. So let's say I have a data set of one month of data of sales. And I'm like, dang, I want to explain the sales for each of these days. Okay. Well, I'm going to add in a variable that says how much I spent every week. Okay, that actually works pretty well. Then maybe I get creative. I'm like, I'm going to add a variable for every day. So I have 30 days of spend I'm trying to predict and I'm going to add 30 variables to explain this. I'm going to put this in a regression model and I'll pop something that fits the data perfectly. And you're like, amazing, this is a perfect R squared. We're explaining everything. Well, the reason is, is that the model found out that it could just train each of your 30 variables for each of the days. And so there's really just a variable for each day. Now what you'll find out is if you use this on the next month of data, you know, a hold out set, it would do very poorly. Because, you know, the model has been tuned for each of the specific days. Now it doesn't know anything about how to predict the future because it's been tuned perfectly to, you know, another historical example. - One fallacy a company could throw out is to say, you know, oh, there's company A and B who want to do some attribution work for us. Company B says their fit is 99% accurate because they've applied these techniques. So it sounds like great marketing, but actually it's not great science. And I think you hit on the key of, well, how do we actually measure that? It's that these should make verifiable predictions about the future. - Yeah. - Can that become part of a feedback loop? Is that something you guys look at? - Yeah, so there actually is not even a hypothetical situation you've put out there. There are people out there say we have .99 R squared models. And in fact, there might be some cases where that's okay. If you just wanted to understand why something worked in the past or explain with some variables, why something worked in the past. And you don't want to use that as a prediction model. - Sure. - Maybe that's okay. Maybe there's some argument for that. Yeah, but if you want to predict the future, that's a very good question. So I've actually looked into it with my colleagues. We thought, hey, would there be a way to go out in industry and say, hey, industry, you think you have a good model? Let's all measure ourselves so we can see each other. And actually, the more we thought about that, that's a pretty hard kind of proposition. As every company is going to have models that are tuned to their particular type of data formulation, they're going to have, these models, you know, while they're simple up on the whiteboard are quite complex when they're implemented in code. And you would have to have a third party and the holdouts that would have to be secret. And I think that there's, in order, basically, I think the problem is in order to, you'd have to pick some general data set. And no one is going to spend as much time tuning their model so that general data set specification is they're going to be to their own models. All right, so where does this leave it? So I think the only way you can really measure yourself then is to say, and this is what Converter is doing, in your dashboard, you say, hey, we think, if you change your spend allocations at the top level, shift it a little bit more over here to TV and a little bit more over here to display. The next month, your overall profit's going to go at 5%. Okay, they say, okay, I'm going to, I did that. And they mark it on the chart. And you come back a month later and you measure what actually happened against what we said is going to happen. And again, this goes to outside of using these fit statistics and everything. What really matters at the end of the day is did you actually help them save some money or increase your profit margins, et cetera? So the first people that are really going to be able to do that, no one's going to care what math you're using anymore. No one's going to care that it's fancy data science, blah, blah, blah, blah. If this stuff is actually making you money, then you're golden. I mean, that's the ultimate verification validation of all of this. So there's a challenge there, too. You essentially proposed at that point an experiment, change your behavior in this way, and we forecast this outcome. So it can be verified. But what can't necessarily be forecasted is when that change is done, and then 24 hours later, it comes out that the CFO is a secret Nazi escapee, then living in South America and avoided Nuremberg and your stock plummets and this and that. And then someone comes back and says the forecast was garbage. And or to put it in a less extreme way, the forecast doesn't hit, and then the analyst says, well, that's because there was a drop in the price of wheat and we didn't anticipate that, and that's why BMW sales went down. Although that's sort of a possible, but probably spurious correlation. So it kind of goes both ways. It's maybe over-panelizing in some case to the modeler and under-panelizing in other cases. I guess where do you fall in terms of the uncontrolled variable, the unpredictable things that the world's filled with that are going to affect what might be a very good mathematical model? - You could never predict everything that's gonna happen. - Right. - So I mean, so part of the prudent process would be to include, whatever you can possibly include and control for in that model initially. And we come across this problem all the time because you're gonna tell some sort of prediction and when it's not exactly right, you're gonna sometimes lose credibility with the model. I think part of the education process is that there really is not, we don't know for sure, but there's some sort of confidence level around what we're telling you. Maybe we say, hey, if you do this, we think it will go up one to 14% and an average is 7% with a 95% confidence. All right, maybe it goes down 20%. Well, there may be there's a 5% chance of that happening. You know, those unknown things. And so I think that's actually a really important point is I think you'll find more sort of metrics and KPIs which are given in ranges. Any of these things coming out of these models are really ranges we don't know for sure, but we do know with some confidence what kind of range we should supply. And a bigger range means less confidence, but you could argue that any type of information is better than nothing. - So when it comes to formulating your model, one of the challenges I see, and we talked a little bit about the accuracy of certain observations you can get. We know cookies aren't perfectly reliable and there are other ways of tracking uniqueness and they have their own margins of error. Certainly that's true of any measurement we take. Probably online stuff is more accurate just because it's so hard coded server logs as opposed to like the billboard example where even if you knew from GPS someone passed it, did they pass it going the right direction? Was it cloudy? Were they even looking at billboards? These sorts of challenges exist. So if we're going to be intellectually honest, we have to say that there's a margin of error that hopefully we have some estimate of and it's going to kind of mount over all of our variables. And the product of the regression can only be as good as the quality of the inputs for the most part. So I guess my question is, can you account for the accuracy of all the measurements that are going in and how can those contribute to, can a model even, what's the convinces that we can even solve this problem when there's so much uncertainty? - Yeah. So there's a couple of factors. Whenever you fit a variable, you're going to get some kind of, say a t-statistic on it. And it's going to help you understand how well that variable is actually explaining the data. And you could look at like, hey, it has a high coefficient, so it must be explaining it well. Then you look and it has, oh, well, the standard error on it is also really high. So maybe it doesn't know it very well. So there's sort of mathematical ways of finding out, if there's problems with the input, 'cause I guess you would assume if there's some sort of problem with the input or it's dirty that hopefully it doesn't actually by chance correlate with the actual output, I guess that's a problem. The other issue is bias, I think. Okay, so let's say we're looking at the user level and we happen to know that some of these users saw TV. So we put sort of a TV exposure on all these guys' click paths. What if we're by chance, but only capturing 10% of the actual exposures? So we're sort of biased. We know where there are, say, TV exposures happening, but we only know for 10% of the people. So our data set's going to be under exposed to this. And so that's actually pretty common. And it's pretty common when you're doing audience-type measurements and looking at how different data sets intersect with each other. And so you do have to account for that bias. There's modeling and math types of ways of accounting for the bias. And there's also, you can account for the bias in the interpretation of the results of the model as well. I'm like, okay, I know that I wasn't quite sure about this one thing or there might be this extra overlap and you need to interpret the resulting model accordingly in those cases. - So what actually brings to an interesting question I was trying to get to around bias of your measurements. - So one challenge that anyone talking about attribution as to face is multi-device problems, which are becoming even more and more common these days. I think I brought three things with me here today, a laptop, a tablet, and a phone. And there's some work being done trying to correlate and understand that I'm one unique person, all these devices, but it would be very disingenuous for anyone to say they've solved that problem. Certainly a hard problem. Yet there are people who are single device people, who have one machine, it's where they get everything from or maybe just a TV and a single computer who are going to be easier to track uniqueness on because they truly are unique at a machine level. Do you face challenges or do you have to factor for things like that that because someone, let's say, didn't install privacy software, they're easier to track. So they're overrepresented in your data set. Yeah, in the case of doing this sort of multi-touch user level type of regression, the artifact that that's going to leave in your data is going to create a bunch of divorced paths that should otherwise be one. And so that's the type of bias that's going to be introduced is that you no longer know the whole chain of events. You don't know the whole chain of advertising exposures. So the data set and the regression is going to be biased accordingly. And you're right, the cross device is a very hard problem. It has not been solved well. And we try to solve for that, but there are ways of solving for it through logins and through like third party, say you log in through your whatever, you log in with your idea through two different devices. Well, now we know, we know about your two different devices, but arguably that doesn't happen 100% of the time. So you're still left with a sort of divorced data set. And I guess the idea is that you would get enough of a signal from that little bit of matching you do have. But again, you're right, these models aren't perfect. And really there's a whole question. We always validate our models against hold outsets and make sure that they're better than any other model that we know in any other previous version of the model. But still, there are a lot of paths out there which aren't very long. Maybe there's one exposure and then there's a purchase. These are simple types of things. And in other words, there are situations where an old school first click or last click type attribution really are not that bad. Maybe you don't have me saying that on the radio here, but. - Well, let's define those, 'cause I'm not sure that everyone will know, I mean, they're kind of self-explanatory, like. - Yeah, yeah, so in the old days, three years ago, you have your markings written, they're like, how do we value all this, our advertising? Well, they'll say, oh, well, we sold $1,000 worth of stuff this day. And that was through a traffic that Google brought us through a pay-per-click. So we'll say, okay, and it costs us 500 bucks to buy that traffic and we made $1,000. So there's the margin of return right there. - And I have that because Google's very good at tracking. They've told me exactly how many clicks they sent me and how many sales I generated from that. - Well, expand our simple universe here. Just a little bit bigger, say there's display ads. And maybe these people are seeing the display ads on their favorite news sites, and then later they search for it and click on that pay-per-click ad. Well, you could, so you could give credit two ways. You could give all the credit to that pay-per-click, that final click, or you could go of all the credit to that display. Who should get the more credit? I don't know, they probably should share it. - It depends on which company you work for. - And really, it depends a lot of times on which department you're in. - Oh yeah, yeah. - If you're in a display department, display gets a credit, or the PBC department, the PBC gets a credit. - That's right, the reason you even did that search is 'cause I first- - And now you're saying, you know, that there's twice as much credit that's being given than there actually should be given. - Right. - 'Cause the sum of all the credit, if you let all the departments look at it, siloed on their own, it's going to be more than actually, should be distributed. - Yeah, so neither of those make sense for one department to get the entirety of credit for that sale. - That's right. And that's what we're talking about here, is using statistical methods to determine that credit instead. - Yeah. - So in a simplified way, maybe I would say, I'll look at, and your regression does this at a scale, essentially, but to kind of frame it for non-mathematicians, I'll look at some people that saw 10 display ads and did one search versus people that saw one display ad and 10 searches and see of those two groups where the conversions are most fruitful. - Yeah, and to extend that, to sort of the A/B testing example, and you could, if possibly think of regressions, just a type of huge tests that we do in the cases where we can't do an A/B test because there's too many variables where we don't have enough tests and regression kind of fills in the gaps, but let's look at two different paths. There's one that had a five different types of marketing touch points on it. And let's say we had 1,000 of those, 1,000, 100 of them converted. Now we have another one, another 1,000, and they're identical to that first 1,000, except that they have this TV exposure, and 200 of them converted. So you could say that extra 100 conversions, incremental conversions, is due to that TV because otherwise the paths are the same. - Comparatively, yeah, yeah. - And that's how, in the A/B case, and really how it works in the regression case once you have those coefficients determined, that's how you determine the incremental lift of each of the touch points. - In that way, you're kind of creating, I think there's more technical term for this, but in lieu of knowing the right word, it's like a after-the-fact experiment. You're saying, like, well, I didn't conduct an experiment, but I do see here that there are two groups that can kind of be represented and is tested and controlled. - That's, yeah, that's a good way of putting it. And you're fitting a very high dimensional plane to the data. It's kind of wiggling in between these data points, I guess. - So there's one, I guess I mean this in a devil's advocate or just a purely skeptical way. Well, let me give you an anecdote first. In the, around 2008, when the presidential election was going on, there were a bunch of big news stories about the fact that the Obama administration was spending money to advertise for in-video game ads, as like while you're playing, I think EA was the big part. Do you play some EA game and there'd be a billboard advertising Obama? And I suspect that for every one person that actually saw that ad in the game, 50 people heard about the fact that the ad was there and that this was a win, not because the in-game advertising was of value, perhaps it was, but it was a win because this was the first time anybody did this at a big scale and they got news which is ostensibly advertising out of it. And I think I see that a lot where the first time something new happens, there's press about the fact that a new thing happened. So there's, there's a, it could be the case that take a big strong brand, something like a Coke or Pepsi or Microsoft, you know, household names. Just making any perturbation is going to have different people take notice of something and garner some positive return just because something new has happened. You know, like let's say BMW puts out a commercial where it's just 30 seconds of a blank piece of notepad and somebody writes BMW a bunch of times on it. It's probably a horrible ad and everyone hates it but people will talk about it. They'll say what's with this crazy ad? Why would they spend money on this? And just because there's buzz and it's different because I made some perturbation, I made some change, that's going to have some net effect on my advertising. So, whereas one might say, oh switch to 10% of your budget away from TV and into magazines because our model suggests that that will be positive for your campaign, it could be just that that shift and the fact that now you're all over every magazine and it's a change, it's not necessarily that the magazine was the benefit. How would you address someone's claim, maybe a competitor come up and say, we have a strategy and ours is just constant change? - Well, you know, what we're trying to do is put Don Draper out of a job. (laughing) So, maybe the guys who are advertising for Obama in 2008 got lucky with that. And I think to me this all, it kind of feels like, okay, there's different types of, sort of much more course PR type exposures you can do. You could send a guy up and have him jump out of the edge of space for your energy drink product. - All right. - All right, so yeah, so that's the, I would say in a whole different world from what we're doing basically. I'm not saying there's a huge value there. But, and I'm not saying that you can't measure that. I mean, you'd use our technologies to measure that. And so, if you could convert that type, I don't think that what we're talking about now, the type of technologies and data science we're talking about is not gonna be able to give you really cool, human, smart types of suggestions, like shoot a guy out of a cannon. - Sure, yeah, yeah. - It's gonna give you answers in numbers. - Or at least six months away from that, so. - Yeah, right, is that what they said in the '60s, the summer they were gonna solve artificial intelligence? But, you know, we can still measure that with our techniques. We can measure that effect if we could fit that as a input into our models. - But if someone made the claim, simply you're advertising at an aggregate works. It's hard to measure, maybe these models are useful, maybe they're not, just shifting your budget around is going to, well, actually, no, it doesn't make the predictive claims. If you attach with it, like you'll have this ROI to see. - Yes. - I don't get that with just the perturbation approach. - Yes, yes, it's gonna, and we're modeling in the elasticities, or in other words, the diminishing return on investment. 'Cause you mentioned this a lot earlier in our conversation is that, okay, maybe it's $1.10 for PPC now, but if I bought a whole bunch of it, it's gonna get less return on my investment. They're 'cause there's, at the end of the day, there's only whatever it is, six or seven billion people on earth now. - Right. - And they can only buy so much. - Right. - This could go into another conversation, as well. - Well, that leads me to something interesting, too, is the, you don't really, to some degree, you have some control over your inputs. You can run certain experiments, and you're making recommendations to advertisers to make changes, and you can see the response to those. But a lot of the data you have is sort of accidental data or organic, it is what it is. So if, let's say a company had been spending $100,000 a month on TV ads, that's probably not enough, but just go with a round number. And you are able to ascertain that that's converting especially well, that's their most valuable channel. The natural recommendation is invest more, and perhaps you could even recommend it at a level, but the only data point you have is $100,000 a month. So you're kind of, in a vacuum, it's like you're regressing against one data point. What will, but there will be some, it won't, it probably won't scale linearly. You won't get double your return for a double your investment, I should say. Or maybe you will, but to some point, you won't. How can you forecast into those dark areas where there is no data yet? - Well, I mean, from a modeling perspective, you can look at how other similar experiments have gone. And so you can understand that's a sort of global way that the diminishing return on investment works. In your example, I guess you just have one data point, but if you spend $100,000 a month, maybe you know how much you spend each day. And you would, again, you would need to, it's really important to interpret the model in terms of how you ran it. So if you really don't have one data point, maybe you shouldn't be making any kind of predictions on the future off one data point, but. - Sure. Well, I meant it more in that, let's say your budget is about the same. So you have some variants, one month you spent 95, next month 105, but essentially all your observations are clustered, which you have many clustered. - Yeah, so I think one thing we haven't talked about is that a lot of the work I'm doing is more than just telling you to spend more or less in the top level channel. So we kind of divide that into the strategic optimization, which you might have the top level executives okaying money switching from department to department. But then you have the actual operators within the channels and they need to know how to spend money on different programs and different networks for their television, on different platforms for different display, on different keywords. And so that's that more day-to-day sort of technical type of optimization. And there are ways of giving recommendations at both of those levels. Depending on the type of business you have and what your objectives are. I don't know if that answers your question. - No, I'm gonna have a call this one. So I wanna ask a little bit about how models can work for different types of businesses. So you have a very small mom and pop shop and they have a consistent business. And the linear regression example you gave earlier, they're a flat intercept that be values pretty constant. And now they've decided they're gonna invest in some advertising. They buy TV ads that are gonna run for a two-week period. They should be able to look at the number of calls they're getting or number of sales they're generating and see a response curve with a nice decay. And you have a very, because it's isolated and it's a singular treatment, you have a really clean data set of which you can make some assumptions. But then look at a company like Coke, Pepsi, Microsoft, any big household brand. And they're everywhere and there are dozens of advertising activities constantly taking place. And I've often wondered what if Coke decided on May 21st, we're not gonna do any advertising that day. We're gonna cut our budget by 1/365th for the year and that's the day we're taking off. Now, of course, even doing that would cost you a lot of money to execute that plan. But just assume there was a switch you could flip. I doubt if we'd look and see Coke revenue have plummet and have this little spike. I think you'd see a pretty constant plot when you looked at that later. Now, if you did that for two days, maybe so. If you just stopped for an entire year, at some point, yeah, I'm almost certain we'd see a decline. But I guess so, I'm asking two questions. One is how can models work or can they work in homogenous way across advertisers? And secondly, how can they account for this complex ecosystem where if my hypothesis is true that one day of no advertising would have no net effect, how can that all roll together? - Yeah, so I actually wanted to mention this a little earlier and what I'm gonna talk about is just the base in our equation, the constant, the intercept. That can be interpreted in the econometric models as being the brand awareness or the brand equity. And in models I've seen for Coca-Cola and these other big brands, you mentioned, these bases, that constant might explain up 90 plus percent of the sales. So everything else that they're doing doesn't matter. And you hear, oh, Coke's one of the most valuable brands on Earth. - Right. - Well, that's why, that when you try to model everything, it turns out that the reason for sales of Coke is that there were sales yesterday of Coke. (laughing) That's actually 90% of the reason. It's just the awareness of the brands. Doesn't have to deal with seeing the polar bear so much. And what it does happen, what does have to do with seeing that polar bear every Christmas year and year. And you're right, if you stopped all advertising, Coke's, profits would go down a few percent. But after, probably they'd probably still don't care after a year. - Maybe, yeah. - Maybe after 10 years, you might not hear about Coke so much anymore. - Yeah, yeah, yeah. - And so there was some decay effect, and I think that would be a whole nother sort of submodel on that brand equity over time. Now, conversely, so if you, you know, all these models have this base or this constant term in them, if you're a super direct response company, you're selling a nose flute on a infomercial, right? (clearing throat) The base in that equation is gonna be really small 'cause no one's gonna know about this before or after. And the only people they're gonna buy are because they saw that infomercial. And so, in fact, the money that you spend on that infomercial is gonna explain 90% of the purchases in that model. - Sure. Ah, yeah, that's really insightful, that your model can account for that, that intercept, that brand equity, I think. - Yeah, and you can, so you can run time series models and run them, say you run them on the last 90 days of data and you run them forward kind of rolling. You could graph that base as time progresses and you can see how that brand equities for that. It's, you know, for a big brand, it's like this big monster that kind of lumbers around and it's gonna move slowly up and down, sort of generally being blown around by everything else that's going on. - At least in economics, we would, I think generally, accept the world as an incredibly stochastic place and there's a limit to the predictability we'll get to. Not just because there are going to be events that no one could have foreseen, you know, an earthquake's gonna happen and, of course, prices, or purchasing of certain things will go up and down in response to that and we didn't know it's gonna happen. But along the way, as we're modeling it, the modeler needs to make some choices about how much more time do I invest pursuing new variables to add into this. And there's no, at least to my knowledge, no clear theoretical upper bound there. Like, we know we can get to an explanatory model that has this quality of fitness. Yep, at some point, we can't invest $10 trillion in the staff of 1,000 to collect every little data point that might fit. How do you guys go about approaching what's the right data set? - Well, I'll never claim that we can do it perfectly. And I think that at some point, you're gonna come up the limit of you're not, we're not modeling the human mind yet, to do advertising maybe perfectly, and this is possibly a scary idea, but we could model the mind and then model how advertising would work on the mind and give everyone their own unique advertising that makes them do exactly what we want them to do. Luckily, we're not doing that yet. And so I think what you're talking about is how much effort do we spend in building inputs for the model? - Right. - What's the return on the effort of including inputs? - Yeah, absolutely. - That's actually an interesting analysis that could possibly be done. I think you'd have to be able to quantify the effort and the value of the input. I guess you could quantify the value of the input by how well it explains the output. It's definitely an inexact game. I don't know what else to say. - Well, there's a lot going on especially nowadays into more personalized advertising. So when we've identified a particular person, a search for, I think the analogy people tossed around when this first even became a thought, I started hearing 10 years ago, was someone searches for flowers. Perhaps certain demographics are interested in gardening. Other demographics are interested in sending flowers. Like 100 flowers would be the best response in one case and in other case some instructions for how to grow something. And if the search system in that case can identify who you are and make a good forecast about what you're actually looking for from this very terse and not fully formed query of flowers, they might give you a better response. And the same is somewhat true in advertising. We've seen things like retargeting, which I think is generally thought to have been a failure in general, where essentially now you, people who are engaging in those activities are probably adding noise to an attribution model, I would guess, because now what you thought was one channel is actually this multivariate sort of thing where for the people that they're personalizing very well to, it performs one way and then for the other people it's performing in some different way, would you just let that fall out at an aggregate or does that fit into a model? - We do two things. We will filter out certain types of traffic. If there is a coupon affiliate type touch point, which happens within like five or 10 seconds of the actual conversion, we know that that's fraudulent or that that's probably not a real deal. - That makes sense. - So you can do that and the other thing you can do is if they truly aren't associated with conversions, the statistics should determine that that is an evaluable touch point and it won't get any weight, right? Because it's included equally on all types of converting and non-converting paths. So, you know, what the statistics see is that, hey, when it's included, it doesn't make a change, so there's no value in it. - So I got one, I was inspired gonna give you just kind of a last softball brain teaser kind of question. I was inspired a little bit by a talk, which I was trying to find this week, I couldn't track down. It was either a talk I saw or article I read or something like that, but about the early guys at Amazon who developed their recommendation system and the joke that the team had was, well, when is our work complete? And we'll know that our work is finally finished when Amazon webpage, when you go to it, shows one product and it's the exact product you buy. And that's how we know our recommenders finally finished. In the spirit of that, maybe one could say that advertising is essentially just content. 'Cause it is, but you wanna get the right content in front of the right people. So if advertising became a quote solved, unquote problem, there wouldn't be advertising anymore. You would just have channels putting the right information on the right channels. And then if that problem could be solved, the sort of game over, there is no more advertising. Or on the flip side of that, I guess the antithesis would be to say that even if we get there, someone who wants to push their product is gonna start yelling on a street corner, hey, buy this. And if they yell loud enough that they attract business, the fact that you're not yelling means you have to go and start yelling. And it's this detente where because others are advertising, I have to start advertising. And there's this multiplayer kind of game taking place where, so I guess in more of the modeling context, the fact that I'm going to make a change will have other people make a change. And that in theory will show up in the future in my data. Can you account for that or is that just the nature of the unpredictability of the world? - Well, I think people will always come up with disruptive technologies based on old or new technologies to advertise to us unless it becomes illegal. So I think this is a really interesting topic, actually. What is advertising going to be like in the future? And what do you want it to be like? What do I want it to be like? - No. - I really think there's going to be a kind of knob and you have this knob associated with it and you can turn it up or down. And when you turn it up, it floods you with advertisement. But you get paid for it or you get reimbursed for it. So I would imagine a family that wants to have their TV, their TV service, all their cell phones, their laptops, their tablets, all paid for. They can turn that switch way up. They don't have to pay a monthly fund for any of this, they don't have to buy any of this equipment. And they'd see bunches of advertisements everywhere in their experience. Now me, I actually don't want to see any of this. So I turn it all the way down and you might get charged for that. You might have to pay to, God forbid, you might have to pay to read your New Yorker without advertising. I actually think I should get paid to read the New Yorker but I don't. So I guess I pay a couple bucks. I can't believe that that's all that's holding up the whole institution. But if you could choose this sort of switch, I would choose to have no advertising in my life whatsoever. Do you think corporations are going to be okay with that? - Oh, certainly not. - Well, do you think that it might be impossible to even get away from advertising? Like it might be somehow illegal to extract yourself from advertising in the future. - Illegal to extract is tough claim to make. I'm not sure, but I actually want to retract my certainly not answer because I thought about it more from the advertiser's perspective. The guy who's running, if I'm running an ad blocker, essentially what that means is I'm paying an advertiser to show an ad and that advertising system, whether it's a real time display network or it's Google or whomever, think they're sending me an ad and only in my client in my browser is it being blocked and they might not know that. So that advertiser's paying for a service they didn't receive. If I were an advertiser and I knew your knob was turned off, I might be happy about that 'cause I'm not gonna waste spend on you. But the flip side is like, no, I want to get through everybody. I want to be that spam artist. - Yeah, and I might add, you framed that idea that way. What about me framing it this way? I was exposed to an ad that I did not choose to be exposed to. Like, what about my rights to what goes into my eyeballs? - Yeah, well, I mean, if I don't like the ads on a website, I can choose not to go there. But if I've noticed the TV I bought when I turn it on, there's an ad for another product that that company owns. And that kind of annoys me, so. - It's probably baked into a circuit somewhere. - Yeah, probably. - It's internet ready, so I'm sure it updates itself. - Oh no, so there you go. Now you have advertising baked into the TV that you can't get rid of. - Right, although I could also make the argument that I chose to buy that TV. Now I wasn't aware of this quote unquote feature when I bought it, or I might have selected it for model. But hypothetically, the free market could give us TVs that don't have garbage built into them. And those manufacturers would make more money. But it would certainly be an interesting time to see what role legislation plays and what advertising looks like in the future. - Yeah, let's hope that it becomes more and more targeted and that we have the right to turn the targeting off if we wish. - Yeah, I think anything that contributes to targeting is in a way a benefit because why would I want to see a garbage ad that has nothing to do with me? If it's targeted to me, maybe I don't like me advertised to, but if it's at least vaguely relevant, there are times when advertising does come off as a service. - I'm always very suspicious of anyone who claims that advertising is a good service for people, but. - That is a good suspicion to have. This is my skepticism about the system. - My only counter example is like ticket fly. I get emails from them. I don't remember asking for them to send me emails, but every once in a while, I'll be like, oh, the whole state is in town, or a man man is playing at the LRA, cool. And I didn't know that. And that's so it was, ended up being news. - So I could say that's almost like you're part of a community. - Yeah, it's a fine line. - Which is a way of advertising. - So I like to kind of end every episode and ask my guest to give me one, what I call a benevolent link, which is something you have nothing to do with, but would like to give some advertising, if you will, to something you find valuable, and one completely self-serving link that benefits you and you get something hopefully out of it. - Well, for my benevolent link, I would point people to the Long Now Foundation, which is a foundation that's very creative group that's thinking about different ways for us to think about time in our place on the planet. So one of the projects is a clock. It's called the 10,000 year clock. And so the idea was, hey, can we build a clock that will run for 10,000 years? - That in and of itself is a really interesting engineering question. And they indeed built several models of this prototype and they have a clock. It's based on Bronze Age technology or can be maintained with Bronze Age technology, which is one of the requirements so that anybody in the future can fix it. With, you know, it needs to be charged. It gets charged off people coming to visit it by standing on a platform. That kind of moves down a little bit. And it has a, I believe it just tracks days and it has this thing they call the, I think the cylinder of time or the cam of time. And there's a needle that kind of falls on it and it's this kind of beautiful, feminine shape which describes basically where the sun's gonna be every day of the year when it looks up at noon with a detector for the next 10,000 years, something like that. And that's very interesting. They have a Rosetta-type project. It's called the Rosetta Disc where they inscribed like hundreds of different languages and on a single disc of sapphire that will last for many thousands of years. I think they're involved in like the, looking at the Bristlecone Pines in like Colorado that are very long lived trees at high altitude. So interesting projects. Self-serving, instead of pitching my company, I would pitch my very close friend, Jeff Jaganich, who has some fantastic art out there, some really abstract stuff. He just visited me and dropped off a piece that I was lucky enough to get off Kickstarter that he spent over a year on and just has a lot of really, really neat arties. He's a true artist, he's a resident in Denver right now. - Excellent, I'll put links in the show notes for anyone who wants to go check both those things out. Well, thanks so much for doing this, Nathan. This was awesome. - Yeah, this was great, yeah. Thanks Kyle. - We'll do it again sometime. - Totally good. (gentle music) (gentle music) (gentle music) (soft music)