In this episode, Jake and Michael discuss feature flags, particularly the freshly-released before hook, and the perils of incorrect eager loading as your application scales.
Show links
North Meets South Web Podcast
The one with feature flags
In this episode, Jake and Michael discuss feature flags, particularly the freshly-released before hook, and the perils of incorrect eager loading as your application scales.
Show links
- Duration:
- 36m
- Broadcast on:
- 11 Jul 2024
- Audio Format:
- mp3
I'm Michael Dorenda and I'm Jake Bennett and welcome to episode 158 of the North Meets South Meets Hopes and Dreams podcast. Meets Hopes and Dreams. Oh, I've got to hear about this. What are the Hopes and Dreams here? I hope and dream that we have something to talk about this episode. Oh, we've got plenty. We've got plenty. Let me start by saying, this last week I had the best mustache I've ever had in my life and it was incredible. It lasted for almost a whole week and today I had to get rid of it and I was sad because, you know, here's the deal. Let me be real honest. I had to shave it this morning because I had like this mustache power thing going on and I was yesterday, I was undefeated. I could not lose yesterday and I didn't want to go to work today and ruin the streak so I shaved it. I was like, I'm done. Let me tell you, can I just tell you real quick? I don't want to break my own, patting myself on the back here but I'm just going to tell you what happened. So on Friday, I went into work last week, had my mustache. By the way, here it is. I do a freedom's stash every year. So like for the 4th of July, I shave a must, I do a mustache. So I grow up my facial hair for June and then July, I did a do mustache. I got eight guys to do it with me this year, which was so fun. So next year, maybe I'll start a Twitter group and see who I can get to do it and just, you know, freedom's stash for them. It's going to be great. Even Australians and Canadians, you're all invited to do the freedom's stash. It's going to be awesome. So did my freedom's stash, went to work on Friday and we have this basketball hoop up on a door, like one of those little Nerf basketball hoops, okay, in our banking department and then there's a literally a tape line on the floor. One of my buddies brought it in and the original idea was we were each going to put a dollar in for the week and whoever got the shot first earns the money. But, and then, you know, we ended up starting getting decent at it. And so anyway, yeah, I've made it once a month. It has to be, here's the deal. It has to be your first shot of the day. You can't warm up, no warm ups, one shot from the line. If you make it, you get to put your name on a sheet of paper that's on the door. Great. So it had been two months since I'd made this shot. I could not get it. And so anyway, went on on Friday. Yep. Yeah, exactly. Yes. Little mini ball. Yep. So on Friday, I went in with the mustache, made it. Nailed it. So excited. Got my name put on the door. Okay. Then I go in Monday, still have the mustache. Make it. Make it first shot. So two days in a row. That's never happened to me before. Well, then there's also another line that's like three feet back from the regular line that we call the money ball shot. Nobody's ever made it first shot. So I had mind you, I just made the regular shot. So I back up, I'm like, Hey, I'm feeling it. Let's go for it. I made the second shot, the money ball shot. Yes. Unbelievable. So then I tell the developers that morning when we're having our stand up, I'm like, Hey, by the way, it's a really good day for me. You know, I feel like I'm going to win the game that we play every day in our development meeting. We have this thing called boom party, which is like bomber man. And so if you win all three rounds, you get the crown for the day in our dev chat. So I told them I called it. I was like, Hey, by the way, I'm going to win this today. And they were not going easy on because they want to kill you. There's no way, you know, it's like on the last round, everybody's against you. I won. I beat every one offering is unbelievable, unbelievable day yesterday. So I had to shave it just to make sure that I didn't screw up the streak today. And yeah, so that was that was the story of the stash this week. It was a good week. It was a good week. Well, yeah, we I got one of those little hoops for my birthday this year from the inlaws. And it's like it's got rubber, like strip on the door. So it's like, not supposed to kind of wobble around. But any time the ball touches any part of that thing, the hole just shakes and wobbles around. So if your shot is not dead in the center of the ring, it's not going in nine times out of it. Right. Because it just like flings the ball out. And we've got two point or meter ceilings. Yeah, which is like really standard. It's like eight feet. What is that? Let me see her. Let me see her. Yeah. Seven feet, seven feet, 11 inches. Yeah. So it's like seven feet, you know, 10 and a half inches. So it's fairly standard for like most times built these days, 2.7, which would be like nine feet, I guess, or 12. I think it's usually like eight foot, nine foot and 12 foot ceilings or whatever. Eight feet. Yeah, almost nine feet. Yeah. But yeah, it's too. There's not much room to get arc. Right. It's, you have to shoot it very flat and you have to be very precise with it. And so it's you want the secret? You want the secret? You know what it is? Don't be sure it will like me. It's called, it's called the Cobra. It's the under Cobra and, and, and it's a backspin with just like this. Yeah, that's, that's the trick. And so that allows you to start low at like your waist. And then you get the backspin, you get that shooter's role and lay it right in there, man. I have to give that a go because it's like, it's easier for Eli because Eli will go right up to the ring. And he's exactly. He's quite. So he can actually get some arc on his shop. The amount of times that boy tells me up, you know, 20 to four or something like that. It's embarrassing. It's hilarious. That's so funny. Yeah. But he loves it. That's good. Yeah. That's good. Yeah. He's on, he's on school holidays at the moment. So we, I had him yesterday, the day we went to the shops. They've got like one of these giant chess, chess boards. He's like, let's play. Oh, cool. Do you like chess? He could play chess. Nice. Oh, oh, okay. Okay. I know very vaguely the rules. Like, I know that the pawns can move like one space and I know that the night can go like in an L shape and queen or, you know, so it's, it's very loose. Like there's no, and he's sick. So it's like, just let him win because you don't want him in the mall. So that was a bit of fun because he. That is fun. I had to, I had to start taking all of his pawns eventually because he just, I'm like, are you sure you want to put your piece there? Like what, what shape can like the night move in? He goes, oh, yeah, I probably shouldn't. I could go. But in the end, he just decided to take, just like, take his bishop or whatever and walk over and just take my king. I'm like, all right. Well, that's been 20 minutes. That's long enough and they haven't been any tears. So let's get happy before something. Nice. Nice. That's awesome. Yeah. I don't want to look it up because like he enjoys it. So I'd like to at least have a knowledge of like which pieces can move where, like a really accurate knowledge of that. And then the rest we can just make up this. I think that like chess.com has, you know, some tutorials and stuff. But yeah, it's, it's pretty simple, rooks moves and straight lines, bishop's room and root move and diagonals the nights move in the L shape. Then you've got the queen can move anywhere. The king can move one space anywhere. And then the pawns can move two spaces forward on their first move and only one space forward on any other move, but they can only attack diagonal. They can't attack directly in front of themselves. So like, yeah, yep, that's every. That's all. Yeah, look up Fool's mate. That's a good one too, to know. So if you don't, if you're playing with somebody who doesn't know how to play chess, you can typically checkmate them in about three or four moves, which makes you feel really fun. It's, it's really good. It's a good move. Yeah. Yeah. So anyway, okay. So shall we talk about the stuff? So I think maybe one thing to talk about real quick is that Tim McDonald put out a new Laravel pennant feature that dropped this week. I've been waiting for so long. Have you? Okay, so so let's talk real quick about feature flags and the idea behind them and what things that affords you as a development team. And then let's talk about pennant. So and then let's talk about the new feature as well. So let's talk real quick about like feature flags. So the typical way that you roll out features without feature flags is you just say, okay, we've been working on this feature for a week. And we're pretty confident it's going to work. And we've tested it locally. And we've tested it in our CI environment. And now we're going to ship it to production. And so you ship it to production and you watch your error tracking service to make sure that you know, nothing breaks in production. And hopefully you've got like a group of people that you typically say like, Hey, Scott, can you go try this out? Because you know, you're a pretty good tester. And you just just try and try it out and see if you could do anything with it. And they break it, whatever. And then you're like, Oh, crap, it is it is broken. I didn't think about that. And then you have to like roll back. And then you have to go fix it. And then you kind of do it again, you come a couple iterations of this and and there you have it. The problem with that is, you know, ideally, you don't have a breakage. And most of the time you probably don't. But there's always this sort of like lurking fear in the back of your mind like, Oh, I don't think it's going to mess up. But I always like want to ship it on like a Monday night, like after everybody's gone. So I can be the first one to test it sort of deal. Well, feature flags removes that barrier. For me, it does at least because what it allows you to do is it allows you to ship a feature to production without exposing it to anybody except for yourself or a group of beta testers or whoever else you want to bring in on that one. And so you basically surround your feature with in Laravel, you actually get a directive with this pennant package. And in the directive, you can say at feature and then you can either pass a string in there. If you name the feature with a string, or there's a class based resolution that you can use. And so you pass in the class there at feature and then the name of the class. And if that feature is enabled for that particular user over that particular scope, it will show it. And if it does, if it is not enabled, then it won't. And so you can use a blade directive and then there's also, you know, a facade that you can do the same thing feature enabled or whatever it is inside of your controllers or inside of your actions or your jobs or whatever it is. And so you can kind of bifurcate your logic and decide what you want to do based on who has the feature and who doesn't. That's the great part about it. And for me, it has helped me ship more often and with greater confidence than I ever did before. Because even if a feature isn't completely 100% bulletproof tested, I know it's exactly perfect, I can ship it and I can iterate on it after getting it into production. As long as I know it's not going to like bork everything, I don't have to like push it to everybody. I can say it's the end of the week. It's Thursday. I'm going to push it out. We'll maybe do some tweaks next week and we'll push it out again. And there you go. But man, it's insanely powerful. How long have you guys been using that sort of development flow for Michael? It's been relatively new to me and I've been loving it. Yeah, we've been using it sort of haphazardly, I guess, for the last five or six months, I reckon. And when I say haphazardly, we've kind of like gated the feature in the back end, but forgotten to hide it on the front end. And so that if you click the link and then get gotten a 403 or whatever, because the feature's not actually active, but the link to that feature was visible in the menu bar and things like that. So, and this is, I guess, part of the problem with having a separate front end is that you don't have the blade directive. So when you're building the back end functionality, you have to remember to like expose that through inertia or wherever else of your API and making sure that the front end is actually built in such a way that it considers the feature flags. Whereas when you're when you're using, you know, Livewire, if you're using blade, it's it's more evident because, you know, you are more likely to be building the front end than if you have a separate back and front end team working on it. So yeah, it's been it's been nice. But when pennant came out, which was what Laracon US Laracon. Yeah. Oh, yeah, right around. No, no, you know what? Actually, I think it was talked about in Laracon, India was early maybe. I think they were talking about it then, but it wasn't released then I think it like sort of released with Laraval 10. I don't know. I don't remember. It's a good question. I'm not sure. So yeah, when when it came out, you know, we started playing around with it. And I remember using it for the Laracon AU website last year, where I wanted to have a feature like to use the feature flags as like a time bomb kind of thing that like show this thing after this day. But the way that the feature flags were resolved by pennant at the time was it would resolve at once and then it would associate it with the user, right? And because the Laracon AU website is more of like a visitor, a guest user kind of thing, there's no user to associate it with. So when it resolved that feature, you know, it was it was before that date, for example, we wanted to show the schedule after a certain date because it had already resolved it, it doesn't resolve it again. Correct. And so, you know, we worked around that somehow. It basically is like you can't use it at that point. Yeah, it's like, yeah. Yeah. Yep. So I think you can't use it in that way necessarily. Yeah. Right. I think the way that we worked around it in the end was to use like the the explicit array storage so that every request, okay, every request would do it. Right. There we go. So with with this feature that that team ship this week, he messaged me and he's like, I finally did it. Oh, he messed it. Oh, okay. Nice. Was to introduce this before hook in a similar way to like Laravel policies work where you can divide a escape hatch, I suppose, to do a test that like, if you have an admin user logged in, they default have permission to do everything in the application. So this before hook in pennant works in a similar way where you could put these kind of things in there and then get it to trigger based on an environment variable or get it to based on, you know, a time switch or something like that. And that way it would always check that before then resolving it from whatever the normal case would be, you know, to use a lottery or to resolve it from the database or whatever else. So yeah, that's going to be really useful in a lot of situations like that where you want to, you know, either globally enable or disable a feature without impacting on the already resolved values in the database because it will skip doing that those checks. Which is huge. So let me kind of go back just a quick step to talk about this. So each feature, if you're using class based resolution, which is what we do, when that feature isn't is first encountered in your code base, what it will do is it will previous to this, it would attempt to resolve that feature. And so the resolve method is called on that feature and it gets past a scope. Now the default scope that the feature gets is the logged in user. That's what passes in to that scope by default. Now you can change that if you want to, if you're using the feature facade, you can pass your own scope. So the scope could be, hey, I don't want the user, I want the user's team. And then the resolve function now gets the team, or you can pass a string. So you could say scope is, and what we've done sometimes is we say scope is global, just a string called global. And then what it does is when it encounters that flag, it will pass that scope in, and then it stores the results of that resolve method, either truthy or falsely true or false inside of your preferred storage driver. So in our case, we use the database. And so if if it looks when it's going to resolve that and sees that that scope, that particular scope has already been resolved, it will not resolve it again, which is why that global scope is really nice. You just resolve it once, and then it's resolved for everybody. However, what Michael said is true, which is once it's resolved once, it's never resolved again. So if you wanted to do something where you said, okay, now I want to enable it for everybody, what you'd have to do is you'd have to wipe out all the values for everybody in the database, which is annoying, because you have to do that in production somehow, you may have to go kill all those database records, or you have to modify the code, whatever. So with this check now, and I think there's one thing to point out here, Michael, that's maybe, I don't know if this is strictly this, but what he said is this is it before, it's performing always in memory checks before resolving it features stored value. That always in memory, I'm not sure what exactly he means by that. Like you would always do it, as opposed to, you know, not like skipping the check if there is a value in your storage driver. Yeah. Yep, yep, that makes that makes sense. Yep. And so in the case that you want to always resolve this particular thing, then it will do that before check will always do it. And you don't have to return a value, right? You can just not return a value. So instead of, yep, you can just return all or not return anything and just return void. In the case that it returns void, it just won't do anything, it'll just fall through to the resolve method. But if you do return a truth or a truthy value from that, it will either enable or disable that feature for that thing. So the time bombs thing worked, like you said there. And then the other thing that's interesting that we talked about, that we talked about too, is like if you had, you know, so the way he calls that is not a time bomb, but like basically a rollout schedule, right? Schedule a feature rollout based to be on like this date for everybody. So we have a beta group that's going to get it, but then two weeks from now, everybody should get it unless we need to specify otherwise, which I think is also really great. So really cool addition there and much needed addition. I think that's, uh, that's, that's pretty awesome. So also the ability to, whether you do this in the database or if you do it in an environment variable, you know, being able to globally toggle a feature, you know, you might have rolled something out and it's, you know, some issue, like maybe it's connecting with the third party and the third party has got some downtime or whatever, rather than persistently hitting that and then degrading the experience of your users, you can just turn off using, you know, either a database flag or an environment variable like to say, you know, this thing, this thing is now off. So that's where I would. That's a great point. Yeah. Yeah. In the before, I'd be like, no, this, this feature, yes, it's a feature in the application, but we're globally turning it off for whatever reason, you know, you love that. Yeah, that's a great idea. So then that makes it really easy to manage. Yes. Because then you can resolve it basically for a scope or globally, right? You could do both. You don't have to choose one or the other. You can, you can do both. Nice. Yeah, that's a really powerful feature. I'm liking this a lot. And that's, and that's a good shortcut, you know, so that you don't have to then go and purge all of the already resolved records. Yes. Which means it doesn't impact on the, the status of, you know, individual users might have something enabled. You don't have to remove it or one have to re-resolve it. That is all for a huge one right now. We're turning this off. And that way, you know, once you flick that global switch again, it will go back to whatever it was before. So if you had access before, you will continue to have access once that before hook, you know, whatever that is, is, is restored to that original state as well. So yeah, I'm very excited for that feature. Yeah, we have had ones where we've, we've turned it on just for a specific group, like manually. We went into the database and edited the value to just return true. And it was, you know, it was particular to those, those users. And maybe that's not a great way to do it, but like we needed to do it that way for a period of time. But then there was a situation where I needed to turn it off and it was like, dang it. Well, I have to either just modify that code and re-resolve it for everybody or just always, you know, I don't know, it was, it was goofy. So I didn't really have a good way to do that. But this is exactly that way. Depending on where you are, if you're not, you know, if you're in an organization that has documented policies around, like I know as part of ISO and as part of SOC 2, like there's the whole, someone has to approve the code. There's got to be, you know, all of these steps that you have to go through, which can delay, you know, just versus just having a switch that you can just turn off. Yeah. And that can then, you know, I think we spoke about this previously, where we can then put the feature flags in the hands of the business rather than correct developers. Yeah. Well, well, you know, there's a compliance issue here or there's some other business reason that we want to disable this feature. So they can just go into some interface and just turn it off. I love that. Yeah. I really like that. Yeah, big fan of this, this edition. Huge. The other thing that's, that maybe we could talk about panic for just a minute here is that these features, if it's not returned from the, the before, typically you are invoking some sort of storage mechanism that you're using to resolve these things out of, right? So it's whether it's the database or whether it's, you know, radus or some other, some other deal, you're having to query or go grab something to figure out if it's enabled or not. So just like any other thing, if you're not eager loading those values, you end up making a query for each one of them. So that's one of the thing you're going to want to probably do is if you have a page with a lot of different feature flags on it, it's a great idea to resolve those ahead of time, because if you don't, you go look at your level debug bar, you're going to see, you know, one query per feature. And so that can rack up the number of queries you've got pretty quickly. One thing that we did to help solve this problem is we really, there's this one page where we're doing a lot of rapid application development on it. And this is one of the only places that we are using the feature flagging pendant stuff. And there's other applications, but in this application, that's where we're using it the most. And so that features directory, the level app features directory, what we'll do is we actually have a little command that scans through that list of classes that's in there, and then says, resolve this for all users ahead of time. Like every single user in the database, or yeah, every single user that we have in the database, resolve it for all of them. Now we've only got hundreds of users, right? So like it's in this in particular case, it's all internal users, and so we can resolve it for everybody ahead of time. And then what we do is we say, okay, now get feature double colon loaded. And what I think that does is it says basically grab the distinct features that are in the database, and give me the list of those. So what we do is we say, bump that feature loaded list up against that list of classes that we just grabbed out of the file system. And any of the features that no longer exist purge them from the database, so get rid of them. They're old, they're stale. And then we cash that list of features that I just that I just mentioned, and we eager load those features every time that that big page is loaded. So we basically ahead of time, when we're doing our pipeline, our continuous integration deployment pipeline, we're in advance, loading all of them, caching all of them, and then asking them to be eager loaded on that page, which works really well. I was going to talk to Tim about that and see if there was like some way to do that. If you tagged a feature with some tag or something like that, and just could say, eager load all the features with this tag, that would be really handy, I think, and actually even better. See, there we go. I think because having the eager load them manually, you kind of have to look through all of the things that are on that page, and you just kind of have to know, Oh, these are the ones where you're using. But it'd be nice to just be able to tag them and then say, eager load all these. I'm sure I could roll something like that myself, you know, just throw it on there. And then yeah, what do you think of this? Yeah, there's precedent in the framework now, or using attributes for that kind of behavior, the model observers and things like that. So yeah, a some some attribute that is well named by the Laravel team, no doubt, that that allows you to do that might be might be an approach to go. And since I'm already doing this, so like, since I'm already in my CI, doing this thing where I'm when I'm deploying, I'm looping through every feature and sort of precaching it, you know what I mean? I could just say, and each one, like, I could have like a global list of loaded features, and then I could have here's the features with this tag, and here's the features with this tag. And so if I said, you know, feature load for scope user, whatever it is, I can't remember exactly what it is, but then you list all the features load missing is what it is feature load missing, and then you pass in the list of features. It'll eager load them for you. So just one query instead of 15 or 20 or whatever it is, you know, so interesting. Okay, some homework for me. That could be, that could be cool. But I've really liked using it, and I feel like it has done a lot of good for me, and really excited about this new one. So pretty cool. Someone someone will no doubt write some kind of filament plug in or an over plug in or something, you know, to manage features in the application that you can just drop into your app and then off you go, which would be really handy as well. Okay, go ahead. On the subject of ego loading, if we're ready to move on from from I have one, I have, I do have one more thing to talk about what features but we can come back to it because there's not necessarily specific to features. So go ahead, ego loading. So we, we have this like one endpoint in our application that is responsible for calculating the state of an application. So I like this already steps, there are multiple steps in the application. Yes, yes, multiple steps, yes. And each, each step has a status, and the status could be, you know, complete or pristine, which means it hasn't been accessed yet. I like that. Oh, that is such a good name. I've called it, I've used fresh before. I hate fresh pristine is better way better. Yes. So each, each step can exist in one of these three statuses. But sometimes you might move between a a visited low. So the steps are different or there's different requirements depending on what you're in. So anytime you open an application, we hit this endpoint that goes through and calculates the state that the application is in where you're up to, how much you've completed, whatever. For, as it turns out, two years, right, this endpoint has existed, it has, this is a persistent bug that has been there for two years, but it presented sporadically. And it was not like a per application thing, it was just like a time of day thing, if you know, the right number of people happen to open an application at the same time, like weird things would happen. And instead of taking like a second or a few hundred milliseconds, it would take 30 seconds or 60 seconds or it would timeout. And because of the sporadic presentation of this bug, it's been very difficult to track down because usually by the time you get a report of it and you look at it, the issue's gone away. It's fixed. Yep. It's fixed. It's gone away. But it's always crept up, you know, every now and then. And to the point where no one was reporting anymore, because it wasn't happening anymore. But yesterday, so we've been doing these, these merging of tenants over the past few weeks. And so we've, we've done six out of seven so far. So we've basically almost considerably grown the size of this single tenant over the last few weeks. And, and yesterday, fortunately, when I was not at work, there were all these reports of like all of these timeouts. And this, this endpoint that, you know, is typically fine, but sporadically not was like getting to the point where every time it was opened, it would cause an error. And I got a message at like eight o'clock last night, nine o'clock last night from my boss, who's like, I have cracked it finally. He's like, please review this PR. And what it boiled down to was we had changed a bunch of stuff from doing some explicit queries to using ego loads or lazy ego loads. So you can do like model arrow load, right? Yes, right. So we were saying, you know, instead of querying the database again, we'll like, just say, go and get me these records. And it was some, you know, there was some nesting. So we were getting like a status dot step dot state or whatever it was. The problem was, as we had merged the tenants together, we had gone from like a few hundred records of a few hundred of these types of records in a tenant to tens of thousands of records in this town. And because this thing was like doing model arrow load, whatever, it was loading, not every single one related to the application. Oh, gosh, every single record that existed in the database of that record type. Now, when you've only got hundreds of records, sure, that's going to present sporadically, isn't it? Because, you know, sometimes you load a couple of hundred records and no one will be doing anything and it'll be fine. And sometimes you'll have three or four people do it. And it's going to go through this every single time. So the solution to this problem was to like explicitly add like where has on the Oh, okay. So like load arrow statuses arrow, where has application ID equal the application that was being loaded. You would think that relationship would just figure that out. Like you would think it would just do that. Yeah, but it but it was not. So it was not like the parent relationship. It was not the application that we were loading from. It was from like the application, a nested child sort of thing that the wizard. Yeah. So it had lost that linkage data. Yeah, we were we were eager loading the records we thought we needed. It's just because they were not scoped to the application that was being requested. It was loading, you know, a few weeks ago, a couple of hundred records and then it was a thousand records and then it was five thousand records until yesterday, you know, you have seven or eight people open tens of thousands of records at once. And it's like churning through each of those things to try and calculate the state at the state. And it was like, so effectively, it was recalculating the state for every single application. Every time one application was open. That's brutal. And and the you know, we got to the point where this was happening so much that like the IOPS were through the roof on on the RDS instance, which then caused the CPU credits to go bye bye. Yeah, so basically, we solved this two year old bug. Well, I say we my boss solved this this two year old bug within, you know, 12 hours out of necessity because it took, you know, the database down, which by and large is very over provision for our needs. It's just this one end point. You had to you had to get all those records in there in order to be able to reproduce it consistently enough to find out what the problem was. Like this is really kind of looking down to the fact that yeah, if it was not for the fact that we merged the tenants together, this this issue might not have ever been so for two, three, five years, whatever, but because we combine, you know, thousands and thousands of applications over over, you know, several years. And it's like, and I looked at I'm like, oh, no, I, I wrote that eagle load to you. Oh, no, yeah. So there you go. It's easy to do. It really is easy to do production database back to zero zero back to zero. Well, the thing is that actually was crashing it all the time. It's just, you know, I had a similar situation today where it was like we we had a worker, a background, sort of lambda thing that was just running jobs and crashed our main application, like our actual like system of record just literally pegged the CPU, hung everybody had to get out. We were like, okay, what was that about? And so go in and restart it and like restart the application server. Like what's going on? Okay. All right. So I was like, pause, pause our lambda there and was like, okay, so everything came back up. We're like, okay, let's let's start it. And then we started it and watching the CPU like pegged, like, dang it. Okay, pause it again. Went back down to 60. Like, okay, start it one more time. Grow, grow, grow pegged. I'm like, okay, well, we know what it is. That's like, you know, so I just had to reduce the number of workers and then it kind of settled down a little bit. But yeah, I mean, sometimes it's just like, you don't know until, you know, I've never had that many jobs in there before. It was like 10,000 jobs backed up. And it's like, you just never know. I've never had to. It's never been pushed that hard. And so whatever, I guess. So I had to limit the number of workers. Yeah, you can plan for like some end state, but a lot of the time you're so busy focused on like, what can we deliver now to make the business get there that it's like, sometimes these issues at scale don't actually present until you're at scale. And until you until you get there. You won't know. And you're just if you always were trying to prepare for that and always trying to like mitigate those risks before they became risks, you'd never ship. That's the problem. So it's like, you kind of just got to get it out there. That's like, and then secondly, like black, you know, black fire, whatever it is, like those those monitoring things to be like, where are the slow queries? Oh, okay, there's the slow queries. I need to go fix those up. Yeah, I mean, at the end of the day, it comes down to how well you respond to those issues and what's your workflow like in terms of addressing it when it happens. And we've got, you know, I've I've prepared data dog in the past because it is like a very heavy, very complex piece of software that, you know, does a lot of stuff. But if not for like the traces in there and like the query monitoring all of that stuff, we wouldn't have been able to kind of narrow down the exact query, which then help us narrow down the exact line of code, which then allowed us to figure out, you know, oh, we were not scoping that query. So instead of taking, you know, 10 to 15 seconds, build this query to execute, you know, isolation, you know, when no one else was using the system, it's now like consistently less than one second. So that's awesome. So yeah, it feels good. It feels good. Yeah, it's good. And like, I wasn't involved in the fix. So that's nice. Like, yes, I introduced it. And if I was at work, I probably would have gotten to the bottom of it. But it's nice that we've grown as as a business and as a team to the point where like anyone can look at any of these things and just deal with it. And so that's, you know, from a, from a team and business growth perspective, it's really good. That's awesome. Yeah. It's nice to have somebody else on the team other than you that can fix the problem, you know? Yeah, shout out to Sam. He has changed his profile picture in Slack to fireman Sam, because he just seems to be putting out fires for everyone lately. So fireman Sam, I love it. That's great. That's funny. All right, dude, let's wrap this one up. What do we got 157? Is that what it was? 158. 158. Thanks everybody for tuning in, hanging out with us. Find a show notes for this episode at north means south audio slash 158, head us up on Twitter at Michael Jordan at Jacob and north south audio. And as always, if you like the show, rate it up in your podcatcherf's choice, five stars would be incredible, amazing, and awesome. All right, folks, see you in two weeks. Peace. [Music]