Don't even need to use it as a CI platform. This is something you can use in for more than CI. I've never thought about it outside the context of CI. That's really interesting. Normally, yeah, it should work. Yeah. [LAUGHTER] I'm a bit too interested. I'm a bit too interested. [LAUGHTER] Go ahead and edit that bit out. [LAUGHTER] Oh, yeah. Let's just clear this green, because I can't edit that part out. [LAUGHTER] It does work. There's nothing wrong with promised hope. [LAUGHTER] Yeah, yeah. I think you might look correctly. [LAUGHTER] Yeah, this is going to be really interesting. [MUSIC PLAYING] [MUSIC PLAYING] [MUSIC PLAYING] [MUSIC PLAYING] [MUSIC PLAYING] [MUSIC PLAYING] [MUSIC PLAYING] So last time we did this, I propose the next time we get together, we will pair up on something specific. And that's exactly what we will do today. Welcome back, Alex. Thank you. Nice to be back. So we are also putting to the test the more the merrier. Welcome, Tom. Thank you. Let's summarize what we want to achieve today. And I have three items on my list. The first item from last time was to speed up each pipeline run. Bad least 3x, but hopefully 5x. Second point is to try out the new TypeScript support in Dagger modules. And three is to learn what we are missing in Dagger to make this easier for others. If we go back to point number one, Alex, can you tell us more about the pipeline that you have? Currently our pipeline takes about 15 to 20 minutes to run. And the main issue is we do a type check and link phase. Then we do a phase where we upload source maps to our integration and production environments so you can get nice stack traces in production. And then we do a phase where we run our component unit and end-to-end tests. And then we do the final stage where we deploy if everything is green. But the problem is that each one of those stages in GitHub actions, it has to run a full MPM run build. And because the asset is not shareable between each part, each phase of the pipeline, we have to rebuild every time. And that rebuild takes anywhere from two to three minutes per build. So it's two to three minutes times the amount of phases there are plus the duration of each phase. And yeah, it's roughly 15 to 20 minutes, which is pretty painful if we've ever had a deployment that's gone wrong. And we need to revert and that pipeline has to run. Typically what we've then done is commented out a bunch of the phases so we can roll back and then add them back in a minute in a future commit. It's just so nice. If we said the baseline was three to four minutes for the pipeline run, and we could give that as a guaranteed time to revert to any of our stakeholders, I think they'd feel a little bit more comfortable than 15 to 20 minutes to roll back. Right. And again, this 15 to 20 minutes is everything from building, glinting, testing, integration tests you mentioned as well, and deploy. Yeah. All of it. So it's like a full CI/CD pipeline where also deployment happens. Where are you deploying? Currently we deploy to Lambda to run our application, but we're working on moving it into our EKS cluster. But yeah, for now it uses something called Arc. Is there a reason why you haven't combined all these steps into a single one? So when we're visualizing it in GitHub actions, we could see exactly what point of the pipeline failed. Their graph thing is quite interesting to look at. Having that visual feedback is kind of nice. Okay. So you're keeping the stages of the pipeline separate so that you can see them in GitHub on the GitHub actions graph. That is helpful for you to know which stage failed. That's why I do that, but that means that every stage has to build. Yeah, exactly. Every stage has to build. And also a GitHub actions, if you only have a single stage that failed, you can just rerun that failed stage after. So yeah, that reduces the amount of time you have to run the pipeline. By using Dagger, what your hope is, first of all, the total reduction of time. So from 15 to 20, we're thinking four to five. Yeah. So that's about a 4x. Okay, 4x improvement. That's what success would look like end to end. And you would still want to keep the separate stages in GitHub, or would you be okay to have the separate stages visible in a different way? Because again, Dagger, it tries to, like you would get a lot more optimizations if you would have a single pipeline because within the DAG, it can figure out what needs to rerun and whatnot. Now, you couldn't use the GitHub graph to see that you need to use something else. That's fine. It's more about as long as we've got something to observe and visualize our pipeline, then I don't see whether it'd be any pushback from the team. We normally run everything through Jenkins. GitHub actions is actually something we're thinking of retiring just because our entire CI/CD pipeline runs in Jenkins. We've got a dedicated server for it. Okay. It feels kind of like the consensus amongst the team is it feels kind of odd to have multiple. And I go ahead, you're a big believer in having redundancy and redundancies, having redundancies. But, yeah, I think in our case at the moment, it's curative and silent in one place. Why is this specific pipeline in GitHub actions? While the other things are in Jenkins, I'm missing that part. Yeah, so it was a rush to have something that worked. So I was due to go on holiday for three weeks when this application got built. And they just said, can you get it deploying to production? And I was like, yeah, but I kind of don't want to do without tests and a proper pipeline in place. And I've used GitHub actions before. Didn't have the most experience with Jenkins. So I just hacked something together. And then you know how it is in business. You put something together and it will stay there and it will stand the test of time just because other things have high priority. So we just never got around to changing it. Did it work? Yeah. That's amazing. That's what matters. It worked, right? So it was rushed, whatever. It was different than what you do, but it worked. So that's what matters. Cool. So I'm looking at you, Tom, now. Tell us about the TypeScript support in Dagger modules. Because Dagger allows you to use different language. So for the module, for its Go TypeScript and Python, the base case is that you can keep the same language as your code base to end all your pipeline, so your CICD pipeline. So if you have a stack, a text stack that's used TypeScript, that would make more sense to you use the TypeScript SDK to enter the CI. So I guess that would be the reason. Then what is it? So it's the TypeScript integration of the Dagger module. A module is a reusable piece of code that will execute some task. So for instance, you can have a Node module that will expose some commands such as build my Node project, run my test, run my lint, deploy it on npm, or like anything you want to do, basically, on a Node project. And then you can have all the modules, like for example, we could have a blend module, we could have a Go module, we could have a module for basically anything, a Jenkins module. And what is cool is that you can combine this module in order to build your custom size that match your needs. But you don't want to code everything from scratch because it takes a lot of time. You have to maintain the code. So you could use existing module to do some part of the boilerplate job, like initialize your repository, run npm command, and just like had your custom logic for your specific CI module. And you can go iterate over it like that and had additional module when you did. One addition which I would make to that is that modules extend the API of the engine. So rather than just being tasks or code that you consume and you run, this is an extension of the Dagger engine. There's more endpoints for the GraphQL API and more capabilities. So modules are giving capabilities to the engine. We will most likely end up writing a module for this specific app that you can either keep private and reuse across other apps in your org. Or you can put it in the Daggerverse so that others can build on top of it. And while we don't aim to create a React module today, I think it's an interesting idea. Okay, I think we're getting close to the screen sharing part. And for this one, I would like Alex to drive. Well, that means that I would like him to share the screen. And me and Tom will be navigators. So we'll be guiding you to build this module. And if we get stuck, we have a couple of options, but let's see how far we can get. I sort of planned out what I would like to do as a sort of bare bones start of this today. So at the moment, we have another phase of this pipeline that's currently unpredictable. Like it sometimes passes, sometimes fails. So for today, I'd rather just stick to the two parts of the test that I know run. The type check that I know runs because the linter wasn't always applying a commit if it had failed. And there's currently some things that the linter can't fix. So we'll keep, we'll admit that from the pipeline today. But if the pipeline could just do this, build type check, run the two sets of tests and then deploy to an integration environment. I think that's a really good proof of concept of this. While this conversation is 25 minutes long, the hands-on session is over an hour long. My suggestion will be to stay here to the end, then watch the video if you want to see all the details. Link is in the show notes. [MUSIC] It's a promise that all is probably just fine. I don't think that was a problem. Yeah, it's always that thing when you start putting your finger at node internals before you look at your own code. It's like, you're doing something wrong. And even that optimization that we did right where we split out the code, I mean, that was just a refactoring. It shouldn't have mattered for this, right? Even if it has some duplication, it should be fine. Because as I said, all this gets translated to calls to the API, and it doesn't matter whether you have one function or two functions. I guess it's because one, I was calling execFire, the container that we built, but one, I was calling it via execFire, the node module. And then, because that's two different calls, maybe you don't get bundled up together, so it's slightly slower. It's doing build, and it's running, oh, there we go. They're both run. Yep, they're both run at the same time. I didn't see that. And then, come on type check, finish your stuff. One minute, 25. Amazing. And then there's the standard L, both of them. It works. It does work. There's nothing wrong with from a store. Yeah. Okay. I can make them correctly. [LAUGHTER] Awesome. Faster is than what you currently have. It's one minute, but is it good or bad? What do you think? I think it's good. So let me show you, this one took seven minutes. So, well, is that one slow? Yeah, you see, it's taken like one minute per every part of the pipeline. And that one minute is like to build and then run type check. One minutes of build and then run for test. So, we've already improved the speed by 100%. 2x. Like half to, yeah. Exactly. It's half what it was before. Yeah. Just for that part. But then we start incorporating things like running this, then this whole step probably becomes about a minute because it's only the length of the actual Cypress test running. So that will already be like a minute less because we don't have to build anymore. That makes sense. So then deploy should also come down by a minute. So the total sum of the pipeline now should be like three minutes rather than seven minutes. Already half that is pretty impressive. There's one thing which I need to mention here. You will need somehow to be able to reuse the cache between runs locally, right? You have the same dagger engine. Yeah. And that's fine. But remotely, whenever the GitHub actions runner starts, it's plain. It doesn't have anything in it. And that's by design. You want deterministic runs, which means the starting point is always the same. Yeah. Right. So you can trust it will be the same. But in this case, what you want, you want some existing cache. So you have two options. You can either connect somehow to an engine that keeps the state, which means not spend the engine every single time in the context of the GitHub runner, which has Docker by default. So you want to run an engine and somehow connect to it. And there's a couple of ways to do that. One that's like most people using production is to use the runners, like do self hosted runners, because the self hosted runners, they connect to engines, big ones, which are provisioned and they just reuse the cache, not just between runs, but also between the various different pipelines. These people have like tens, even hundreds of pipelines at this point that all use the same cache. And then you want those engines to basically reuse them between different runs. Okay. You don't see you get like a bunch of things there. The other option is to use GitHub actions as is, but then get Dagger Cloud so that the engine when it starts, it connects a Dagger Cloud and it pulls down the cached operations from Dagger Cloud. I would like us now to take a step back from the actual demoing and the coding and like, you know, we were in the weeds for a while and it was great because we found some amazing things. I think we ticked all the three boxes and I would like us to share the key two or three key takeaways. And one is fine as well. Each of us, we go around. What have we learned? So for me, my key take was like, I really enjoyed discovering the whole thing, right? Like how it integrates with everything through your eyes, Alex, to see your experience and to see the questions that you have and the things that we could do better. That was really, really good. The format was great. I really enjoyed that. And it just shows that we are much closer to having this thing finished than I thought we were. So for me, this was the first real test looking TypeScript, checking TypeScript in action, the TypeScript support in Dagger modules. And your feedback was amazing. So I really enjoyed that. First of all, just thanks to both of you being here and holding me through a little bit on the process of getting this all set up, but just like what my experience now versus when I first picked it up 18 months ago, probably for the first beta release with Q, it's just like so wildly different in terms of how intuitive that API is. I feel like now even without the two of you here, I could probably look at the docs on that new site that's not deployed and still start working my way towards something that would function. And the fact there's continuity between all the SDKs and the API is the same as Tom mentioned earlier. The reason we can use root from the JSON file is that in different languages, the directory name where you execute the program from differs and because the API needs to be had continuity between all the different platforms, that's the reason it's designed in that way. You have to be explicit in your actual implementation file rather than in the configuration file. I think just like the quality of the product and the design decisions around it, it's great that you're just thinking so much about how you can build a product that's going to scale and have a consistent experience regardless of where you're working. The fact that when we were blaming mode and saying promise that all doesn't allow parallelism, the fact that when we mentioned that to Tom, he was like, "Oh, actually, I need to take away and research this and make sure that there's like, work out why it doesn't work." It just shows that you're taking pride in delivering the highest quality tool you can for this. I'm very grateful to you guys as a team and looking forward to seeing how this expands in part two. That was a really interesting experience because it's the first time I can see someone live discovering the type of SDK and I get from that really interesting information, especially on first, you were like, "Oh, I'm not used to do class-based function, so I know that now I have to work on the top-level function integration because you are not the only one. Indeed, I can see also that the way you go through the doc, you understand how the SDK works, but it's very interesting because first I was guiding you through the first step. At the end, you were basically doing almost everything alone on your own. I think that gives me a pretty accurate idea on the learning curve of the type of SDK in an hour and a half, two hours you are able to be on your own. The idea now is to have as much luck as possible to reduce the time, improve the experience you did, and make sure that you can, from the first step, from the first minute, understand everything about the type of SDK. So yeah, a really good experience, thank you for your time. That was really amazing and thank you, Geralt, for your help. Of course, well, don't you love when a great idea comes together and each of us had a contribution to this. What I'm very keen to do now is get together again in a few weeks, I don't want to say next week, but in a few weeks, and when we do get together next time, what would you like to tackle next? Like how do we see this continuing? So I think, as I said earlier, having like dependent parts of the pipeline, so type check has to run before we run the unit tests and the component tests, and then finally actually getting it deployed into our integration environment, and then being able to sort of drive it live on screen and prove that it's all working, I think that would be a really nice way to get this done and wrap up, and also maybe a tree shaking of dev dependencies, if that's something that we can do with the production build that's built out of fact, that's my aim. Yeah, that sounds great. I know that for us to be able to complete the deployment, we will need some sort of an arc module so we can interface with AWS with the Lambda part. Also we need to some shape or form, we need to get like those secrets in and then get them decrypted. We've seen in your pipeline, in your GitHub actions workflow, that you do quite a few R commands, it needs quite a few N secrets and things like that, so that, I think that is the part which on its own will take a while to figure out, and I think we'll need to do some work before. It may change between now and next session, because we, I've just finished porting from Arc to Express and we're going to deploy it into our Kubernetes cluster. So if we've done that by the time we come around to our next call, the landscape that deployment might change significantly. That'll be really cool. The one thing which I'm working on separately of this, because again, we have a bunch of EKS and a bunch of Kubernetes workloads too, I'm trying to keep the CI and CD part separate. What that means is that CI finishes with the artifacts being published. It's a little bit like CD, but I'm not calling it CD on purpose because people think deployment. In this case, it's just delivery. We publish the artifacts, everything's done. On the CD side, there's something else running in the clusters, which keeps watching these dependencies that they update. When there is an update, they somehow consume it, and then they deploy it. There's tooling that has been mature, and has matured over the years. For example, I'm going to mention ArgoCD, because that's the one that I'm looking at, which has some really complex deployment strategies and rollout and a couple of more things that I know many do. Same for flux. Many other tools as well, because you need to build a lot of knowledge, so if you were to use Dagger for that, I go to the knowledge of everything goes through the square hole. Any peg will fit through the square hole. I think there is some benefit to use a tool that was purposely built for deployment, especially in the context of Kubernetes, rather than trying to make Dagger do that, because then you have security concerns that you have all sorts of interfaces, permissions, it just gets really, really hard to do that. So one day, who knows? But today, I would say Dagger should stop with publishing those artifacts and something else. Should consume them, maybe check that they will be signed correctly, things like that. So that's where I stand when it comes to deploying into Kubernetes. I think at the point of saying I have this build and I'm now going to go publish an image, then Jenkins picks it up or whatever, and then that's responsible for setting up all the Helm charts applying them to Kubernetes and then finishing the deployment. Yeah, I think that's fine as well, just keeping it as a CI pipeline where another stage takes over. Well, not if you're using Jenkins. If you're using Jenkins, we can do that in Dagger. I'm thinking a more complex deployment system, which, as I mentioned, does like canary deployments, and if this is built into the system, then not Jenkins, because we can definitely automate that part. If you have rollouts, for example, do A/B testing, you're using metrics that are built in from the platform to consume those metrics, and then the deployment system decides whether it's OK to proceed or not. Like if you have all that complexity and I know that Argo supports it, then replacing that, I'm not sure it will be worth it. But if you're doing that in Jenkins, I think we should. I think we should do the CD part as well. Yeah, it is just Jenkins. OK, then I think we can do the CD part as well, for sure. That's the piece which I was missing. Cool. Thank you, Alex. Thank you, Tom. This has been a lot of fun. I'll see you next time. Thanks, both. Have a great day. Have a good day. Thank you for joining us today. If you're curious in the behind the scenes of the show, you can subscribe at makeitwork.carehard.io and get a link to the retrospective of the first three months. Until next time. [MUSIC] (upbeat music)
