Archive.fm

Make It Work

Access your Kubernetes pods anywhere

How does Michal Kuratczyk, Staff Software Engineer at RabbitMQ, access Kubernetes workloads securely, from anywhere? Regardless whether it's a Google Kubernetes Engine (GKE) cluster or Kubernetes in Docker (KiND), Tailscale is a simple solution for this particular use case. This also makes it easy to share private services with all devices on a tailnet, including with friends that want to access them on a smartphone.

Watch the demo 🎬 Access your Kubernetes pods anywhere


If you want to watch the full, 32 minutes-long video, go to 🎁 https://makeitwork.gerhard.io

LINKS

EPISODE CHAPTERS

  • (00:00) - INTRO
  • (05:12) - DEMO STARTS
  • (06:11) - RabbitMQ in Kubernetes
  • (07:32) - Tailscale in Kubernetes
  • (11:59) - Magic DNS
  • (13:31) - Let me connect to it
  • (15:33) - Is this the last RabbitMQ 3 minor?
  • (17:12) - An alternative way to expose a service
  • (19:11) - Works on any tailnet device
  • (22:04) - How do we continue?
  • (23:26) - Have you tried upgrading the operator?
  • (24:23) - Can we try it?
  • (25:43) - DEMO ENDS
  • (25:54) - Exit nodes & subnet routers
  • (28:50) - OUTRO

Duration:
33m
Broadcast on:
12 Aug 2024
Audio Format:
mp3

How does Michal Kuratczyk, Staff Software Engineer at RabbitMQ, access Kubernetes workloads securely, from anywhere? Regardless whether it's a Google Kubernetes Engine (GKE) cluster or Kubernetes in Docker (KiND), Tailscale is a simple solution for this particular use case. This also makes it easy to share private services with all devices on a tailnet, including with friends that want to access them on a smartphone.

Watch the demo 🎬 Access your Kubernetes pods anywhere


If you want to watch the full, 32 minutes-long video, go to 🎁 https://makeitwork.gerhard.io

LINKS

EPISODE CHAPTERS

  • (00:00) - INTRO
  • (05:12) - DEMO STARTS
  • (06:11) - RabbitMQ in Kubernetes
  • (07:32) - Tailscale in Kubernetes
  • (11:59) - Magic DNS
  • (13:31) - Let me connect to it
  • (15:33) - Is this the last RabbitMQ 3 minor?
  • (17:12) - An alternative way to expose a service
  • (19:11) - Works on any tailnet device
  • (22:04) - How do we continue?
  • (23:26) - Have you tried upgrading the operator?
  • (24:23) - Can we try it?
  • (25:43) - DEMO ENDS
  • (25:54) - Exit nodes & subnet routers
  • (28:50) - OUTRO
Michal, welcome. - Hey, long time. - 2020, how to monitor RabbitMQ. Very happy to be doing this again. And this time I'll be learning something new. What will I be learning? - We'll see whether you learn anything new, but yeah, we've been talking about taste scale, which is a VPN, but unlike all the other VPNs that I know, this one actually works and it's easy. I had to set up IPsec, I guess 25 years ago, almost for the first time and then a few times later on, then I remember when an open VPN appeared and see, it was just so much easier, so much better. So that was a big improvement. Then I didn't have to set up any VPNs, I had to use some corporate VPNs and that was almost always some kind of a problem at least every now and then. Sometimes they just work, other day they don't, you never know. I always try to keep them off just in case. Only turn them on when I need to access whatever I need to and you know. Yeah, now taste scale is in terms of startups and personal usage among the geeks, I think they are becoming the fact of VPN. So yeah, I've been using taste scale for, I guess two, three years now, mostly to access one other machine. So I would generally work from a MacBook and then I had a bare metal Linux box under my desk where I run benchmarks and do some other things. And yeah, usually it's on the same network at home, but if for every reason I'm not at home, I want to still be able to access that machine. Draw days where basically, well, I use my MacBook to drive all the work, but they actually work for the day, basically all happens on this other machine. So that's how I started. I wanted to have an option to access that machine remotely. And setting up taste scale was just so easy. Like honestly, when we discussed that we should record something about it, I was like, I should learn what taste scale really is and how it works because I have no idea. (laughs) And you set it up, (laughs) it works, you install it in two places, you log in. That's it. (laughs) Everyone then, I wanted to have the same thing with a Kubernetes cluster. I also have a, well, sometimes I access other clusters, but generally speaking, I have a GKE cluster where I run other tests or other kinds of tests. Yes, sometimes I want to access something that is running into that cluster. And specifically, I had a need to access the locking, which is a lock collection tool. And every now and then I want to clear that lock instance. Until then, I was using the Grapphana UI to access locky. But yeah, there's the lock CLI, I guess. I have an alias for it. (laughs) I don't actually remember what the actual tool. But it's actually called locky because yeah, it also specifies some format. Yeah, so basically I wanted to access locky that is running inside the GKE cluster. But I didn't want to make that locky public. And I didn't feel like setting up HTTPS and authentication and all of that just to every now and then run the CLI command. So I was searching for task and Kubernetes support a few times in the past. And people were asking for it, but there was no solution. And at some point, once again, I had the need and I'm like, okay, let's be searching again. And this time, yeah, there is something. I'm not sure exactly when it was added. There's now Kubernetes support in beta. There's a Kubernetes operator for Taysca. And basically, you can just deploy Taysca to your Kubernetes cluster. And you can very easily expose basically any port or any service that you have there in a few ways. I think I will show two ways today. Based on the use case and the details of how we deploy stuff, one can be easier or the other. But both just work, both are easy. There's not much of a difference really. Yeah, and this way, it feels like a direct connection from my laptop, the thing running in a GKE cluster that is completely not accessible from the outside. - Yeah. So by the way, I have never set up the tail scale operator. So this will be new to me. Even though I do use tail scale in some of my Kubernetes clusters, I am not using the operator. - What's the other option? Like I didn't really investigate anything else than the operator. - I'm using Talos OS. And Talos OS has a system extension called tail scale. So this is baked into the operating system image. When the OS boots, it boots with tail scale already running. I think we can go for the demo. We can see what it looks like, how you use it. And then I'll be asking questions as we go. - Taysca, the thing we are talking about. In the docs, you can find a Kubernetes operator. Let's set it up. For the demo, I would just use a kind cluster. So normally on a MacBook, when you use Docker, you can't really access directly the things running in Docker because Docker for desktop on a Mac is basically a Linux VM. And you don't have a direct network connectivity between your host and your VM and the containers running in that VM. Taysca is a potential solution. One of many, the port forwarding. You can do Metal LP, you can do some other things. But it is a way to easily access things running inside your Kubernetes cluster or just inside Docker on the VM inside your MacBook and potentially access it from many other places. And the concepts are the same. It's really like I use this exactly the same manifest to set up this connectivity with kind that I use for my GKE cluster. Since I work on Ruby and Q, that's my use case. For those who don't know, we used to work together on Ruby. - Yeah. - So I also use the Ruby and Q operator to easily deploy Ruby and Q. And here I have a simple manifest to deploy a Ruby cluster. But before I can do that, I need to deploy the operator, which is basically QPC to apply. Check that it is now running. And now if I apply this manifest, the moment I should have a Rabbit and Q node running here. So far, nothing to do with Taysk, right? It's just my use case for later at Taysk. So in a moment, we have a Rabbit cluster. I have one more step that is related to Rabbit. It's just to set up a user so that it's easier to log in later. So if the Rabbit is running-- - Yeah, running. - I would just create an account with simple user name and password for those who use Rabbit and they expect guests to always be there. When you deploy, we have an operator get a generated user name and password by default. So I just added the guest user that you expect to that. All right, so I have a Rabbit running in my kind of cluster. Now, if you read through the Kubernetes operator Taysk, Kubernetes operator dot, there are a few installation options like a Helm chart. Personally, I just follow the static manifest file because honestly, it's just the easiest. You basically grab a file, everything for here and you set up two things, a client ID and a client secret. I already have that file, you need to trust me that what I have in my Taysk.yaml is exactly that file with these two values. I just don't want to share that whole screen. But that's all I did, honestly. So now if I do-- - By the way, by the way, I want to mention something, in Tayskale, you could set up a client secret that is single use. - Right, actually, yeah, that's true. - Which means that I know that for convenience, you're just like reusing it, the same one that maybe expires sometimes in the future and you can use it more than once. I think it's called multi-use secrets, but you can also do one-offs. And whenever I screen share, I always use single use secrets. - Cool, right. Before I apply that to my Fess, actually, let's go to my Tayskale console. As you can see, I currently have two S or four devices on my VPN, Tayskale VPN network. My MacBook that I'm working on that Tayskale operator running in GKE and two things that I expose from the GKE cluster like that. But we can just ignore them for now. I didn't want to delete them really because I would need to really delete them because otherwise they reappear because they're running there. - Yeah. - So they are here, but we can basically, only this MacBook will be actually, actually. So now if I deploy the paste in my Fess, we check that Tayskale name space. Okay, if it's running, I guess if we stay on the page, it will appear in a moment, but I can just refresh and it's here. You can see the little skit, which is a hint that this is running in Docker. So that's it. Static manifests with two secrets, basically. And we have the operator running. Now I want to access my rabbit cluster. Now that I have Tayskale operator, it should be easy. So as I mentioned before, there are two ways. Probably the simplest way is to add an annotation to a service. And then the Tayskale operator will basically discover that this annotation is there and will expose that service just because it has the annotation. And I guess let's try that. So if I edit the, so my rabbit is called RMQ and because of that, I have two services. One is just called RMQ and that's the one that I care about. So let's edit that manifest annotations. The important it is put through in quotes. - Yeah, that will close. This is Yamo, remember that. Very important. All right, cool, Tayskale.com/expose, set to true as a string. - And that's it. - Okay. Now, nice. I'll refresh this page. Hopefully, with five machines. - Very soon. - Ah, there it is. - Default, default, because it's in the default namespace in my Kubernetes cluster. - Now that's really interesting. So every single service gets its own IP address. - But if so, yeah. - That's really interesting because by the way, the system extension in Talos doesn't work like that. You get a single IP address for the host where Talos is running. And then if you expose a service, you need to expose a port. You need to set the port. And then you go to that, to the host's private IP address, Tayskale IP address. And then you use the services port. But this is really cool. Every service gets its own IP address. Nice. Okay. - Yeah, and yeah, here are the details. It has an IP address. Basically, all the Taysk IPs are 100 something, something. All kinds of details that I've never really looked at. The important bit is this host name. So now, Tayskale, apart from seeing the VPN for you, it has a feature called Magic DNS. And basically, it creates host names for the machines on your Tayskale network. And you can just use these host names on all those devices as long as you have the Magic DNS option in the name. - You have machine details, addresses on the right-hand side. Top, there you go, have the short domain. That's the one that we'll be using and the full domain. But again, both of those will only work in your tail net. And that's the full one which contains the tail net domain. So it's the short domain that we'll be using here because you have Magic DNS. And this host that you're on, it's on the same tail net. - Now that I have all those details here, I can grab this host name and just put it here. And there it is. This is the management UI of RabbitMQ inside my Kubernetes cluster, inside my Docker, inside my VM, inside my MacBook. And that's crazy. - Again, and there it is. This port number has nothing to do with Tayskale. That's just the management UI default port for Rabbit. - Yeah, so that's basically it. Like I have a few additional options to show. Honestly, that's basically that. That's it, like that's my user. - Okay. (laughs) Let me try and connect to it. Do you wanna share it with me? See, that works? - Yeah, let's try that. - Let's try that. - Let's try that. - Here we go, here. - Okay. - I guess we can do this. - Generate, copy, invite, like. - And okay, guess, guess. I'm looking at now, then you have to trust me. And let's see. Let's see if I remember how you do this. So let's go to exchanges. Let's add a new exchange. Let's do, let's do this. Make our heart and direct, add exchange, cool. So you go to exchanges. - There it is. - You see it, great. So I was able to create the exchange. There you go. - That's the one. - So definitely works. - That's unbelievable. That's how we know it works. (laughs) - This is a feature I've never used before. But yeah, if you are working as a team together, like let's say you have, you're developing a microservice and you want to test it against somebody else's microservice or something like that. And yeah, obviously, ultimately you should probably deploy that somewhere and hopefully you have some CI to test that it works together. But if you want to just quickly expose your thing to somebody else, you can do that like that. There are also commands such as taste case serve and taste scale, something else I don't remember right now. But basically you can expose like a file on your devices or a folder and just give a link to someone and that person will be able to download the file. So let's say like some large log files were produced or you have some kind of like, like before I be like, sometimes when I talk to someone, I want to share a file with data which is corrupted in some way or something like that. And this file can be like half a gig. And there's always this question like, how do you share a file like that? And yeah, with this, you can just expose a folder, give a link and they download that little machine. - Now, before we look into more tail scale options, I'm very curious about something. Is this the last rabbit in queue minor? - That's what we said on the blog, yes. - All right, okay great. - There is a, which blog, show me or show us, shall I say, correction, show us. So if we go, first of all, we have a new website. I don't know if you'd say that. - Ooh, the dark hole. - Ooh, okay. I like it. - That's awesome. - I love the tagline, one broker to feed them all. - To be honest, I am not too bad. - Whoever came up with that. - I like it. I like it very much. - Okay, right. - And if we go to 3.14, it's here. - And that's what we say. - I think I know that guy. - Yeah, that's, that's the guy. (laughing) - Okay, 3.30 is here. - I clearly use this photo a lot. (laughing) - I like it. How long ago was that, by the way? Because I don't see any difference. - How many? - 2012. - 2012, so that's 12 years ago. All right. What's your secret to not aging? - I don't think I should have. I'm certainly bolder now. But yeah, it's a picture that was taken by surprise on a platform at the train station in Doncaster. I was just waiting for a train with a colleague and he took a photo of me and I just really liked it. (laughing) - Nice. - Okay. - All right, so one more thing I wanted to, well, actually there are probably two things I want to show. So one is an alternative way of exposing a service. So right now I added an annotation to a service that existed, which is probably the most convenient option. If you can't do that. But I can imagine scenarios where, for whatever reason, you cannot add an annotation that's the service that already exists. Like it's not you who created it. You don't really own it. It can be overwritten by some CI, anything like that. Just modifying manually a service is not necessarily an option. So another way, if we do get YAML service, we get the details of the current service. We delete all the stuff that shouldn't be here when we create stuff like that. It has this annotation. So we definitely want to delete that. But basically we can delete almost everything here. I guess this, this, this. That's pretty much what we need. Now the trick is, we set the type to load balancer and the load balancer class to a test scale. And I will change the name of this service. And that's pretty much it. I apply this to create a service like that. And, okay, there it is. And if I access that. That also works. Nice, okay, that's really cool. Okay, so you can either annotate the existing service or you can create a new service of type load balancer and the class, the load balancer class scale. That's very cool. Now that we did all the magic with sharing so that you could access something on my laptop, I guess that's not so impressive anymore. Okay. So right now I'm, I have the test scale installed on my phone already. But I deleted it from the machines on my test scale network. So when I turn it on, it might ask me to log in, okay. Connect. Cool. So the iPhone 12, me as a new machine here. Yeah, now if I go here and I do HTTP and default or Q. There it is. Wow. No way. That's really cool. UI is not pretty optimized for mobile. He's there. Look at that. We can see our exchange. Very, very nice. Not very well, but it is here. Cool. Yeah, it is there. Definitely. Nice. So this is where we're saying that any device would work. And by the way, I just noticed something. I noticed the name of your tail net. That's a really cool one. How did you get that really cool tail net? I don't know what you are talking about to the others. Okay. So if you go back to where the machines were in your tail scale. And if you were looking at the details. Do you see the full domain? Tail. 3D, duh, exactly. Like duh, that's a really cool thing. I know, but it's a really cool, rather than they cool. And by the way, same thing. I'm able to access it from any. That's really cool. When you share it with anyone, that person can access it from any machine that is on that's using tail scale. Yes. Within reason, the way you said it sounds like major security risk. But remember, this is like only on my VPN network, everything is entered and encrypted. I explicitly shared it with you. And then on top of that, there are ACLs that you could use to control. I've never used them. But there is a bunch of options where you can say, like who can access that? And like there's from which devices, from which IPs, I don't know exactly what's possible. But yeah, that's really cool. Okay. So how would you like to follow up with this? Like what do we do next time? You mean not about tail scale or? No, no, no, I'll still tell you. Like how do we add on top of this? So one thing that I consider playing with is a router inside your Kubernetes cluster that gives you access to everything in that cluster. At least that's my understanding. I've never done that yet. So potentially, maybe it will be convenient. So if I, for example, so I said that I run all kinds of tests in my GKE cluster, sometimes there's like a dozen rabbit clusters running in parallel. And every now and then I would like to access the management UI, for example, one of them, or maybe two of them. Anyway, right now I would need to expose each of them explicitly one by one. With that, with adoption, I could probably grab an IP address or a given call service, I'm sure, and just access it as if that IP from a GKE cluster was on my network. So that's something I would try at some point, whether that's more convenient or not. But following the talks was actually, for the first time, I want to do something with Dayscape. And I wasn't sure how to do that. - I'm wondering if you tried upgrading the operator, like any issues when you try upgrading the operator? - I have not like explicitly done that really. The way it is manifest that I grabbed is configured. It uses the unstable tag. So every time I deploy it, or potentially, then every time I restart it, it will fetch the latest version. And either check whether it actually pull the latest version or not, but I've never had any issues. - Yeah. - I'm wondering if you delete the operator, will you still be able to access the services? - Well, I'm pretty sure that it will be possible. - So right now, if we look at the task in namespace, there are two services, so for, well, two pods, for each of the services I exposed, there is a dedicated pod that does some magic. And I guess this loop is up here when I did the operator. - Can we try it? - Sure. - I'm really curious to see what happens, because if every, so if that means that in every pod, there must be a tail net demon, sorry, a tail scale demon. I think it's tail D or whatever it's called, tail scale D. And that's what gives one IP per service. That's why every service basically appears on your tail net as a new machine. But if you delete the operator, maybe, just maybe. - Yes, if I just delete the operator using the same manifest file, it will delete the whole namespace. So that would destroy everything I can try to delete the operator deployment. - Oh, I see. Yeah, that's right. - But then I would expect everything to keep working. - Ah, so, yeah. - Yeah, that's right. So that's really cool, because that means that you can upgrade the operator and not affect any of the running machines, right? Running pods, so that will still continue working. - I guess, so these pods are still there. And if I refresh, that's what this works. - Yeah, nice. Okay, so the operator's not tied. That's very cool. I'm assuming it would be similar to the rabbit empty operator, right? Deleting the operator doesn't affect the class. - As long as you don't do this to the user, it'll be wrong. - Exactly, yeah, that's right, yeah. So I'm assuming the same approach was used here because we still have the machines running. That's really cool, that's really cool. Okay, exit nodes, so let's talk about that. So you mentioned the subnet router, and that's something that I haven't looked into either, but I am using exit nodes. So a bunch of my machines that using tail scale, I registered them as exit nodes. And what that means is that when I connect from my phone, exactly as you mentioned, and I'm traveling, I can say use my home machine as an exit node. And that means that exactly like the VPN use case, that's my gateway, I'm using my home's public IP address. It appears that I'm home all the time. What I do, I say, hey, phone, please connect, use my home exit node, I know everything's encrypted, and then that's what I use. And it works well, but I haven't tried the subnet router. - Yeah, the way you describe the things that I'm not sure. What's the difference there? - So yeah, so the subnet route as far as I know is that you're able to access hosts on that subnet through the subnet router. So I suppose this would be like you have a like LAN, and you're like within the LAN, and then you can route requests to any hosts in that LAN using a tail scale machine. For example, configuring your home router. Now again, don't configure it in such a way so that the host that you're configuring from loses connectivity because that would suck. But that would be like a use case for the subnet route, again, in my understanding. But I would like to revise this because this may have changed. I've never used it myself. Would you use this, for example, to connect between clusters? - I believe you can. I have never tried, but I think it's in the docs, or maybe I listened to a podcast, an interview with someone from TSK. And I think it was mentioned either in the docs or in the podcast that you can, or actually if you even go here, like this whole banner, right? Like it mentions, I believe, access to databases, right? So like let's say you have your app running inside Kubernetes and your database is not Kubernetes. And you want to access it over a secure network because it's like, you use some kind of a database as a service thing, which hopefully is not stable. But maybe you want something on top of that. And yeah, you can set up a basically a VPN tunnel between your application and your database. - That sounds really cool. I can imagine this being useful. If, for example, you want to do some sort of like a migration. So rather than clustering across Kubernetes clusters, maybe you want to set up a replica on another cluster. And you do that by just simply pulling down, like doing like a live replication. And you do that like one off, you do it and then you stop the wherever you're migrating from. - Well, this is really cool. I love like how many views we can bounce really, really quickly. And-- - I want some blue-green upgrades. Like you use the tail that DNS the AMEN. - There you go. - It just happens that the thing behind it is different. - Yeah, that's it. That's it. That would be really cool, exactly. The other thing, which again, I use quite a bit and maybe you do too, is I don't run Docker on my Macs at all. Like I don't run because of some things that you mentioned. So my Docker always runs on my Linux bare metal host. And wherever I am, I said the Docker host to be the tail net, the machine IP address. And I'm using the 2375 port. So wherever I am, I'm connecting to Docker remotely. And when I'm in my LAN, it's really fast because, you know, it's a LAN. But when I'm remote, none of that changes. It just uses tail scale. And it's always the same Docker instance. And I don't have to worry about, oh, is it running? Is it using battery? Do I need to upgrade upgrades? I hate upgrades. Always upgrading, always being slow, pushed me over the edge and I just stopped using it. And I'm not going to upgrade to this thing every single time I want to use it on a Mac. So no more Docker desktop, just using the open source one on Linux, you know, no extra VMing, none of that. And I connect to it wherever I am. So that's good. - In my case, I don't keep my bare metal machine always running. So it's actually not very convenient if I have to turn it on every time. - Why is that? Is it noisy? - It is noisy. - Using it too much. - Right now I don't really have a good place to put it there. - It is noisy, okay. Well, let's talk about the fanless. - I know. - I'm actually next time. - I browse through the, I don't remember the name of the vendor that you recommended at some point, but I certainly spend some time on their website later. Yeah, we'll see maybe one day. - Yeah, so as soon as that thing is like properly fanless and I don't mean like Macs because they do have fans, they can eventually spin up. Like there is no fans, not even the PSU. It doesn't have a fan. It's just like blissful silence, you know, all the time constant. So I know exactly that switch, what it felt like and it was a worthwhile upgrade. And then once you have that, you know, it frees up a bunch of resources on your Macs and you know, now electricity is a worry and there is some costs associated with that, but I'm able to keep mine within like 60 Watts per hour. It's like having a light bulb on all the time. Anyways, that could be maybe not to follow up potentially. Cool. All right, anything else that you want to wrap up on? - No, it's just, if you have a need like that, just give it a go, it's really so easy. Like there's, I totally understand the reservation. I'm like, oh, I need to set up a VPN like that. No, just like you install it, you log in and you can pick like Google account, a bunch of other options, GitHub account, whatever you want, whatever is convenient. And once you install it in two places and log in with the same account, I think that's it. It's been a while since I did the initial setup, but I think that's it. We just log in to the same account from two different devices. That's all. You can already access it. - So just to clarify, this is not sponsored by Kelsky. - Not in the way. - And we do not work for Tailscale. We are just Tailscale users and we love it. - Yeah, that's the only thing that's happening here. Cool. Michael, thank you very much for today. - Thanks a lot. - I really enjoyed it. See you next time. (upbeat music) - If you enjoy this podcast, you can watch the full video at makeitwork.garhard.io. You can also watch the screen sharing part on YouTube for free. Find the link in the episode description. If you're listening to this on Apple podcast or Spotify, I will appreciate your rating. And if you have a few minutes to spare, I look forward to your feedback. Thank you for tuning in. I'm really excited for the next episode. (upbeat music) (upbeat music) (upbeat music) (gentle music)