I've heard rumors that you're knee-deep in Raspberry Pi's. You could say that you're embedded in Raspberry Pi's, sitting there coding. What's new? What's happening? I'm all nerves, that's what's happening. You're all nerves. It's just the bundle of nerves. No, I've definitely been doing a fair bit of embedded recently. And embedded is such a weird thing, because it barely means anything clear to anyone. Because to an embedded engineer, it may very well mean like microcontrollers. And to someone on the outside, it's just like, yeah, I don't know, electronics maybe. And usually there's a distinction like, oh, I'm doing embedded Linux, because there's a big difference between what you do on a microcontroller and what you do on an embedded Linux device sometimes. And sometimes people treat them much the same, because they write everything in C or whatever. But yeah, I've been doing a decent bit of nerves recently, and that means diving into like a world where people just know part numbers off of the top of their heads. They're working a bit with Frank Hunt-Lith, and he does binary in his head, which is weird. Like I understand how people end up learning that just by looking at enough hex and doing enough hex to binary conversion that they just build an intuition for it or build a road skill for it. So yeah, it's a strange world where there's all device trees and deaf configs and compiling Linux kernels. Like, I thought I over-provisioned my work computer as like, I don't need like an AMD Ryzen 5950X for my web development. But it's nice to have. And like, I do some video stuff, so I guess it's excused. I keep maxing it out, because I keep compiling the Linux kernel. So now you need a thread ripper, or what's the really first one? Yeah, so thread rippers are apparently about the ideal, because if you go up to the Epic server CPUs, you can get more threads, but you get lower clocks. Which can be worth it. Like I think an Epic could probably be a thread ripper for compiling certain things, but it's also that it's a pain in the neck to try to run a server chip in a desktop. So thread rippers are really, really popular for kernel development and similar embedded things. So yeah, there's just a lot of things that are very different from what I'm used to with the web. I've had two different cases where one of my clients have had a bit of a challenge with a real-time clock, and it's boiled down to after trying a ton of things in software. We end up looking at the, not the data sheet, but the actual circuit, and then look at the parts, and look at what like, oh, there are a few options for this part, which part did we actually put down? Oh, this part. Well, that doesn't have the pin we were trying to use. Oh, no. That kind of thing. This is not a thing that happens in web development. The physical device is missing. Okay. Or, oh, yeah, that was revised away in hardware. It's like, no one cares about the hardware. It's all the same. It's just CPUs, right? Yeah. Kinda. Yeah, it's fun, and it's funny, and I'm learning a ton. So, probably going to get to poke a bit with AI at the edge. Oh, so, well, machine learning inference on low-power devices. Yeah, that sounds much better. Edgy AI. Yeah. Yeah, most definitely. AI edging. Are you going to trim the edges of the AI? You'll see. So, I got one of those coral TPUs at one point, and I haven't used it for much, but I've tried it. So, that's Google made what they call tensor processing units TPUs, and then they spun off a company called Coral that could sell those to consumers as well or sell those to system integrators and stuff. So, there's a USB dongle you can buy, that's a small AI accelerator. If you connect that to a Raspberry Pi, it should, for example, be able to do face detection in an image much, much faster and more consistently. So, when I ran it on my MacBook, the MacBook could produce between two, it could produce results between two milliseconds very rarely, but usually up towards eight and with great variety of variants, while this one was just always four milliseconds steady, which has, it's uses when you like, oh, we want to achieve a particular frame rate on the analysis or we want to kind of follow the frame rate of the video or whatever. So, those devices have their uses, voice assistants or just your recognition or just seeing if there is a face in the picture, that kind of thing. And of course, spying on citizens, that's also an important part. Absolutely. Yeah. And Raspberry Pi released the AI kit for Raspberry Pi 5, which is a much stronger TPU. So, the core role is like six years old, so it's kind of dated, and this one has more tops, trillions of ops. The weird thing about tops is they don't tell you at what precision, so you can double your tops by halving your precision. Oh. So, that seems legitimate to me. So, they could do like eight bit tops or ops, and they're really quick, but really bad. Well, like eight bit tops would be pretty reasonable in an embedded form factor, I think. So usually the big GPUs do 16-bit, they might do 32, but I think generally they tend to do 16-bit and count at 16-bit, but I've seen like when you quantize large models to try to fit them into smaller sizes, and that's essentially when you reduce the precision. And I've seen two bit integer, I've seen models that are originally like 32-bit float or 16-bit float be quantized down to two bits. That sounds seriously cursed, does it work at all? It does workish, they get really dumb, really, really dumb. They do weird things and do poor jobs, but they produce words, but it's like, yeah, yeah, they'll do something. Well, I think you can go down towards four and still get decent results. And at like eight, I don't think you necessarily notice all that much that you went down from 16. That's really good because eight is so much smaller. But you get that kind of marketing friendly unit when it comes to tops because they can go, hey, this chip does five tops at eight bit or whatever, and then it's like, or maybe we just say 10 tops at four bit, but we don't mention the four bit. Yeah. Wow. Yeah. Marketing messages. It's a lovely thing. Absolutely. Yeah. But I've been poking around with secure booting, Raspberry Pi CM4s. That was part of my talk for Nervs Conf, actually, which I hope will come out sometime soon. It was kind of security centric. Cool. Did you get it to be secure or was it all just still kind of work in progress? So I have the secure and the boot part, but I don't have a root file system yet. Well, if you don't have a root file system, it should be more secure, right? Yes, but also very useless. Yeah. But we weren't talking about usability, we were talking about security. And you're only optimizing for one variable at the time. Yeah. Yeah. I'm optimizing for, actually, it shouldn't turn on. That would be the best for me. Yeah. Absolutely. This is a secure rock here. Have a secure rock, please, give me my money. Yes. Yeah. And the sound would be more secure if I just gave you it in pieces instead of a silicon. Yeah. Because that way, you can arrange it to do math at your leisure. Oh, many small rocks. You can do it. You have like, it's like an abacus, or you can play all kinds of fun games with them. What do you put into the term embedded? What do you think of us embedded? I'm a bit, from my background, I haven't done embedded for real, but I've done it at school and embedded for me is everything you can't run Linux on. So as soon as you can run Linux on something, it's out because then you've got Linux. It might be a strange computer, but you can like, yeah, maybe it should have a terminal too, somehow, or a serial port or something. Yeah, they usually do. Yeah. So you can have a shell of some sort. Because then it's, I think it's about how hard it is to work with it and how much real time it is. Because if you've got Linux, you haven't got the true real time. No. And real time is, I think the reason why I think real time is so important is because I made a project, the Chalmers, with some other happy students, and we wrote quite a lot of C code to make basically a bad audio cable. The idea was to take audio in real time, digitize it, it's not the word, it had an ADC and a low to digital commercial. Yeah. So we got it in numbers. This horrible, horrible dev board we used was, I think it was 8-bit, and it didn't have a floating point processor. Oh. So you can't do much fun with sounds, and it had a way to low clock speed or clock frequency. So we couldn't do that much fun with it. So we mainly made horrible noises, which, okay, that was fun. And then, from analog to digital signal, horrible noises, then digital to analog and out into a loudspeaker. And from that experience, I have a very clear image of what's embedded for me. It's something where real time is needed or very much appreciated, at least. So microcontrollers, et cetera. Something where you can actually read the data sheets and get a good idea of the whole system. Maybe that's possible for a Raspberry Pi? I don't know. The Raspberry Pi data sheets feel kind of limited, but you can definitely get a very complete data sheet for like one of the boards that we're looking at bringing up. Mostly some other people are helping or doing the bringing up, but I'm mostly poking at it occasionally and trying to build some cool stuff. But it's a TI AM 625 series board. Is it an ARM thing? Yeah, cool. And that one has both ARM A cores, I think they're called, the normal CPU cores that you'd run general workloads on. And then I think it includes two separate M cores, which are essentially microprocessors, or one microprocessor, two cores, I'm not sure, which means it has both. And that's kind of where the lines start getting very blurry. But I've definitely, from listening, starting to listen to embedded podcasts and stuff, there's definitely the sense that embedded is microcontrollers. But most of the embedded devs that mostly do microcontrollers kind of agree that, yes, it's still sort of embedded work, but it's a different kind of embedded work when you get to embedded Linux. Because, well, it's called embedded Linux. Like a single board computer is not the same as a server, because you're doing a lot of the same stuff you do with other embedded devices. It's like this is I squared C, it's UART, it's SPI, it's GPIOs, and all of that stuff. Like you're talking to, you're talking to devices fairly closely, usually, and you might be writing drivers rather, for embedded Linux you might be writing a driver rather than just talking to the device without abstractions, I guess. But it's pretty close to the same thing. And a lot of the people doing embedded on microcontrollers now use something called ZephyrOS, which is an RTOS, so it's a real-time OS, but it's very similar to Linux. It has a bunch of K-configs, and I think it has def-configs, and it has a bunch of similar concepts to Linux and the driver system. But it's still not memory mapped and weird timings, it's very much still hard real-time, but it has a bunch of nicer abstractions. So for example, if you use two separate boards that both have a Wi-Fi, there's a Wi-Fi abstraction, which you can code against, and the underlying driver will be different between the chips, but you don't really have to care about it unless you're doing something very specific, and then you might be using the vendor SDK or something. Is that what they call a HAL? I guess that's a HAL, yeah, hardware abstraction layer. Yeah, it's one of the best acronyms. Somebody needs to make a HAL 9000. Oh, yeah, the new standard for health. Absolutely. It's going to be great. Yeah. It seems like there's sneaky, embedded work in other non-embedded systems, so I was listening to one of the Oxide, and I was going to say Oxide in front simply, but Oxide computer employees. She was on a podcast talking about her work with Arm Trust Zone, because it was looking for conversations about Arm Trust Zone, because I'm trying to understand it, and she had spent some time trying to interface with it, deal with it on some NXP board that they were using. I don't remember exactly what that part was for, but they built their own switch, I think. They built a few pieces of hardware that go in their rack that is particular. Yeah, they constructed their own networks, which I think afterwards they say, "Don't do this. It just don't, but we had to," kind of. Yeah, because they wanted kind of security down to the wire, like they wanted to know the whole secure process. I think the microcontroller stuff was probably part of their boot process, so I think there's a microcontroller in there doing something, or being part of starting things up. Like Arm Trust Zone is a mechanism for securing things inside of an SOC or microcontroller, and they found a bunch of undocumented things in an NXP part, and I think they ended up doing a DEF CON talk about it, which also led to NXP eventually shipping a mitigation after much complaining. It's very good. Everything that ends up in a DEF CON talk or a CCC talk is good. I might have to revise that statement eventually, but until such a tool, I'll have to revise it, I will stand. I will hold this opinion for a little bit stronger, fair. And at different levels in many types of computers, I think there are things that feel very much like embedded development, mostly, I guess, driver development. This is very much similar to embedded, or is maybe embedded. So I think that's kind of also a place where the line blurs. To me, embedded devices have been, like most embedded stuff I've poked at and the embedded and JSON stuff I've poked at have been kind of Linux level, so that's been in the embedded sphere for me for a long time, mostly because I come from Elixir and we have nerves, which is like this IoT oriented framework, and that builds on Linux, but the way it builds Linux is, of course, very different from Debian or whatever. So you have build route, which is a project for custom Linux for whatever you need. I think they might use build route in Linux from scratch now, because I was searching a bunch about build route and ended up in a Linux from scratch guide, and I remember when I was a young one, Linux from scratch seemed like the coolest idea ever, like build your own district from scratch, because it's a long, long as guide that will let you do that. I think I've started only three or four times, but I was like 20 years ago, so when I was a kid, I had time. But I think that uses build route or there's options for using build route, or maybe there was just like an article about the possibility, I don't remember, but it mentioned build route, which I thought was neat. And build route is like a cool project, which lets you just essentially pick what parts you want in your Linux distro, but compared to app, it's not like it just installs them. It builds them into your system, because the idea is not that you add a bunch of things while the system is running, but rather you ship new firmware, you add, you burn the firmware disk or flash it to some memory somewhere, and then you run it. So it's fairly immutable. Cool, so a bit like the idea behind Docker et al. Yeah, so I've definitely considered, so there are nerve system images for vulture cloud VPSs, so you can actually just ship a nerve system to the VPS, and it's like, you get a very, very bare bones OS, which is nice for performance reasons and like repeatability and stuff. But like, I think Alpine and Docker and all that have probably covered that space pretty well. But on the other hand, it's definitely cool to be shipping like, oh, 12 megs of OS image instead of like two gigs, which is also why people don't want you to use full-fat Debian or full-fat Ubuntu for your Docker images. Absolutely. There's a project called Wolfy, I think, or Wolfy Dev or something. It's named after a very small octopus that's incredibly cute. That's the idea is to, it's a bit like Buildred or something. The idea is to have very small images that you can build your stuff on, and it shouldn't come with all the bells and whistles that even, well, Alpine is small, but there's still some stuff from it. They put in one bell and two whistles, it's upsetting. At most, 1.5 whistles, and then you can put your Python code or whatever on it and deploy it. And ruin it? Yeah, absolutely. All for ruining things, ruins are cool. I'm getting decently familiar with Buildred and decently comfortable customizing Buildred. It's not like I'm good at it, but I get there. I'm not good enough to figure out exactly what I need to do to run weird hardware because I have a piece of weird hardware and it keeps fighting me and I'm very upset. But yeah, I like how Nervs gives you this opinionated stack of like, okay, but we have a thing for building your Linux that you can customize if you need to. And then on top of that, we just start the beam. And the beam brings up the fancy stuff that you need. So if you need networking, the beam will orchestrate the Linux parts. So it will say, oh, hey, bring up this interface. Oh, here, have a VPA's applicant. But I know a previous Nervs networking iteration actually tried to build the networking layer with like networking, state machines and stuff. I mean, they had it working, but it was a lot of work to develop and maintain. So now they just delegate to Linux and that seems to work well. That's incredibly ambitious. I'm glad they did it. Yeah. Yes. Yeah, that's also some work I got to do, which was digging into vintage net and figure out that you're familiar with WPA's applicant, right? Way too familiar. Yeah, because it's a pain to get Wi-Fi working on Linux sometimes, especially in the 2000s. Yeah, but also now with Edderome. Okay. Yeah, you can still make it horrible if you really want to. So Edderome, does that have some kind of special off stuff to it? Yeah. 802.1x. Could be there are certificates involved and there are many different conflicts and some might work and some might not work. And it mostly seems to be a function of time. But it's like a username and password plus some certificates. Yeah. Yeah. So it's PFM Shapp V2. Yes. Yeah. That one. It's one of the more popular E apps, but it's also horribly broken from a security perspective. So it's not recommended really. What you should be doing is EFTL less. But then you need to have a device certificate. So every device should ideally have a certificate. Which is something you can do in IoT pretty neatly. So this is, this is some of the stuff I covered in that talk. But also some of the stuff I've been digging into and some of the stuff I'm implementing for vintage net. So vintage net is the nerves networking libraries. So this was the moment I realized, Oh, WPA applicant is not just for Wi-Fi. And this is also the moment that that became a problem in vintage net because it was only being used in the Wi-Fi part. So it was built into the Wi-Fi part. And I wanted to do wired 802 1x over ethernet. Why don't you have enough problems in your life? So client work, but don't the clients have enough problems in their life? Yes. But they need more because security says like that Star Wars meme more, more security. Yes. Yes. That. Yes. So if you've gone into like a serious network environment, like a university, for example, and you plug in an ethernet cable and nothing happens, or maybe you just get very limited access. That's usually 802 1x. It puts you on like a guest network or refuses access unless you authenticate. And that's where you're, for example, using P opm shop, V2. And you can also do EFTLS, which means you have a certificate that matches that kind of means you have a private key that is expected and compatible with whatever, whatever auth system is on the other end. But fundamentally like, oh, someone signed or someone generated a certificate for you. So usually the university or whatever would have CA certificate authority that has handed out a certificate to you that is valid for some period of time. You have that certificate, and if you pass that, if you use that to sign your TLS session and whatnot and shake hands and stuff, they can know that you are one of the approved ones because you have the certificate, but you have to keep that certificate really secure, especially if you're doing like hardware devices, because you don't want someone to just go in like pop DSD card out of out of your Raspberry Pi secure device and go, huh, well, let's, let's remove the certificate from this one and let's put it in an orange Pi and plug that in instead and now it can pretend to be the Raspberry Pi. Exactly. That would be silly. Yeah, that would be the wrong Pi in the network. Yes. Yeah. So then you get into like secure elements and how to do, how to try to secure secrets on a device that might suffer a physical axis, which is involved. Yeah. Because most security experts I've listened to say that once the bad guy has physical axis, your yes to go home. Yeah. Yeah. I mean, that's the conventional wisdom, but it's also not quite how we, how things work because, for example, iPhones are generally considered fairly well secured. I don't know what the reputation of Androids is in terms of hard to extract things from. Yeah. I think it also depends on which manufacturer it is. But in Apple's case, like they have specific hardware, which is decent at protecting keys and all the access to like disks and stuff requires that to be cooperating. So you have to authenticate in one way or another to be able to decrypt things and use them nice and like they call it the secure enclave. But I mean, it's a security chip and some security chips have known vulnerabilities such as the popular a tech series, which you can shoot with lasers until they get give up and then they can give you the secret. That's true for many humans, too. So I'm like, shill, but there are other devices that might, as far as we know, do not give up their secrets if you shoot them with lasers. Okay. Fair enough. But I'm also fine with like, it's reasonable that some companies would say, okay, we're okay with people taking this device, soldering off the park, putting it in a special rig, shooting it with lasers to do fault injection and having like a 30% chance of extracting the key without ruining, without completely burning the security chip. I would be totally cool if I was a hardware company. I would be totally cool with that if the bad guy had to live stream all of this because when I could eat popcorn, I really enjoy this attempt to steal my keys. It would be exciting. Yeah, absolutely. It's like, well, they succeed. They have like a one in three chance. Yeah. But it is like an attack mixture. Yep. So there are things that don't have that sensitivity. There's also a bunch of things like those chips are not good at securing non-certificate secrets, really. They have limitations, have weird and bad limitations. Is it about size or what's the? So for example, they have, so what you want when you want to secure like a password is that you want to do symmetric encryption instead of asymmetric. Yep. So AES is a typical one. Yeah. These chips, recent versions support AES. But, but the everything you passed to and receive from them, that is the decrypted clear text that you would receive back from them. So the down and encrypted secret, like they'll keep the secret real, like the key to the secret, they will keep really safe and secure inside the chip. But they'll pass the secret over an unencrypted I squared C bus. Oh. So you could just get that with some electronic electrical probes and, yeah, like read the data. That sounds so much easier. A very slow I squared C bus as well. Yeah, but still. It's like not difficult. Oh, okay. Yeah. That sounds so much easier than doing the laser thing and desoldering and stuff. Yeah. I guess like a transport protection mechanism, it just doesn't cover the AES feature. For some reason. So good. Yeah. I think I've found a way to use the chip to protect a secret that does have the transporting protection. So it's essentially like this scramble, the transport with a nonce or something, which just makes it impractical to eavesdrop on. Yeah. And I think there was a mechanism like the, the age Mac generation system that does use the scrambling and could actually be just be treated like a, like a secret generation thing because it means it, it's holding a secret key and it generates a hash that is fairly high entropy as far as I understand it. And so essentially you feed it in like, oh, I want my SQLite encryption key. So you send it the word SQLite and then you get a hash that is based on your secret and shouldn't be reversible. So I think you can essentially use it that mechanism as passwords. I haven't tried it yet, but someone that works at microchip that makes these suggested that that was doable in a forum post. And then someone asked, how do I do that? And they said, uh, consulting services are found over here. That's application implementation and stuff. We don't help you with that in here. Yes. So the embedded world is also much more proprietary than I'm used to. Yeah. Not everything is open. If you want full data sheet for the attack 608 series ships, uh, it's also like, oh yeah, you can reach out and sign the NDA and order like 4,000 parts. And then you get full data sheet. It's very much like a quest. Well, well, actually you can argue with them until you get to sign the NDA without ordering a think, but yeah, it's a little bit special. Yeah. So what I'm hoping to do instead of, uh, the attack chips, uh, but like the attack chips are cheap and useful and practical. So I'm probably going to encounter them multiple times and it's good to know that there are mechanisms you can use in abuse to achieve those goals probably, but my kind of ideal what I want to achieve for the nerves community like, Oh, this is a good feature to have. This is a good tool to establish is to use arm trust zone instead, because if you have arm trust zone and good security peripherals inside of it, as they say, that's what they say. But if you have some secure memory hooked up to arm trust zone and you have a crypto accelerator in there and all that stuff. So the trust zone is just a different mode of the processor where the, there's a bit set to one for secure, and the processor treats everything differently, essentially. This, yeah, this sounds like memory protection, but embedded, yeah, it's something similar to that. It's just like, uh, instead of privileged, it's secure. Yeah. And the, the idea is there are pieces for hardware that the system will not let you access outside of secure mode. And if those are baked into the SOC, it gets really inconvenient to try to extract anything from it. And you can also build them in, in sort of secure security oriented ways that make futzing with them hard. So then you can do all of this, but not have, for example, an exposed I squared C bus that someone can stick probes into. And you can, you can also have it do more stuff because you're doing it on the main processor. So you have like at least one full core to, to process with, but probably, you can probably do multiple as well. So you could squirrel away as long as it has some kind of persistent storage, which they usually do. You can squirrel away secrets inside of the SOC instead. There's usually even stuff you can burn, like for uses and stuff, which is not all, not beautiful, doesn't feel great, but if you want to make something really security sensitive, then burning some fuses is a pretty good idea for lock, really locking down a device. Yeah, it sounds like security on a device that bad guys potentially could have physical access to is a bit like walking in cold winter weather. In what way you need good socks. You need a route of trust, if that's what you mean, yes, yeah, that too. This is where I make a joke about the Yekker route of trust. It's always there. (crowd murmuring) [BLANK_AUDIO]
