The Bold Blueprint Podcast
The Bold Blueprint Avideh Zakhor Surround Yourself with Positivity
Hey Amazon Prime members, why pay more for groceries when you can save big on thousands of items at Amazon Fresh? Shop Prime exclusive deals and save up to 50% on weekly grocery favorites. Plus save 10% on Amazon brands like our new brand Amazon Saver, 365 by Whole Foods Market, Aplenty and more. Come back for new deals rotating every week. Don't miss out on savings. Shop Prime exclusive deals at Amazon Fresh. Select varieties. Decisions, decisions. Wait a minute, are you still looking for cars on Carvana? Yeah, decisions, decisions. When I use Carvana, I found the exact car I was looking for in minutes. Bought it on the spot. Electric or full diesel. Decisions. Come on, you've been at it for weeks. Just buy it already. You're right, crossover it is. Decisions, decide it. Whether you know exactly what you want or like to take your time, buy your car the convenient way with Carvana. [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO] I went ahead and showed some pictures, let me just revive chapter three as well. [BLANK_AUDIO] And we went ahead and showed you some pictures on how we use Laplacian filtered images to enhance the imagery. So in particular, what we talked about was that if we have, come back to the paper here. So if we have a simple isotropic second order derivative operators that we can process a signal by g is a process signal by the original minus the del squared of f of x comma y. Then you can enhance the edges and going back to the computer, this was an example of an image that we started doing that. This is an image of the north pole of the moon, this is applying the Laplacian filter to it by itself. And you can see there's some white edges, white dots here and a white edge here. And then this is the scale version of that and then this is f of x comma y which is the original minus del squared f. So as you might or might not be able to see on the screen there, this picture has a lot more sharper edge than the other one. So what I'm going to talk about today is kind of two more tiny bits about image enhancement and then move on to edge detection. The two pieces of enhancement that I want to talk about is unsharp masking and high boost filtering plus using a very quick overview of essentially chapter four of Gonzales and Woods. I'm not going to go over too much into the details of chapter four simply because we already covered all of those Fourier analysis, convolution theorem, correlation to all those things earlier in the semester. It's almost redundant to do that again. I might just flip through some of the pictures in the book to show you how we can use a frequency domain filtering to enhance pictures, but that would be very brief. So today we'll continue talking about enhancement. In particular, the technique I'm going to talk about is what's called unsharp masking and high boost filtering. And what this technique is, is nothing but increasing the high frequency component of a signal. So I'll start with unsharp masking. And basically what that does, it says that the unsharp mask version of a signal, this is the process, can be written as the original signal, minus the blurred or the average version of a signal. So it's f of x comma y, which is the original, and then minus f bar of x comma y, which is really the low pass filter or the blurred version of the signal. So what happens if you start with the original and you move the blur, you're essentially removing the low frequency part, so you're kind of enhancing the high frequency portion of that signal. And then a slightly more general version of unsharp masking is high boost filtering. And how that works is, you start with the high boost version of f of x comma y, which we denote by f of hp, is nothing but a scaled version of the signal, minus the blurred version. Okay, so a is just a scaling factor. And generally speaking, we pick a to be larger than one. So, of course, you can relate them these two things together. It's pretty straightforward. Let me just do that. Can you roll up, please? So the other half is shown. Thank you. So we can combine these two equations as well. So you can write down the high boost. You can relate high boost to f s, the short mass version of x comma y is just nothing but a minus one f of x comma y plus f sub x of x comma y. And so, depending upon how you obtain the blurred version or the low pass version of the signal, you could think of this as a high boost filtering as a linear operator. And you could have the impulse responses that look something like a plus four minus one minus one minus one zero zero zero zero. You can think of that as the impulse response of your high boost filtering operation. Or alternatively, you can you can you can, depending upon how you again, how you implement for you could have another version of it. So this this is essentially n one and two in the language of what we've covered previously. Okay, alternative response for the high boost filter could be a three by three filter, which would be a plus eight minus one minus one minus one minus one minus one minus one. So, switching back to the, so these are possible impulse responses for doing h b high boost filtering. Okay, it's as it turns out that all of these things are linear operators nothing, nothing fancy. So switching back to the computer, what you see up there is an original kind of a SEM picture of a, I think it's a tungsten. Let me just make sure that that I say the right thing. Yeah, it's a SEM image of a tungsten filament. That's correct. So, I showed that already last time. This is what I really want to show. So think of this image 3.41 c, which is this guy. This has been turned a little bit darker, and that's the starting port for this experiment. And this is what you obtain if you just apply the Laplacian operator to this original signal. This is what you obtain if you apply the, this filter here with a equals one to this signal. And this is what you obtain if you apply that same filter but with a equals 1.7. So think of this, sorry, think of this thing, this filter a plus eight with a bunch of minus ones. So the first case we let a equals one, you get this, the second case you get a equals 1.7, which essentially has the effect of what? If you think about the filter that we're dealing with, if a is one, then you get nine here together with one, two, three, four, five, six, seven, eight. So the DC value of this is just one. When a is 1.7, the DC value of the frequency response of this filter is slightly higher than one. And what would you expect it would do intuitively? It would make the fifth picture a little bit brighter, right? If the DC value has been increased. And so that's exactly what's happening is, is you're applying some sort of a high pass filter, but it's also become a little bit brighter by applying this, this operator. Okay. Um, that's all I want to talk about in terms of high boost filtering or, or on sharp masking. The idea is, you know, pretty straightforward. These are all linear operators that, that one applies to, to systems. The last thing I want to talk about here is computing the first derivatives for enhancement. So using gradients for enhancement, we've already talked about the del squared F operation in the previous lecture, just to remind you, the second order derivative was this d squared F dx squared plus d squared F dy squared. And we talked about that quite a bit last lecture. Now, what I'm going to talk about is the first derivative instead of a second derivative. So, I talk about the, the first derivative of a two dimensional signal F of x, y as a V of F of x, y dx as a vector made of two elements, D of x, y dy. Okay. So this is the gradient along x. This is the gradient along y. And, and we can talk about the magnitude of the left, which is g x squared plus g y squared, and then square root of. And many times, if you compute that quantity, and if that quantity is larger than a threshold, then you can kind of find the discontinuities in an image. And we'll talk about that in just a second. So this can be used for edge detection. Hey Amazon Prime members, why pay more for groceries when you can save big on thousands of items at Amazon Fresh? Amazon Brand, like our new brand Amazon Saver, 365 by Whole Foods Market, Aplenty and more. Come back for new deals rotating every week. Don't miss out on savings. Shop Prime exclusive deals at Amazon Fresh. Select varieties. [Music] How many times this is too compute intensive to in practice, and so people come up with operators like to simplify computationally. They come up with approximation, which is magnitude of g x plus magnitude of g y. And they could come up with any approximation to it, for example, to compute g x. You can compute using a three point average or a two point average. You can compute g y more or less the same way. So at this point, I'll show you an example of applying these kind of operators, a Sobel operator, that approximate g x and g y. I'll talk about Sobel in just a second, but let me just motivate the discussion as shown here. This is an optical image of a contact lens, and there's a defect at about this point, and again, I don't know if it shows up on your screen or not. And this is applying the Sobel edge operator, which is a particular way of computing this magnitude of del f that we just talked about, and once you do that, you get something like this. So these can be used for edge detection, these kind of operators. So to get into this discussion a little bit deeper, I'm going to now switch gears and formally talk about different methods other than this that people use edge detection. So switching back to paper, there's three basic methodologies for edge detection. One of them is the gradient method, which is almost what we talked about here, and I'll talk about it in a little bit more detail. The second one is Laplacian, which we also talked about, the del squares, f stuff, and the last one is, it's called L of G, or L of G, which is Laplacian of Gaussians. And this is something that Hildreth and Moore came up with in the 80s, and I'll talk about that as well. So to motivate this discussion, I'm first going to talk about the, we've talked about this just a little bit last time, but think of a one dimensional case. Suppose I have f of x, and there's a transitional transition from light to dark or dark to light at approximately x0. So why are gradient methods useful in detecting edges? Well, intuitively speaking, because if I compute the derivative of this thing and it's large, then that might be an indication that a big change is going on. So the derivative operator, which is the f dx, if this thing is large, then it's likely to have an edge. And indeed, if I plot the derivative with prime of x, right around x0, I'm going to get a peak like this. So one extremely, at least in the one dimensional case, one extremely simple minded way of detecting edges, if you roll up, please, is to say, start with f of x, compute the derivative, that's f prime of x. Take the magnitude of that. Let's try this one. Yeah, this is much better. Take the magnitude of that. And then go to a box where you threshold it. If it is larger than some given threshold at x equals x0, if the answer is no, then x0 is not an edge. However, if it is yes, then we have to make sure what? We have to make sure just because the derivative is larger than a certain value, it doesn't mean necessarily it's a local maximum. So we want to make sure that at x0, f prime of x isn't that indeed corresponding to a local maximum. So you ask yourself, is f prime of x at x0 a local max? And if the answer is no, still x0 is not an edge. However, if you get a local maximum, then it is an edge. So this is a very simple minded way. You're basically checking to make sure that the derivative of f of x at a particular location, not only is it higher than the threshold, but it has a local max. Another way of making sure that that's a local max is you want to make sure that the second derivative goes through zero. That's obviously a possibility. And again, I'll come back to, let me come back here to chapter 10 and kind of show that in action if possible. So this shows kind of the derivative operator. What you see on the left column is show figure 10.7 of Gonzales and Woods. So what you see on the left is a patch of an image that moving from left to right is increasing, but what we're doing is we're adding noise to it. You're adding random Gaussian noise, and up here, there's no noise. Here, the variance of the noise is 0.1, here is 1, and here is 10. So you can see this is still a ramp going up, but it's a lot noisier. In the middle column, what we have is the derivative of this signal. So in here, it's nice and clean. Here, we've added a little bit more noise on this. Visually, you don't see the noise on this, but when you take the derivative of this, you see all this noise added. Here, the noise is even more, and here's the noise is almost bearing it. And what you see here is kind of the second derivative, so it's the derivative of the derivative. And as this thing goes up, you get a bump here, and as this thing comes down, you get another bump there. The same thing here, so when the signal is very clean and no noise added, these doublets, or this thing is extremely easy to tell us that there was an edge at this point. However, when a noise gets added, those doublets get buried in noise, and now by the time you get down here, you can't even see a lot of these things. So the point being is that every time you take the derivative operator, either first or the second order, it's a huge amount of noise you're accentuating all the noise that was already there. Therefore, that's kind of a motivation when we get to Laplace and O'Gossians to low-passable to the image initially to get rid of a bunch of the high-frequency noise and then apply these derivative operators. So that's motivation for technique number three, which is Laplace and O'Gossians, but I thought it's important for you to kind of be aware of it. Okay, so coming back to paper here, this is a very simple one-dimensional edge detection scheme. And the question is, can we easily extend it to 2D? Yeah, it's not that hard. What you do is now you have f of x comma y, and it's an image, and we're going to talk about the derivative of it. Can you roll down please? Along two directions. One is around the x direction, we call that Ix, and another one along the y direction, Iy. So the gradient of this thing, del f, is just exactly what I talked about a second ago. You can write it in terms of a vector, gx over gy, or you can just say, if the f of the x comma y over dx, ix plus the f of x comma y dy times Iy. So it's a vector, that's the point. Now the definition of this edge detection in this two-dimensional system can be, can you roll up please? As follows, you start with f of x comma y, you compute the del operator, which is this vector computation, compute the derivative along x, compute the derivative along y. And once you have that, you pass it through a magnitude box, you say, what's the magnitude of this vector? And if it is larger than a threshold at a particular x0 than y0, then you proceed to the next stage. If it is not, then you say x0 than y0, not an edge. And here, instead of trying to figure out whether you have a local maximum, whether that point is a local maximum, you go through a box that's called edge-tinning. And this box wants to ensure that at least one of the components of the derivative is much stronger than the other one. So here's the detailed description of an edge-tinning. Basically what it does, it looks at two conditions. If this, the magnitude of delta f of x comma y has a local maximum at x0 comma y0 in the horizontal direction, but not vertical, then we declare x0 and y0 an edge when this happens, when the magnitude of along x at x0 comma y0 is quite a bit bigger. It's bigger than k times the magnitude of the f along y at x0 comma y0. So you want to make sure this condition of computing the local maximum, which we had in 1D, is not a very easy one to compute in 2D. So instead, you cannot replace it by this, and you want to make sure that at least if it is a local maximum in the horizontal direction, you want to make sure that really the FDX is a lot bigger than the FDY. And k, you can choose anywhere around 2 or 3 or something like that. That way you can declare that you've got a horizontal edge, and you can justify having an edge. But if this condition is not satisfied, then you don't declare that an edge. So this is the edge-tinning process. Since it gets rid of a lot of false edges that you would otherwise get, and then there's the reverse of this condition along the vertical direction. So, and the condition B is just the opposite of that. If del f of x comma y has a local max at x0, y0 in the vertical direction, but not horizontal, then we declare x0, y0, an edge if the opposite of this thing happens, which is the FDY is much larger than k times the FDX. All of these things evaluated at x0, then y0. Okay. So this whole system that we've talked about, if you roll back up here, this whole system that we've talked about here, this whole system that we've talked about here, this whole system that we've talked about here. This is what's referred to as a non-directional edge detection. And why is that? Well, because I'm looking at the magnitude of del f. I'm not trying to find edges along the horizontal direction separately from edges on the vertical direction, or vertical direction, 45 diagonal direction, any of that. By the fact that I'm taking the magnitude here, I'm not worried that I'm not worried that the direction of the edges that I'm detecting. Now, most of the time, to get reasonable results, people want to do directional edge detection, and I'm going to spend, so this was a gradient, what we've talked about here, this is a gradient method where you simply didn't care about direction. So the next thing I'm going to talk about is directional gradient edge detectors. Okay. So how does that work? Well, you're basically looking for, basically what that means is that you have a bias towards a particular direction. You're either, for example, want to find horizontal edges or vertical edges, so you want to compute, for example, the vertical edge, then you care about df dx. And if you care about horizontal edge, vice versa. You look at df dy. Okay. And let's just spend a few seconds or minutes to talk about computing these derivative. How do we go about computing df dx and how do we come up with approximations with those? And again, we talked about that a little bit last time, but not too much. So df dx can be approximated as f of x comma y minus f of x minus one comma y divided by t, which would be kind of the distance between the pixels. And these t's all cancel out and drop out and disappear, so we don't have to worry about it. Or alternatively, you can talk about it as f of x plus one comma y minus f of x comma y divided by t. That would be another way. Or another alternative, if any of these are good enough, you can talk about f of x plus one comma y minus f of x minus one comma y divided by two t. So if you look at any one of these operators, again, you can simply see that these are nothing but convolving original f with another filter with some impulse response. So what would be the impulse response of this filter? It would be something that's n one as a function of n one and n two. It's one here, and it's minus one at this point. This one is just the opposite, n one and two. It's again one here, sorry, minus one here, and one there. And this one is going to be an impulse response that's plus one here and minus one there. Is that clear to everybody? That's all we're doing. We're convolving the f with these kind of a filter. And so it's not that difficult to see that it's not that difficult to kind of invent your own set of approximations to how you want the derivative operator to be computed. Hey Amazon Prime members, why pay more for groceries when you can save big on thousands of items at Amazon Fresh? Shop Prime exclusive deals and save up to 50% on weekly grocery favorites. Plus save 10% on Amazon brands, like our new brand Amazon Saver, 365 by Whole Foods Market, Aplenty and more. Come back for new deals rotating every week. Don't miss out on savings. Shop Prime exclusive deals at Amazon Fresh. Select varieties. Today we're going to be talking about the Home Depot and the Home Depot. Now we're going to be talking about the family. We're going to be talking about the Home Depot. We're going to be talking about the Home Depot and the Home Depot. Today we're going to be talking about the Home Depot and the Home Depot. We're going to be talking about the Home Depot and the Home Depot and the Home Depot. And in particular, what's the most famous name we have associated with edge detector, which is simply just a guy putting his name on the filter because he came up with the definition of a derivative and it worked well. It's called Sobel. I mean, you've had Sobel edge detector. Has anybody heard of that before? Good, okay. So, I mean, basically, when I put this down, you're going to say, what? Did he put his name on that? Well, I could have done that, which is exactly what anybody's reaction could be. But kind of a famous directional edge detection operator is Sobel. And by the way, one of the most famous edge detector techniques was done by Kenny as his -- I think it wasn't even his PhD thesis. It was his master's thesis at MIT. And his method works very well, not only because he has very good filters, but also of the post-processing that he does, is after he does the linear filtering, kind of the linking that he does to the edge points in order to create a clean image. And I think, in Berkeley, we still use Kenny's edge detector in a lot of the computer vision projects. But Sobel is not -- Sobel edge detector is simply just a filter. What it says is to compute the FDX, convolve F with H of X, where H of X is a filter that is of this form. It's one here, two here, one here, minus one, minus two, and minus one, and zero everywhere else. Does that make sense? Yeah, it's taken to derivative respect to X, this minus that, this minus that, this minus that. And he's giving twice as much weight for the current pixel than the ones below and above. Makes sense. And that's all there is to Sobel edge detector. He doesn't tell you how to link, he doesn't know how to post-process, none of that. It's just the description of this derivative filter that's called the Sobel edge detector. The same thing with DFTY, you can do as a function of N1 and N2. Now you have the opposite of it because you're taking the derivative respect to the Y direction. So you'll have plus two here, minus two there, and then one here, sorry, one there, one here, minus one, zero, zero, zero. So this is H sub Y. So DFTY in this case is equal to F convolved with H sub Y. Okay. And you can either apply these operators separately to an image and threshold them so you can apply this to an image and try to find vertical edges. Just apply this to the image. If the magnitude of DFTX is larger than some threshold, declare that as a vertical and it also achieves local maximum and declare that as a vertical edge. Similarly, you can declare this as a horizontal edge. Or you can plug each one of these DFTXs into this big system we had compute the magnitude of DFTX, DFTY, and then threshold it and do these kinds of operators. And I'll show you examples of this in just a second, but let me in the interest of not going back and forth 50 times. Let me just write down one more kind of famous edge detector. And if that's Robert's, just to have served this guy our dues. An edge detection on the various filters reminds me of filter design, filter windows. And when we teach DSP classes in like E123, everybody came up with a window, got his name on a function. So Robert's edge detector is again nothing but another definition of H sub X and H sub Y. So for Robert, H sub X was minus one here and one here. And H sub Y is minus one here and one here. Okay, so let me now stop at this point and show some pictures. And some of the pictures that I want to show are actually not in the, not in electronic. It's off of Jay Lim's book. I'm just going to show some pictures straight up here. Okay. If you can focus on zooming in as much as you possibly can. So you've probably seen this picture before. On my monitor, there's a lot of flicker going on. Oh yeah, yours too. But that's not part of the image. Flicker is part of the electronic system, the display system. So this is a picture of a building and it's a particularly good one for directional edge detection, right? Because it's got windows that go to horizontal. It's got edges that go vertical and it's got, because of the parallax effect, it's got things that go at an angle as well. Okay. So right off the bat, this is, if I apply the vertical edge detector to this, the FDX and threshold that, using sobells, I get. Hey Amazon Prime members. Why pay more for groceries when you can save big on thousands of items at Amazon Fresh? Shop prime exclusive deals and save up to 50% on weekly grocery favorites. Plus save 10% on Amazon brands, like our new brand Amazon Saver. 365 by Whole Foods Market, Aplenty and more. Come back for new deals rotating every week. Don't miss out on savings. Shop prime exclusive deals at Amazon Fresh. Select varieties. We wear our work, day by day, stitch by stitch. At Dickies, we believe work is what we're made of. So, whether you're gearing up for a new project or looking to add some tried and true work wear to your collection, remember that Dickies has been standing the test of time for a reason. Their work wear isn't just about looking good. It's about performing under pressure and lasting through the toughest jobs. Head over to Dickies.com and use the promo code Workwear20 at checkout to save 20% on your purchase. It's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century. Sorry, that's not exactly so bold. The soba level is the next page. He calls it a vertical edge detector. Let me see what he exactly means by vertical. He just means, yeah. By vertical, he just means a filter like this. So, one, one, one, minus one, minus one, minus one. And then, horizontal, it'll be this one. So, this is a simple, non-sobel, non-roberts, something extremely straightforward that you can come out with. So, again, this is what you get. As you expect, you've detected a lot of the vertical edges, but it's not a terribly good representation of what's going on in this image, right? The edges are broken, they're not connected. Now, if you apply that simple horizontal edge detector, which is this guy to it, you get this picture, which has more of the horizontal edges. Now, if you apply the sobel edge detector, not the directional one, but you compute the FDX and the FDY using sobel, and then you take, square them up, take the square root of it, you get this picture, as a result of sobel. And you can see this is a lot better representation. You can actually make out the building and the stairs and the various other things that's going on. So, this is essentially applying this system, if you can zoom on for just a second. This is applying this system where we compute the FDX and the FDY using sobel edge detector. And what we get is something like this. And this picture here is the same thing, but using Robert's edge detector. And you can see that Robert's edge detector is in many ways inferior to sobel, but it has one advantage, and what's that? Not visually, it comes out in the definition. Robert's edge detector has only two points, sobel has six points. Robert's edge detector has coefficients that are plus one and minus one, so you can do it with additions, whereas sobel has a two. Well, actually having a two is not so bad because multiplying by two involves just shifting to the left, right? But why do you care about the computational complexity? Well, if you have to do things real time, for example, when I was at MIT, there was a company, I forgot the name now. But it's not Pixar, but it was Pick Something. Essentially, they were in the inspection, automated inspection business. They would build vision techniques, they would inspect, print a circuit board, this and that the other. And none of these operations are implemented in C on a computer, because it's way too slow. You have to either have A6 or you have to have general purpose video signal and process program video signal processes like TI chips to do all these things. And in that case, if your images have 1,000 by 1,000 pixels and they're coming at you at 30 times a second, it makes a difference whether you're using, for example, Roberts, which has only two taps, or whether they're using sobel, that has six taps. It's three times as much computation, one versus the other. Anyway, so yeah, Sobel does look better than Roberts, but it also is a little bit more time consuming to compute. And let me just show you what I really regret about Jay Lim's book is he shows the bad directional DFTX for not Sobel. But then when it comes to Sobel, he combines DFTX and DFTY and shows this image. So to undo that effect, let's go back to the screen here, because Gonzales and Woods is a nice picture regarding that. So what you got there is an original image on the upper left, this is this guy. Again, people always use house pictures because it's got a lot of good directional kind of edges. Part B is the magnitude GX, this is just DFTX. If the magnitude of DFTX is larger than certain threshold, you get that. This is magnitude DFTY and this is now DFTX plus DFTY. And as you can see, this guy detected the vertical ones, this guy detected the horizontal ones. When you look at the magnitude of this plus magnitude of that, if that's larger than threshold, you get both of them here. And this is kind of showing what happens if you felt that the signal that you got here had too many edges and was too noisy. For example, look at the picture up here. It is true that the roof up here has a little bit of a texture to it, but I don't really think you can see that. No, this monitor is too coarse, but for those of you who have the book in front of you, there's a bit of a texture right here. And the edge detector, as you can see in these pictures, it's definitely detected. And this pattern, by the way, is striped, but it's not very strong. It's a little bit. So edge detector has detected those. And as you can see, there's white areas here that shouldn't have been white, etc. Visually this looks good, but there's a lot of garbage there. And one way to improve that is to get rid of the picture to begin with by a 5x5 average input. And then apply DFTX, DFTY, and then magnitude of DFTX plus magnitude of DFTY. And as you can see, a lot of the garbage edges by doing that process has disappeared. Okay. The last thing I really want to say about directional edge detectors and the gradient methods in general is that you can only have filters that are tuned to detecting vertical and horizontal edges. You can also have filters that detect 45 degree angle type edges. For completeness, write that down. So this is directional diagonal rather, diagonal directional gradient filters, if you will. So for solbow, if you're interested in diagonal, then the H guy, I call it HD diagonal, N1 and 2, looks something like minus 1 minus 1 1 2 minus 1 minus 2 0 0 0. You have a 0 going along this direction, and then you've got this minus that plus 1 plus 1 plus 1 minus 1 2 minus 2. Okay. And so this is one diagonal, and then the other one is the opposite of this going the other way where the zeros are here, here, here. And now you have a plus 1 minus 1 2 1 and then minus 2 minus 1. So this is diagonal going this way. So again, I apologize. I don't know why, but Gonzales doesn't have an example of a case where you strictly speaking detect the diagonal edges. But if you just were interested in 45 degree edges, you can apply these filters and apply the thresholding operator to that. Alright, before I move on to Laplacean, are there any questions about this gradient based technique? Gradient, you just compute the FDX. The first order derivative, you either look at the FDXs by itself and threshold that, or you look at the FDX magnitude plus the FDY, or variation of that. And how you compute the FDX has to do with whose filter you fall in love with it. If you want to do Soul Bowl, you want to do Roberts, you want to do this, that, or the other. So let me now move on to Laplacean. So the idea here is if F of X looks like this, and then F prime of X is going to, let's say at X naught has a transition, F prime of X is going to do this. And F double prime of X, Laplacean means we're going to hire the derivatives. Can you zoom out just a tiny bit? F double prime of X is going to do this. Again, zoom out just a tiny bit. Thank you. And the same thing here, if F of X goes down at a particular point from a high intensity value to small value, then F prime of X is going to do something like this. And then F double prime is going to do F of X naught, something like this. So here I'm showing the transitions to be smooth, and that's why the doublet is not so short as we saw before. It smoothly goes down and up. But the truth of the point is, the zero crossing of F double prime of X corresponds to when there's an edge change. If this thing was very sharp, then this thing was boom, and then boom, and then boom, then this thing would have been much sharper, and this thing would have been much narrower. Everything would have been tighter if this was a sharper edge. But the bottom line is, when the derivative is crossing zero, then there's an edge happening. So I'm sorry, if the second derivative is crossing zero, then there's an edge happening. So essentially, in the Laplaceian edge detection technique, we care about the zero crossings of the second order derivative. So the story to, so the morale is that the zero crossing of F double prime of X corresponds to where possibly an edge is. So now, how do we extend this to 2D, not too difficult? Again, you're looking at del squared of F of X comma Y, which is del of del of F of X comma Y, the derivative of the derivative, which is d squared F of X comma Y dx squared plus d squared F of X comma Y dy squared. And you play the same game as before. You say dF dx is nothing but F of N plus N1 plus 1 comma N2 minus F of N1 comma N2. Oops, I switched my notation to N1 and N2. It's the same as X and Y, so don't worry about it too much. And then d squared F dx, dx squared, is nothing but F sub X of N1 and N2 minus F sub X of N1 minus 1 comma N2. And if I now combine these two things together, I can write d squared F dx squared as the following, it's F of N1 plus 1 comma N2 minus twice F of N1 comma N2 plus F of N1 minus 1 comma N2. And then finally, now that I have this expression for d squared F dx squared, I can also write down the expression for the whole thing, d squared F dx squared plus d squared F dy squared, which corresponds to F of N1 plus 1 comma N2 plus F of N1 minus 1 comma N2 plus F of N1 comma N2 minus 1 minus 4 times F of N1 comma N2. So once again, all these operators, the derivative operators, is a linear operator. And if this is what it's doing to my signal, it's almost like I'm convolving it with a filter, with a linear time invariant or linear shift invariant filter. So this is the same as convolving F with some H, where the H, as a function of N1 and N2, can you roll down please? It's nothing but minus 4 here, 1, 1, 1, 1, 1. Hey Amazon Prime members, why pay more for groceries when you can save big on thousands of items at Amazon Fresh? Shop prime exclusive deals and save up to 50% on weekly grocery favorites. Plus save 10% on Amazon brands, like our new brand Amazon Saver, 365 by Whole Foods Market, Aplenty and more. Come back for new deals rotating every week. Don't miss out on savings. Shop prime exclusive deals at Amazon Fresh. Select varieties. We wear our work day by day, stitch by stitch. At Dickies, we believe work is what we're made of. So whether you're gearing up for a new project or looking to add some tried and true work wear to your collection, remember that Dickies has been standing the test of time for a reason. Their work wear isn't just about looking good. It's about performing under pressure and lasting through the toughest jobs. Head over to Dickies.com and use the promo code Workwear20 at checkout to save 20% on your purchase. It's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century. And once again, this is a particular implementation of H we came up with because we chose to approximate our DFTX this way and this other one this way and we got this. Now, if I look at the figure 8.33 from JLIM, you can come up with other ways of the derivatives and come up with other Hs. And let's turn back here and zoom in please. So here's for example, if the approximated my derivative operator slide a different way. This is the example we just derived. Minus 4, 1, 1, 1's around. Here's if you have a different approximations, that means you have a different approximations. They're all valid. It all depends how you approximate the derivative operator. So the next question that comes up is how could I build a system that then uses this del squared operator in order to detect edges? And your first instinct might be, okay, well, so the question is, zoom out please, how to use this del squared F operator to detect edges. And you might say, okay, well, start with F of N1 and N2 and pass it through a del squared F operator and look at the zero crossings. And if there's no zero crossing, it's not an edge. And if there is a zero crossing, call that an edge. And if you do that, you get terrible results. Let me just convince you of it. So if you can zoom into this, so if I just do what I said, this is what you get. This is the cameraman picture. That's what you get. Terrible. I mean, edges are all over the place. And then why do you think that is, I mean, intuitively, why would just do that? Because taking a derivative enhances noise, accentuates noise, if you look at the frequency response of all this impulse response side right down, they're a high pass. And if there's noise in the picture that's high frequency noise, the high pass filter is going to boost the noise, and that's why you get all this garbage. So one way to fix that, and again, we also talk about applying Gaussian filters before we do this as a way of fixing it, and that's the third method, but until we get to the third method, there's still ways to fix that. I mean, if you look at it, what seems to be the problem? Well, there's all these kind of constant intensity regions like here, here, here, where the signal is almost constant, and yet we're getting a ton of garbage. So maybe looking at this, we could get an idea of saying, okay, the areas of the signal where the variance is not high, i.e., we have constant intensity, we won't declare it as having an edge. So not only does this del squared f have to have zero crossings, but for that local region, we have to have local variance of that to be higher than a certain threshold for us to call this an edge. And that's exactly what the tricks would come out with. So we're going to estimate the local variance, sigma squared f, and then, and if the answer here is yes, and only if, in this box, only if sigma squared f is larger than some threshold, if it is not no, then not an edge. But if it's also larger than an threshold, then it is an edge. And you might be asking yourself, well, how do I compute sigma squared f? Well, it's kind of along the lines of the local processing stuff that I talked about last time, which is you go around an m by m, like 3 by 3, or 5 by 5 window of a pixel computed. So it's the mean over those 5 by 5 region, compute the variance over the 5 by 5 region, and that's how you compute the local variance. So to compute the local variance, first I define local mean, which is m sub f of n1 and n2, is 1 over, this is all for a 2 m plus 1 by 2 m plus 1 region. So that's what I mean by local. So it's 1 over 2 m plus 1 squared of double summation of k1 and k2 going from minus m to plus m of f of k1 comma k2. So the local mean, and the local variance is defined in terms of local mean, sigma squared f of n1 and n2 is nothing but, again, 1 over 2 m plus 1 squared of, actually, can you roll up just a little bit? Thank you. My hand has no room in this double summation again. Hey Amazon Prime members, why pay more for groceries when you can save big on thousands of items at Amazon Fresh? Shop prime exclusive deals and save up to 50% on weekly grocery favorites. Plus save 10% on Amazon brands, like our new brand Amazon Saver, 365 by Whole Foods Market, Aplenty and more. Get back for new deals rotating every week. Don't miss out on savings. Shop prime exclusive deals at Amazon Fresh. Select varieties. We wear our work, day by day, stitch by stitch. At Dickies, we believe work is what we're made of. So, whether you're gearing up for a new project or looking to add some tried and true work wear to your collection, remember that Dickies has been standing the test of time for a reason. The work wear isn't just about looking good. It's about performing under pressure and lasting through the toughest jobs. Head over to Dickies.com and use the promo code Workwear20 at checkout to save 20% on your purchase. It's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century. So, this is not from minus M to plus M. It's N1 minus M. To, oh, really messed up here. K1, you know what? I'm going to redo this local mean. I have little space and I just keep spiraling down a hole. It's terrible. It's my own fault with my handwriting. So, let me redo local mean and local variance. Okay, local mean. Now I have a lot of space, which we call M sub F of N1 and N2 is 1 over 2M plus 1 squared summation of K1 ranging from N1 minus M to N1 plus M and K2 ranging from N2 minus M to N2 plus M of F of K1 comma K2. So, that was easy. Now, local variance, which we call sigma squared F of N1 comma N2, it's again 1 over 2M plus 1 squared F of summation of K1 ranging from N1 minus M to N1 plus M of K2 ranging N2 minus M to N2 plus M of something squared. And what is that thing squared? F of K1 and K2 minus the mean. M sub F of K1 and K2. Wow, you know, conceptually, it's extremely straightforward. I mean, you're going around a pixel, you're competing the mean around it and you're competing the variance around it. But putting it in equations, it's a lot of notation. Don't get scared by the notation, though. So, compute the local mean around 2M plus 1 and our window, just so that you remember, is 2M plus 1 by 2M plus 1 centered around a pixel in 1 and 2. All right, so if you do all of that, you can rescue this disaster that we have getting this terrible looking image. And now, I'll give you a chance to copy everything before I remove it. By the way, all of these things is going to get scanned in PDF format and be available on the web. Now, if I do that, I get this picture, which is a lot cleaner than this picture. This was the original, this was without doing the sigma squared F, without looking at the local variance, and this is with that. So, you can see that the edges are a lot more meaningful. There's still islands of bad things, but we can live with those. Okay, that's all I have about the Laplacian method. Now, the last thing I want to talk about is Laplacian of a Gaussian, and I'll explain that in just a second. Any questions on this? Let me pause. And so, the third method here is edge detection via Laplacian of Gaussian. And what's happening here is that basically Marne Hildreth in the 80s, by the way, Marne died very young at the age of 32 in mid-80s, and there's all sorts of tributes to him, and I was at MIT as to, you know, how much he contributed to computer vision, et cetera, et cetera. This is one of Marne's central contributions to computer vision. They kind of noticed the fact that if you talk about edges in an image, it makes sense to talk about edge at different scales. And we'll talk about the notion of scale and multi-resolution image processing, and wavelets, and pyramids, and stuff like that, a little bit more in detail when we get to compression. But this notion of scale was actually something that different technical communities arrived at from different perspectives. For example, the computer vision people came out with a notion of scale by saying, "Oh, we had it in 1982, that's what Marne Hildreth did, and what it means is you start with an image, you convolve it with a low pass filter of a certain radius, let's say." And then you convolve it with a series of low pass filters as the diameter of the low pass filter changes, and that they call looking at an image at different scales. Intuitively speaking, that's all scale means. Now, when you go to wavelets, you can come up with mathematical theories of if your filter satisfies certain conditions, then certain other nice things happens, and blah, blah, blah, blah. There was a lot of detail to it. However, in plain old computer vision, the notion of scale is just low pass filter in your image with a series of low pass filters so that you get a series of blurred version of the same function. And when you do that, you can also subsample. If you've low pass filter, your signal with a sufficiently large low pass filter, blurred it enough, you can also subsample. In that case, you can then build a pyramid. At the bottom is the original image. Next layer up, the image on half the number of pixels, half the number of pixels. And why aren't these pyramids useful? Does anybody know? Many times if you want to do a task, you might want to do the task in the low resolution part of the image, which is much more like 32 by 32 version of the signal, which is much more computing intensive, and then propagate the signal from 32 to 32 to the higher resolution, 64, and then 128, 256, 512 to 1000. So the first time Bert and somebody else whose name I forget built an image pyramid, their motivation was to do motion estimation. Hey Amazon Prime members, why pay more for groceries when you can save big on thousands of items at Amazon Fresh? Shop Prime exclusive deals and save up to 50% on weekly grocery favorites. Plus save 10% on Amazon brands, like our new brand Amazon Saver, 365 by Whole Foods Market, Aplenty, and more. Come back for new deals rotating every week. Don't miss out on savings. Shop Prime exclusive deals at Amazon Fresh. Select varieties. We wear our work, day by day, stitch by stitch. At Dickies, we believe work is what we're made of. So, whether you're gearing up for a new project, or looking to add some tried and true work wear to your collection, remember that Dickies has been standing the test of time for a reason. The work wear isn't just about looking good. It's about performing under pressure and lasting through the toughest jobs. Head over to Dickies.com and use the promo code Workwear20 at checkout to save 20% on your purchase. It's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century. We'll do the motion estimation at the top of the pyramid where the images are small, and then use the motion estimate at the top to propagate that and refine it as we go down the pyramid, for example. In the late 80s, Stefan Mala came out with a more formal mathematical notion of wavelets. Again, he was from the computer vision society, but at the end, he formalized everything mathematically about wavelets and scales and various other things. We'll talk about the details of wavelets later on, but for now, what you need to know is that the meaning of Laplace and Gaussian is this. First of all, what do I mean by Gaussian? Well, Gaussian is a filter that has this kind of impulse response. Is h of x comma y is e to the minus x squared plus y squared divided by 2 pi sigma squared. And as it turns out, what can we say about the frequency response of a Gaussian? If this is the impulse response, what's the frequency response? Anybody? A Gaussian too, exactly. So then h of omega x comma omega y is 2 pi squared sigma squared to the minus 2 pi sigma squared, omega x squared plus omega y squared over 2. So what do I mean by Laplacian of Gaussian, which sometimes it's Gaussian, which sometimes it's written as l-o-g, is you first start with your sigma f of x comma y. You convolve it with h of x comma y. This is the Gaussian. And then you apply the Laplacian operator to it to the Laplacian of a Gaussian, which is the same as all of that. You can also simplify as f of x comma y convolved with del squared h of x comma y, because linear operators you can commute, which is the same as f of x comma y convolved with, if I were to expand this, d squared h of x comma y dx squared plus d squared h of x comma y d y squared. And just if you can tolerate a few more equations, you can also write down, and you can roll up for just a little bit, so del squared h of x comma y in the space domain can be written as e to the minus x squared plus y squared over 2 pi sigma squared, the whole thing over 2 pi sigma squared times x squared plus y squared minus 2 pi sigma squared. And the Fourier transform of that thing here is minus 2 pi squared sigma squared e to the minus pi sigma squared omega x squared plus omega y squared over 2 times omega x squared plus omega y squared. Enough of, so just so that you can see what these things look like, look at figure 8.36 in j limbs, just so that you know how this function looks like. So essentially, I mean, once again, we're convolving our signal f of x, we get another filter. It's all linear time invariant, linear shift invariant kind of operation, nothing new. And so in the time, in the space domain, this h of x, this function here, del squared h of x comma y looks like this. If you can zoom in as much as you can. Hey Amazon Prime members, why pay more for groceries when you can save big on thousands of items at Amazon Fresh? Shop Prime exclusive deals and save up to 50% on weekly grocery favorites. Plus, save 10% on Amazon brands, like our new brand Amazon Saver, 365 by Whole Foods Market, a plenty and more. Come back for new deals rotating every week. Don't miss out on savings. Shop Prime exclusive deals at Amazon Fresh. Select varieties. We wear our work, day by day, stitch by stitch. At Dickies, we believe work is what we're made of. So, whether you're gearing up for a new project or looking to add some tried and true work wear to your collection, remember that Dickies has been standing the test of time for a reason. Their work wear isn't just about looking good. It's about performing under pressure and lasting through the toughest jobs. Go over to Dickies.com and use the promo code Workwear20 at checkout to save 20% on your purchase. It's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century. And the Fourier transform of that looks like this. This is Omega X, Omega Y. It's kind of at zero. It has a dip. It's kind of like a high pass, kind of like a band pass kind of a thing, right? Why is that? Because you're applying your first convolve on your signal with the Gaussian. That has the effect of blaring it. That's kind of a low pass filtering. And then you're applying a bell squared, which is kind of a high pass operation to that low pass. Okay, just so that you know what these signals kind of look like, right? What you see up here, A, B, and C, is the low pass filter image with Gaussian of sigma squared equals four, sigma squared equals 16, and sigma squared equals 36. So as you move to the right, you would expect the signal gets more and more blurred, right? And I really wish that Jay Lin would show one more picture before he shows what's below, but he doesn't. I really wish after the show in the Gaussian, he would show the Laplacian applied to it. But instead, he skips a step and he shows the result of applying Laplacian to these after you go through a thresholding operation. Rather, the zero crossing operation. Okay, so if you then apply, so this is the applying Gaussian, if you apply Laplacian on top of that and then detect all the zeros, you get this, this image, this for this and this for this image. So you can kind of see what we mean by the notion of scale. So this picture is detecting edges at this scale. And as you can see, there's a lot of garbage and you can probably remove a lot of these garbage by applying similar tricks that we talked about earlier. Just so that you've gotten kind of a slightly different kind of flavor of how these pictures look like, let's switch back to the computer. And this is the del square H operator filter again. I like this 3D plot from Gonzales a little bit better than Jay Lim's. It kind of shows the fact that it goes up, but then it goes down. Sometimes people refer to this as what? It's del square H in the literature. It's the Mexican hat. Don't ask me, well, I guess it does look like a little bit like a Mexican hat. And this shows a cross section of that. Where it crosses in the x, y axis, where it crosses zero has to do with sigma, where sigma is the variance of the Laplace in you. You convolve to this. And this also shows the intensity, a different representation of this where white shows height, dark black is this negative part. And then this region is the first come back to zero. And this is some kind of a digital approximate discrete time approximations of these del square H. And I'm going to end the lecture by just going over this picture, which I think is a fairly nice one and off of Gonzales some woods. It tries together some of the concepts that we've talked about. So what you have there is an original image on the upper left. So this is the sobo gradient. And it's just shown for the sake of comparison. OK. And the book just causes sobo gradient. I was just wondering, is it DFTX, DFTY? It's probably its magnitude of DFTX plus magnitude of DFTY. Where DFTX and DFTY are defined by sobo. OK. Next, what you have down there is the spatial Gaussian smoothing function. This is the Gaussian filter that we applied. This is the Laplacian mass that we applied. And this is the signal that you get by applying Laplacian of a Gaussian that del square H guy to the original. OK. So it kind of looks like as the high pass version of the signal. You start with this, you blur it with a Gaussian, then you apply the del squared F operation, and you get this. This is the thresholding operation applied to this signal, right? And after you threshold it, then you can get the zero crossings by looking at the places where it's black and white and draw the zero crossings. So the thresholding is not really necessary, but it's used as a step to compute the zero crossing of this signal. You first threshold it, and then you find the zero crossings of this thresholded signal. So you can see as compared to this sobo guy, this thing has a ton of noise, a ton. So if you're purposed of edge detection, I mean, this is one thing we didn't really sweat about too much. Why do we do edge detection? I mean, edge detection is kind of a first step in many image analysis, image understanding systems, computer vision systems. You do edge detection to kind of get a broad picture of what's going on in a picture. No pun intended here. So you would like to have images as the edge detectors if you don't want your system to get thrown away. Obviously, what is a good edge detector? Very much ultimately depends on what your goal is. What are you going to do with that edge detector picture? If missing an edge could be consequential, could you have severe consequences like a plane crashing or people losing lives, then yeah, maybe you want to have this kind of an edge picture. If your goal is to make sense out of the picture to say, yeah, you know, let's say you wanted to do recognition. For example, yeah, you can now build higher level reasonings and abstractions and machinery to then detect that this is a lizard climbing of a tree, whereas this would be extremely difficult to do it from here. I'm going to end this discussion by saying that this operator is the first step in edge detection and there's subsequent stages where people do what's called linking and this is just a very, very simple example of starting with an image, computing the GW, computing the GX, and this is a result of linking the edges that were here and here, and that's what I was referring to when I talked about Kenny's thesis. He wasn't one of those guys, he said, here's a new filter of Kenny, Kenny filter like Sobel or Roberts, he didn't just do that. He came up with techniques after we do the edge detection using these filters to link them together to get this picture out of this, while combining these two and applying additional, it's essentially secret massaging and processing this to get something that looks like this. Okay, I'll see you all on Friday. Hey Amazon Prime members, why pay more for groceries when you can save big on thousands of items at Amazon Fresh? Shop Prime exclusive deals and save up to 50% on weekly grocery favorites, plus save 10% on Amazon Brands, like our new brand Amazon Saver, 365 by Whole Foods Market, Aplenty and more. Come back for new deals rotating every week, don't miss out on savings, shop Prime exclusive deals at Amazon Fresh, select varieties. We wear our work, day by day, stitch by stitch. At Dickies, we believe work is what we're made of. So, whether you're gearing up for a new project, or looking to add some tried and true work wear to your collection, remember that Dickies has been standing the test of time for a reason. Their work wear isn't just about looking good, it's about performing under pressure and lasting through the toughest jobs. Head over to Dickies.com and use the promo code WorkWear20 at checkout to save 20% on your purchase. It's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century.