The Bold Blueprint Podcast
The Bold Blueprint Avideh Zakhor When you view setbacks as learning experiences,
Hey Amazon Prime members, why pay more for groceries when you can save big on thousands of items at Amazon Fresh? Shop Prime exclusive deals and save up to 50% on weekly grocery favorites. Plus save 10% on Amazon brands like our new brand Amazon Saver, 365 by Whole Foods Market, Aplenty and more. Come back for new deals rotating every week. Don't miss out on savings. Shop Prime exclusive deals at Amazon Fresh. Select varieties. We wear our work day by day, stitch by stitch. At Dickies, we believe work is what we're made of. So whether you're gearing up for a new project or looking to add some tried and true work wear to your collection, remember that Dickies has been standing the test of time for a reason. Their work wear isn't just about looking good. It's about performing under pressure and lasting through the toughest jobs. Head over to Dickies.com and use the promo code workwear20 at checkout to save 20% on your purchase. It's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century. Announcements. There's a homework here today. There is a proposals due on the 17th and I gave a handout we got possible topics. Last time if you didn't pick that up and you would like to pick it up go to Rosita in room 253 to pick that up. There is going to be another lab assignment on enhancement which is I think one to the last assignment due on the 24th of March and Cindy reminded me to put that up on the webpage. Two weeks I think should be plenty and that's about all the assignments and announcements I have. Any questions or comments? Just as a quick show of hand how long on average you did take you guys to do. How many people took less than five hours to do this? Any questions or comments about the term project? I thought about another quite interesting topic more in the main stream of image processing. For those of you who haven't picked one, aren't happy with what I said before and that has to do with multi-frame image enhancement where you first align and register multiple frames and then try to enhance the resolution of an image and that's used all the time for video printing. It's used all the time when you take multiple pictures of the same scene and you want to come up with one image that has much higher resolution than it's of the same scene. So if any of you are interested in anything corporates and harder enhancement restoration stuff that we're talking about in class so it's kind of more in the mainstream of image processing. If you guys are interested in finding out more about it come talk to me. I can point you to some references that you could potentially use to do that topic. Any questions or comments? We have 14-page, 14 blank sheets and no more. So, I did a conserve as I write it. Actually, Cindy, would you do me a favor and try to get a little bit more blank sheets given my handwriting? I don't ever know. So what I'm going to talk about today Oh, you got it. I mean you had it on your own. This is your own stuff? Oh, okay. Very good. Thank you. Oh, you usually go through 18 so that you don't have to end up dealing with this in the middle of the lecture. So what we talked about last time was using histogram equalization for image enhancement and what we're going to talk about today is looking at some local enhancement techniques whereby you look at a small region in order to, rather than looking at the entire image and the entire histogram on image trying to regard what transformation to apply to make it look good, you want to do things in slightly more localized way. Use the local statistics to either do contrast enhancement or do histogram recordization in local way. Then I'll talk about enhancement using arithmetic and logic operation in particular using subtraction averaging. Then I talked a little bit about spatial filtering which is really nothing but the linear shifting variant filters we've talked about earlier. Then I talked about non-linear filtering which is order statistics, things like median filtering, min/max filters. I talk a little bit about sharpening filters that means high-pass filters, again linear timing variant that make the image look better. In particular I'll talk about using the derivative operator make images look sharper. Generally speaking sharper images look better so to enhance things you want to sharpen things up and the last thing I'll talk about is on sharp masking and high boost filtering which are related to each other. So let me get going and the first thing I've got to do is bring up the zip page from Gonzales and Woods. Open. Evaluation. Okay and we want to go all the way to picture number 23. Oh that's chapter 4. Okay. One one. One chapter. Sweet. Okay. Okay so I think we have everything just about ready. So the first kind of extension of what we talked about last time that I like to mention is local enhancement using local histogram. Okay so the basic idea here can you yeah we're done here great. So the basic idea is that you want to define some sort of a square or a rectangle neighborhood around each pixel and then and then move the center of this rectangle across your image. I call it a window across the image and you end up doing localized operation and so at each location you do something that depends upon the characteristics of the image at that location. So that's what what we mean by local. So at each location for example in this case we compute histogram in in the window that you've generated. Okay and then and then do histogram equalization. So the picture is this you have let's say a three by three window or a five by five window you center or done this pixel you compute the histogram you equalize that and then that you can either do what's called blocked by block processing which means that you then move this sent to the next block next to it another five by five and repeat that same thing or you can determine the value of the mid-picks that just you can do histogram equalization and after you're done doing histogram equalization that only nails down the value of this one pixel and then you have to drag the windows from one by one pixel and do that thing again and each time you do all of that calculations you end up getting you know you end up equalizing the histogram for that one point. So either one is possible. Of course the second method I just talked about is much more computation intensive and it takes a long time to do. But just to give you an idea if you can switch back to the to the screen here just to give you an idea of how this technique works. Suppose that you start with an original image that's shown here and if you do global histogram equalization you get something like this and if you do the local histogram equalization you get an image that looks like this and you can see there were these little squares inside these squares that were hidden and by doing the localized histogram. Hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon Fresh. Shop Prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon Saver 365 by Whole Foods market a plenty and more. Come back for new deals rotating every week. Don't miss out on savings. Shop Prime exclusive deals at Amazon Fresh. Select varieties. We wear our work day by day, stitch by stitch. At Dickies we believe work is what we're made of. So whether you're gearing up for a new project or looking to add some tried and true work wear to your collection remember that Dickies has been standing the test of time for a reason. Their work wear isn't just about looking good. It's about performing under pressure and last thing through the toughest jobs. Head over to Dickies.com and use the promo code Workwear20 at Checkout to save 20% on your purchase. It's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century. Gram equalization you were able to revive them pretty easily. Okay and in this case the window that you're dealing with is seven by seven. Next what I want to talk about is using local statistics for enhancement. So use local stats for enhancement. So here what do I mean by statistics? Well the two easiest or most common parameters used for statistics are in statistics are mean and variance. And the beauty of image processing is that you have a lot of pixels therefore it's very easy to compute statistics. You have a lot of data points in other words. So you can you can come up with things like global mean defined which we define and denote by m sub g summation from i equals 0 to l minus 1 of r sub i probability of r sub i where r sub i is corresponds to the intensity. Intensity level i and we have i ranging from 0 to l minus 1 we have l levels of intensity. You can also come up with the global variance which we denote by var sub g which is nothing but summation from i equals 0 to l minus 1 of r sub i minus the mean that you just found square times probability of r sub i. Okay now you can you can at the same time come up with the idea of local mean around a region. So local mean is defined as follows if suppose I have a I want to use s of x y as a neighborhood or sub image that's centered around x y then the local mean is defined first of all we denoted as m of s x y am standing for mean it's the summation for all the pixels s and t that are in this that are member of this region s x y r s comma t is their intensity value and probability of r s t is the probability that they take that value and then local variance is s squared s x y again these are all fancy mathematics for really nothing you're looking at a small window and you're completing the mean and the variance of the intensity around that that little window is the summation again of s and t in s x y of r s t minus m s x y the mean that you just computed a squared and then probability that r takes on probability that of intensity level or at location s t okay here is an example of of a situation so here's an example of a problem or a situation where we might want to apply a technique like this suppose that we have a picture if you can switch back to the camera that looks like this this is the SEM of of a image of a tungsten filament that's magnified by about 130 times and I don't really I have a hard time looking at the monitor here and I would probably bet you definitely have a hard time finding it here but you see these these things wrapped around very easily but there's a shady thing there's a shade of white here do you guys see that on the monitor hardly hardly well I mean actually that's the beauty of image enhancement by the time we're done with this example you will see it okay so so here is kind of the goal of what we want to accomplish our goal is to enhance these dark areas because there's something going on here why leaving this light area as unchanged as possible you want you want the the bright area not to change that much you want to keep that but at the same time you want to enhance it so you can easily imagine whatever you want to be doing has to look at the local mean of a particular pixel whether you want to change a region has to do whether its local mean is high or not if the local mean is high you don't want to change it because that's our mission somebody has told us don't don't mess around with the bright parts of this image but try to enhance the dark parts so what you do is is adaptively you compute the mean around each neighborhood and depending on whether that mean is high or not you you want to enhance it okay and and the second thing you want to do is if if if if if you found kind of a dark region and you found that the contrast is low and what do we mean by contrast being lower a very simple definition is that the variance is is small then you want to enhance that you want to boost boost the variance okay you want to I said you want to boost the signal you want to multiply by a large number like five or ten or four or some some number that brings the brings that out okay so there's two things going on if the mean is too small that means you're in the dark region and the contrast in that region is small that means the variance of that small region is small you want to boost up the signal so how do we do that what we use exactly to come back to the paper we use exactly these local mean and local variance quantities that we found okay so coming back here the goal is to enhance the dark areas while leaving the bright area as unchanged as possible okay and so to do that we first detect regions that have both the fault they have both of these characteristics characteristic one is they have to be dark and what does what dark mean that means a local mean has to be small small compared to what compared to the global mean and what do we write mathematically what does that mean it means that MSXY was definition we wrote in the previous page is smaller than some constant threshold k naught mg this was our global mean this is our local mean and this is some constant k naught that we arbitrarily choose between 0 and 1 and we wanted to have you want it to be dark but also it has to have low contrast okay so I wanted to have no contrast so if it is it is it is a dark region if you go back to the to the computer it was a dark region like this and it's got high contrast already you don't want to touch that too much all right so come back to the paper so low contrast which means that sigma s x y the variance the local variance is smaller than some constant k2 times the global variance so this is global bear this is the local bear and this is some constant k2 larger than equal 0 some constant this is also a constant and this is all good but but it turns out there's a bunch of regions here that that are completely dark and they're flat that means there's no object in them so if you end up enhancing that then you you might end up boosting noise so if the local variance is really really small which means that it was probably just a flat region with a little bit of noise added on how you want to leave that untouched so we're gonna add a third condition we don't want that we want the enhanced air hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon Fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon Saver 365 by Whole Foods Market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon Fresh select varieties we wear our work day by day stitch by stitch at Dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember that Dickies has been standing the test of time for a reason their work where isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to Dickies calm and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that is made Dickies a trusted name for over a century because they have low contrast but not too low of a contrast okay so if you can go back so condition C is wanting to have not too low of a contrast because we want to leave the flat regions unchanged flat regions that really didn't have anything in them to enhance unchanged so Sigma S X Y you want it to be larger than K1 verg where this K1 is a very small number like you know 0.01 or something and this this K2 we want to pick as kind of a larger number something like 0.4 okay and finally we pick this K0 again for this particular example as 0.4 so you can see this number it's very very low so you you just want to make sure that there's a little bit of contrast happening it's low contrast but not zero contrast not extremely flat so to summarize not that we we we figured out which regions we want to touch regions that are dark low contrast but not too low of a contrast then you can write down the processed image G of X, Y as a function of the unprocessed image in the following way is it's a scaled up version of the original image if these three conditions are satisfied in other words the local mean smaller than K0 global mean and the local variance is smaller than some constant times the global variance but larger than some constant times global variance and it's 0 otherwise so this is kind of the overall strategy one one can pick in terms of enhancing what this okay so how does this thing work in practice if if I go switch back to the camera I'm gonna show you three pictures none of these is enhanced picture yet so don't panic so this picture is the same as what what we had before except except that for every pixel we've computed we've replaced it by its local mean so again on your monitor it doesn't come across too well but if you were looking at the book or if you were looking at this high res monitor that is sitting in front of me this will be what how would this compare to this if you replace these brief super its local mean what effect does replace my local mean have smoothing exactly it's the same because you're replacing a pixel by average of the pixels around it averaging is a low pass filter operation with h of m1 and n2 is 1 1 1 1 1 1 1 1 1 1 so it it has effect of low pass filtering and the bigger the region is it you're doing the average in the more blurred the thing becomes and in fact that's one of the things I'm going to be showing to you in like I don't know ten minutes or something so so this is what you get by just low pass filtering local mean processing this is computing MS X Y for every pixel and replacing that pixel with that is computing the variance for each pixel okay and as you can see it's it's this is kind of local standard variation so these are the parts where you could easily see those things problems going on and these this is a binary picture that's obtained by looking at this equation that I wrote processed is if condition ABCs is satisfied and do nothing if those conditions all three are not satisfied this is a binary picture showing where conditions A B and C are all satisfied so if so is there a white pixel or completely black pixel if it's white it means condition ABC was satisfied it's black and it wasn't satisfied so you can see these these stripes appearing here that weren't there before and now if you apply this equation that I wrote down this is what you get and you see these oh sorry you see these these stripes appearing which I hope on your screen also does appear okay so so depending upon what's going on you might want to play additional games and and and another actually if anybody doesn't really have Jay Lim's book and not there's another thing about it there was a really good example in Jay Lim's book as well I usually bring both of my books I didn't this time because I thought I'm just going to use nice power points but anyway there is there's a picture for those of you who have the book there's a picture of an aerial image taken by an airplane from a local region where clouds are covering a certain region so what it is is that the the region where there's clouds the intensity values are high because clouds are white but the contrast is low so if you look at your image and look at the way it was acquired then you can devise these equations and these systems to figure out which areas you want to boost which areas you want to unboost etc so there's again I just thought about it now it's a dramatic example it's one of the good examples in Lim's book not all of them are very good so maybe next lecture I'll bring it out to show it to you but but here's another example where you want to boost contrast in in places where the intensity where the local mean is high that means there's a cloud okay any questions alright so let me move on if you can switch back to the camera down here let me look up to the move on to the next set of techniques so enhancements are using arithmetic or logic operations so what are some logic operations we all know about where you can do and you can do or you can do not for those of you who are circuit designers you know that these three operations are functionally complete that means that if you can do these things and then you can accomplish any any any logical operation that you ever needed actually Cindy isn't it true that if you have a NAND gate is that's all you ever need to do something the NAND and the NOR itself is functional complete NAND and NOR together not just NAND oh that's that's exactly what I remember so I don't I think and or and not might be just too much not quite sure but but from circuit design we know that NAND by itself is also functionally complete but it's very hard to kind of it kind of hard to imagine what a NAND process does to the pictures it's a little bit easier and or not as I'll explain in just a second so the not operation is the same as it's the same as just a negative transformation that we talked about many many weeks ago it's just you have a pixel you've got eight bits eight bit per pixel and you just revert the ones the ones in that pixel become zero the zeros become one and so it's kind of like building a negative and then or are kind of masking operations and so let me tell you let me let's switch back to the camera and let me show you what I mean by masking operation so here's a picture of I don't know it seems to me it's the Capitol Hill but the book doesn't really say what it is and here's on the top I'm doing the and I have an and image mask so what does it mean that mean this this is my mask it's black everywhere all the all these pixels have 0 0 0 0 0 as their intensity and every pixel in this region has one one one one one as its intensity the highest the brightest possible way now if I and each pixel here has eight bits as well so if I and this mask with this essentially I am grabbing this part of this building and I get this right and and you can imagine that if you didn't design your and mass to be all ones in this region and you know have I don't know four of them one and four of them zero you can get funny other logical operations at the end and here I'm doing the opposite this is an or mask where all the all the pixels here are completely white and the pixels that are completely black and I or this with this image and again I get I get the top of this thing it's just that the rest of the image here is black the rest of the image is completely white but you can you can play funny other games with this as well okay hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon saver 365 by Whole Foods market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon fresh select varieties we wear our work day-by-day stitch-by-stitch a dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the dickies has been standing the test of time for a reason the work where isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to dickies calm and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that has made dickies a trusted name for over a century coming back to to the paper I can also do another one of the arithmetic operations I can do is image subtraction as a mathematical operation so what does it mean is I have I have I process or enhance an image to get G of X from a Y by starting with F and then subtract him some some mask from it okay and the best way of looking at this this example for this again switch back to the computer is this this is the fractal image that we had I think I showed it to you last time or and the upper left and in here every pixel is quantized to eight bits here I have set the four least significant bits of every pixel in this image to zero and there's absolutely no wonder that visually you don't see any difference between the two things right I mean and if you really looked at it very very upfront and very closely and all that I mean I look at it here I can let me not lie do I see a difference I mean a little bit of a difference I actually my eyes aren't really the best I will contact mens is there like five and a half and they're not ever on the spot anyway I honestly I can't tell the difference in the book either so so what we've done is subtracted we've set the fully significant bits of this to zero and now we subtract this from that and we get this which again you see nothing but now I apply histogram equalization to this image and I get this so finally you can see this is the difference between these two things and so so the difference operation reveals the difference between two images that are important where does it get used the most in image processing community when you go to conferences I know you haven't been to too many of them or any of them maybe but where does it get used to used when people do compression and they say oh I did this on this and this to this video and and this is the reconstructed texture or it doesn't have to be video it could be just an image and and people say oh I can't tell the difference between the original reconstructed and say subtract when you subtract ideally what you like that to that signal is to look kind of whitish you don't want to discern any lines or features in that subtracted image that was in the original if you do that means your compression wasn't very good if you don't and it looks kind of like whitish then that that's a sign that that you've done a very good job right because if you see lines that means some line got got blurred out in the reconstructed images okay so another example of another real-life example of subtraction it's shown here so what we have here it's called mask mode radiology okay so what they do is they take an x-ray of a part of the a part of a body of a patient which is shown on the left okay I think this is of a of a head of somebody's head and supposedly here is like the spinal cord don't ask me why this is a head it doesn't look too much like a head to me either but then what they do is they inject a contrast medium like I think in this case is a iodine something something what's it called iodine medium iodine medium into the patients and and then they start taking a series of pictures with the game with it's an it's an x-ray video type thing so you get a series of videos coming on you subtract the mask image before you injected the dye from these images that you get after you inject the dye and by looking at this sequence of images you can figure out how the dye is propagating in the guy's head how the dye is propagating in the guy's head right and so you can see that these these bright areas exactly correspond I think to the veins or the forgot the differences in arteries and veins but the bloodstream let's say of this guy what's another example where you inject something the contrast agent and your grams right you I forgot what they inserted either here or here and they pass a tube and then when it's the right place in your heart they blow a balloon that opens up some arteries that were clogged before because cholesterol or fat or whatever the reasons were so there's another example where you take pictures and by subtracting these two from each other you can you can the surgeon or whoever is doing the injection can can see quite well what's happening actually when they do a spine injection I think they also put a contrasting agent especially for necks because you want to make sure the needle goes in with this it doesn't touch the spinal cord that the consequences are great if it does and and the x-ray that you use for these things if it is it's a TV x-ray where you're getting lots of pictures as a function of time it's very low dose because otherwise I don't know you get exposed to too much radiation but but but the point is the subtraction reveals what's what's happening there okay any questions okay so number the next thing that I'm going to talk about really briefly along the lines of enhancement using mathematical logic operation or mathematical operations is enhancement using averaging and actually I hesitate for saying that why do I hesitate for saying enhancement using averaging it's it's kind of an oxymoron term right why that's right because because in that case enhancement blurs it but but for some situation you can you can get a better picture right by doing well sorry sorry there's two kinds of averaging the example I'm going to talk about now is not local averaging of the pixel but averaging over a series of pictures right so for example in an astronomy you're you're stuck with imaging in very very low light levels and the sensor noise becomes very dominant but if you took multiple pictures of the same thing then then and then average them out you can you can average out the noise but the signal doesn't get averaged out and then you can drag the signal out of the out of the noise what would be another I'll write down the mathematics of this in just a second but before I go what's another example of artistic form of averaging where you've seen posters people take pictures of the same scene and then they average it out what's the most famous poster you've seen on that the picture of like Bay Bridge right if I if at night or some bridge if I took hundreds of pictures of it at every point in time let's say thousands even at any point in time there's a different car at a different location of the bridge and they each have it their own headlight so it's a black strip but the first image there's headlights here here here the second one here here here now if you superimpose these are on top of each other or averaged them out there's another word of saying that you get a straight line which is a superimposition of all the lights that ever occurred and all the thousand images showing the path of the cars I mean you guys seen this what am I talking to it okay so here's another example that that out or you can do a long exposure that's very true can you have as long as like an hour I think yeah I think the picture would look too blurred yeah but the 10 seconds is not enough I think to do enough averaging yeah anyway so you can do that it's true you can generate pictures I mean how the pictures were generated I mean with exactly what image processing is all about is from now on after you take the scores you never quite trusted source of an image because it could be processed like we process these guys it could have never been captured but what what is the definition of a picture when you look at the picture did that come straight from a camera did your eyes visually see that scene and the answer is many times no because of all the processing we end up doing on these things it's kind of a equivalent actually analogous to when you buy a CD why is it that when you go to your favorite artist concert it never sounds as good as the CD why do you think that is it's not that because during that recording he paid a lot of attention and he slept a lot the night before and the musicians were allowed none of that it's not not that because they got recorded and they got post-processed they applied fading they reduced noise they boost high frequency all of those things and on top of that it's in a controlled sound room but it's walls are designed appropriate hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon Fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon Saver 365 by Whole Foods market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon Fresh select varieties we wear our work day by day stitch by stitch a dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the dickies has been standing the test of time for a reason the work where isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to dickies calm and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that has made dickies a trusted name for over a century you know a million tricks went into making that CD sound fantastic now when you buy that when you go to the concert you're hoping that you can have that same experience but well you could argue though well you know a million tricks was played but it was recorded on a media and while as long as it was recorded there was temporal sampling there was amplitude quantization so so you might you know when I was a younger and I used to go to concerts I would say really I want to go to the concert because I want to hear the low-distortion high fidelity version of my favorite artists and then you go there and you're so disappointed and a huge stadium with the acoustics of terrible even from the stadium let's say it's Zellaback Hall the acoustics are terrible and besides on top of that none of the tricks that were played for post-processing that audio was ever applied so that that's why I never go to concerts anymore that it's buying the CD and live with you know the amplitude quantization and this screw sampling oh I love those because they're good and besides you know you can listen to it at your home as as you're reading a paper or or something like that okay so the same thing with images when I'm gonna show you a picture you know of this babe rich thing that in real life you might never observe that with your own eyes right it could be a result of a lot of manipulation and that and that actually by the way that causes a lot of issues for forensic type cases in in in courts where forensic experts start with images and then they manipulate it to enhance something versus another and you know it gets very close to this boundary of being ethical versus unethical if you if you process those if you process images okay actually the most famous cases is you can take a picture of someone that in the compromised position and just remove the head and put someone else's head and there you've compromised whatever you didn't like okay so enhancement using averaging so what do I mean by that so I've obtained some image g of x comma y which is f of x comma y plus some noise and strictly speaking you see we deal with noise and restoration and in the next lecture when we get to the restoration chapter before now let's just let's just for enhancement purposes let's talk about it anyway so we assume we obtained a series of noisy images okay and which we call g sub i of x comma y and and what we do averaging what it means is you just build the average of that by summing k of m summation from i equals 1 to k of g sub i of x comma y so if you do that then what can we say about the expected value of g bar and the variance of g bar well the expected value and and fundamentally what I'm assuming here that my noise is uncorrelated from pixel to pixel with itself and with the image and it's zero meaning that means if I if I take an average of it the the meaning is zero so what happens if I take the expected value of g bar I bring this in and it's the expected value of f which f is deterministic so it comes right out but because the expected value of this is zero this is zero mean what I get is f of x comma y so averaging is actually a good operation they expect a value of this is let's call this and if you're if you're dealing with estimation theory let's call this our estimate we have these and from these we want to come up with an estimate of the image this is how we do it so the mean of this estimator is the is the signal we were after is that good or bad good and how about the variance of our estimate g had of x comma y you can show is which we write by sigma squared of g bar of x comma y is given by one over k sigma squared eta of x comma y okay and so what happens is that as you add more terms so so what does it mean it means that this g bar is a random variable right and it's scattered all over the place it could be here it could be here et cetera but it's mean on average we expect it to be centered around f of x comma y and the variance it means how far away it is from the the mean the variability around that of that estimate goes down with with k that means when k the larger k is the narrower this is the more likely we're close the final g bar that we come up with is close to this so think of g bar as a random variable which has a certain mean and variance and we're saying the mean of it is f and its variance shrinks as k becomes larger so variance of g bar shrinks as k goes up and that's intuitively that makes sense is if you have more pictures and then they do more averaging than you expect the why is that well because the noise values cancel each other up you're you adding a random variable but then the signal components add to each other and so it was the signal levels so so here is an example if you can switch to the camera and again I don't know how well it shows up in the pictures there actually let me recommend because I thought about switching the room to somewhere where we have a much more high resolution projector but for those of you who can't make it because of the many schedule changes we did at the beginning may I message us that moving on may perhaps you bring your books with you castle man how many of you have bought cast on I mean not cast on like Gonzales and Woods almost all of you have bought it if you can bring it to class then at least for this portion of the course then we're not at the mercy of this projection system that we have anyway so what you have on the left is is okay I'm first to admit that this is a canned example and I and I switched to Gonzales and Woods because it has fewer canned examples than limb but this is a canned example this is an astronomical image and and this guy on his own he adds got additive Gaussian noise with zero mean and standard deviation of 64 gray levels so if you if you then average that eight times you get this averages 16 times you get that sorry this is the image this is after you just corrupted with with gas additive Gaussian noise of 64 gray levels with the standard deviation of 64 gray levels this is this is what you get if you average eight of these guys this is what you get if you average 16 of these guys this is what you get if you average 64 of these guys and it's what you get if you average 128 of these does it show in the picture that this looks closer to the original then let's say this one does okay so some things come true but it's much better so the more averaging you do the better you end up getting. Also the next picture is kind of an interesting thing it's the it's the difference between if you did one two hundred sixty four yeah so this is the difference between an image that was eight time average and the original image and this is pictorially the difference and this is the histogram of it this is the the same thing but for 16 time average this is for 64 time average and it's 128 times average and as you expect this signal becomes darker and darker meaning that if you do 128 times of averaging you should look similar to the original image more and more right so that's why you get black almost nothingness and as a result this histogram also gets shifted as we move on to the left this Gonzales guys is a is a big fan of histograms even when you go to the restoration stuff he keeps showing histograms it's not a bad thing but but it's not everything but but regardless you can see that the histogram of the difference becomes darker and narrower and shifted to the left as you do more averaging so why was this a candid example because he didn't actually get 128 astronomical images and averaged him he started with one that was good and he artificially added noise and that's why I don't like the example but at least the image itself is not synthetic it was a real image hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon Fresh shop Prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon Saver 365 by Whole Foods Market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon Fresh select varieties we wear our work day by day stitch by stitch the Dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the Dickies has been standing the test of time for a reason the work where isn't just about looking good it's about performing under pressure and last thing through the toughest jobs head over to Dickies calm and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century questions okay so next we move on to spatial filtering and a spatial filtering is nothing but linear shift invariant filtering of a signal that's the stuff we covered at very very beginning of the semester we even have done filter design on it which is what the thing is you handed out today so so basically what what do we have you have an image and this is an example of an FIR filter finite impulse response filter you have a 3 by 3 mask you you multiply so this is your your actual filter with different weights at that for these nine the nine values of the filter are the input essentially the impulse response of that FIR filter and so you multiply this this weight by this this weight by this this weight by this this by this this by this this by this this by this etc to compute and then add them all together and average it out in order to compute the output value at location in the middle and then you drag the window and you just keep kind of doing that it's nothing but FIR filtering so let me for the sake of completeness write that down to remind you for those of you who might have forgotten that so come back to the paper spatial filtering basically this is linear shifting variant FIR filtering where the output G of X comma Y is the convolution of the filter coefficient double which is this and the original image so s goes from minus a to a t goes from minus b to b okay so you can have you could you can have depending upon how you design your filter whether to be low pass or high pass you can have a blaring kind of effect or you could have a sharpening kind of effect so if WST corresponds to a low pass filter we expect to get blurring and actually blurring is not too bad for regions where there's noise but the problem is it also blares the sharp edges and you don't like you don't want to do that on the other hand if WSNT is high pass filter then you get sharpening now what's the problem with with applying a high pass filter to an image it sharpens the areas that you want sharpen like the edges get sharp and etc but what what problem is it if you apply high pass filter for image anybody like a differentiator an example of a high pass filter what's what's the problem with a differentiator yeah exactly the amplifies noise if you had noise in your in your image then then it gets it gets amplified so so the low pass filter it blurs edges what removes noise removes unwanted noise here it sharpens edges but accentuates noise and for that reason many times to improve the quantity of fixtures it's better not to do linear shifting variant operations at all or minimal and if you do it you ought to be aware that these problems could occur okay so here is an example of a applying a square averaging filter of different sizes to an original image so this is your 500 by 100 image and it's actually a pretty good image that this guy Gonzales has come up with there's little squares whose size increases from left to right these are the random noise areas this is wider than to go down here it's it's more pronounced as you go down and this is a letter a and for most times we have intelligibility in the images you want to be able to weed these things kind of reminds me of when I go to the optometrist but anyway and and these are equally spaced bars that if you end up learning you get terrible terrible blurring so so the image is 500 by 500 to begin with and here we apply it 3 by 3 f r filter and all the time when I tell you when I said 3 by 3 that means all the coefficients are 1 then we here we apply 5 by 5 9 by 9 15 by 15 and 35 by 35 so as as you can see this letter big letter a in the middle gets blurrier and blurrier as you move down these lines are discernible here discernible here by a time you get here you can't you can't really tell the difference between the lines the squares keeps getting blurred as you move on you can't even by the time by time you get here you can't even make out this letter by the time you get here you can't even make out these were all A's okay so but but what can we say good about this picture well it's we move the noise kind of all together this was the noisy part we're trying to get rid of okay and and for me I mean those of you you don't remember overhead projectors because you probably how many have seen overhead projector in your lives before good they all okay you're not all right all right everybody uses this this DLP thing we forgot but anyway what would be a good way of using an overhead projector to to emulate that just turn that knob the focus knob but what is that that's exact job of doing the blurring right or in your camera if you're taking pictures if you're out of focus then you get the blur image actually in your own eye if you become near side of the far side essentially the place where the light the light the focal point of your eye instead of falling right on your retina which is where this high sensitive rods and cones that takes the image and senses your brain it occurs either in front of it the focal point or or or behind it which means the picture on the retina is out of focus so I mean why am I going a long way what does this remind me of this reminds me of when I wake up in the morning and I don't have my contact lens on it's all there is right that's that you see this blurry kind of a thing and then when you put your contact lens in you turn from this kind of to that so focus being got to focus in any system electronic optical eyes biological causes a lot of blurring um you can also want one example where the averaging can be useful is you start let's say with an image from Hubble telescope there's too many stars here you you you you blurried out by doing a 15 by 15 averaging and in the in the process of doing that you got rid of a lot of the little noises and then you do thresholding so you end up with big stars so here's one example where you can use this for enhancement purposes okay okay so linear filters we haven't really talked about high pass but so far we're talking about low pass filters this problem of blurring one good example of enhancing images is for both enhancement and restoration is to apply nonlinear filters for example median filters let me come back to the paper and tell you what I mean by median filter or generally speaking using what's called order statistics and the best example is median but you can also have things like max or min and I'll explain that in just a second well what how does median filter and work where you look at a five by five or three by three window you compute the median of intensity values over that window and what does median mean I mean since you're students and you're not not you but other students are so great conscious we all know what median means median is a number at which half the class is below you have the classes above you right so so so I rank order these 25 numbers and I figure out the point at which the intensity value at which 50% of the pixels are smaller and 50% are above and so I replaced them the intensity packs pixel for the middle point with that median value and then I shift the window by one hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon Fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon Saver 365 by Whole Foods Market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon Fresh select varieties we wear our work day by day stitch by stitch a dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the dickies has been standing the test of time for a reason the work where isn't just about looking good it's about performing under pressure the last thing through the toughest jobs head over to Dickies comm and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that has made dickies a trusted name for over a century and I keep doing that okay and how about max well max is the same thing you look at the window but instead of computing the meaning you compute the max or the 0th percentile and for me and you can sorry 100th percentile and for me and you compute the 0th percentile but by far median filter is the best way of doing this and medium filters are very good as we'll see when we talk about restoration but we can also see it here for removing what's called salt and pepper noise and I'll show you what I mean by just a second sometimes they refer to that also as impulse noise okay and a lot of times when there's shot noise in your electronic or optical system you get this salt and pepper noise so here's an example of of a printed circuit board okay no no this is not a canned example so so we're on a good side of Gonzales again so so here's the original printed circuit board that's that's corrupted by salt and pepper noise I don't know if he added it or if he it was captured as is this is if what happens if you do 3x3 averaging it's still quite noisy and then if you do the 3x3 median filter you can see that if a lot of the noise went away you can see the sharp boundaries here the the grainy appearance that here has gone away so this is a much cleaner version of what we had before okay any questions okay so next I'm going to move on to sharpening spatial filters okay so let me let me go back to the to the paper for just a second before we go back to this so the objective is to what to highlight the fine detail in an image or enhance some detail that was played so highlight or enhance some detail so if you have then if I'm dealing with spatial filters that's another code word for filling is an LSI filter and it's an FIR but this time it's instead of being low-pass it's going to be high-pass filter okay and one of the best kind of techniques to kind of to enhance details and it's to apply derivative operators okay either first order or second order first order means the FDX second order means D square FDX square okay and what are what are some kind of desirable how do we define these derivative operators given that we know what derivative means in my calculus mean calculus terms but we don't really know what it means in terms of when you're dealing with images which have discretized values so ideally what you want is for example the the first order derivative what what you what properties if you have it in discrete domain do you want it to have well if it isn't applied to a flat region and image you want it to be zero if it is applied to an onset of a ramp when there's a change in an image you want it to be non-zero if it is a longer ramp with a finite slope first order derivative is non-zero the second order derivative is the weight of change of the slope and if it is applied to a ramp for example because this slope of a ramp is constant you get zero so I mean all the usual intuitions that that we have with derivatives so if you have a picture what's the best if you if you have a one dimensional signal okay what's the best way of defining the derivative at all if is f of x plus 1 minus f of x is one possible example definition or you can just say if it's f of x minus f of x plus minus 1 that doesn't really matter and the second order derivative one possible definition is f of x plus 1 plus f of x minus 1 minus 2 f of x and how do I get this will I apply the derivative operator to this right so applying the derivative operator to that you quickly get this thing so just I know this is a trivial example but for the sake of reminding you what these why we do the derivative operator and how these things behave on let's take a let's take a look at this picture on the computer again so here's an example of a picture and I think this is a synthetic image and what this guy has done he's taking a horizontal cross section along this line and if you do that this is the profile kind of that you get so you start off here it's it's bright the intensity goes down kind of linearly then it becomes completely so that's this part then it becomes kind of flat black for a while that's this part then you hit the dot and it goes boom up and down and then it's black again for a while which is this one then you hit the line it goes up and down and unlike what you would think for a line it's not a square thing but it's kind of a triangular thing and then it's flat again black which is this region and then you hit the white ramp which is this thing okay so you apply the derivative operator in order to kind of detect the changes that's going on in an image and and what he's showing here is kind of the caricature or the illustration or the graphics of this thing okay so this ramp here is the same as this ramp this one here and then this guy is the same as this this peak is the same as this that's the isolated point this is the flat segment this is the line the thin line which is showing here and then this is the step so if you look at the image strip along here these are the intensity values that you get that what you see here is first derivative this is a second derivative so first derivative along here is minus one because it's a constant negative slope and then becomes zero corresponding to this region and then it's plus six boom goes up and minus six boom comes down zero zero zero and then another bump but this time not six but one or two and positive two going up and then minus two coming down minus one zero for a while and then boom a big slope of seven and zero okay now this is the second derivative is so the second derivative is the derivative of this guy so a minus one minus one minus one a bunch of zeros here it becomes it's a doublet it goes positive plus six minus twelve and then plus six again so it kind of goes like boom boom and then boom okay and then another thing so in general you can see that this which one do you think is better for image enhancement application for kind of enhancing details first order or second order actually this doesn't show you this example is not the one that answers you this question but in general for enhancement applications you better off using the second order derivative and when we get to the edge detection part of the course which is later on talk about segmentation analysis edge detection you'll see that you can use either first or first order derivatives or second order derivatives for doing for doing edge detection but for for many enhancement applications second order is is much much nicer but but nevertheless and and and you can see that the second derivative has has a lot more zeros it only it only flares up when an activity is going on like this or or another kind of activities going on like at the thin line like that so coming back to the picture to the paper here what's what's the what second order derivative shall we use the simplest isotropic one is is the del square operator which is the Laplacia which is d squared f dx squared plus d squared f dy squared and for this guy you apply this you can apply that so d squared f dx squared can just be f of x plus y comma y minus f of x minus 1 comma y you just add a y minus 2 of x comma y and d squared f dy squared you can do f of x comma y plus 1 plus f of x comma y minus 1 minus 2 f of x comma y and now to compute del square f all you have to do is add these two things up can you roll up please great so for del squared f, you just add these two things up and what you end up getting is f of x plus 1 comma y plus f of x minus 1 comma y plus f of x comma y plus 1 plus f of x comma y minus 1 minus 4 times f of x comma y. So if I were, so what does that mean? That means if I, this is L squared, an operator applied to f, it gives me this. So is this guy L squared f and f y r filter? Absolutely. And what kind of impulse response does it have? Well, minus 4, 1, 0, 1, 0, 1, 0, 1, 0. That's exactly what it is. This minus 4 is this one. These ones are here, etc. So this is n1 and 2, etc. Okay? And if you, you can show that if you include diagonals, then the impulse response will become minus 8, 1, 1, 1, 1, 1, 1, 1, 1. So right now these diagonals did not contribute to the second order derivative. You use that to get this. Okay? Okay, so what does all of this have to do with enhancement? We're kind of running out of patience and running out of time. We have g of x comma y is nothing but f of x comma y minus the squared f of x comma y. The enhanced version is the original minus the kind of a high-passed version of the signal. Okay? And why do I have a minus sign here? Because this is a negative here. So I want minus, minus, I want to get plus here and then subtract these things from it. So let me show you an example of this operation. Yeah. So let me show you an example of this particular operation applied to the picture of North Pole of the Moon. That's the original picture. This is what we get if we just apply the Laplacian. That means the del squared f. You apply the del squared f operation on top of this. And as you expect, what do you get? A high-pass filter, the high-pass signal. The places where there's sharp boundaries, it kind of detects it. And there's a bunch of scattered white points here showing that there's a boundary change here. Do you see these white stuff on that screen? Okay? Now, this is the same image. This is the same picture as this, but just scale so you can see it better. Just scaling operation. And this is the image that you get by f of x comma y minus this. Okay? So you can see that there's quite a bit of sharp, does this come across in the screen? I think this doesn't. There's quite a bit of sharpening up here along these things as compared to here. And this edge is kind of much sharper. Okay? So, I guess... Another variation of this thing, just for the sake of completeness, let me just show it to you. So another variation of this thing, so this is the mass that we just derived, right? Minus 4, 1, 1, 1. And then I said if you add diagonal, you get 1, 1, 1, 1, everywhere else. Now, you can kind of do the negative of this. Sometimes people in the literature refer to the negative of this thing. Okay? So what does a negative of this thing mean? Minus 4 becomes 4 minus 1 minus 1 minus 1. So in the... And this is kind of the negative of this thing. So if you use these guys, then instead of having f of x comma y minus del squared f, you end up having f of x comma plus del squared f. But you shouldn't get tricked by that. What is multiplying the filter coefficients by minus 1 due to the filter and the frequency domain, any one? It shifts it, right? So minus 1 times times the filters, let's see. It's the same as multiple. That's correct. The magnitude stays the same and the phase, that's correct. The phase only changes. It's equivalent to multiplying either the j pi and the frequency domain. Okay. So the point to make is that you can also come up with variations of these as well. So this is 4 here, 8 here, etc. You can come up with a filter that is 5 or that's 9 here. It doesn't really matter. But when we make one more observation about this filter, if the sum of the filter coefficients here is here, here, here, and here is what? Zero. And what does that mean? If the sum of h of n is equal to zero, that means the dc value of the filter is zero. That means the frequency response at omega equals zero, zero is zero. And what does that mean? It means it's a, not a low pass filter, but a high pass filter. So that's exactly what we would expect it to be. This is, this is a, they're both high pass filters. And coming back here, this is the last thing I want to show before we break up for today. I'll talk about unsure of masking and high bus filtering next time. But so, so here's another SEM picture. I think it's, it's of a, of a tungsten filament or some, such thing like that. And down, down here, yeah, down here, I apply this, this Laplacian. And here I apply this other Laplacian to it. And as you can see, this picture is, is quite a bit sharper than, than this one. And this is, this can be, it's not, this is, if you think about it, neither one of these is strictly speaking a high pass filter, because at the seat of value of this thing is one, the value of this thing is three, six, eight. This is also one. They won't have the same dc, but this thing is the next century. So you can play with the center values of these things in the shape of your filters in order to bring, bring out details, contrast and various other things in your picture. And all these are, we have to remember, is linear shift invariant filters. All we're doing is the FR filtering that we've talked about, dealt with, you've done home works on, et cetera, et cetera. I'm going to stop right now. Next time we'll, we'll do unsharp masking and high bus filtering. And then move straight on to, to restoration. Hey Amazon Prime members, why pay more for groceries when you can save big on thousands of items at Amazon Fresh? Shop prime exclusive deals and save up to 50% on weekly grocery favorites. Plus save 10% on Amazon brands, like our new brand Amazon Saver, 365 by Whole Foods Market, a plenty and more. Come back for new deals rotating every week. Don't miss out on savings. Shop prime exclusive deals at Amazon Fresh. Select varieties. We wear our work day by day, stitch by stitch. At Dickies, we believe work is what we're made of. So, whether you're gearing up for a new project or looking to add some tried and true work wear to your collection, remember that Dickies has been standing the test of time for a reason. The work wear isn't just about looking good. It's about performing under pressure and lasting through the toughest jobs. Head over to Dickies.com and use the promo code Workwear20 at checkout to save 20% on your purchase. It's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century.