The Bold Blueprint Podcast
The Bold Blueprint Avideh Zakhor Cultivate a Growth Mindset
Hey Amazon Prime members, why pay more for groceries when you can save big on thousands of items at Amazon Fresh? Shop Prime exclusive deals and save up to 50% on weekly grocery favorites. Plus save 10% on Amazon brands like our new brand Amazon Saver, 365 by Whole Foods Market, Aplenty and more. Come back for new deals rotating every week. Don't miss out on savings. Shop Prime exclusive deals at Amazon Fresh. Select varieties. We wear our work day by day, stitch by stitch. At Dickies, we believe work is what we're made of. So whether you're gearing up for a new project or looking to add some tried and true work wear to your collection, remember that Dickies has been standing the test of time for a reason. Their work wear isn't just about looking good. It's about performing under pressure and lasting through the toughest jobs. Head over to Dickies.com and use the promo code workwear20 at checkout to save 20% on your purchase. It's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century. Are there any extras left over? Okay. So let me just quickly go over the selling points of the hander. I won't I won't explain everything because otherwise it would take too long and what's the point of the handout if I go over everything already. So let's see. There's two choices. You can either do a term paper on a topic of your choice and I'll explain what are the things that I expect in a term paper or you can do a project. In either case the topic of what you work on needs to be mutually agreed. It can come from you. There's a bunch of suggestions here and kind of the process that we go through is you send that you submit a one-page proposal not this Friday but next Friday. I'll take a look at it and then the following Wednesday if I feel that the project or the paper or the topic you're proposing is too far off from the main topics of the course or it's too ambitious or it's not ambitious enough or it's too vague I'll give you feedback. So we'll kind of try to converge on this project choice by the end of the by mid-March or maybe March 24th or so before the spring break begins because March 24th is the Friday before the spring break and if you end up so what I've done here is I've suggested a number of projects and also a number of paper type topics. Many times there's a for example section one it's paper or project. I'm suggesting something and you can either do a literature review on it or you can actually do a real project. There's there's going to be a presentation on the last Friday of the class which is Friday May 5th if I'm not mistaken. Is the last of instruction May 12th or May 5th? I think it's probably May 5th. I'm almost sure of it if it is not we will just make it the last day of the class and depending upon how many groups you've divided yourself the presentations might take just one day or might take two days that week. I also reserve the right to possibly have the presentations occur during the schedule final time for the class. We'll just meet here and everybody's a presentation. Don't buy your plan tickets yet or don't don't bank on the time of the presentations yet. Well as we get closer depending upon how fast and how much of the material I need to cover we've covered. We might we might use the last lecture of the semester or we might just end up using the schedule final time. I've done it both ways in the past. So if you end up doing a project it's very important many of you are doing projects that you've already done something on and it's an existing project that you want to enhance with something related to image processing. It's very important to in your proposal to let me know exactly what you've done in the past where you are now and exactly what you hope to accomplish as part of this course. So that it's clear that you're not going to just recycle something you did for your master's thesis three years ago as a term project for this. So we need to have a little bit of a check sign balance. At the same time I don't want this project to get out of hand. Roughly speaking let's see how many weeks by the time you start working on this which is let's say next Monday we have approximately seven or eight weeks to the end of the semester right and I don't want you to spend more than five to ten hours per week on it. So the total amount of time you invest in this should be no more than five times seven is thirty five or upper bounded by seventy somewhere in that range. It's supposed to take roughly the same amount of time it took you to do the homework because and and there's probably one or two other lab assignments so take take that take subtract some time off of that. So don't don't think of it as a humongous thing that you have to do. The other thing to notice is that I would strongly encourage you regardless of whether you do a paper or a project even if you do a paper to team up with each other because many of the topics I'm proposing has have a very broad and a lot of components so what might make sense is to say okay you know you make a team of three to five people and one person does investigates this branch the other person investigates this branch and and even if you want to compare the results it turns up being a project and it's much easier to have more people on your side. But at the same time you have the flexibility of doing a single investigator type project. I'm not gonna frown upon it but I would encourage the team up with your classmates. Initially I put three topics done on paper last night and then when I woke up this morning I said gee what if they really don't like those topics so I added two more. For a class of size 20 I think five topics is enough. However if you don't like any of these and you don't have any idea what other topic you want to work on come talk to me I can still I can just kind of interview you a little bit find out what your interest are and then I can suggest something else okay. So don't despair if what you read here is just way over your head you have no idea what it determines me and how it works etc. Come talk to me either after class or an office hours I'd love to talk to you about about this stuff. Now so for projects it's very important as I said to delineate what you've done already and what the increment that you plan to do next is. For papers let's say you pick a topic that you want to work on the kinds of things that you want to do I've written it here but basically you want to do a literature search you want to know what the state of the art is in this field for this problem what are the techniques people use you want to provide a bit of a critique I mean one of the things you'll do maybe not so much as a graduate student but once you get your PhD and leave Berkeley's scientists are always evaluating each other's work I mean God knows how many trips we make to Washington DC on this NSF panels to read umpteenth proposal rank them up critique them this was good this was bad I would solve this problem this way there's a flaw with this approach so you or you review other people's papers even if you're not reviewing proposals you you keep when you submit a paper others review it for you when you sit in your office you get bombarded by papers to review for others so so besides just reading the literature once you read enough of a paper in an area you should be able to judge the quality of papers you know first of all you do that by by seeing how well the paper is written but also in the context of that field at the time that paper appeared did it really offer a significant novel incrementer to the field was it just a recursion of what was there already or was there a delta step up and sometimes these things get lost because the papers kind of get piled up you know you look at 20 papers forgot to read the forget to read the dates and to get to read the the precedence of the papers so generally the researchers who come up with an idea first and kind of lead the research community in that direction from that moment on art those are the ones that get the most credit and those are the papers that should get kind of rated the highest what was there before that paper and what new thing did that paper suggest that made hordes of people follow that and continue that line of work you know kind of break through I just you know after 20 years after hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon Fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon Saver 365 by Whole Foods Market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon Fresh select varieties we wear our work day by day stitch by stitch at Dickey's we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the Dickey's has been standing the test of time for a reason the work where isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to Dickey's dot com and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that is made Dickey's a trusted name for over a century their people is published the significance of that original people kind of goes down people can forget about it the students only read the most recent people do they forget who proposed it first but it's very important to kind of critique papers in terms of what was there before and what the paper introduced and then the third component I want to make sure is in your papers is kind of it's it's kind of a thought process or a hypothetical process is imagine yourself wanting to do a thesis in that topic and I'm hoping that that topic is interesting you know that you can for a few minutes or few few weeks you can imagine that what what would you do what what are the air what are the approaches you would pursue what are the techniques you would pursue of all the techniques proposed to solve that problem which ones do you think are most promising where would you put your energy resources and time if you were going to spend the next I don't know three to five years doing a you know PhD thesis on this okay so having so so I hope you have a kind of a clear idea the besides the proposal the term paper is due I said that somewhere here did I put this down for that to do it at the end yeah do I say that I can't find it myself oh okay good I should have put all the deadlines in one paragraph okay good I did I did pick me 15 so Monday May 15th at Rosita's office at 9 a.m. and they the the people who turned tend to do a look just a literature review a report that no more than 20 pages double spaced people who do the project report that's no more than 10 pages double Swiss he doesn't mean you have to hit 10 and 20 it has to be upper bounded by that so a five page report or a 10 page report is it's just as acceptable as 10 and 20 again I don't want you to make a monstrous big thing out of this thing something that you feel that that exposed you to a part of image processing and made you learn something so that's that's all I have to say but logistics under any questions or comments okay so I will very briefly go over each of the topics that that I mentioned some of them you know some of the things I put down on this paper is really verbatim some of the proposals I've recently written and got funded so or not fun that I can't remember it's a mixture of everything so so some of them come across more like as a proposal but and there's many many subtopics within there we propose to do this we propose to do that so don't get thrown off by that language you can pick any one of those areas and dig into it deeper or you can pick a collection of the subtopics and dig into those so to don't don't don't despair so topic number one is kind of in the intersection of multimedia and networking there's a new thing called overlay networks that people have come up with in the networking community that facilitate many functionalities that are not easy to implement on the internet basically internet was designed with quality of service not in mind which is why we have such a hard time sending audios and videos and other real-time communication stuff on it so I first list a bunch of approaches that people have applied on the traditional internet and now I'm asking you given that over the networks are gaining more popularity what are some ways that we can use these techniques to come up with ways of taking advantage of overlay networks to facilitate multimedia communication so if you if you don't have a very strong networking background haven't taken one 22 or haven't taken possibly CS 2 68 this is probably be a hard topic for you to to tackle paper number two is is also kind of an intersection of multimedia and networking it has to do with wireless lens and video or multimedia communication or video transmission over those and so there's a lot of recent buzz about coslayer design where you let what do I mean by coslayer where you can think of communication over networks as having a bunch of layers all the way from physical to link to transport to I started the IP layer which is the routing then the transport and then the application layer and so people are suggesting ways and these abstractions are made so that to make the design of the systems easy so that you can also plug and play different systems to against each other and people have come up with ways of linking these different layers in order to facilitate wireless multimedia communication and I've put down one two three four five six seven papers a lot of them is this little like a lady who's actually whose name I can never pronounce even though I just for the recommendation letter for him I still can't pronounce his name it's a it's a Dutch name but nevertheless you know his first name Reginald this is first name so six of them are by him and the seventh one is by Magda al-Zerty who's also quite famous in the field if you again if you haven't taken this simple course like 122 this probably be a hard topic but but very worthwhile to investigate or to write about number three three and four is is what I added this morning when I just woke up I said look these guys might not have too much choices so number three is is to be a paper or a project is content based search and retrieval for video and images so as I'm sure you know by playing with search engines like Google Yahoo and Amazon and MSN there's there's and with the intent of I don't know mp3 songs and videos and videos on hand holds videos on your vcast cell phone videos on your cellular cell phone which is a little just announced yesterday they launched radio services there's lots of multimedia data sitting around in the world and the question is how do you do the same way as we do Google searches on text how do you do search on retrieval on multimedia data and the the latest issue of IEEE Signal Processing magazine which I just got in my office yesterday is a whole thing is devoted to this so you can have the reason this is you want to look at the latest because they all refer to the older articles so it's a fantastic topic for you to do a literature survey or paper on now if you want to do a project on that most recently missed announced a competition where they whereby they provide it's a fairly structured competition with lots of boundaries and definitions and stuff like that so they're providing people with a bunch of data and I forget how many hours of video it is and they've already segmented it into shots for those of you are not familiar the shot has a specific definition a meeting in video it's if I'm taking if I have a camera if I'm taking a picture a person a shot is set of contiguous frames that correspond to the same locale if I just then move to a different part of the room and that's a different shot so the shot segmentation of these videos have been done and they have a bunch of competitions I think three or four different categories of competitions like kind of like olympics where one of them is they provide a set of keywords the mic is off it says on oh power off off no on is it on now I didn't change anything is it working okay well that that might provide a lot of motivation for people to attend but we don't want to penalize them actually they can't hear us so we can say anything we want about them no just you're okay mute on power off power on so I didn't change anything oh this is the mic as well okay I'll try not to move too much okay so hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon saver 365 by Whole Foods market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon fresh select varieties we wear our work day by day stitch by stitch at Dickey's we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the Dickey's has been standing the test of time for a reason their work where isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to Dickey's dot com and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that is made Dickey's a trusted name for over a century yeah so so so they they released a ton of video I forgot I think a hundred hours but but the video that's been pre-processed so it's divided already into shots and and what you can do is and then they've also released a bunch of keywords I forgot 40 keywords and these videos also have audio associated with them some of them actually in English some in Arabic some in Chinese don't ask me how they pick those languages and the competition is on you you supply the keywords the one that I am interested or I find most interesting let's put it this way there's three others as you supply a keyword and you're supposed to retrieve pieces of multimedia video and audio that are synced together that refer to that keyword the keywords are things like I don't know fire terrorism I can't remember all the details some of them are some of them you definitely have to use the speech recognizer and the transcript of the of the click in order to recognize this for example democracy look how can you look at the video and say oh that's about democracy you can't it's it's the words and the audio that's important in detail in classifying that as a as a as a democracy piece right and then there's some other ones that are more action oriented like I don't know fire like that's their visual you know I see flames all over the place so and I'm just drawing blanks with all the other keywords but there's 40 of them and so if you want to do a project you can download these things and I've been happy to help you I don't put a lot of detail in this mainly because I did this in the morning but but if there's interest from any of you I can put you in touch with the right people to get access to the data to do some interesting stuff about it topic number four so that it has to do it's a fairly mathematical topic for those of you who are interested in more theoretical stuff has to do with signal expansions along over complete set of basis functions and applications to video clothing so there's been a lot of work on signal decomposition on over complete basis functions and the application to video one of them is by Ralph met on myself in which was published in 97 and since then we've published a lot more but it's called it's based on a technique called matching pursuits and but most recently folks at Donahoe who used to be at Berkeley but unfortunately he moved to Stanford in the stat department he has come up with a technique called basis pursuit and I'm happy to say he came up with basis persons while at Berkeley Stanford and you can go to his webpage and look at some of the papers actually you know what I noticed the the word before done the space before the word Donahoe on that web address and the paper should have a tilde and it doesn't that's on page four so you might want to add that tilde there so so this precise matching pursuit best best basis pursuit is yet another way of doing things and then a third way of doing sparse signal approximations okay why are these things important is because they enable us to represent signals in a most compressed efficient way and you'll learn more about these when we take get to the compression part of the course so fortunately there's a bit of a kind of a shift or a mismatch between what we cover now and what some of these things cover so so that the third technique is called SBL which is a sparse Bayesian learning and people like Tipping and Delgado and Raoul have all and the references are at the end of the paper have all come up with with with doing it so if a group of you wants to do a project and on top of all these I have a grad student here who is working on SBL and so SBL is kind of the newest and the most in my opinion the newest and the most exciting on these techniques now to get SBL to work you need to do very very large matrix inversions and we've already gotten a bunch of clues as to what techniques for matrix inversions you should use and we could even use the millennium system in the CS cluster to do that but you really don't have to do it you can do it on regular solaris systems and it it turns out it takes about for a 128 by 128 image it's an iterative SBL is an iterative process and it takes about 10 minutes the size of the matrix that you have to invert is 10 thousand by 10 thousand for a 400 by 100 image so as you can see the size of the matrix grows square of this dimension of the image but even for that for that size image it's it's no more than 10 minutes to do an inversion on today's computers and fast processors so if you can decide to do a project it'll be fantastic to kind of implement SBL for a long image on on a computer and and investigates its its compaction or properties this is something nobody really has done I mean if you look at Delgado's paper they they go as far as an 8 by 8 blocks in an image so would be interesting to know if you go to larger if you apply this technique to an entire image either 200 by 200 or 100 by 100 using you know fast inversion algorithms how much performance improvement in terms of compression you can get out of it or if you don't want to get your hands dirty programming you can do a paper in this area read read some of the stuff I've listed here and kind of provide an expose as to what the pluses and minuses of these schemes are and you know the usual three things I ask in any paper the last topic is it is really very blue sky open and that this is the one that I just cut and paste from my proposal so it has to do with the project that's been going on in my group in the last three four years on 3D sitting modeling and most recently we've kind of gotten additional support from the government to continue that work and the gist of the project is that we we built a system a ground-based system that that collects laser data from the from the facades of the buildings then we hired airplanes and helicopters to do laser scanning of the top of the roofs so from that we built a three-dimensional texture for the roofs and then from the ground-based laser system we built the three-dimensional models for the facades of the building and the key issue in this whole project is registration how do you register and fuse this multiple modalities of beta there's there's there's texture which is intensity to be collected from ground and sky then there's geometry to be collected from sky and and ground sky and ground and so how do you fuse these things so we've come up with methods to do that and I've listed the five four five papers by Brewer and myself that that addresses this where to go from here well there's essentially section 5.4 and 5.5 are the possibilities and again what I've put down here is very much image object recognition target recognition centric for example what I'm talking about here Okanada we have it so so there's let me backtrack this method that I just described you can go to our web page download 100 megabytes worth of zip data which is a 3D model of downtown Berkeley and for those of you who are video game generation you can plug it into your cosmo player which is a VR remote viewer and interact with that model the same way as you interact with your video games like you play Madden football or baseball and there's all these 3D things you've got straight not in your head and by the way your generation is supposed to do a lot better in this 3D aspects than our generation because you play so much more video games I actually I played video games too but mine were strictly 2D so my 3D sense it's not as developed as even my 12 year old son but that's a different point anyway you can download these 3D models you can plug it into your cosmo player and there's a rotation button and there's a zoom button and there's a pen and tilt button and you can fly over the model like fly over Berkeley and you can come down walk over Berkeley hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon Fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon Saver 365 by Whole Foods Market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon Fresh select varieties we wear our work day by day stitch by stitch at Dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the Dickies has been standing the test of time for a reason their work where isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to Dickies calm and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century you can drive through Berkeley whichever way you control this it's kind of like a flight simulator if you will and and if you don't like to do any of these things again on my webpage you can download the video that shows all of these things and watch the video in any event so so the models are there and what section 5.4 and 5.5 talks about is okay now what are some interesting things we can do those in those models or if you sprinkle them for a lot with lots of surveillance cameras and network of cameras you can do a lot of tests that people have traditionally done an image processing with a single camera without the knowledge of this surrounding scene around them you can do them much better by having a network of cameras sprinkled throughout this 3D world whereby each camera also has inherent and intrinsic knowledge of the 3D model where it's sitting with respect to the world and carry out traditional tasks such as tracking an object such as surveillance did anybody pass this this part of the street in the last 24 hours who did how fast are they moving where did they go this camera can track it for this piece and then we laid that information to the next camera I said oh yeah that's the same person I track you not track it etc so there's a bunch of tasks that you can do can we can we sprinkle the cameras in such a way that that we have complete knowledge of what's happening in a particular region without missing a car passing by or pedestrian passing by even with the 3D modeling aspects there's a lot of loose ends that interesting projects of paper can be done if if you model downtown Berkeley and don't simplify your models you end up getting huge amounts of data so applying simplification techniques that preserve the architectural details and keep the buildings as a box like thing rather than a skier type of thing if you apply traditional simplification for example to the geometry you you'll end up rounding the buildings making it look like a ball and all kinds of bad things happen if you don't simplify it the models are too detailed to be viewed with any realistic VRML viewer like cosmo player so you're kind of stuck between the rock and a hard case so coming up with simplification techniques that that effectively simplify while keeping the building architectures as square and Manhattan like as possible is kind of important indoor modeling of buildings that could be a very interesting term project what are some pre kind of a literature review of what people have known for indoor modeling as opposed to outdoor modeling and again I have ideas on how to do that once you have the 3D modeling you've got sprinkled a bunch of cameras you can you can then build models of the moving objects using these cameras transmit this to some sort of a central station and have those videos and models be inserted inside the pre-existing static 3D model that you've already constructed so you can imagine that we put a bunch of cameras and that's on shadow cavernous right and but and we already have a model of that shadow cavernous region which we do and and people are in some CIA headquarters or some whatever are watching the 3D models but then you can insert the dynamic objects or the moving objects from that region into the 3D model and have people view the traffic and the moving objects remotely in that model so there's a whole lot of things you can do here and I've listed some of them if you there's almost too much listed here so if you find yourself not he doesn't mean your paper has to add this all of these things right you can pick just a small aspect of it a piece of it and investigate that okay and at the end there's references I it's kind of a mixed thing I put some papers for some of the topics in the middle and the middle of the text and some of them at the end but it shouldn't really matter too much so let me stop at this point and see if there's any questions questions comments okay how many people have already more or less decided what they want I think we did the show of hand before have more or less decided what they want to do oh wow excellent okay so those of you who are still thinking feel free to you know contact me come come to the office hours and we'll get we'll get going on it any questions or comments at this point oh sure I'd be happy to talk and you don't have to but if you want us to converge faster yeah yeah 5 to 14 we'll cross let me let me look at so when was the 15th of me oh okay yeah we can we can accommodate you okay so I'm gonna go back and talk about the histogram equalization which was a method for sharpening or method for enhancing images so what we talked about if you can if you can possibly put the camera on the on the monitor yeah okay so what you see in these pictures are this this pollen SEM that with different levels of darkness and different things added to it and it shows the histogram corresponding histogram for this image and as you can see the the most appealing one is the bottom one it has the right level of sharpness and contrast and if you look at the histogram you can see it's more or less flat so what we decided what we concluded last time it's it it's a good what's a good thing to do is to if you start with images whose his ground histogram is not equalized like this one which is shown here if it is too dark or it's too bright or it's a low contrast image which has a compacted histogram here one way to quote-unquote fix that image or to enhance that image is to make transform each of the pixels using a simple transformation technique or devise a transformation technique that that will will flatten the histogram which is shown here and so what I'm going to talk about now is is today is really a way of coming up with the pointwise transform R is the intensity back to of a pixel and we're going to and we're assuming it's normalized between 0 and 1 we're going to come up with this curve T of R which match which which maps R sub K particular K intensity level to S sub K so that that pixel gets transformed to that to that value and then thereby causes enhancement so the question is how can we design this T of R and the answer is that we can do that but just looking at the intensity values of the pixel itself so it's a signal adaptive transformation if you will we build this transformation by looking at the histogram of the image itself and and I'll come back to this picture in just a second but if you if you apply this histogram equal I talked about it a little bit less than what if you apply the histogram equalization to each of these dark or bright or low contrast images using the white transformation then you can end up with these images that hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon Fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon Saver 365 by Whole Foods Market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon Fresh select varieties we wear our work day by day stitch by stitch at Dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the Dickies has been standing the test of time for a reason their work where isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to Dickies calm and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century that are all enhanced and good so so there is light at the end of the tunnel and as an example for for these four images and these are the four transformations as input versus output this is R this is S these are the four transformation number one number two number three and number four that were applied to these four images that that resulted at the end to get a histogram equalized one so you want to come over the technique that no matter what's wrong with your image the final the final process image looks good right so so so what is what is that methodology I will I will talk about two things one of them one of them is called histogram equalization whereby we changed the we reply transformation where the resulting image has a uniform probability distribution histogram and then the second method is called histogram matching where you have a desired histogram you want the output image to have a desired histogram or desired probability distribution function and and now what transformation do you apply in order for the final output to have that distribution so those are the two things that I want to talk about and in an informal way in the last lecture I already talked about both of those how we do that just kind of the mechanics of it but but in today's lecture I'm just going to provide a little more mathematical analysis of that that technique so so today's lecture is really on mathematical aspects or derivation really of histogram equalization and and just so that because last time's lecture was on Friday and it's been five days since you saw this let me just briefly before I get into the mathematics let me just briefly describe to you or remind you what the intuitive understanding of this thing was here we go so excuse me so if you can zoom in please as much as you can thank you okay so if you have if this is the probability distribution of the input intensity and it's kind of skewed this way goes down this way and and so this is piece of F the input image and this is piece of D of G the desired histogram at the output and it kind of looks a little bit more centered and more equal spanning the whole range and in this case we're just assuming we have 15 values of in 16 values of intensity and we're assuming that we're dealing with a 8 by 8 or 64 value point image okay so the first thing we do is we build the CDF cumulative distribution function for this probability and as you all know CDF is just the integral of P of F and we also build the CDF which is the integral of this distribution and these are shown here so this is the CDF of the given input and this is the CDF that we would like to achieve and so as I explained last time if you can zoom out just a little bit and I don't really know whether you can read the numbers or not so the way it works is so at the end I want to come up with the transformation for example I want to know input intensity 7 should get mapped to what output intensity and the way I do that the number on top of these bars I know you can't read it but essentially shows the height of each bar for example at 7 the height is 56 at 6 the height is 52 at 5 the height is 48 the what that means is that 48 pixels have intensity of 6 and smaller smaller on the input image so this is this is what we mean by the CDF of the input image by a time to get because this image only has 16 intensity values by the time you get to intensity and input intensity 15 you've reached 1 which means in this case is 64 that means all 64 pixels have intensity 15 or lower right so this is the shape of the input CDF and this is the shape of the CDF the desire so so let's see we want to figure out for example what the input intensity value 6 what does it correspond to so you go up look it up here it's 48 that's the height of this bar then you come down here and you look for 48 and you see well that's is the bar that correspond that has high 48 and you come down and look it up and intensity is 10 so input intensity 6 is matched to 10 and you keep repeating that for every input intensity in order to find out what output intensity it's meant and this is kind of the transformation you get and after after you apply this transformation to this input distribution this is the PDF of the input that you get of the output that you get which is quite a bit better than what we had before it used to be like this now it's a little bit more centered so what I want to talk about today is kind of the mathematical derivation of why we do what we do where did it come from I mean it's kind of I told you what we do but I never told you how come it works and why it works okay so the setup is the following for this derivation I'm gonna consider a continuous values of intensity okay rather than rather than discretized and I'll discretize this in just a second as well okay and I call that R okay so R is just a gray level of image to be equalized and R has been normalized we assume to the range 0 comma 1 so 0 corresponds to dark and 1 corresponds to bright in other words dark IE black bright IE white okay and now the goal is to design a transformation and we call that transformation as you saw on the screen T of R equals S for the range R between 1 and 0 and also S between 1 and 0 that that that we're gonna that that results in a particular probability distribution function that I'll just talk about in just a second so we're assuming to begin with again these are set of assumptions this is a transformation that does histogram matching or equalization this is the thing that we're after mathematically or numerically if you will we're after this transformation and we're gonna make the following assumptions about T of R and the reasons kind of become clear in just a second can roll up just a little bit so so here here are the assumptions assumption A is that T of R is single valued and monotonically increasing and I'll explain the reason for these assumptions in just a second in the interval R between 1 and 0 okay and so before I even do that I'd like to go back to to this picture if you can search the computer for a second so what what do I mean by it's it's single valued that means hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon Fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon Saver 365 by Whole Foods Market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon Fresh select varieties we wear our work day by day stitch by stitch a dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the dickies has been standing the test of time for a reason the work where isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to dickies.com and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that has made dickies a trusted name for over a century. For every value of R there's a single output or single corresponding T of R okay so the curve for example can't come up here go back up and then come back down this way there's not two values associated to there's not two T of R's associated with a particular value of R let's put it that way so the the single value why do we want this is single valued because we want the inverse transformation to exist so this is needed single value that's is needed for the inverse transformation to exist so so here's our let me just draw it here might as well do that so we don't we don't want this thing to do this here this is not single valued for this value of R not we've got two we've got two values of S okay we also we also want this to be monotonically increasing so why why do we want is to be monotonically increasing well in that interval because otherwise what happens so so this is not single valued and this is not monotonically increasing if you can go down a little bit so okay so what what happens is why don't we want T of R the transformation that takes input intensity to map it to the output intensity to kind of do that because as we because there is some intuitive thing that we want to preserve the increasing order of black and white in the output image the same as input because for example what happens is that if the input image R not gets mapped to S not intuitively you would like higher values of R not to get mapped into higher values of S not otherwise there's a reversal of brightness going on in the output that I mean artistically it could result in a lot of fun image you could sell it and make a lot of money or something but from a intuitive point of view you want the same way as the same way as the input intensity values progressively increases from left to right you want the output intensity also to follow that curve but how that curve is is whether it's going to do this or this or this whatever that's what we're after we we want it to be increasingly increasing we don't want it to go up and down etc so the reason for that is because we want to preserve the increasing order from black to white in the in the output image and so this is condition A so we're assuming that our transformation satisfies these conditions and then condition B is that we want T of R between 1 and 0 for R between 1 and 0 and and why do we want that to be true because we want the if the input intensity R ranges between 0 and 1 we want the output intensity S which is T of R to also vary between 0 and 1 because we want the output to be in the same range of intensity values as the input levels so intuitively the reason for this is we want the output gray levels in the same range as the input levels okay so clearly again if you switch back to the computer for just a second yeah sure yeah I'm just a little curious from last lecture about what this transforms actually doing so I mean is it going to take some intensity R in your original image and then multiply it by this value of S or something not multiplied map it it's a table look up it's a table look up okay so for example this is the best way of intuitively explaining it here when the transformation is all said and done okay so it's gonna it's good this is the final transformation you store that as a table look up you say all the pixels that have input intensity one are gonna map to up with intensity three all the pixels have input intensity two are gonna get mapped to five so you store this table and you just get the input image look up the intensity of his picture and then look it up in this table and figure out what the output intensity is is that clear that that's exactly what we're doing and by doing that now the histogram is much much better shaped okay so coming back to coming back to here go back to the to the computer so this particular transformation satisfies all of these things it's it's monotonically increasing it's single valued and it ranges between zero and one just like we wanted it to be okay so um so having you know set up all of these things what can we say about histogram equalization okay so T of R is S and furthermore T inverse of S is R and remember R is the input R is the input intensity picks input intensities and S is the output intensities for a given pixel for each pixel okay so there's there's a major result from probability theory that that says the following so result from probability theory that says that if T inverse of S satisfies condition A that that are just listed then the probability distribution of S and R are related in the following work PDF of S is related to PDF of R to this relationship okay so that's straight from probability and then rolling up the camera one more time and so based on this we can now figure out why we build the CDF so consider the CDF thing that we've been talking about as a transformation for a long time so if S is T of R CDF is in the continuous domain is essentially the integral of the PDF P R of omega P R of W D W ranging from 0 to R okay so this is the CDF of the probability distribution of the input pixels and now we can apply straightforward calculus D S D R what is the derivative of this transformation and by definition what is it what is the derivative of this with respect to R because the integral here is all the way to R it's nothing but P R of R okay so if we if we have a transformation which is CDF which is the one I just showed you an example of we just all along then this derivative is P R and now if I plug this into this relationship what do I get I get P of S of S is P R of R times one over P R of R which is just one for S in between one and zero so what does it says it says if we apply this transformation the output is gonna have probability distribution function that's uniform hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon Fresh shop Prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon Saver 365 by Whole Foods market a plenty and more come back for new deals rotating every week don't miss out on savings shop Prime exclusive deals at Amazon Fresh select varieties we wear our work day by day stitch by stitch a dickies we believe work is what we're made off so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the dickies has been standing the test of time for a reason their work where isn't just about looking good it's about performing under pressure the last thing through the toughest jobs head over to dickies.com and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that has made dickies a trusted name for over a century. So in words what we have is if the T of R is just a CDF or just the integral of input PDF which is piece of R of R then the then applying T of R results in an image whose PDF is uniform which we call piece of S of S is uniform and this is exactly what we mean by equalization histogram equalization histogram you can think of as the same as PDF when we say it's been equalized that means it's become a uniform equal for all heights uniform it's become a uniform PDF okay so so that's that's kind of an interesting exciting result that that's why we use that's why we start with PDF and kind of go to the CDF domain. Are there any questions? Now the the next thing I wanted so so this is all good for continuous and in case you're terribly worried that it doesn't work because pixels have discretized amplitude let's let's talk about let's just briefly extend that to the discrete case so your mind is at is at piece so so now consider the discrete case okay so now we have R sub K which is the discrete intensity values for K ranging from zero all the way to L minus 1 and and in that case probability of R sub K probability that we get intensity value R sub K is N sub K over N where N sub K is just number of pixels that have intensity in the image that have intensity or level R sub K. See the beauty of images I mean deep down what's going on that the beauty of images is that it's got so many pixels like if you got a hundred but hundred image you've got 10,000 pixels and you can easily build PDFs from the image itself and that's why you can you can kind of mix histograms and PDFs in a nice way when you're dealing with communication system design for example you talk about statistical ergoticity theory or something you don't get so many instances of this statistic that you're dealing with and therefore a lot of the statistical averaging things that are much harder to deal with than we're dealing with images anyway so so in the case that R sub K takes some discrete intensity values and N sub K is the number of pixels that have intensity R sub K then and N is often and here K ranges from zero all the way to L minus one and and what we have now S sub K is the transformation of R sub K which instead of being the integral of the PDF for R it's the sum summation from J equals zero to K of P sub R taking on the value R sub J this is the transformation that results in in in in a uniform histogram or in an equal equalized histogram which we can also write down P sub R as summation from J equals zero to K of N sub J over N again all K ranging from zero to L minus one and so and so what the transformation now that we have R sub K S sub K is going to look something like this again some variation of this you look up a given intensity value you look up in this transformation to see what the output intensity value should be between zero and L minus one zero and L minus one and boom now you have a table look up method of doing this okay so does anybody have any questions regarding the discrete case it's very much analogous to the continuous case the integral is just being replaced with a sum and the last thing I want to talk about is is histogram matching so so far we've been dealing with this problem whereby we have an input image it has a PDF we have a PDF for its pixels and we want to transform it so that the output image has equal has a uniform PDF distribution what if you don't like uniform and I'll show you an example of this in just a second what if you really rather have a different histogram for practical reasons for reasons that come from the application domain how do you do it then how do you design the transformation then and actually the answer is not that hard so let me just derive that also so this is called histogram not equalization but matching okay so T of R is S and rather than making P of S uniform we are given we want the P of S to match a desired if you will PDF a desired given PDF it's it's kind of like an inverse problem here's a PDF of the output go design the transformation so that the output will have that kind of a PDF so so here's how how we solve it we say R again again I'm gonna do it in the continuous domain because transferring it to the screen domain is trivial it's just a matter of replacing integrals with sums so R is the pixel values before equalization and Z is the pixel values actually let's not call it equalization before matching because now strictly speaking we're not doing equalization and we're doing matching we're modifying histogram but not to equalize it but to match it to some other thing and this is pixel values after matching okay so then we're given we can compute from the given image we can compute piece of R of R from the given image okay and we can we know or will given piece of Z of Z this is this is somebody has come to you for some reason you believe that what distribution should look like like piece of Z that's the one you got to play so how do the question is what is the transformation so the goal is what is the transformation from R to Z how do I design that okay so so once again and this might sound a little bit surprising to you once again I'm gonna build the CDF for R and I'm gonna build the CDF for Z and then do the matching in the CDF domain and let me expand on that a little bit more a little bit more details so the approach that I'm gonna take is I'm gonna build SST of R which is integral from zero to R of piece of R of W DW this is the CDF cumulative distribution function of R and I'm gonna build a new transformation G of Z which is the CDF of Z it's zero to Z piece of Z of T DT okay and this is the CDF of Z and our goal remember is to make it's to make because we want P of R because we want the the output to have distribution of Z are essentially the histogram matching the telling histogram matching corresponds to G of Z to be equal to T of R that that's that's really what that's really the transformation we have to design T in such a way that after we apply T to R we get we get this guy D of Z and so in other words we want Z to be the inverse value of T of R so what does it mean mathematically well or algorithmically here's R and what we're gonna so so here's R and and here's S oh and it's gonna be so so now I'm gonna plot the CDF of R 0 1 0 1 so so this is just a CDF of my Intel intensity values or and I'm also going to build a CDF for Z and I call this V so this is Z 0 1 0 1 I call just G of ZV just for the sake of completeness so I'm given this kind of CDF that's what my image has and this is the kind of CDF I want to have this is where I am this is where I want to go so what should be the transformation between R and Z to make this happen when I have an intensity value are what do I transform it what value of Z do I transform it to such that the resulting image will have this kind of desired CDF well the answer is pretty straightforward let's say I want to know what what the mapping for or not is I pick a random value or not and I want to know what what value how to find Z not the map for the map for R not and how do I do that well the easiest thing to do is to go to this curve go all the way up until you hit the curve and then look up this value whatever T of R is T of R not is and because I want T of R not to be the same as G of Z not I go to this axis now and I look up where T of R not is and then I go all the way here until I hit this and I find the corresponding Z not okay so so this is this is the procedurally this is the step we take which is not that different from the one I talked about earlier at the beginning of the class is just more mathematics gonna associate it with it okay and I and I keep doing that for every R sub I then I do that for R sub one so so now I might have having done that I'm remember I'm building a table now so my lookup table will have you know R not gets mapped to Z not how about a new value like R one well here's R one this is T of R one and then I come here and I find T of R one and I get maps to Z one this is how I find it okay and I keep doing that for every intensity value until I have this table lookup and and I end up getting a this this is called this now I have matched the the histogram of P sub Z of Z which was the desired okay so let me show you an example is this thing mostly clear okay so let me now show you a real life example where what we don't want to just equalize the histogram we want to do a little bit more so if you can show go back to the computer please okay so here's the image of Mars moon that that NASA got using the global surveyor and this is the histogram of that picture okay and and you can see that you know this region basically is way too dark and this region is too light now if you if you just knew the good old histogram equalization in other words make the PDF of the output be uniform we get this picture which is already better than what we started off with we had this as input this is output okay so and and how did we get this this picture well this is the input gray level this is the output gray level this is the transformation we applied using the histogram equalization method so that's the transformation and at the end when we're all said when all is said and done the histogram of the output looks something like this so there's no intensity values between zero and I don't know 150 or so and and you might say gee there must be something wrong if you really equalize histograms it's gonna be uniform but it's not because these mappings that we deal with are not necessarily on two and one to one and various other things and so this is the best that is term equalization can do so you look at this image and they say you know I really don't like I really don't like the fact that this is too bright and as such I want to I'm gonna come up with the desired histogram that's kind of more dark and this is kind of the the histogram that you come up with you shift things to delete it's more or less equal here but then you shift things you want to have a big this is a desired PDF of the output that you come up with and now this shows the g function g of z and this shows the g inverse and now if you apply the histogram matching where you're matching this kind of a PDF you get this kind of the histogram as the output and this kind of an image which looks a lot better than if you simply did histogram equalization so as you can see the histogram before this is before the histogram was shifted here it was from 150 onwards and now it's shifted a little bit to the left to 64 and if you're still not happy with that you can move things to the left even more question so the thing we're tweaking is g of z right to make the that's what we're trying to match no strictly speaking yeah yeah you come up with it with the p of z you come up with it this is the same as g of z which is the integral of it that's right so you might so so yeah you've you've you've changed one tweaking to another tweaking right depending on what how you pick g of z you could get different images right and in this case intuitively we just looked at this and we saw this is too bright and said okay now we're gonna make our g of z which is the desire probability distribution function moved to the left a little bit this is this is the desired that we came up with and that desired results in this g and this g inverse the transformation and the final histogram looks like this I think I'm gonna stop now I will continue this discussion on on Friday and one announcement I forgot to say don't forget to hand in your lab assignment on Friday hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon saver 365 by Whole Foods market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon fresh select varieties when you need meal time inspiration it's worth shopping king supers for thousands of appetizing ingredients that inspire countless mouthwatering meals and no matter what tasty choice you make you'll enjoy our everyday low prices plus extra ways to save like digital coupons worth over $600 each week and up to $1 off per gallon at the pump with points so you can get big flavors and big savings king supers fresh for everyone fuel restrictions apply