The Bold Blueprint Podcast
The Bold Blueprint Avideh Zakhor Remember
Hey Amazon Prime members, why pay more for groceries when you can save big on thousands of items at Amazon Fresh? Shop Prime exclusive deals and save up to 50% on weekly grocery favorites. Plus save 10% on Amazon brands like our new brand Amazon Saver, 365 by Whole Foods Market, Aplenty and more. Come back for new deals rotating every week. Don't miss out on savings. Shop Prime exclusive deals at Amazon Fresh. Select varieties. [Music] Let me start with a couple of announcements. There's homework due on Friday at the same time. There was some confusion as to whether 420 is an assignment or not and the answer is that it's not so you don't have to hand that in. Any questions or comments? There's like a lab part I guess that you never mentioned it in class either on Friday. Some of us just find out about it so that's also due Friday. Let me understand. The lab part is in the web page but I didn't mention it in class. I wrote down a bunch of problems from the book. It's not those problems that have labs it's a separate lab. It's a separate lab you know with a picture of a turtle. I see it's a picture of what? I guess it's a turtle. I haven't looked at it yet. Let me ask you this. How many problems did I assign? You assigned five out of the book. Including 420? No. Excluding 420. You didn't mention 420 last week. So what's the problem with 420 minutes Cindy? Probably 420 was in the handout. The handout that was the homework five description. I understand that. No let's not make the lab do this Friday that's too much. That's great news. I didn't I didn't really know that that was happening. How did that lab think it on the web page? Because in 414 we use we have a transformation and we use that transformation to design a thin filter and that's the lab. No forget about the handout for now. Just Friday's thing that's due is just whatever numbers I wrote on the board last Friday that's it. We can't just pull things out of the hat last minute. I didn't realize that that was got to the web page. Who put that on the web page? You and you and Rosita. When she asked me should I put it on the web page I thought the numbers that I wrote on the board I said yeah put those numbers on the board but I guess I didn't realize that it was the that let's just do this. What's on that handout that's on the web that's the following Friday. It's not that you don't have to do it ever. You'll do it but not immediately not this Friday. I think there's enough homework for this Friday and I guess Cindy and I will discuss after class whether 420 is in or not. I guess the handout has 420 plus the lab component. Are those still related to each other? They're separate. Cindy and I will discuss whether 420 will be on or not so don't yet do that the handout problem said until we've resolved that issue. For this Friday just hand in the five problems off the book that I wrote down on the board last time. Okay so are you guys done with that? How many people have finished that? Excellent and how many hours roughly did say less than five more than five? You can't remember and and I'm on fire okay and what what has happened to our air conditioning? Do you know what is this temporary or permanent? Okay okay well the heat together with my extreme headache that's might not make a good lecture but let's get started anyway. What we talked about last time let me mark the pages here. And I would like to to also briefly log in to this computer because I'm going to show some pictures off the off the so we want a one displayed of two displayed up top so one displayed there. Control all delete and and how do I log in here? This is my own account oh yeah I can log into EECS fantastic. Okay it'll take a while for the login process to happen but then then we'll have access to to something I put online. So what we talked about last time was my pages are all gonna fly off. Our additive and subtractive color system right? We talked about the fact that some color systems inherently are additive. For example when you have a TV gun system with the red gun, blue gun and and green gun and the the lights the right proportions of these guns get added together to produce the correct intensity or the correct color on your TV screen. Alternatively there's the subtractive color system and that's your what you have in your printer system where you have cyan, yellow and magenta and in some most systems also black and the way the subtractive color system works is that the model for it is that you start off with white light and each time you pass through this each time you have a input like cyan dot then you go through a band pass filter corresponding to cyan whatever that that remains is the color that you end up with. And many times in nature we have pigments in objects that when light hits it the object absorbs all the frequencies except for the color of that pigment in which case it reflects it and that's how you see things. And we talked a little bit about the human visual system and the fact that you know there's the model of the of the human eye and there's the retina where the picture falls into it in the back of your eye and there's a portion of retina called fovea where there's highest concentrations of rods and cones and that's really responsive for our vision at that part. So what I'm going to continue talking about today is a little bit more about let me just make it not oh I know what it is okay so what I'm going to continue oh I see then you don't get any wind because if it keeps moving all the papers keep from I'm not used to flying papers as I lecture there's one more variable you have to keep track of in addition to to various other things okay so what I'm going to talk about today is some more stuff about modeling of the human eye and I must preface all of these things by saying that in general the human ear is a lot better understood than the human eye there's there's linear time invariant kind of filtering that happens in the cochlea and and we've developed things like a hearing aid that amplifies the signals and and produces produces electrical signals that's connected to the nerve and in general the ear is a lot better understood than the eye there's no equivalent of cochlea for example for the human eye and let me also motivate why we want to understand the human visual system can anybody kind of can anybody make a guess or give a comment on that I mean other than other than oh I see you guys are from vision science well other than these guys who are getting a degree in it and they really are deeply interested in it but for my image processing system point of view why do we want to why do we care about human visual systems okay well tell me your reasons anyway you can tell any reason hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon Fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon saver 365 by Whole Foods market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon Fresh select varieties we wear our work day by day stitch by stitch the Dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work wear to your collection remember the Dickies has been standing the test of time for a reason the work where isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to Dickies calm and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century money oh I didn't know that where's the big money oh I see I see so you're thinking about therapeutic kind of reasons actually there is you know the basic thing that fixes your eye and that that's a good thing but with that say all right all right but but let's say we're not really interested in in any like biological surgeries or health things or anything like that which is which is kind of what your focus is what from an image from an engineering image processing point of view what why is it beneficial to understand yeah I'm so you have to speak up because of the noise I can't I can say just your imaging system is not over engineered like you don't have to hire a resolution you don't waste processing power on something like that right right because you build a lot of imaging systems and if you know something at the end of the day when you build the imaging system it's the human human operator or the human user who uses it and if you if it's kind of like you guys studying for the final exam right if you know what the final exam is all about then you can go study for it and get prepared for it if we know the ultimate test that the resulting pictures from an imaging system will be subjected to IE be observed by a human observer knowing the characteristics of the human observer can help us design that system better for example if I'm designing a compression systems and I'm sending not only do I want to save a natural resource IE the bandwidth and the spectrum and all these other things given that that's that the finite resource and you want to preserve it as much as you want and given that almost all compression systems will introduce distortions into the final image or video that you end up compressing it it'll be good to put all the distortions in parts of the image that the eye won't notice ideally you want the thing to be compressed in such a way that the the final image looks indistinguishable from the original and even though from a mathematical point of view you subtract two images there's a difference between them there's a finite amount of mean squared error between the two images original and and encoded if the human eye doesn't see that error it's fantastic it didn't matter so the reason you want to understand a human visual system is is because because the ultimate judge in many imaging systems is the human now I said in many and in many systems it's humans but it's not in all can you think of examples of imaging systems that the ultimate judge or the processing that we do for images is not for the sake of humans and for the sake of machines yeah can't hear you Alan okay can you use the mic so everybody can I don't know if I'm there because of this or everybody lithography okay can you expand on that and look for the rest of the classroom might not know what photography is all about also basic lithography is how they make integrated circuits you create a mask and you try and lie on and then do processing on it but then the there's I mean there's diffraction there's the light system is not perfect whatever you design on the mask mm-hmm there's not necessary corresponds to what gets the final product of the integrated circuit so you do what you mentioned before where you have an image that you desire image and you design the mask such in such a way that you get that desire image as close as possible all right in that case you want to design a good image so that you get a functioning integrated circuit so so in that case it's not for human eye to consume it good other examples computer vision systems where you're asking the computer vision system or machine vision system to accomplish a goal to track a car or to camp the number of cars in a highway or to inspect them and I see or to inspect the printed circuit board for inspection processes there there you're capturing images and you're trying to deduce something from them you trying to interpret them or find out something you know is this is the picture of this circuit board telling me something whether the circuit board is functional or not or all the starter joints properly done or not in that case the images are not ultimately consumed by the human observer but more than the machine but having said that a good part of and then again if you want to learn more about computer vision you are to probably be taking CS 280 but I said that a good part of of the of the stuff we talk about in this course ie image enhancement image restoration all of those things an image compression in almost all of those things the the the human observers is at the end of the chain and and and not only for image but also for video and in fact modeling the human we have we've made over the last few years some strides as to what metric in an image to look for in order to predict what what the human eye would have liked so let's say you show images to human observer A versus B and person likes A over B and let's say there's A original and then B and C are the process you show all of them to the human observer and the person says I like B better than C in terms of replicating A in terms of faithfully representing A and so one of the biggest question that comes at this is let's say for image compression one of the biggest question that comes out is what metric is the human eye computing in order in order for us to have these preferences of I like this image better than the other image and people so far have come up with extremely poor metrics like mean square error and and this is even worse for video because then there's a temporal aspect that's been added to the whole thing and so in general being able to predict what a human observer would have liked helps us design these these algorithms in a better way so that we can then cater it to the final observer so having said that let me just start today's lecture by talking a little bit about human visual system so really really roughly speaking one can come up with with the following block diagram for for the human visual system we have the peripheral level and then you have the central level so the peripheral level is the hardware in your eye that converts the light signal into neural signals so if we go back to the to the diagram of the eye can you zoom in please that I showed last time here here's the light and the picture falls on the retina this is the full via with the highest concentrations of rods and cones and and this is an electrical signal this this light signal gets converted into electrical signal that gets sent into this optic nerve to the brain for higher level processing so it's this nerve kind of signal that neural signal that we're talking about here so generally speaking the generally speaking the the central level part is not well understood at all on the other hand the peripheral level is slightly better understood and there is there's two ways people try to understand this whole system as a as a whole one of them is through physiological studies and and what that means is you just cut the eye open and kind of like what you do in your biology classes as in high school when you dissect the frog you kind of dissect the eye and see what what is inside and the other one is a psychophysical studies where you you ask observers or human hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon saver 365 by Whole Foods Market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon fresh select varieties we wear our work day-by-day stitch-by-stitch a dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the dickies has been standing the test of time for a reason their work where isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to dickies calm and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that has made dickies a trusted name for over a century and subjects to look at things and tell you what they've observed and I think the you guys are in the vision science program right okay you know that you guys are in vision science so you guys have stand client on your in your department right where you smiling yeah yeah so stand guys like stand client in vision science for example spend a lot of their time doing this this psycho physical studies they they present the observers with stimuli and they asked them to to answer questions and great things and various other stuff and so that's that's kind of the domain of vision scientists right actually nowadays there's there's a modality that combines the two of these things and what modality is that exactly what kind of imaging exactly functional magnetic resonance imaging where you you show things you show objects and pictures to to to a human subject but then they're under MRI and you're imaging their brain as they're looking at something and it's a combination of the two things is that it's psychological in the sense that you're presenting them with stimuli like staying client would would have done the last 40 or 50 years but at the same time it's trying to understand something about the physiology trying to see you know when you look at this kind of picture what part of the brain gets stimulated and the other part are you doing any work in that area yourself right right and guys I know some people that stand for Brian Wanda who does quite a bit of work on that actually did functional MRI is a new thing and it's affecting all kinds of fields just a week ago I was reading in the paper that they're using functional MRI to figure out when people lie supposedly when you lie a different part of your brain gets exerted and so they can tell that kind of as a lie detector test so having having said that then one very simple model that one can think of in for the human eye is is the following light coming in passing through some sort of a nonlinearity and and this could be some sort of a log type function and then passing through a linear time invariant system h of omega x omega y the LTI or LSI system and then outcomes the new signal so this is this one simple model for the peripheral level and throughout today's lecture I'll show you the result of some experiments that support this kind of a model many times you can if you think about the eye having a linear time invariant filter do you think it's low pass band pass or high pass do we have evidence to know it's band pass right you have difficulties in looking at very very small spatial frequencies and you have difficulties in looking at very high special frequencies so I I not that I brought up this issue I want to show figure seven point twenty three in Jay Lim's book that shows the frequency response of of h of omega x omega y as measured by researcher by the name of David something can zoom in please so when when I was a student trial bill Shriver at MIT I was used to say the eye acts as a differentiator for some frequencies as an integrator for other frequencies what he really meant was the eye has asked as a band pass filter here in this part is acting as a differentiator the frequency response of a differentiator is a straight line going up if you can see response of an integrator is a line coming down so so in the horizontal axis here you have the special frequencies and cycles with degree and on the vertical axis you have the magnitude of h of omega x and omega y and actually in the 70s there was a professor at Berkeley Dave sacrosan who published a seminal paper with his student Manus and sacrosan that that also came up with a better characterization of that unfortunately he died of bone cancer extremely unexpectedly in in a matter of few years so um so having having said that I'd like now to spend a little bit of time talking about some visual phenomena specifically that explain some of the properties of the of the human vision so the first thing that I want to talk about is what's called Weber's law and and and and and it's this is explained both by the way in NIMS book and in and guns all of some woods but here's I'm going to draw this here so here's here's the experiment that that you end up doing you you start with some patch that you're showing to an object and it has that hatch at some sort of an intensity which we call I out okay and then inside this patch you have a region that initially this is just called I sorry inside this this region you have a patch that initially has intensity just I and then you gradually crank up the intensity inside this patch by delta I so initially delta I is zero and and then you gradually increase it and you ask the human observer to look at this thing and raise their hand when they see it just what's called JMD just noticeable difference of course you repeat that experiment many times with each subject and you also have lots of different subjects so not everybody not every subject is going to raise their hands at all at the same time and also not all the subjects are going to behave the similar way but you can come up with some sort of rule of thumb and say look 75% of the subjects raise their hands 75% of the time and this is kind of what what we observe and and if you if you do this experiment the the plot that you end up getting for this thing is is something like this if I put up the plot delta I over I as a function of log of I then I get a pattern that looks like this and this flat region is somewhere in the range of 1 to 3% okay so in other words the main phenomenon that I observe is that delta I over I is approximately constant and and because there's a there's a log here if I let if I let delta I go approximately to 0 then I can rewrite this as saying di over I is approximately constant which is also D of log I okay I'm a little confused just kind of confused about what this graph is showing okay so what it's saying is forget about these two ranges okay it's saying that for a large region of the intensity region it takes one percent change in intensity for us one to three percent change in intensity for us to notice that the circle in the middle has a different intensity than the background ignoring these two ends okay that means that you for if you're looking at if you're looking at this patch hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon saver 365 by Whole Foods market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon fresh select varieties we wear our work day by day stitch by stitch a dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the dickies has been standing the test of time for a reason the work where isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to dickies calm and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that has made dickies a trusted name for over a century let's say perfect intensity for a large range values of intensities but just one to three percent you you'll notice it and and that that range is fairly constant okay now there's a there's a different kind of the fact that this however at the two ends namely here and here this this doesn't hold true that you you end up being because the eye essentially doesn't have infinite dynamic range when there's only when the intensity level is extremely small it takes much more delta I for us to be able to notice the change and when the intensity level is very very high it takes much more delta I for you to to notice that in other words there's a saturation process going on the eye doesn't have infinite dynamic range right now we can also do is that clear we can also do an adaptation experiment and so let me talk about that and the way the adaptation experiment works is again you you have a patch of intensity I not constant intensity and and you let the human observer kind of get used to the eye not and then you have a circle in the middle and you you half of this circle you show the intensity I in the other half you show intensity I plus delta I okay so this is the left this is the right and you ask yourself initially delta I is zero so you just see a circle of constant I and then you gradually increase delta I and you ask yourself under what what value of delta I do you see the difference between right and left and and here the difference between this experiment and the other experiment is that the by far the majority of what the user what the observer sees is intensity I not so you your eye has adapted to that intensity level I not and if I were now to plot the the result that I would get out of this thing is something like this as a function of log of I I can plot delta I over I and so so if I'm looking at a particular intensity for example I I too then the plot is gonna look like this and this is for I not equals I too so your maximum what this is telling us is that the maximum sensitivity occurs at an intensity at an intensity I that coincides with the background okay so the fact that the that the bottom here occurs at I too and and I to is equal to I not means that you have that that the that the sensitivity to intensity is highest near to the level that the observer is already adapted to so let me write that down so sensitivity let me write it here sensitivity to intensity is highest that means delta I over I is lowest near the level that the observer is adapted to adapted to because I not is equal to I to and if I just move that to to I 3 then the curve would look something like this so this is I not equals I 3 and if this was the the original curve that I had from the previous plot that kind of looked like this then these curves are gonna look I'm gonna get a family of curves that are gonna for example I want this would be the curve corresponding I not equals to I one this is I 2 this is I 4 I not equals I 4 this is I 5 I not equals I 5 so that the the bottom of this curve identifies the same it has the same trajectory as as the curve that that we had before so so what this is telling you is that the sensitivity of how much change in intensity you can see is very much dependent upon the ambient intensity that your eye is already has already gotten gotten used to okay and that if you change for what what it's telling you is that if I'm in this part of this curve for example if if I'm surrounded with intensity I 2 it takes a lot more delta I over I for me it takes a lot more delta I for me to to see the difference between the right half and the left half of this circle then then when I'm here so the more to the the the level your eye gets adapted to is the one that you're most sensitive okay is that point clear okay and the the the the last kind of thing I want to talk about is is this spatial frequency response a little bit more so suppose we do the following experiment we come up with an image I of X comma Y which is I not of Y times cosine omega of X times X plus some sort of a constant okay and this constant is chosen in such a way that this this quantity is always positive so constant is to ensure I is always positive okay and if I plot that I get this figure 7.22 in in the J-lims book and I'm just gonna show it to you if you can zoom in please okay and and and so this is exactly what what we expected it to be right the spatial frequency as we move from left to right increases right it's lower frequency here and it's high frequency at this end and furthermore the intensity increases as we go from the bottom of the picture to the top of the picture that's that's how we designed it there's an eye not of Y and then there's cosine omega X of X so that's kind of what this you can zoom in so if if the frequency response of the eye was completely flat then then the sensitivity that you would have across the horizontal axis would be constant so do you do you see if I if I cut if I cut this thing along the horizontal axis let's say if I look at the line here do you see things with the same sensitivity to the right and left as you see in the middle no which part do you see do you have an easier time seeing the middle right the reason being this is too low of a frequency this is too high of a spatial frequency here is when we discern the black hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon Fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon Saver 365 by Whole Foods Market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon Fresh select varieties we wear our work day by day stitch by stitch a Dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the Dickies has been standing the test of time for a reason the work where isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to Dickies comm and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that is made Dickies a trusted name for over a century the most so this is can be used kind of as an evidence of the fact that the eye acts as a band pass filter or the eye has some sort of a band pass factors we're not very good at seeing extremely low spatial frequencies we're not very good at seeing extremely high spatial frequencies but somewhere in that mid-range we're pretty good at seeing high frequencies and when you go to image coding that's a you you you're encoding DCT discrete cosine transform coefficients of the script for your transform coefficients which are almost never used for compression but cosine is the one that's used but that's the reason you can easily discard extremely high frequency coefficients and your eyes won't see it first of all for natural imagery those have small values anyway to begin with and second of all the human eye is insensitive to those range of frequencies so you can safely discard those frequencies and without worrying too much as to whether or not you've affected the the human the human visual experience so what what I want to talk about next is the MAC effect and this is in a picture that I want to use from from this book the Gonzales and Woods that's another visual effect so let me then go to the website where I stored this thing what happened do we want to start wizard or the classic anybody classic okay okay so I like to go to this picture here and spend a few minutes talking about that so what we have here is a sort of a staircase function where as you move from left to right the intensity level increases okay and what what what we claim is that the perceived brightness looks something like that there's there's undershoots and there's over just before the transition from one band to the other and there's overshoots immediately after the transition so the perceived brightness looks something like this so essentially what that means is that the the the the perceived brightness just before the transition here this looks a little bit darker than it really is into the human eye and right after here looks a little bit lighter than it is to the human eye just because it's next to the next to the other band so this is what's referred to as the as the MAC band effects okay and again it is it's just something that you perceive showing you that the brightness of the images that you're looking at are not necessarily a hundred percent correlated with what you actually perceive and the eye can get fooled in many different ways or many different kind of directions to show the same thing there's also other kind of a perception type things that that you can play with what you have here is three squares where the intensity in the in the middle square is constant for all three of them at the same time this one looks a lot darker than this one simply because this is next to a very dark region so it looks bright this this square here looks bright whereas his one because the the surrounding region is is as much brighter this looks much darker but in fact they actually if you were to measure the intensity levels of these things they'll all be the same so again it's it's the games that the human eye kind of plays with with your eyes and here's some other well-known optical illusions there's no there's no square here but your eye immediately interpolates and thinks of this as having a square here this line here is exactly the same line length as this one but because of the way the arrows are the two and this looks is perceived to be a lot longer than this one in this picture all of these diagonal lines are actually parallel with each other exactly parallel with each other but because of the the cross lines that people have put it looks like the lines are diverging here and they're converging from here to here etc and here's the last part yeah right and this is the last the last illusion where you've got a bunch of lines and this gives you the illusion that there's a circle here but in fact there's actually none so we talked moving on moving along here we talked a little bit about the frequency band last time so this this axis is energy of photons this is the frequencies this is wavelengths and meters but just to kind of recap there's ultraviolet at this end infrared in between and in this range and somewhere in the middle there's a visible spectrum which is blown up here ranging from violet all the way to red violet blue green yellow orange etc and then to the to the left you have at much smaller wavelengths you have x-rays hard x-rays gamma rays soft x-rays and on the right you have radio waves I think we talked about this last time I'm gonna skip some of these things this shows kind of a an imaging system generally speaking you have a scene element there's a illumination source like the lights or the sun or whatever the source of lighting is and the light from that illumination source hits the object and then it gets reflected it goes through the imaging system it gets reflected and the imaging system then does the they have some sort of an internal image plane that digitizes this and produces a and an output image so this output image is digitized in two ways right one of them is there's finite size pixels here therefore you you're doing spatial quantization the same way as you have a time signal and you sample it you're doing temporal sampling here you're doing spatial sampling and the second way is that each of these let's say CCD sensors quantizes the amount of light that it it's hit with to a finite number of levels so in that case you're quantizing the amplitude of the intensity so you there's two sets of quantization just like with the discrete time signals you sample a signal you have quantization in time because because of the sampling that you did and then that that gives you a digital signal sorry a discrete time signal then you convert that discrete time signal to a digital signal by further quantizing the amplitude and what you see here is the the quantized what you see there is is output of the quantization for both the spatial quantization and the amplitude quantization there's one other thing I'd like to talk about this picture and for that I would like to get the camera back down into right here okay so so as I as I mentioned when you're doing image sensing and acquisition hey Amazon Prime members why pay more for groceries when you can save big on thousands of items at Amazon Fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on Amazon brands like our new brand Amazon Saver 365 by Whole Foods market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at Amazon Fresh select varieties we wear our work day by day stitch by stitch at Dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true work where to your collection remember the Dickies has been standing the test of time for a reason their work where isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to Dickies calm and use the promo code work where 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that has made Dickies a trusted name for over a century there's there's two things okay there's the illumination source and illumination source can be I mean what we're dealing with image processing so we're dealing with visible light but in many other applications it could be radar which is another source of electromagnetic it could be infrared and we we took infrared lasers and we used that for city scanning one of my projects at Berkeley could be x-way whatever and then and then there's reflection or absorption or transmitters okay so this happens by by by the elements in the scene all of them reflectance absorption transmission and and it's this element it's this thing that we measure in order to say something about the scene okay and so when we're dealing so this is in general this is this has nothing to do with just visible light and image processing what this has to do with all modalities of of imaging so all modalities now in particular what we have in imaging image formation is characterized by two components one of them is amount of source illumination that's incident on the scene and the other one is the amount that's reflected okay so we can write this down as what we get f of x comma y which is the the sensed signal or the sensed image is i of x comma y which is the illumination times r of x comma y which is the reflectance so what are typical values of i of x comma y in real life well it's between it's between infinity and zero but that doesn't really it's a positive quantity but that doesn't really mean anything well we can give we can put some numbers to it in a sunny clear day all i can be as much as 90 000 luma per meter square luminance per meter square and in a clear evening it could be as little as let's say 0.1 luminance per meter square it's a unit of light right it's it's there's a different unit that people used to use in the past called i forgot what's called candle that's right the LUMO i think is the mks lumen is the mks version of that i believe and then the the reflectance is also can can take on a range of values so that that essentially tells you that that the illumination could take on a very long large set of values actually and one of the one of the projects that people do a lot in in computer graphics and computer vision and image processing is is the following is that you've taken a picture or multiple pictures of a scene with a particular lighting condition and now you're interested in from those pictures to derive another set of images under different lighting conditions as if you've taken it under that lighting condition let's say for example Paul Dubavic at USC he went to i forgot where the ruins in Athens or something like that some historic sites archaeological archaeological sites and he image the whole scene you know at 10 a.m. in the morning and given the sun location and given the characteristics of the lighting illumination source then you can play some games and some mathematical and physical kind of algorithms in order to derive the pictures you would have gotten on their moonlight for example so that's that's a that's a quite a active area of research and and similarly r of x come on y the reflectance can can be anywhere between one and zero okay um and you can you either reflect all of the light or none of it it's a fraction really and as you all know what is what is the reflectance dependent on how much light you reflect on the color right if you're wearing a dark color like black it is on the color and the material that you're wearing if you're wearing a black thing then you absorb out the light you hardly reflect anything if you're wearing a white clothes or or in snow for example you you reflect everything and you absorb nothing so um it's is of the order of 0.01 let's say if you have black velvet if you wear black velvet 0.15 if you have stainless steel it's 0.80 if you have a flat white wall at 0.9 if you have a silver plated metal and it's 0.95 for like snow which is white it hardly reflects anything it hardly absorbs anything okay so um so it's it's it's this function f of x comma y that in this picture if you can switch back to the pc it's it's this so so if the light hits the object it reflects some of it and what you're collecting here is really just just f of x comma y or quantized in both the space domain and in amplitude domain so along the same lines if i were to cut this this object with a line from a to b then and and look at the intensity that i've captured in the in the continuous tone image assuming there was absolutely no quantization this is the intensity pattern that i would get okay i i start off as being very bright white area so this is high and there's this noise showing that even though to your eye it looks completely white this region in reality there's it goes up and down if you have a measurement device it has some noise then then you hit the object so it goes down and as it reverse from this point to this point the intensity gradually increases and then it decreases to this point so it goes up and then down and by the time you're down this point is supposedly darker than this point so this is why at this point is lower than that point then comes comes right back up now here is what shows the same function there but after it's been sampled sampled in this in the in the space domain right so now you um you you have discreetness along the x-axis here and here is after you've quantized it also the amplitude and so this is the quantization scale and as you can see it's but at the time you're done with that you you get a function like this so most images that we deal with have been both sampled in the space domain and quantized in the in the amplitude domain and so this is a continuous image that's been projected into a sensor array and this is the resulting image after sampling in quantization okay and again we're exaggerating here in most times if you if you sampling in the quantization is so coarse you're going to have a lot of trouble so the next thing i want to talk about is is the notion of spatial resolution and the notion of amplitude resolution okay so if you have an image and generally speaking uh you you can hey amazon prime members why pay more for groceries when you can save big on thousands of items at amazon fresh shop prime exclusive deals and save up to 50 percent on weekly grocery favorites plus save 10 percent on amazon brands like our new brand amazon saver 365 by Whole Foods Market aplenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at amazon fresh select varieties we wear our work day by day stitch by stitch at dikies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true workware to your collection remember that dikies has been standing the test of time for a reason the workware isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to dikies.com and use the promo code workware 20 at checkout to save 20 percent on your purchase it's the perfect time to experience the quality and reliability that has made dikies a trusted name for over a century you can have different spatial resolution resolutions you can capture an image as a thousand by thousand pixels the spatial resolution refers to the number of pixels that you've got it can be a thousand by thousand or that same image could be could have been captured at 500 by 500 or 250 or 10 etc etc so a given image can be shown at different spatial resolutions it can also be shown at different number of amplitude levels for the quantization and it will all have different appearances in the early days of the internet when people have pictures on the web pages and the pictures would have taken too long to get downloaded generally what they just did is they just reduced the spatial resolution so if you had a rose that you wanted to have a new website it was thousand 24 by 1024 and at that time the high speed internet wasn't so prevalent everybody had dial up and this would have taken five minutes to to download then you just come up with a 128 by 128 version of it and and that's what you would have so what i've shown here is that same image that has been subsampled successively into smaller and and smaller kind of versions and in real life what is what's one of the things that you got to do before you subsample an image if you don't want to introduce aliasing low pass filter the same way as with temporal signals time domain when one dimensional signals you have to low pass filter before you're subsampling that's the same thing as you have to do there so so this was an example of changing the spatial resolution what you see here is an example of keeping the spatial resolution but but replicating the pixels so this is the same thousand 24 by 1024 picture as we had here up here but here you've what you've done is the effective spatial resolution of this thing is really just 512 by 512 in a sense that you've taken every four pixels here and you've averaged that out and you have a pixel which covers the same amount of area on the screen as this one does but the intensity for this super pixel is this is the average of four pixels here and you keep doing that this is 256 128 64 and 32 now here each one of these pixels as you can see that this is fundamentally a 32 pixel by 32 pixel image but i have chosen to blow it up so the size of each pixel is is much bigger than it was in this picture here here as i shrunk the pick as i reduced the spatial resolutions i also shrunk the pixel size for display purposes here i chose to keep the pixel sizes the same and the end result is the same the amount of information here is exactly equal to the amount of information in this tiny image there but as you can see this is not a very pleasant image to be looking at so in general for every pixel size and for every amount of information in the image is not very high it's not advisable for for visual quality reasons to to display that the large size here's another interesting picture where we're again keeping the special resolution constant but what we're changing the dynamic range and this is a medical image so the dynamic range is this 256 levels here i believe this is the original this is the 256 sorry this is original with 256 levels of amplitude quantization so now we're quantizing the amplitude here's the same thing with 128 levels of gray 64 32 16 8 4 and 2 okay so is as you go so let me go back so by the time you're here let's say moving from here to here or even from here to here you don't see that much perceptual change all the way from 256 to about 64 is more or less the same you see some bad things happening here at 64 but but by now you you're beginning to see some what's called contouring artifacts and at at at i believe this is 16 and at eight the contouring artifact is even more at four it's this and at two finally you you reduce it to a binary image and that's quite disturbing so the number of amplitude levels you need for most black and white images is definitely no more than 256 levels but it can be as low as 64 or so meaning the eye the eye's quantization can can live with with somewhere around six bits or so and the last yeah the last picture that i want to show is let's talk about what's called isopreference curves okay so i'm going to deal with three classes of images an image which is of a face and this is an image with a very low level of detail this is an image with medium level of detail and this is an image with high level of detail okay and the question that i'm going to ask myself is there's the there's the number of pixels n that we can change so the same way as when we're dealing here we changed n from 1024 to 512 256 128 64 32 and in the next experiment we changed the amplitude levels k from 256 128 64 etc now i'm going to change n and k simultaneously and i'm going to show it to the observers and i'm going to ask them to tell me what their isopreference curves are in other words as i change n i change k simultaneously in such a way that they say okay that the image didn't change that much if i do that this is the curve that i get along this this axis is n along this axis is k which is the number of quantization levels so a couple of interesting observations first of all for the crowd image which i'm going to go back and show it again which is it has lots and lots of details in it right you get some sort of a almost vertical line here which means that it almost doesn't matter that you're reducing the k level from 5 to 4 at this particular n value which is somewhere around 92 or so or 100 pixels the isopreference doesn't doesn't change that much it's the picture looks just as good at 5 bits per pixel as almost as it does at least to the viewers as as 4 bits per pixel for for the other two pictures face and cameraman first face is is this one lena it's also called lena and this is the cameraman which is in front of mit that's the kresge auditorium um for those on the other hand you see that the curves kind of slide down this way look at this this region here and what that means is that as you increase the spatial resolution the number of pixels and you can you can reduce the contrast without the observers complaining or noticing degradation in the quality of the image that they're looking at so kind of one one one effect makes up for the other one you can you can increase the spatial resolution but at the same time decrease the amplitude resolution and not not people notice it so much and that can be kind of to some extent that can be contributed to the fact that the perceived contrast goes up when when you decrease the k okay this study was done by the way many years ago by tom Wong over at the University of Illinois at Erbana champagne and um the last thing that um that i want to show is the fact that even even if you reduce the information content of an image to 32 pixels by 32 pixels which is kind of what we have here this is one this is the 128 by 128 version of the original 1024 by 1024 you can see there's pixelation happening this is a 64 and this is the 32 so the the information content of this picture is no more than 32 numbers by 32 numbers is it that's it even then the best way to show it or presented is not in this way and you can do basic bilinear interpolation or any kind of other low pass filtering to get this picture from this this picture from this this picture from that in order to to give it a good appearance even though your web page could only afford to transmit the 32 pixels by 32 pixel image when it gets to the receiver if you have if you have a little applet you can have your applet do smoothing over that and again that brings me to another point which is an active area of research and image processing is to reduce the spatial resolution and get something that has less information but do extra processing at the other end to enhance the edges so this bilinear interpolation we did on this it was a very extremely dumb simple mind that we are doing things and one can imagine doing a lot more sophisticated processing on this to get a better picture than that so the reason formation loss as you smooth out your pixels but you can you can you can do smart processing at the other end to undo some of the some of the damage i am going to almost stop now except that i have one more one more thing that i want to talk about and that is the some empirical observations let me just write this down instead of flashing that page because then at the end you won't have it in the lecture notes so if you can bring back the camera okay so um here's some impure to wrap up to this lecture here's some empirical observations of empirical observations about that that are typically exploited in image processing systems okay number one and and most of what i'm going to say is is going to be obvious to you but sharper images look better we all know that right that that's the trick people are using to get us all to buy an HDTV screen TV because it's sharper if you have noise if you're introducing noise into your system for whatever reason either because your sensors are imperfect or because you have to do compression or whatever it's best to introduce it in in textured regions rather than regions that have uniform background because noise gets lost in the middle of all the texture but if you have uniform background and you see you see noise then it's it's a lot more noticeable noise essentially is what high frequency in signal processing language noise is high frequency so if the spectrum of the noise is far away from the spectrum of the signal if the signal is low frequency and the noise is high frequency it becomes very noticeable but if the noise is high frequencies and the top of texture which is also high frequency you don't tend to notice it so much so the same noise in uniform background region is more visible than noise in edge areas sorry I don't mean edge areas I mean textured regions sorry and textured regions have a lot of edges anyway the next the next property that you want to exploit is that the same noise in dark areas look more noticeable than noise in bright areas so same noise in dark areas are more noticeable than in bright areas and let me show you a picture of that in Jay Len's book so there's a nice example called here we go figure 718 and if you zoom in here as much as you possibly can so this is a picture of some building and we've added noise to both the uniformly across the entire image again I'm not really sure if this comes across or not but the noise in the the bright areas like it's almost unnoticeable the noise added here and in the in the darker regions it's quite noticeable that would be the reason when you go shopping for like cars or actually we just ended up buying a dishwasher a week ago because of dishwasher and the guy asked what color do you want and I said something that doesn't show dirt and doesn't show anything he said that said you got to buy a light one a white don't buy a dark color one the dark color shows the the white particles on a dark color thing or fingerprints on the dark coloring is a lot more noticeable than on the white one so and this applies to your to your clothing as well you can spill I can spill a lot of coffee in my in this color shirt and it doesn't show as I'm sure you don't see it but if I were to wear my dark black shirt you'll see a lot of junk on it okay number four same amount of artificial noise is a lot worse than looks a lot worse than the natural noise okay and what do I mean by artificial noise when you do dct you're introducing blocking artifacts that are rectangular and that really stands out but if you have a camera with ccd sensors and there's shot noise from there you know that's kind of a natural noise too many four times heated and too few didn't hit it etc that that looks a lot less noticeable and then the last thing is that images with unnatural aspect ratio attract a viewer's attention so images with unnatural aspect ratios attract viewers attention so that that's again why when HGTV was introduced in this country we ended up using aspect ratio of 16 to 9 which is more unnatural and there was an excuse for consumers to widen the darn system otherwise people would say oh I paid the same amount of money I get something that just looks like my current TV it just has more pixels I'm not I'm not buying that so they changed the aspect ratio to 16 to 9 from 4 to 3 and in order to kind of grab our attention and buy things and that's also why when you go to the movie theater what's it called the cinema scope it's long and narrow and it's an unusual aspect ratio so you tend to kind of pay more attention to it but that's all I have to say and on Friday we're going to embark on image enhancement so that we'll have our first discussion on that and then what we'll do on Friday too hey amazon prime members why pay more for groceries when you can save big on thousands of items at amazon fresh shop prime exclusive deals and save up to 50% on weekly grocery favorites plus save 10% on amazon brands like our new brand amazon saver 365 by Whole Foods Market a plenty and more come back for new deals rotating every week don't miss out on savings shop prime exclusive deals at amazon fresh select varieties we wear our work day by day stitch by stitch it dickies we believe work is what we're made of so whether you're gearing up for a new project or looking to add some tried and true workware to your collection remember the dickies has been standing the test of time for a reason the workware isn't just about looking good it's about performing under pressure and lasting through the toughest jobs head over to dickies.com and use the promo code workware 20 at checkout to save 20% on your purchase it's the perfect time to experience the quality and reliability that has made dickies a trusted name for over a century