# Lecture 1: Images and image filtering

Lecture 2: Image filtering What is an image? A grid (matrix) of intensity values 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 = 255 255 255 20 0 255 255 255 255 255 255 255 255 255 255 75 75 75 255 255 255 255 255 255 255 255 75 95 95 75 255 255 255 255 255 255 255 255 96 127 145 175 255 255 255 255 255 255 255 255 127 145 175 175 175 255 255 255 255 255 255 255 127 145 200 200 175 175 95 255 255 255 255 255 127 145 200 200 175 175 95 47 255 255

255 255 127 145 145 175 127 127 95 47 255 255 255 255 47 255 255 74 127 127 127 255 255 255 74 74 74 95 95 95 74 74 74 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 (common to use one byte per value: 0 = black, 255 = white) Images as functions An image contains discrete numbers of pixels Pixel value grayscale/intensity [0,255] Color RGB [R, G, B], where [0,255] per channel Lab [L, a, b]: Lightness, a and b are color-opponent dimensions HSV [H, S, V]: Hue, saturation, value

Images as functions Can think of image as a function, f, from R2 to R or RM: Grayscale: f (x,y) gives intensity at position (x,y) f: [a,b] x [c,d] [0,255] Color: f (x,y) = [r(x,y), g(x,y), b(x,y)] What is an image? A digital image is a discrete (sampled, quantized) version of this function f (x, y) x y Image transformations g (x,y) = f (x,y) + 20 g (x,y) = f (-x,y) Super-resolution Noise reduction Image denoising Why would images have noise? Sensor noise Sensors count photons: noise in count Dead pixels Old photographs What is an image? A grid (matrix) of intensity values: 1 color or 3 colors 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 = 255 255 255 20

0 255 255 255 255 255 255 255 255 255 255 75 75 75 255 255 255 255 255 255 255 255 75 95 95 75 255 255 255 255 255 255 255 255 96 127 145 175 255 255 255 255 255 255 255 255 127 145 175 175 175 255 255 255 255 255 255 255 127 145 200 200 175 175 95 255 255 255 255 255 127 145 200 200 175 175 95 47 255 255 255 255 127 145 145 175 127 127 95 47 255 255 255 255 47 255 255 74 127 127 127

255 255 255 74 74 74 95 95 95 74 74 74 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 (common to use one byte per value: 0 = black, 255 = white) An assumption about noise Let us assume noise at a pixel is independent of other pixels distributed according to a Gaussian distribution i.e., low noise values are more likely than high noise values grainy images Noise reduction Nearby pixels are likely to belong to same object thus likely to have similar color Replace each pixel by average of neighbors Mean filtering 0 0

0 0 0 0 0 0 0 0 0 0 0 10 10 10 0 0 0 0 0 0 10 20 20 20

10 40 0 0 0 10 20 30 0 20 10 0 0 0 0 10 0 30 40 30 20 10 0 0

0 10 20 30 40 30 20 10 0 0 0 10 20 10 40 30 20 10 0 0 0 10 20 30

30 20 10 0 0 0 0 0 10 20 20 0 10 0 20 0 0 0 0 10 10 10 0 0

0 0 (0 + 0 + 0 + 10 + 40 + 0 + 10 + 0 + 0)/9 = 6.66 Mean filtering 0 0 0 0 0 0 0 0 0 0 0 0 0 10 10 10 0 0 0 0

0 0 10 20 20 20 10 40 0 0 0 10 20 30 0 20 10 0 0 0 0 10 0 30

40 30 20 10 0 0 0 10 20 30 40 30 20 10 0 0 0 10 20 10 40 30 20 10

0 0 0 10 20 30 30 20 10 0 0 0 0 0 10 20 20 0 10 0 20 0 0 0

0 10 10 10 0 0 0 0 (0 + 0 + 0 + 0 + 0 + 10 + 0 + 0 + 0 + 0 + 20 + 10 + 40 + 0 + 0 + 20 + 10 + 0 + 0 + 0 + 30 + 20 + 10 + 0 + 0)/25 = 6.8 Mean filtering 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 10 10 10 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0

0 10 20 20 20 10 40 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 0 0 0 0 0 0 0 0

0 0 0 0 0 0 10 30 40 30 20 10 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 40 30 20 10 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 10 40 30 20 10 0 0 0 0 0 0 0 0 0 0 0 0

0 10 20 30 30 20 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 20 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 10 0 10 0

20 0 10 10 10 0 0 0 0 (0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 10)/9 = 1.11 Mean filtering 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 10 10 10 0 0 0 0 0 1 4 0 0 0 0 0 0 0 0 0 10 20 20 20 10 40

0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 0 0 0 0 0 0 0 0 0 0

0 0 0 0 10 30 40 30 20 10 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 40 30 20 10 0 0 0 0

0 0 0 0 0 0 0 0 0 10 20 10 40 30 20 10 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 20 10

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 20 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 10 0 10 0 20 0

10 10 10 0 0 0 0 (0 + 0 + 0 + 0 + 0 + 10 + 0 + 10 + 20)/9 = 4.44 Mean filtering 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 10 10 10 0 0 0 0 0 1 4 8 0 0 0 0 0 0 0 0 10 20 20 20 10 40 0

0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 0 0 0 0 0 0 0 0 0 0 0

0 0 0 10 30 40 30 20 10 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 40 30 20 10 0 0 0 0 0

0 0 0 0 0 0 0 0 10 20 10 40 30 20 10 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 20 10 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 20 0 0 0 0 0 0 20 10 0 10

0 20 0 0 0 0 0 0 0 0 0 0 0 10 10 10 0 0 0 0 0 0 0 0 0 0

0 0 0 0 (0 + 0 + 0 + 0 + 10 + 10 + 10 + 20 + 20)/9 = 7.77 Mean filtering 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 10 10 10 0 0 0 0 0 1 4 8 10 8 9 6 4 0 0 0 10 20 20 20 10 40 0 0 0

4 11 13 16 11 12 7 4 0 0 10 20 30 0 0 0 0 6 14 19 23 19 18 10 6 0 0 10 30 40 30 20 10 0 0 0 8 18 23 28 23 17 8

2 0 0 10 20 30 40 30 20 10 0 0 0 8 16 26 31 30 20 10 3 0 0 10 20 10 40 30 20 10 0 0 0 10 18 27 29 27 17 8 2 0 0 10 20 30 30 20 10 0 0

0 0 8 14 22 22 20 11 8 3 0 0 0 10 20 20 0 0 0 0 0 0 20 10 0 10 0 20 0 0 4

11 17 17 12 6 4 2 0 10 10 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Noise reduction using mean filtering Filters Filtering Form a new image whose pixels are a combination of the original pixels

Why? To get useful information from images E.g., extract edges or contours (to understand shape) To enhance the image E.g., to blur to remove noise E.g., to sharpen to enhance image a la CSI Mean filtering Replace pixel by mean of neighborhood 10 5 3 4 5 1 1 1 7 4.1 Local image data Modified image data f S[f] A more general version 10 5 3 4 5

1 1 1 7 7 Local image data Kernel / filter A more general version 0 10 5 7 0 5 11 6 8 3 9 22 4 5 1 2 9

14 6 7 3 10 15 12 9 7 Local image data Kernel size = 2k+1 A more general version w(i,j) = 1/(2k+1)2 for mean filter If w(i,j) >=0 and sum to 1, weighted mean But w(i,j) can be arbitrary real numbers! Properties: Linearity Properties: Linearity Properties: Linearity Properties: Linearity Properties: Shift invariance f f Shift invariance Shift invariance Shift, then convolve = convolve, then shift Output of convolution does not depend on where the pixel is

f f Convolution and cross-correlation Cross correlation Convolution Cross-correlation 1 2 3 1 2 3 4 5 6 4 5 6 7 8 9 7 8 9 w

f 1*1 + 2*2 + 3*3 + 4*4 + 5*5 + 6*6 + 7*7 + 8*8 + 9*9 Convolution 1 2 3 1 2 3 4 5 6 4 5 6 7 8 9 7 8 9 w f 1*9 + 2*8 + 3*7 + 4*6 + 5*5 + 6*4 + 7*3 + 8*2 + 9*1

Convolution Adapted from F. Durand Filters: examples * Original (f) 1 1 1 1 1 1 1 1 1 = Kernel (k) Blur (with a mean filter) (g) Source: D. Lowe Filters: examples * Original (f) 0 0 0 0 1

0 0 0 0 Kernel (k) = Identical image (g) Source: D. Lowe Sharpening Source: D. Lowe Sharpening What does blurring take away? = Lets add it back: = + original detail Sharpening Sharpening What does blurring take away? = Lets add it back: =

+ original detail Sharpening 0 0 0 1 1 1 0 1 0 1 1 1 0 0 0 1 1 1 Sharpening filter * Original

0 0 0 0 2 0 0 0 0 - 1 1 1 1 1 1 1 1 1 = Sharpening filter (accentuates edges) Source: D. Lowe Another example -1

-1 -1 -1 -1 -1 -1 -1 -1 -1 -1 1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 1 1 -1

-1 -1 -1 -1 -1 -1 -1 -1 1 1 1 -1 -1 -1 -1 -1 -1 -1 -1 1 1 1 1 -1

-1 -1 -1 -1 -1 -1 1 1 1 1 1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 1 -1

-1 -1 -1 -1 1 1 1 1 1 1 1 -1 -1 -1 -1 1 1 1 1 1 1 1 1 -1

-1 -1 1 1 1 1 1 1 1 1 1 -1 -1 1 1 1 1 1 1 1 1 1 1 -1

Another example Convolution is everywhere Horse Why is convolution important? Shift invariance is a crucial property Why is convolution important? We like linearity Linear functions behave predictably when input changes Lots of theory just easier with linear functions All linear shift-invariant systems can be expressed as a convolution Non-linear filters: Thresholding Non-linear filters: Rectification g(m,n) = max(f(m,n), 0) Crucial component of modern convolutional networks Non-linear filters Sometimes mean filtering does not work Non-linear filters Sometimes mean filtering does not work Non-linear filters Mean is sensitive to outliers Median filter: Replace pixel by median of neighbors Non-linear filters Takeaway Two general recipes: convolution cross-correlation Properties Shift-invariant: a sensible thing to require Linearity: convenient Can be used for smoothing, sharpening

Also main component of CNNs Next up Back to linear filters Signal processing view of filtering Filtering for detecting edges etc

## Recently Viewed Presentations

• Recovery. PC stores deplete quickly in around 8 seconds. Pc stores replenished - 50% in 30 seconds and 100% after 3 minutes. Oxygen stored in myoglobin depleted after bouts of exhaustive exercise, but can be fully relinked within 3 minutes...
• MarketingSherpa Membership: 8 Tough Lessons Learned From Our Test Launch Anne Holland President MarketingSherpa Monday, May 7, 2007 Membership launched 2/22/07 Screenshot memberhome #1.
• Linguistics Morphology: Word Formation Processes (Yule, 2003 & Jarvie, 1993) M.C. Rafael Velasco Argente Spring 2012 What's Morphology? Morphology refers to the study of how words are created in a language There are two processes involved in Morphology: Inflection and...
• South Carolina Geography. What is the coastal zone? 185 miles long and 10 miles wide. Atlantic Coast. Myrtle Beach, Georgetown, Charleston, and Beaufort ... What are SC's 3 river systems? Pee Dee River System. 1. north coast. 2. no lakes....
• Long interest in planning for and conduct of coalition warfare. How to explain the phenomenal success of the wartime Anglo-American coalition. Specifically examine whether the informal Anglo-American collaborative efforts prior to Pearl Harbor made a vital difference in terms of...
• From client briefing to design to construction to FM Product models and information management Support in early project decisions Demonstrate benefits of product models Prioritize agenda for further R&D Integrated Tools in PM4D Simulation and analysis tools 3D model of...
• Introduction. Operation and Maintenance (O&M) is a large contributor to the life-cycle cost of wind farms, therefore the prediction and optimization of maintenance activities provides a significant opportunity for life-cycle cost reduction.
• Create a lookup field — a field that provides a list of choices. ... and then entering a field name. All you have to do is click and type, and we'll show you how. For some tables, you can save...