# Submodular Dictionary Selection for Sparse Representation Volkan Cevher

Submodular Dictionary Selection for Sparse Representation Volkan Cevher [email protected] Laboratory for Information and Inference Systems - LIONS & Idiap Research Institute Joint work with Andreas Krause [email protected] Learning and Adaptive Systems Group

Sparse Representations Applications du jour: Compressive sensing, inpainting, denoising, deblurring, Sparse Representations Which dictionary should we use? This session / talk: Learning D

Which dictionary should we use? Suppose we have training data Existing Solutions Dictionary design functional space assumptions <> C2, Besov, Sobolev, ex. natural images <> smooth regions + edges induced norms/designed bases & frames ex. wavelets, curvelets, etc. Dictionary learning regularization

clustering <> identify clustering of data Bayesian <> non-parametric approaches Dictionary Selection ProblemDiSP Given candidate columns sparsity and dictionary size training data Choose D to maximize variance reduction:

reconstruction accuracy: var. reduct. for s-th data: overall var. reduction: Want to solve: Combinatorial Challenges in DiSP 1. Evaluation of Sparse reconstruction problem! 2. Finding D* NP-hard!

Combinatorial Challenges in DiSP 1. Evaluation of Sparse reconstruction problem! 2. Finding D* (NP-hard) Key observations: F(D)

<> approximately submodular submodularity <> efficient algorithms with provable guarantees (Approximate) Submodularity Set function F submodular, if D

D D D + v Large improvement + v

Small improvement Set function F approximately submodular, if A Greedy Algorithm for Submodular Opt. Greedy algorithm: Start with For i = 1:k do Choose Set

This greedy algorithm empirically performs surprisingly well! Submodularity and the Greedy Algorithm Theorem [Nemhauser et al, 78] For the greedy solution AG, it holds that Krause et al 08: For approximately submodular F: Key question: How can we show that F is approximately submodular? A Sufficient Condition: Incoherence Incoherence of columns:

Define: (modular approximation) Proposition: Thus is (sub)modular! Furthermore, is approximately submodular! An Algorithm for DiSP: SDSOMP

Algorithm SDSOMP Use Orthogonal Matching Pursuit to evaluate F for fixed D Use greedy algorithm to select columns of D Theorem: SDSOMP will produce a dictionary such that Need n and k to be much less than d SDS: Sparsifying Dictionary Selection Improved Guarantees: SDSMA Algorithm SDSMA Optimize modular approximation

instead of F Use greedy algorithm to select columns of D Theorem: SDSMA will produce a dictionary such that SDSMA <> faster + better guarantees SDSOMP

<> empirically performs better (in particular when is small) Improved Guarantees for both SDSMA & SDSOMP incoherence <> restricted singular value Theorem [Das & Kempe 11]:

SDSMA will produce a dictionary such that explains the good performance in practice Experiment: Finding a Basis in a Haystack V union of 8 orthonormal, 64-dimensional bases (Discrete Cosine Transform, Haar, Daubechies 4 & 8, Coiflets 1, 3, 5 and Discrete Meyer) Pick dictionary of size n=64 at random Generate 100 sparse (k=5) signals at random

Use SDS to pick dictionary of increasing size Evaluate fraction of correctly recovered columns variance reduction Reconstruction Performance SDSOMP <>

perfect reconstruction accuracy SDSMA <> comparable the variance reduction Battle of Bases on Natural Image Patches Seek a dictionary among existing bases discrete cosine transform (DCT), wavelets (Haar, Daub4), Coiflets, noiselets, and Gabor (frame)

SDSOMP prefers DCT+Gabor SDSMA chooses Gabor (predominantly) Optimized dictionary improves compression Sparsity on Average hard sparsity <> cardinality constraint <> group sparsity

matroid constraint Conclusions Dictionary Selection <> new problem dictionary learning + design dictionary Incoherence/ <> approximate / restricted singular value multiplicative

submodularity Two algorithms <> SDSMA and SDSOMP with guarantees Extensions to structured sparsity in ICML 2010 / J-STSP 2011 Novel connection between sparsity and submodularity Experiment: Inpainting Dictionary selection from dimensionality-reduced measurements Take Barbara with 50% pixels missing at random Partition the image into 8x8 patches

Optimize dictionary based on observed pixel values Inpaint the missing pixels via sparse reconstruction Caveat: This is not a representation problem. Results: Inpainting Comparable to state-of-the art nonlocal TV; but faster!

## Recently Viewed Presentations

• Polly Mullins-Bentley, ActingExecutive Director, CHFS, Governor's Office of Electronic Health Information (GOEHI) KHIE /GOEHI Overview. KHIE Update
• SQLLite and Java CS-328 Dick Steflik SQLLite Embedded RDBMS ACID Compliant Size - about 257 Kbytes Not a client/server architecture Accessed via function calls from the application Writing (insert, update, delete) locks the database, queries can be done in parallel...
• The radial-inflow admission of the working fluid is from one or more . ... Total-to-total efficiency of the turbine vs. channel width, at fixed 100°C total inlet temperature and total inlet pressure corresponding to a 10 K super heating level....
• Jun 25, 2014
• Changes II. Rapid, near-universal adoption of adaptivity (e.g., AVS) "setup violation" becomes hazy; removes "DC" part of timing margin. Path-based analysis with SI enabled is needed earlier in flow
• Looking Through a Different Lens . Set timer for 5 minutes. Get yourself a paper bag with prompts, role, and school. Set timer for . 10. minutes. Break out into small group (no larger than 7) with other "teachers", "administrators",...
• -Don't drive yourself crazy.- A strong UKCAT score may work in your favour however a low UKCAT score wont be too detrimental. Traits Medical/Dental schools want to see from you. Team work- What Team work skills have you witnessed during...
• Earthworm Dissection. This is the "anterior" end. (You would say it's the head, or the front.) It's the part you'll be concentrating on in the dissection.