Interactive Object Recognition Using Proprioceptive Feedback Taylor Bergquist,

Interactive Object Recognition Using Proprioceptive Feedback Taylor Bergquist,

Interactive Object Recognition Using Proprioceptive Feedback Taylor Bergquist, Connor Schenck, Ugonna Ohiri, Jivko Sinapov, Shane Griffith and Alexander Stoytchev Developmental Robotics Lab Iowa State University, Ames, IA, U.S.A. E-mail: {knexer | cschenck | ucohiri | jsinapov | shaneg | alexs } @ iastate.edu

What is Proprioception? It is the sense that indicates whether the body is moving with required effort, as well as where the various parts of the body are located in relation to each other. - Wikipedia The Importance of Proprioception

Full Empty The Importance of Proprioception Hard Soft

Exploratory Behaviors in Children [Power, 2000] Lifting: Weight, Gravity, Effort [http://www.subflux.com/blog/index.php?cat=8]

Shaking: Weight, Inertia, Contents Dropping: Gravity and Physics Pushing: Physics and Objects Crush: Compliance, Flexibility Five Exploratory Behaviors

Lift: Crush: Shake: Push: Drop:

Robot Platform Objects Used in the Experiments 50 household objects Different materials: metal, paper, plastic, wood, etc. Some objects have

contents inside of them (e.g., pill bottle) All are graspable by the Barrett Hand Torque Data Preprocessing Joint torque data was sampled and recorded at 500 Hz using the robots API The raw data was filtered to remove outliers

J1J7 Feature Extraction Joint torque data (500 samples/second in 7) Some way is needed to compress the data Discretize to obtain a sequence Pi of tokens from a finite alphabet J1 J7

SOM Training the Self-Organizing Map Problem Formulation Proprioceptive sequence Object Recognition

Model Predictive Models k-NN and global alignment Emphasizes temporal structure of the proprioceptive sequences Multinomial Nave Bayes and n-gram Emphasizes distributional structure of the sequences

k-NN k-NN: memory-based learning algorithm With k = 3: ? Test point

2 neighbors 1 neighbors Therefore,

Pr(red) = 2/3 Pr(blue) = 1/3 Uses Needleman-Wunsch global alignment as a measure of similarity between two sequences A guided tour to approximate string matching, by Navarro, G. in ACM Computing Surveys, v.33, 2001 Multinomial Nave Bayes (N-grams)

Sample Sequence 1-grams 2-grams 9 11 10

1 4 3 0 5 6 8 1

1 Multinomial Nave Bayes (2-grams) Sample Sequence 2-grams where n(wt, di) is the number of occurrences of word wt V in

the feature vector di. 1 4 3 0 5 6 8

1 1 Multinomial Nave Bayes (2-grams) Sample Sequence 2-grams where n(wt, di) is the number of

occurrences of word wt V in the feature vector di. 1 4 3 0 5 6

8 1 1 Evaluation Ten-fold cross validation 2500 total interactions, 250 per fold Evaluated on recognition accuracy, where: # correct predictions x

100 % Accuracy = # total predictions Chance accuracy: 1/50 = 2% Recognition from a Single Behavior

Recognition from Multiple Behaviors How to combine predictions from multiple behaviors? assume that all behaviors are equally useful weight behaviors according to their accuracy Object Recognition Results What happens when the robot uses information from multiple interactions with the same object?

Multimodal Recognition # Behaviors # Modalities Single Single Multiple

Multiple Multimodal Recognition # Behaviors # Modalities Single Multiple

Single Multiple This paper (proprioception only) Multimodal Recognition # Behaviors # Modalities

Single Multiple Single Multiple Follow up paper (proprioception + audio)

Interactive Object Recognition Using Proprioceptive and Auditory Feedback Submitted to IEEE Robotics and Automation Magazine (under review). Audio Data (for the same dataset) Audio data was recorded during data collection and transformed into spectrograms Raw Sound: Discrete Fourier

Transform: Multimodal Training (Two SOMs) Training a self-organizing map (SOM) using sampled joint torques: Training an SOM using sampled frequency distributions:

Multimodal Feature Extraction Discretization of joint-torque records using a trained SOM is the sequence of activated SOM nodes over the duration of the interaction Discretization of the DFT of a sound

using a trained SOM is the sequence of activated SOM nodes over the duration of the sound Multimodal Recognition Audio sequence

Proprioception sequence Proprioceptive Recognition Model Auditory Recognition Model Weighted Combination

Multimodal Recognition Results Related Work Audition

Kubus, Krger, and Wahl, 2007 Richmond and Pai, 2000 Torres-Jara, Natale, and Fitzpatrick, 2005 Sinapov, Weimer, Stoytchev, ICRA 2009 (3 objects) (4 objects) (4 objects)

(36 objects) Proprioception Natale, Metta, and Sandini, 2004 (7 objects) Accuracy vs. Number of Objects

Multiple Modalities + Multiple Behaviors Conclusions and Future Work Conclusions: Robots can and should use proprioception as a source of information about the world Better results can be obtained by combining multiple interactions and multiple modalities Future Work:

More complex behaviors and more objects Integrate proprioception with more modalities (vision, haptics, etc.) Take Home Message Number of objects recognition accuracy

Number of behaviors recognition accuracy Number of modalities recognition accuracy Thank you

Any questions? THE END Object Recognition Results The dotted lines show the best and worst case for the weighted and unweighted combination.

Recognition from Multiple Behaviors How to combine predictions from multiple behaviors? Previous work (Sinapov, Weimer, Stoytchev, ICRA 2009): assume that all behaviors are equally useful Choose the object Oi that maximizes B P(Oi | PB), where B is an exploratory behavior performed on the object. This assumption fails to hold for this work; weight behaviors according to accuracy instead

Choose Oi to maximize B P(Oi | PB) * wB, where wB is the estimated reliability of the model given a sequence from behavior B.

Recently Viewed Presentations

  •  There is a lot of free space in

    There is a lot of free space in

    Robert Boyle There's moreā€¦. Charles' Law At constant pressure the volume of a gas is directly proportional to temperature, and extrapolates to zero at zero Kelvin. ... You heat 1.437 g NH3 in a stoppered 250 mL flask until it...
  • Cell Communication: - WELCOME TO AP BIOLOGY

    Cell Communication: - WELCOME TO AP BIOLOGY

    Cell communication: neurotransmitter released at synapsesat the ends of axon on presynaptic cell Dendrite of the postsynaptic cell. 1. Action Potential . arrives. Presynaptic membrane is depolarized. 2. Depolarization . opens . voltage-gated Ca. 2+ channels.
  • Ultimate Recipe Planner - University of North Carolina at ...

    Ultimate Recipe Planner - University of North Carolina at ...

    Informal description. Purpose of the application: Constraint satisfaction on a meal Process description: Given a desired amount of macronutrients (protein, fat, carbohydrates), return a "basket" of food items to comprise a meal that fits the constraints within a certain fitness...
  • Chapter Two The Information Technology Manager

    Chapter Two The Information Technology Manager

    Performance - Refers to program speed and response time. A front office program requiring a user to execute eight steps to print a guest folio impedes organizational efficiency and customer service. Supportability - Refers to the ability to service and...
  • Chordates: Birds Tiara Jackson Shayna Brown AP Biology

    Chordates: Birds Tiara Jackson Shayna Brown AP Biology

    The Name of Your Subgroup Kingdom: Animalia Phylum: Chordata Class: Aves Body Structure External Bird Anatomy Internal Bird Anatomy Type of Covering Contour Feathers Contour, or vaned, feathers are the most specialized of all feathers. Contour feathers form the outline...
  • Comparative and Superlative Comparative and Superlative 1 Comparative

    Comparative and Superlative Comparative and Superlative 1 Comparative

    When we compare things we use a different form of the adjective: fat fatter fattest We say that fatter is the comparative form of fat - we are comparing them. ... Times New Roman Arial Optima Arial Black teachit powerpoint...
  • Childrens Future Food Inquiry Under the leadership of:

    Childrens Future Food Inquiry Under the leadership of:

    CONTEXT. 3.9 million children are living in poverty in the UK but little is known about how children experience food insecurity, how it impacts on their lives and what could be done about it. The Trussell Trust reports giving out...
  • The Nisonger Center

    The Nisonger Center

    Nisonger Center. Diagnostic Features (p. 37 & 38) Criterion B - adaptive functioning " Additional sources of information include educational, developmental, medical, and mental health evaluations. Scores from standardized measures and interview sources must be interpreted sing clinical judgment. "...