WP 3.3 Natural Language Generation and Speech Synthesis

WP 3.3 Natural Language Generation and Speech Synthesis

Information Extraction Tools and Methods for Understanding Dialogue in a Companion R. Catizone, A. Dingli, H. Pinto and Y.Wilks University of Sheffield LREC 2008 Morocco Senior Companion Developing a range of companions to assist the elderly

Photos QuickTime anddecompressor a TIFF (Uncompressed) are needed to see this picture. News QuickTime and a TIFF decompressor are (Uncompressed) needed to see this

picture. Yellow pages The Senior Companion is about Multimodal dialogue for building a picture of QuickTime and a Photo - JPEG decompressor are needed to see this picture. through his photographs

the user Behind the scenes using the dialogue to tag photos Octavia family holiday Venice 2007 summer Photos

A Photograph application for reminiscing about personal photos. The user begins with a set of photos which will be input into the system in advance. The system engages in conversation with the user about the photos. The system will create a life narrative by extracting key facts from the user about his/her photos and the people, places and events that are represented. The information about the photos can later be retrieved and

displayed in a future user session. Semantic features extracted during the dialogue will be automatically associated with the photos ( may include audio recordings) Life Narrative through Photos Life Narrative - Dynamically builds segments of a persons life that correspond to relationships with: People and Places with respect to Life events (Birthdays, marriages, etc) Journeys and Travels

Special memories News News reading Live from BBC RSS feeds Retrieved through categories Sports, business, international, etc. QuickTime anddecompressor a TIFF (Uncompressed) are needed to see this picture. Yellow pages (future)

To incorporate helping with everyday interactions such as yellow pages assistant for finding information. Example for finding a plumber User: Hello Morgan, my toilet is broken, I really need to find a plumber. Can you help me. System: Sure I can,the broken toilet is at your home on Hayfield Rd right? User: Yes System: Let me have a look and see what I can find. User: Ok System: Yes, Ive found a local plumber in your area. Would you like me to get him on the phone using Skype. User: Yes please System: Ok here you go

QuickTime and a TIF areF (U Q nenc uic edom k T ed preim to sse se ed a e ) dnd th e a is co picm

tu pre re s s . o r QuickTime and a TIFF decompressor are (Uncompressed) needed to see this picture.

QuickTime and TIFF decompressor TIFF (Uncompressed are (Uncompressed) needed to see this picture. are needed to see t The System

Fusion Napier Interface Input Queue Avatar DialogAct Tagger NLU

GATE Dialog Manager NLG Output Manager Interface Layer TripleStore Senior Companion Architecture

Modules Interface + Avatars Face identification (Open CV)

Fusion ASR Natural Language Understanding Dialogue Manager Knowledge Representation/Reasoning TTS Interface Multimodal - speech and touch (using a touchscreen tablet) Includes photos, avatar and text box Currently designed to discuss users photos and read news.

Displays multiple photos at a time Photo management Photo selection Avatars Experimenting with different avatars Morgan (AAA) Crazy Talk

Woman Ken Dodd Lion A computer Interface (2) Interface (2) Please read me the news OpenCV Face identification software

Finds the coordinates of all of the faces in a photo. Cannot recognize the same face in more than one photo, but investigating face recognition software: Polar Rose Photo Applications Also use digital photo metadata such as Date and time GPS coordinates Photos that have annotations (Facebook) More information to start makes for more interesting dialogue

Samuel Carter Thomas Clark Ed Bloch Olivia Ford Fusion Merges speech and pointing input This is my daughter Octavia ASR Dragon Naturally Speaking

requires 10 minute training session accuracy is high - up to 99% application integrated with the SC full integration of the code using the Nuance SDK planned for next version Natural Language Understanding

Sentence splitting POS tagger Parser Annie Named Entity Recognizer (GATE, USFD) person names (10,000) locations (Web trawl) 60 relation names mother, brother, sister, friend, etc. Dialogue Act Tagger (ALB). Populates Ontology instances. To use an IE approach for semantic interpretation

NLU: The task so far 1. Identify people and their relationships to the user and each other 2. Identify locations The process Utterance Gate Dialogue Act Tagger Annie Syntactic Parser

Ontology Processing Syntactic and Semantic representation Gate Noun Chunks Verb Phrases I went to meet Lynne, my old flatmate, together with my sister Francesca. PRP VBD TO VB NNP PRP$ JJ NN RB IN PRP$ NN NNP

Annie Relationships I went to meet Lynne, my old flatmate, together with my sister Francesca. Persons Syntactic Parser I went to meet Lynne, my old flatmate, together with my friend Francesca. Who is the sister and who is the flatmate? ((((I))((went)(((to)(meet)(Lynne)(,)(my)(old)(flatmate)(,)) ( (together))((with)( (my)(sister)(Francesca))))))(.)))

(((I)(went((to meet Lynne , my old flatmate,) ( together)(with( my sister Francesca))))).)) (((I)(went((to meet Lynne , my old flatmate,) ( together)(with( my sister Francesca))))).)) Syntactic bindings will help us identify that! Ontology Reasoning Allows us to infer new knowledge Sister(My) = Francesca Flatmate(My) = Lynne

If we have in the ontology: Mother(My) = Mary We can infer through the properties of the Sister relationship that Mother(Francesca) = Mary Daughter(Mary) = Francesca Knowledge Base Relationships Ontology Almost 40 person classes (mother, father, etc) Almost 40 relationships (has friend, etc) Will be adding more

Locations Ontology Most continents, countries, regions, cities Plus several places of interests per region top 10 things to do in cities Will be adding more DAMSL Dialogue Act set * ASSERT ("This is my sister". Yes and no are also considered asserts, where they are responses to yes/no questions, thus abbreviated assertions.) * OFFER ("Shall we look at another picture?") * COMMIT ("Okay I'll do that")

* EXPRESSION (All social expressions such as "you're welcome". Also things like "wow!" and "great!") * INFORMATION REQUEST (open question) * CONFIRMATION REQUEST (yes/no question) * REPEAT REQUEST ("Pardon?") * ACTION DIRECTIVE ("Show me another one." All imperatives.) * OPEN OPTION ("We could look at another picture." Stating an option in a way that doesn't demand an answer.) * OPENING ("Hi") * CLOSING ("Goodbye") * ANSWER (An answer is invariably also an assert. Yes/no answers are asserts.) * BACKCHANNEL ("Uhuh") * REPEAT REPHRASE (Expressing understanding by paraphrasing)

* COMPLETION (Completing the utterance of the other speaker) * NON-UNDERSTANDING ("I don't understand") * CORRECTION (An assertion that corrects a previous assertion) * ACCEPT (Accepting a proposal) * REJECT (Rejecting a proposal) Semantic specification Each dialogue begins with a user object and a picture object User object User/Person objects have

name, people relations,age QuickTime and a Photo - JPEG decompressor are needed to see this picture. Picture object Name Relations Age location, occasions, people, dates)

Person: Zoe Occasion: Hilary Duff concert Location: Birmingham Person: Roisin, snooks Location Occasion People Picture object Information Extraction in the SC Why do we want to use IE for Semantic

representation? User utterances are unstructured Using GATE IE tools to create templates Relationship between the Named Entities and the significant Events. Categorize events into meaningful classes Information Extraction (2) Example sentence That is my daughter Zoe on the right

Simple Example template : Relation Person1 Related-to Person2 Related-to : is-relation is-relation: is-daughter, is-mother etc. is-daughter: lexical string: is my daughter Filled IE Template Relation : [Person1=Zoe], [Person2=Roberta], [Related-to=is-daughter]

SC Ontology for Inference SC Ontology Here are my daughters, Zoe and Octavia in New York City Infer using the family relations Ontology that: -Zoe and Octavia are sisters -Roberta is the mother of Octavia and Zoe Dialogue Manager (1) Manages discussion of Users photos with respect to

Location Time it was taken Occasion People in the Photo (exploits positional information of people) Name Age Relation to the user

Photo Management Show me all the photos of my mother Photo selection News Reading and stopping Choice of politics, sport or business Dialogue Manager (2) Accepts pre-annotated photos to allow the system to engage in more interesting conversation more quickly. Handles basic photo management tasks:

Show me all the photos of my mother Please move on to the next photo Responds when same person is mentioned in more than one photo (using the users name) Remembers user information from multiple sessions Generation : template based Dialogue manager (3) Dialogue Manager adapted from COMIC DM General purpose control structure that does the dialogue planning

Stack based system Conforms to common behaviour of conversation: Discuss topic 1, move to topic 2, go back to topic 1. Augmented transition networks, called Dialogue Action Forms (DAFs) for handling domain sub-tasks. References the Dialogue History, Knowledge base and the User Model Dialogue Action Forms GUI editor for creating DAFs Composed of nodes and arcs containing tests and actions

DAFs pre-stacked, but can be overidden by matching indexing terms (semantic classes, significant words) Essential for mixed-initiative conversation SC Dialogue Manager Stack System start Run Greeting DAF Pop greeting DAF Push photo DAF Run photo DAF

Run people DAF Pop people DAF name age people relation event event

date date occasion occasion greeting photo

location location goodbye goodbye goodbye goodbye People DAF

DAFs (1) Machine Learning in the Senior Companion Using first Senior Companion prototype to generate more data, augmenting WoZ data gathered by NAP and AAA and hand annotated. Plan to tile (as in Hearst) the data to seek segmentations corresponding to topics and dialogue moves. Plan to generalise across a set corresponding to same topic or move to generate a draft Dialogue Action.

www.companions-project.org QuickTime and a TIFF (LZW) decompressor are needed to see this picture. Thank You Evaluation of the Senior Companion Accuracy Is the information that the system discusses accurate Important when the user returns for repeat sessions

(system needs to remember and recall collected information at the appropriate time). Does the system discuss things in an efficient way? (not ask for clarification when the information is already known) User satisfaction Self-assessment: May include some testing of the users level of contentment with the system while running.

Recently Viewed Presentations

  • Powerpoint template for scientific poster

    Powerpoint template for scientific poster

    You may use this template for educational and non-profit use. Please acknowledge its source, and please send feedback to: [email protected] If you are using site or template for a course on Blackboard or WebCT, please give me Guest access, or...
  • In parallel…

    In parallel…

    The costs of Ebola vaccination will vary depending on the vaccination strategy used to reach the target population. The costs of vaccinating a target population in vaccination campaigns can vary considerably depending on the type of campaign strategy used. For...
  • 7 Ways of Thinking About Art

    7 Ways of Thinking About Art

    The Intentional Fallacy Wimsatt and Beardsley Supposed Romantic fallacy of focusing on author psychology The poem belongs to the public… Appeal to intentions either misleading or redundant… (cf Stanley Cavell - boxer) Scrutiny vs Retrieval Polar opposites = Wimsatt vs...
  • Enhancing Person-Centred Care through ResearchBased Drama Christine Jonas-Simpson,

    Enhancing Person-Centred Care through ResearchBased Drama Christine Jonas-Simpson,

    Christine Jonas-Simpson, Ph.D., Sherry L. Dupuis, Ph.D., Gail Mitchell, Ph.D., Colleen Whyte, Ph.D.(c), Jennifer Gillies, Ph.D., and Jennifer Carson, Ph.D.(c) We declare no conflict of Interest. I'm Still Here Clip from ISH The Long Good-Bye The 36 Hour Day the...
  • 10

    10

    A&P Flix™: Muscles that act on the hip joint and femur:An overview. PLAY. Muscles Crossing Hip and Knee Joints. Most anterior muscles flex femur at hip, extend leg at knee (foreswing of walking) Most posterior muscles extend thigh, flex leg...
  • מונחים בסיסיים באנטומיה

    מונחים בסיסיים באנטומיה

    Prefixes and suffixes used in forming medical terms PREFIX REFERRING TO: PREFIX REFERRING TO: a-, an - without, absence of cyst - sac filled with fluid ab - away from derm - the skin acr -, ac - sharp, point...
  • GETTING PASSIONATE ABOUT OUR PURPOSE (Part 1) Matthew 28:16-20

    GETTING PASSIONATE ABOUT OUR PURPOSE (Part 1) Matthew 28:16-20

    LEADING WITH THE MIND OF CHRIST Philippians 2:5-11 Philippians 2:5-11 (NIV) 5 Your attitude should be the same as that of Christ Jesus: 6 Who, being in very nature God, did not consider equality with God something to be grasped,...
  • What foods contain protein?

    What foods contain protein?

    Describe the functions of protein in food production . Explain the different uses of eggs in food production. Define the role of protein in the body. What foods contain protein? High biological value protein = protein from animals.