The University of Arizona Partners with Carnegie Mellon University to Create A Bilingual, Bicultural Roboceptionist

The University of Arizona is working with Carnegie Mellon on a novel bilingual and bicultural roboceptionist, a computer with a face and natural language interface.

The $1 million, three-year grant from the Qatar National Research Foundation — shared between Carnegie Mellon University in Qatar (Prof. Majd Sakr, PI), Carnegie Mellon’s Robotics Institute in Pittsburgh (Prof. Reid Simmons, co-PI) and the University of Arizona (Prof. Sandiway Fong, co-PI) —  will fund fundamental advances in human-computer interaction. Carnegie Mellon University will provide the robotics innovations, while the University of Arizona will supply the language technology.

Sandiway Fong, who is an associate professor of linguistics and computer science, spent part of his sabbatical leave in 2009/2010 at Carnegie Mellon University's Qatar Campus, where an existing version of the roboceptionist, called Hala, currently greets visitors in English and Arabic.

“They invited me to join the team because they saw opportunities to generalize and improve the limited language capabilities of Hala,” said Fong. “They needed Hala to be more flexible in dealing with language input from users. And that is where the University of Arizona comes in. We will provide both the language-specific and inter-language-related cultural capabilities so that this robot can be, not just bilingual, but bicultural.”

Fong explains that a bicultural robot is not one that merely switches between English and Arabic modes, as in the current implementation of Hala, but rather has both modes simultaneously active in order to spot and deal with potential cultural ambiguities and misunderstandings.

“You may speak Arabic, but you may choose to converse with the robot in English,” said Fong. “You may be conversing with the sensibility and the cultural background and the idioms from the Arabic world. This robot needs to understand both.”

In another example, the phrase “week after week” as in “I’m looking for the group that meets week after week?” means “every week” in English but can map to “every other week” in some Arabic dialects. Only a robot with both lexicons active can compute that this phrase is subject to cultural variation and ask the user for clarification.

Culture affects, not only the syntax and semantics of an interaction, but also the structure of the interaction, from the way greetings and closings are performed, to the form of the language used, to politeness strategies.

“In American culture, we quickly greet someone and then we tend to get down to business,” said Fong. “In Arab cultures it is rude to actually get down to business right away. There is much more turn taking in greetings. Hala will know this.”

Hala, who has a backstory and personality, will adjust her responses based on cues from the visitor, essentially building a model of the user throughout the interaction: Is this person high status? Is this person American? Is this person in a hurry?

Under the conditions of the grant, two-thirds of the money will be spent to support research activity carried out in Qatar. Part of that money may be available for University of Arizona students to travel to Qatar and carry out work on the project in the form of internships. The remaining one-third will directly support work to be carried out in Pittsburgh and Tucson.

"This is wonderful recognition of Sandiway's work in this area and a great opportunity to showcase our growing strengths in computational linguistics and human language technology,” said Michael Hammond, head of the UA linguistics department. “We're hopeful that this is only the beginning of a productive relationship with Carnegie Mellon and Qatar."

Fong has already begun work on the project, and has hired incoming linguistics graduate student Samantha Wray, who has lived in Yemen and speaks Arabic, to help.

Right now Wray is working on creating a database of questions to train the robot. Most natural language parsers are trained on Treebank corpora, i.e., parsed sentences from newswire text. Unfortunately, news stories do not usually contain questions, greetings or dialog. Wray has found transcripts of interviews from Al Jazeera that could be data-mined for additional corpora material. The researchers will diagram, parse and generalize the data to help “teach” the robot how to understand human input in both Arabic and English.

“I'm very excited to be part of the project because I think that in developing these sorts of programs and in working with these sorts of robots, we eventually learn more about how we has humans function, how we learn, and how we interact on a communicative level,” said Wray.

Fong hopes to hire another student to help with the grant and also to provide internships for students in the master’s program in Human Language Technology (HLT), which he directs.

Fong is excited, not only about this project, but also about the possible applications of this work for other human-computer interaction systems. He thinks the technology could be applicable to computer help-agents, multicultural information kiosks, tour guides, and automated international call centers.

“We foresee a future in which robots will help bridge gaps between people of different cultures, acting as intermediaries, to enable them to communicate more naturally and effectively,” said Fong.

Contact:  Sandiway Fong, associate professor of linguistics and computer science at the University of Arizona,, 520-626-5657