More Lively Conversations with Deaf and Hard-of-Hearing People? Instant Subtitles for Spoken Dialogue

Transparent Panels Allow for More Natural Conversation

The National Museum of Emerging Science and Innovation (Miraikan), located in Koto City, Tokyo, offers a wide range of exhibits related to science and technology, from the wonders of life and the universe to the latest technologies. In June, August, and December 2021, tours of the permanent exhibition were held for DHH people by the museum's Science Communicators, who communicate information about science and technology in an easy-to-understand manner.

The Science Communicators had "See-Through Captions" in hand-transparent subtitle panels that display what speakers say in real time. When a speaker talks into the microphone, the tool automatically converts speech into text, displaying the content of what was said on the transparent panel like subtitles.

This innovative new tool was created by five students belonging to the same lab at the University of Tsukuba. See-Through Captions' development began three years ago. It all started when Shitara Akihisa, who deaf (mixed hearing loss), joined the Digital Nature Group lab at the University of Tsukuba. Suzuki Ippei, one of the developers, reflects on those days:

"We were discussing what the best way would be for Shitara and the 40 other lab members to communicate. We explored orthodox methods like written communication and chat apps, but found that while they conveyed exactly what we wanted to say, they took too much time. Voice-to-text apps are an easy way to converse that we do still use, but it's difficult to see the speaker's facial expressions while communicating. That's why we decided to develop our own tool that would allow us to communicate more naturally."

There has been research into using smart glasses to display subtitles as one attempt at achieving more natural conversation. Words are displayed as text in front of the speaker's face using AR technology that projects the text onto the real world as a CG image.

However, since only the person wearing the smart glasses can see the text, the speaker is unable to know whether what they say is being converted correctly. "We felt it was necessary for both parties to be able to see the text for smoother conversation," says Suzuki.

At that same time, Japan Display Inc. provided them with a prototype of a transparent panel for another research project. They wondered if it could be used to display text, and by combining technologies used to display text on PC monitors with Google's voice-recognition technology, they developed See-Through Captions. As text is displayed on both sides of a transparent panel, speakers are also able to check it.

From a Tool Supporting Conversations Between DHH and Hearing People to One Used in a Wide Range of Scenarios

Thus, a tool supporting face-to-face communication with DHH people that also allows facial expressions to be seen was born. To date, two versions of See-Through Captions have been developed: a stationary type and a handheld type. They have been introduced on a trial basis at Miraikan and Tsukuba City Hall's general information counter, where the team has interviewed users to continually verify and improve the tool to make it easier to use.

At a tour held at Miraikan, some DHH participants said that "the transparent panel felt more natural than expected. It's better than using a smartphone or tablet (for voice-to-text)."

Shitara, one of the developers, who is deaf himself, says, "I'm not saying that using See-Through Captions is necessarily the best way to go, but I hope it becomes an option." Suzuki also feels that they have gotten a good response through the practical experiment at Miraikan, and that the tool will be able to support communication not just for DHH people but across a wide range of situations, like for elderly people who may have difficulty hearing conversations.

"When we did the trial introduction of See-Through Captions, we lectured the Science Communicators on the kind of sentences and speech patterns that are easy to convert into text, and had them practice beforehand. That's when I realized that even the technology we have today can be quite practical when combined with a little human ingenuity. It'll even be convenient for those who can hear alongside those who can't. I think this tool will not only contribute to achieving diversity, but also support smooth communication for hearing people."

See-Through Captions transcends the barriers between DHH and hearing people to provide comprehensive conversational assistance. Should this project develop further moving forward, there may be opportunities to see it not only at Miraikan, but at various locations around Tokyo. Hopes are high for its potential as a next-generation communication tool in facilitating interaction and the exchange of information among people.