New Apps for Visually Impaired Users Provide Virtual Labels to Explore Images

A team from the University of Michigan in Ann Arbor has developed two free tools for visually impaired iPhone users. One can read the labels on control panels while the other identifies features in an image so that users can explore it through touch and audio feedback.
285
Haptic controls
VizLens uses a smartphone’s camera to view control interfaces, such as the one on this microwave, and read each label. When a user touches the button in the camera’s view, the smartphone can read out the label. // Photo courtesy of U-M Human-AI Lab

A team from the University of Michigan in Ann Arbor has developed two free tools for visually impaired iPhone users. One can read the labels on control panels while the other identifies features in an image so that users can explore it through touch and audio feedback.

VizLens is a screen reader that can function in the real world. It reads labels at the direction of the user, who points at buttons of interest on control panels.

With it, users can use smartphone cameras to operate a variety of interfaces in everyday environments, including home appliances and public kiosks. The research is funded by U-M with additional support from Google.

“A blind user can take a picture of an interface, and we use optical character recognition to automatically detect the text labels,” says Anhong Guo, assistant professor of computer science and engineering at U-M, who led the development of both apps. “A user can first familiarize themself with the layout on their smartphone touchscreen. Then, they can move their finger on the physical appliance control panel, and the app will speak out the button under the user’s finger.”

ImageExplorer helps visually impaired individuals better understand the content of images. Guo’s goal is to offer visually impaired people the ability to use their phones when alt text is missing or incomplete, as AI-generated captions often are not sufficient.

“There are a number of automated caption programs out there that blind people use to understand images, but they often have errors, and it’s impossible for users to debug them because they can’t see the images,” says Guo. “Our goal, then, was to stitch together a bunch of AI tools to give users the ability to explore images in more detail with a greater degree of agency.”

Upon uploading an image, ImageExplorer provides a thorough analysis of the image’s content. It gives a general overview of the image, including the objects detected, relevant tags, and a caption. The app also features a touch-based interface that allows users to explore the spatial layout and content of the image by pointing to different areas. ​​

Hundreds of visually impaired participants have experimented with VizLens and ImageExplorer, offering feedback to Guo’s team, which is continuing to develop these tools. First discussed in 2022, ImageExplorer is a much newer concept than VizLens, which made its debut in 2016.

Some of its details need further refinement, and different tools within ImageExplorer sometimes give conflicting information, according to the developers.