UC Irvine startup ReadWithUs uses AI-generated image descriptions to help blind parents read picture books with their children.
The boy was partway through a page of “Dragons Love Tacos” when his mother broke in with a comment about the picture. One of the dragons, she said, had smoke coming out of its ears.
He paused. “How did you know?” he asked.
His mother is blind. Until then, they had shared the text, but not the pictures. Now she was hearing the illustration described through an earbud.
“It told me on my earphones,” she said. “Isn’t that cool? Now I know what the picture is.” The mother was testing ReadWithUs, an app developed in Stacy Branham’s lab at UC Irvine.
In picture books, the words may tell readers that a dragon loves tacos or introduce a girl and her big red dog, but much of the story happens elsewhere, in facial expressions, gestures and visual jokes that never appear in the text. For blind parents, story time often comes with an extra layer of planning and adaptation.
Branham, an associate professor in the Department of Informatics, has spent years studying how blind people use technology at home and with the people around them. The app uses AI to turn the text and pictures into audio, so a blind parent and a sighted child can read a picture book together.
A Book Built for Two
For years ReadWithUs was a research project. Then Branham got a phone call from Melissa, a blind mother who had been reading with her daughter using the app.
“ReadWithUs changed my life,” Melissa told her.
“She literally said that to us, and I think that was the moment I decided I needed to give this a real try,” Branham said. After that call, she started thinking seriously about how she could reach families outside the research setting.
Before a book appears in ReadWithUs, it is scanned into the system, where AI extracts the printed text, picks out the important parts of each illustration and generates descriptions of the images. The software then arranges everything in an order a screen reader can follow.
Once the book is in the library, a parent moves through it with familiar screen-reader gestures. A three-finger swipe turns the page. A one-finger swipe moves to the next item. A parent can listen privately through an earbud, then speak to the child in their own voice.
Most reading tools for blind people were built with other goals in mind. Print braille books make the words accessible, but the selection is limited, and the books can be expensive and slow to obtain. They also depend on a parent knowing braille. Even then, the pictures often go undescribed. Some accessible apps break a picture book into fragments and lose the two-page spread a child sees. E-books do not always tell a screen reader in what order to read the text and images, which can make a picture book confusing to navigate. Audiobooks drop the text and pictures altogether. And each tool imagines one reader. A picture book often has two.
That second reader is what drew Branham to develop the ReadWithUs app. Most accessible technology focuses on the workplace and on one person completing one task. Branham was more interested in life at home, where tasks are often shared. She has young children of her own, and she wondered how a blind parent would read a picture book with their child. So, she started asking blind parents.
A National Science Foundation (NSF) CAREER Award, a five-year grant for early-career faculty, gave her the means to act on what they told her. She hired graduate students Cameron Cassidy, who is blind, and Alyson Yin to help build the app. Branham describes the goal as allowing families to enjoy “the age-old tradition of telling stories, visual and textual, with their children.”
AI makes that possible at a scale that would be difficult to reach by hand. But it also makes mistakes. In one case, the system placed a Peppa Pig character in a scene where the character did not appear. Other problems are harder to spot. A description has to decide what deserves attention and when. Too much detail can bury the story. Too little can miss the point or give away a joke too soon. ReadWithUs keeps a human in the loop who can remove an errant pig and think about the overall story. Reviewers can rewrite descriptions or give the AI new instructions and have it try again.
After the Research
Melissa’s call came at the point where many academic projects wind down. “I could have collected the research awards and moved on,” Branham said. “That’s how the system is set up. The papers get recognized, and the prototype ends up in a digital trash can. After that call, I didn’t want that to happen.”
UC Irvine Beall Applied Innovation’s Proof of Product (PoP) funding gave Branham a chance to push ReadWithUs further. It supported another quarter of work by Cassidy and paid for an iOS developer with accessibility experience to build a more durable version of the app.
As part of receiving the PoP award, Branham also went through I-Corps, an NSF customer-discovery program that required her team to test its assumptions by talking directly with potential users and partners. They eventually conducted more than 120 interviews with blind parents, publishers, librarians and others who work with accessible books. Those conversations narrowed the likely audience to parents already comfortable with screen readers and also raised questions about how to build the library and how much editing the AI descriptions would need.
4%
Branham’s interest in accessibility began with a different kind of exclusion. She recalls that women made up about 4% of her computer science class when she was an undergraduate at Virginia Tech. But when an email arrived inviting her to join a women in computing group her freshman year, she deleted it, unsure why the group was necessary. By her junior year, she was the organization’s president. Although she had the aptitude for computing, she often felt as if she did not belong, an experience that made her more attentive to the people technology routinely leaves out.
Branham’s postdoctoral work increased that awareness even more. Her advisor was blind and had helped invent the screen reader technology blind people use to navigate computers and phones. By then, Branham had earned three degrees in computing, yet she had never really thought about how blind people and people with other disabilities used technology. That experience led Branham into research with blind and low-vision people, which became a central focus of her work.
Cassidy has been one of her main collaborators on ReadWithUs, and working with him is part of what makes the project meaningful.
“He’s a computer scientist who’s blind, and he’s making technologies that are going to help him and other blind people,” she said.
Branham is equally quick to credit the wider community around the project. Over the years, hundreds of blind parents, teachers, grandparents, aunts and uncles have tested prototypes, sat for interviews, offered feedback and introduced the team to others. Those exchanges shaped ReadWithUs and showed the team what the technology still needed to do.
The Next Page
Technology built for blind people has a history of reaching everyone else. Branham points to early “talking books,” created for blind readers on vinyl records. They helped establish the idea that a book could be listened to, and now many sighted readers listen to audiobooks every day.
Branham and her team are considering a similar path for ReadWithUs. The technology might also help bilingual families move between languages while reading together. A grandparent with declining vision might enlarge the type or increase the contrast of the text. A child could listen to a book alone when a parent is busy.
With the concept established, Branham is beginning to work through what growth will require. Some of the questions are about her team. Cassidy is graduating, which means Branham will need to replace a collaborator who brought both technical expertise and lived experience to the project. Other members of the team are staying on. Yin, an expert on image descriptions, continues to contribute, while Isabelle Bart, an industry mentor from the regional I-Corps program, remains an adviser. Melissa has also taken on a larger role as the first member of the company’s new advisory board.
“I feel like I have the support system with the knowledge to help me go to the next stage,” Branham said.
Other questions are about money and structure. Funding for accessibility research has become harder to count on, and Branham is weighing whether ReadWithUs should operate as a nonprofit, pursue a separate for-profit arm or combine elements of both. There are also basic market questions still to answer, including how many tech-savvy blind parents with young children might use the app. Branham has even considered taking a sabbatical to give herself more time to work through what comes next.
Even with those questions unresolved, Branham plans to keep going.
“I think it is a privilege and a responsibility for people with the resources that we have at UC Irvine to take the hard-earned insights from those who lack the same platform, like blind parents, and convert that into something that makes their everyday lives more equitable and more joyful,” Branham said.
While reading “Dragons Love Tacos,” a boy wanted to know how his blind mother knew there was smoke coming from a dragon’s ears. The answer was an earbud, though it had taken years of research, more than 100 interviews and a decision to keep building.