StorySignStorySign

Learning to read can be difficult for any child, but for children who are profoundly deaf it can be an insurmountable challenge. StorySign is an AR app created to help deaf children overcome this difficulty and open up the world of books to them. Children scan the pages of physical story books with their device to reveal animations of our character, Star, signing the relevant passage of the story with the correlating word highlighting on the page as she signs.

We encountered challenges when considering character design and animation for deaf children; often in animation, we strip back features to their most basic, or drastically change proportions, but for this project it was important to ensure Star was both ‘humanistic’ but also a character kids would be excited by, and one we could position as an identifiable role model to a wide community of deaf children.

When animating, we motion-captured real sign language performers in order to get the most accurate performance possible. The process involved suiting up our actors in a motion-capture rig with a high fidelity marker set-up that recorded their full body and facial performance from over 70 infrared cameras placed around the studio. These performances were then correlated with the books’ metadata, and therefore associated with the real books using a machine-learnt AI image to text system, ensuring a seamless experience for young users.

Books are available across 14 languages and the app – available for both iOS and Android - is supported by International Deaf Associations and charities around the world.