Reflex Arc: gesture recognition can take you places

The brief

JISC TechDis are the UK advisory service on technology and inclusion, supporting the education, training and skills sectors.

Back in 2011 they asked Jonathan to judge a gesture recognition contest. Not happy with simply judging, Jonathan put together a team to win it.

The brief? To create a digital application using gesture recognition, to help learners with disabilities who lack independence due to an inability to communicate by speech or due to lack of motor control.

Our approach

Working with two companies from his BBC days – Reflex Arc and Gamelab UK – Jonathan put together a proposal for the contest, securing two phases of funding to create a sign-language recognition system to help autistic & learning-disabled Makaton users make an easier transition into independent living and employment.

Leading the project’s user-research, Hassell Inclusion worked with specialist colleges across the UK to investigate the user-contexts in which sign-recognition would be useful for students who use Makaton. The user-research identified ways of motivating our target users to use the evolving recognition technologies being created by Reflex Arc and Gamelab UK, resulting in gameplay concepts to engage users with signing in familiar scenarios.

When we started working with Hassell Inclusion to help students with communication difficulties, I never thought it would give me experience and skills that would help me create innovative mainstream VR experiences for the likes of Reebok. This is where thinking differently has got us. It’s been quite a journey...

The outcomes

The resulting uKinect sign language eLearning games enabled these young people, along with their new non-signing colleagues, to learn workplace-specific sign vocabularies to help them transition from college to work environments.

The games used an engaging animated avatar called Boris who demonstrated and responded to signs via an innovative Kinect sign-language recognition system.

As well as being useful for these learners, the ‘Boris Games’ also demonstrated what was possible through sign-language recognition and won the TIGA Games Industry Award for Best Education Initiative in 2013 and was runner up for the Japan Prize 2015.

 

 

 

 

 

The legacy

Over the next few years the innovations that we created in response to the needs of these initial groups of people enabled us to create similar breakthroughs for other groups.

Through Microsoft, whose Kinect technology was at the heart of our gesture recognition, we worked with:

  1. Guide Dogs, who were looking for ways of making mobility training more fun for blind and partially-sighted young children. Working with a school for vision-impaired kids in North London, we created the ‘Nepalese Necklace’ games which took their body and spatial awareness exercises, and made them controls for motivational 3D audio-games, turning ‘boring’ exercises into fun activities. The games transformed the kids’ feelings towards their mobility training and also improved their concentration on the rest of their schoolwork. They also won our team Guide Dogs Partner of the Year award.
  2. A group of stroke outpatients in Shepherds Bush, London, who were looking for motivational games to help them with their rehabilitation. We were inspired by a project which used Kinect games to help stroke rehabilitation, but wanted to create specific games which used movements specifically recommended by their therapists. If they could use these games in their homes to drive their recovery of function without requiring occupational therapists and physio-therapists to be present to guide and motivate them, this would free up therapists to concentrate on higher levels of care, making more effective use of their valuable time. We secured funding from InnovateUK and, over the course of the next 6 months, we proved that we could do this. Our project was voted best presentation in its category at the Collaboration Nation event showcasing all Innovate UK funded projects in 2014.

Since then more work by Gamelab UK and Reflex Arc has taken the results of this innovation further into rehabilitation, adding VR components, and beyond…

Intrigued? Check out our services...

Contact Us About Our Services

  • Innovation Programme

    Key: Continually Evolve

    A programme to help your digital teams think differently by engaging with disabled people's needs, inspiring them to deliver innovations in your proposals and products.

  • Benefits of Digital Accessibility Workshop

    Key: Expand Awareness

    A short workshop to help you build the case for, and put inclusion and accessibility right into the heart of your digital products, understand what it takes to embed this across your organisation, and how to use the benefits of digital accessibility to meet your organisation’s business goals. We can deliver a bespoke workshop, or you can attend a public workshop - both can be delivered remotely.

     

  • Inclusive User Testing

    Key: Enable Process

    We provide two options, we can do inclusive user testing for you, or add testing with disabled people to the user testing you’re already doing. And we can also train your user testers to do this themselves.