Kinect student project helping children with autism



An audio system which uses game development tools and Xbox 360 motion sensor Kinect is helping children with profound and multiple learning difficulties to interact through natural gestures.

The Kinect Audio Project (KAP) was created by Patricia Afari as part of her MSc Computing Studies at Goldsmiths, London and is being trialled at a school for children with special educational needs.

KAP is designed for users with autism spectrum conditions, mobility, learning or speech difficulties. And, Afari tells us, the inspiration for her workcame from a disabled friend.

“A good friend of mine, Lorenza Brookes, is a keen gamer and registered disabled,” South London-born Afari says. “After a while she finds it hard to keep playing games, especially the likes of fast-paced firstperson shooters.

“She chuckled and said, ‘Could you make something I can use?’ There are lots of people like her who struggle physically to grasp objects. So when Kinect came along I realised the potential for people to control games unencumbered.

“The original intention was to make a physical glove, but KAP means that when people enter a room and move in front of the camera the camera detects your hand and the virtual [one] jumps onto it.”

That immediacy makes Afari’s creation especially useful for people with autism and mobility problems, providing exercise, entertainment and learning experiences designed to aid social interaction. And by helping children to interact and recognise their own movements Afari hopes to encourage independence in the children using her software.

The technology is currently being trialled at South Downs Community Special School, Eastbourne. Afari visited the school over a period of several weeks on the invitation of musician and software designer Tom Smurthwaite, who has 12 years’ experience working with people with profound and multiple learning disabilities.

Afari observed children and sought advice from teachers on how she could develop software to enrich the young peoples’ lives. “I noticed they had a multi-sensory room with musical instruments,” she says. “I saw that the children with very severe disabilities couldn’t hold the instruments and I wanted to find a way for them to participate without holding anything.

“My initial version incorporated other software and took too long so I redesigned it using C#. The teachers wanted it to be very simple and now it’s one application out of the box.”

The latest version of KAP will undergo final testing next week. That promises to be an emotional moment for Afari, who recalls how she felt when she first saw children using her prototype software. “It was really touching,” she says. “It’s so moving to see them come into the room and then come alive. I feel really passionate about this. It’s a simple thing, but it can make a huge difference to their lives.”

Autism is the collective name for a range of developmental conditions which affect a person’s communication skills, imagination and ability to interact socially. Estimates of the number of people with autism vary, but it is believed that at least 1 in 100 people are on the spectrum, which includes those with Asperger syndrome. That means the potential number of people who could benefit from projects such as this is vast. And KAP – which uses Firelight Technologies’ FMOD Ex Programmer’s API – could become a required tool in special education, according to Afari’s supervisor Tom Smurthwaite.

This is not the first time Kinect has been used by scientists looking to better understand autism and support young people with special educational needs. The device was at the centre of a project to detect the telltale signs of the autism in the United States, as was reported by the New Scientist in May. A team from the University Of Minnesota used Microsoft’s hardware to monitor children aged between three and five for early signs of autism. The sensors’ cameras tracked the children and fed data to PCs which calculated the average movement levels in the room. Any child who appeared to be hyperactive or unusually inactive – two possible signs of autism – were examined in one-to-one sessions by doctors. At the University of Michigan Dr David Chesney tasks students with creating Kinect games designed to engage children with autism.

Given global interest in the field and Afari’s passion for the project, this may not be the end of the KAP story. Afari is keen to refine the software further, she says. “In future I hope to make a music game, a gesture mixer. I’d like to explore theKinect microphone array. Maybe I could get movement and voice in there, with people recording themselves and mixing. This is just the beginning.”

If the call comes from a mainstream developer, Afari is willing to answer however. “I’m open to work on any project that wants to be inclusive. If there is a game which can be adapted to use the Kinect so anyone can play, that’s what I’m interested in. I don’t mind where I am working as long as inclusivity is important to the project.”