Oxford Brookes University
Complex theoretical mathematics at Oxford Brookes University has helped to solve a gaming challenge and deliver a realistic and natural playing experience for Sony’s new WonderbookTM.
WonderbookTM is an augmented reality system developed by Sony for PlayStation 3 which allows the player to interact with the game by using a physical book tracked by a camera. The tracked location of the book and its pages create a virtual pop-up book on the screen, allowing the player to interact with the virtual aspects of the game in real time. But many existing AR systems are unable to identify human elements – which meant if the player had their hand over the book in the physical world, on screen it would look like their hands were below the book. This is because the virtual contents would be overlaid on top of the entire book surface including their hands.
Researchers at Oxford Brookes University set up a Knowledge Transfer Partnership with Sony Computer Entertainment Europe with the aim of creating a robust human hand tracker and segmenter, which would calculate the position of the player’s hand and segment it in a live video in real time.
The problem they had to solve was how to identify which parts of the images belong to the player’s skin. There were too many unknowns that had to be incorporated including the wide diversity of skin tones and the varying light levels that could be present in whichever room the player has set up their PS3 in. The researchers turned to statistical machine learning and computer vision techniques to analyse videos of a diverse set of players interacting with WonderbookTM under different illumination conditions. As the system needs to run in real-time and due to the large amount of data needed to be processed, algorithms were created which could make fast calculations needed while using a minimal amount of computer time.
In 2011, Sony signed a deal with JK Rowling to extend the Harry Potter franchise into its own content. Their first Sony WonderbookTM was the ‘Book of Spells’ from the World of Harry Potter series, complete with new writing by the author, released in November 2012. The technology has taken story-telling ‘to a whole new level’ and further Sony WonderbookTM titles will be released including Walking with Dinosaurs and the next in the Harry Potter series ‘Book of Potions’.