World’s Tiniest Violin Uses Project Soli — Created by Design I/O, World’s Tiniest Violin is a ‘speed project’ that uses Google’s Project Soli – Alpha Dev Kit combined with the Wekinator machine learning tool and openFrameworks to detect small movements that look like someone playing a tiny violin and translate that to control the playback and volume of a violin solo.
The team used the Project Soli openFrameworks example provided with the ofxSoli addon and searched for the signal that seemed to correlate closest with the tiny violin gesture. In this case it was the fine displacement signal, which then they fed the delta of to Wekinator via OSC.
Theo (Design I/O) then had to train Wekinator on what types of finger movements corresponded to playing the violin and which ones it should reject. So he recorded a few different finger movements and assigned the value of 1.0 on the slider. The slider to 0.0 and recorded gestures were then set which didn’t correspond: like pulling your hand away from the sensor, or just holding it there without moving your fingers.
After a few minutes of recording these gestures, the ‘training’ was initiated and they were then able to send back an animated value ranging from 0.0 to 1.0 representing how much Theo’s hand looked like it was trying to play a tiny violin. The last step was to map that number to the volume of the violin sample that was being played back by the openFrameworks app.
This project represents only a very small amount of what Design I/O think is possible with this technology, but it was remarkably quick to hook up and using a tool like Wekinator really made working with Machine Learning incredibly accessible and easy for working with such complex data.
Project Soli is a brainchild of Ivan Poupyrev who is currently directing a technical program at Google ATAP and was at Disney and Sony before that.
