Google Creative Lab asked us to design an Android Experiment using the Cinder open source graphics platform. We set out to use modern mobile graphics technology to expand on our work with real time video filters from a decade ago. In the process of doing so, we somehow opened up a metaphysical portal that lets you travel deep into the hyperspace of your own selfie.
(OK, so it’s really an Android app that uses your phone’s camera, motion sensors, and graphics hardware to generate a reactive 3D universe that responds to your movements.)
How did we get here? Our first step was to build a software pipeline that used the live pixel buffer of the camera as an input to a series of shaders. After a few experiments with mapping pixels to different geometries, we settled on the idea of a three-dimensional landscape modulated by the camera feed, and navigated by device movement.
After a test where we flipped the camera inward, we decided that the app should be a mirror. The added body interactions were just too much fun. The self-reflexive experience of navigating a world generated by your own image as you navigated that world—and our personal fascination with selfie culture—led us down the rabbit hole of Selfie x Selfie.
Next we tried buffering the last few seconds of video. By mapping the history of the camera feed to the shape of our geometry, we could create ripples in our scene. Very satisfying ripples. At this point, we had several compelling graphical treatments and dreamed about a way to stitch them together into one mathematically generated surface. To accomplish this, we created a separate software tool that allowed us to map filter parameters to revolved curves drawn in Adobe Illustrator. This let us send continuous ripples out across the generated terrain.
From there we followed our intuitions, simultaneously coding and designing as we went. The more it made us do ridiculous things with our bodies and laugh out loud, the more we knew we were on the right track.