Visual Music Systems
Turns out the first TV screens were really small—and the original movie screens weren’t that much better if you were in the back row. VR environments are anything but small and what designers and artists need to figure out is how that affects the ways that worlds and scenes need to be developed.
To figure that out we need to take a step back and think about our eyes. The fovea is this tiny part of your retina (less than 1%) but processing its data takes up over half of your visual cortex. That’s because the fovea is responsible for our acute vision—for doing things like reading a book or watching our film. Because the fovea is so small, it actually only sees a tiny fraction of the world at once. Because that sliver is so small, our eyes are constantly moving so that the fovea can see everything. Those quick movements from one visual fixation to another are called saccadic movements. There are actually some cool graphs of just how much your eye jumps around like here. Our brains stitch together these quick images into what we see as a full field of view. There are a few other quirks of the eyes worth mentioning. One is our fixation on human faces—when they do eye tracking studies most of us tend to spend more time focuses on faces than anywhere else. That’s actually not true for individuals with Autism. There was a great early eye tracking study about Autism where people watched Who’s Afraid of Virginia Woolf? and the researchers tracked what in the scene they were focused on. Another is that while our focus is jumping from spot to spot we are mostly blind and what we think we are seeing is what our brain has stitched together.
Now back to screen size. The small screen size of TVs shaped the way that directors focused their films. Directors would zoom into one face and then jump to zoom into another face. That’s basically a mimicry of what our fovea does when we are in a room full of people. The directors would also use a lot of quick cuts where the scene would jump from one image to another. Again, that is similar to what our fovea does when it jumps from one fixation to another with blindness in-between the fixations. The directors’ styles of directing for TV felt natural for viewers because it mimicked the saccadic movements of their eyes.
The thing is this only really works on small screens that mimic and fit better within our foveal focus. When directors do sharp cuts and intense face zoom ins on a large movie screen, it tends to look awkward. Our eyes don’t naturally blow up a face to eight feet tall nor does our entire field of view jump quickly from one view to another. While these directing styles look awkward on a big movie screen, they begin to get painful on an IMAX screen. There are some IMAX directors who get that they are working with a large field of view and not just that small, central, foveal area. Other directors don’t get it and you will see fast scene cuts or extreme zooms on this huge screen. It doesn’t work and is, at best, annoying and, at worst, headache inducing. As the viewing screens takes up more of the viewer’s visual field, you have to start providing more visual continuity. As the screens get bigger, good directors will start to transition from sharp cuts to gradual shifts and smooth transitions. James Cameron does an excellent job of this—his films work really well on large screens like the IMAX because of this.
VR is at the other end of the extreme from a TV. You certainly can’t zoom in intensely on a face in VR. Can you imagine a 20 foot face filling your vision? You can’t even do a traditional soft cross fade as it would give the viewer a headache. Instead every scene has to somehow grow out of the previous scene and slowly morph into a new scene. There has to a continuous sense of reality. If you want the viewer to see one set of images at one time and another set of images a second later, part of your design challenge is how to smoothly morph what the viewer sees from that first set of images to the second set. How do you keep in continuous? You can’t introduce discontinuities without sickening your viewer. We have been working on techniques to provide these transitions in ways that let us maintain visual continuity for the viewer.