The sensation of movement in visual music is what gives the viewer a sense of rhythm and the physical sensation of being in the environment. Yet, convincing the viewer’s brain that he is moving and that the virtual world is standing still, is a challenge. For VR developer’s working with realistic imagery, this isn’t a challenge. If the viewer is standing in a room and the walls start to move, his brain is going to assume that he is moving and the walls are standing still. Likewise, if he is standing in the middle of the room and see a mouse racing across the floor his brain will quickly decide that he is standing still and the mouse is moving.
The challenge comes for developers working with abstract imagery. The viewer’s brain has a more challenging decision to make when the whole environment is just abstract shapes. In visual music, we want the viewer to feel that they are moving because the range of emotional communication and physical sensations are totally different for the two concepts. If you feel that you are moving up and down through objects, your body’s balance system kicks in and you feel it in your stomach. If you think the objects are moving around you instead, you miss out on those sensations. There are a whole range of sensations that are tied to your inner ear and your sense of movement, which is why we have had to figure out how to convince the viewer’s brain that he really is moving.
If all that exists in your visual world are three balls and you move straight forward between them, your brain doesn’t have enough information to decide if you are moving or if the balls are moving. The brain’s default is generally that the balls must be moving, not you. Because you don’t think you are moving, you don’t have any sensation of your own speed. But if instead of having 3 balls, you have thousands of balls all intricately arranged in a 3D environment and you see the balls closer to you approaching more quickly, your brain will quickly deduce a 3D model of the scene and decide that you are moving. Your brain perceives the consistency of the movement of the balls and the increased speed of the closer balls and it gets enough cues to convince it that you must in fact be moving.
We play with this all the time in our visual music. We have a whole class of visual instruments called environmental instruments that are designed to express themselves while being anchored. They are placed at fixed places in the environment and stay rooted. As the viewer moves through the scene, the brain detects the relative motions, builds the 3D model, and infers the viewer movement. Because of the number of the environmental instruments and their spatial relationships, our viewers perceive that they are moving and not the world. We are able to have the environmental instruments move slightly—think trees swaying in the room but we have found that there are fixed limits to their movement. If we have them move too much, the viewer stops feeling that she is moving and instead begins to perceive that the environmental objects are moving towards her.
Because of the regularity of our environmental instruments, we are also able to add in dancer objects. The dancers move relative to everything else like birds moving in trees without changing the viewer’s perception that she, and not the overall environment, is moving through space.