When we create visual music we aren’t creating in 3D—we are creating in 4D. In addition to creating in an up and down, left and right, back and forth world we are also creating a world that changes over time. What you see and think of as a 3D model is only one slice of time, like a CAT scan shows 2D slices of a 3D reality.
Our visual music environment 3D is changing dynamically as the viewer moves through it. In traditional 3D modeling and animation people create a 3D model of a place and then after they have created the model, they create a path through it. The 3D model is a fixed thing and what changes is where you are at any moment in time. Think of moving through a room—the animators created a 3D room and your path through it. As you move through the room you see different things because of where you are standing but the room itself does not change. These rooms are fundamentally static, which doesn’t work for music.
In music you care what people see at a given moment of time. If the song’s tempo has increased, you want the viewer to experience that increase. When the tempo slows, you want the viewer to experience that slow down. That means that the environment itself, the room itself, has to change as you move through it. If we tried to build our visual music environments the traditional way, the viewer would have to run from one part of the room to another to find things that match the audio music.
Because we care about what viewers see at each moment in time, we build our visual environments backwards. We start with questions like, how fast do should the viewer move during this section? When in the song should they crest the hill? Is this the right moment for turning and spinning? Those sensations are your path through the visual environment and correspond to the rhythm of the audio track.
Then we decide what you should see at each moment in time around the path—should there be a dancer figure? What colors should you see? How large should objects be? What should be slightly ahead of you? What does the near future look like?
The answers to these questions conrol the synthesizer that constructs the 3G models for each moment in time, so the 3D model is constructed according to what we want to feel at each instant. Instead of building a visual environment first, we create our path first and the build the environment around it. We construct our visual environments based on what we want the viewer to see at each second of the song.
We also get to play with how the viewer constructs reality. The spatial reality in our visual environments are a function of time, meaning that what you see is constantly changing. One example is turning your head. With your HMD you look left, then right, and then left again. While you might expect that what you see when you look left again would be the same, in our world that isn’t true. Because it took time for you to look left and right, what you see when you look back to the left will have shifted. In our visual music environments, reality itself is constantly changing throughout time.