A visual instrument can be played as a solo performance by making it large and placing it in the scene near the camera. As with audio music, a composition can also include multiple instruments performing at the same time. This is done using “placements”, which specify where each instrument will appear in the composition. Instruments can be placed near each other and interact in close harmonies, or they can be spread apart to fill the environment with different elements at various locations. A single instrument can also be placed at multiple positions and orientations in the environment.
Visual arrangements work in four dimensional spaces: a placement not only specifies a location, but also a position in time – when it will appear and leave. The arranger can specify different layouts to be used at different times during the composition, just as an audio music arranger varies the use of sections of the orchestra at different times. The same instrument can appear at different locations at different times, or even be visible in multiple places at the same time.
The VMS Arranger is our tool for creating arrangements and is integrated into the OVC-3D. It allows the arranger to set placements and assign the instruments that play there. It integrates with the synthesizer to auto-generate icons for each scene element, so that the arrangement is displayed as a graphical storyboard. The control system is hierarchical, so that multiple instruments and placements can be viewed as a single object, which can then be opened to display the individual elements.
The time axis can be static or dynamic. When working with a prerecorded audio composition, the timing points are fixed: the arranger knows the exact time that each placement should occur in the composition. When performing live, the arrangement has to adapt not just to variations in tempo, but to spontaneous changes, such as when the length of a solo varies or the band segues into a different composition.
