Visual Music Systems


We experimented with most of the commercial motion controllers, game controllers, and user interface devices, but found nothing that matched our requirements. What we wanted was:

Precise tracing of 6D hand position over a large range of movement:

  • We liked the Razer Hydra, which is now out of production. It uses magnetic transducers to detect translation with 1 mm accuracy and rotation within 1 degree in a 2M wide x 1M deep x 1.5M tall working area.
  • We experimented with but rejected accelerometers as they are inherently unable to detect position precisely. We did not like systems with smaller spatial ranges, or optical systems that used single light sources so that where shadows can interfere with the detection of position.

Expressive linear controls with haptic feedback for fingers:

  • We did not like any of the systems where fingers move in free space without haptic feedback or constraints (LEEP, Kinect, etc.).
  • The process of motor learning is faster and more effective when the musician can associate sensory feedback about the position of each finger with the action produced by the finger being in that position.
  • The Razer Hydra is a bit better, as it provides a single linear trigger control for the first finger, but the trigger control is limited to a short range of movement and the other fingers do nothing but hold the wand in place.

We have not yet worked with either the Oculus Touch or the new Vive controller, but they should have similar characteristics to the Hydra. We expect to obtain good position tracking, but the finger controls are limited in that most of the fingers will still be needed to hold the wand in place. These controllers will work adequately as an introductory controller for playing visual instruments, but more advanced users will want to have independent use of all fingers, as we provide on our proprietary controller.

To solve this, our controller uses a special palm grip and two straps to clamp the controller securely to the palm, so that the thumb and all fingers are free to operate sensors. The palm grip is contoured to provide a secure triangular attachment, while providing clearance around the thumb muscle so that finger and thumb movements are not constrained.

We created two different custom variations of the controllers. Both varieties use the 6D position detection from the Razer Hydra. We used their base station and rebuilt their inductor boards to fit inside our controller. Both varieties use a thumb control panel with 4 buttons that provide linear pressure outputs. The two units have different finger controls:

  • Slider controls:¬†Each finger has a spring-loaded control with haptics similar to a trumpet valve.
  • Linear FSR controls:¬†Each finger operates a linear force sensitive resistor that detects both position and pressure independently. The finger slides within a track shaped by bumpers that constrain the movement and provide haptic feedback for position.

We often use the standard hydra instead of the Linear FSR Controls for the left hand, when we do not need the additional control vectors.

Each controller has different strengths. In our system the two hands work cooperatively– each hand performs different functions similar to the approach used in a guitar or violin in contrast to a piano or drums.

We use the slider controls for the right hand. In our system, the right hand usually controls the amount of different energy flows in the system, somewhat analogously to the way that the pick hand of a guitarist controls the amount of sound coming from each string. The trumpet-like controls provide increasing resistance for higher values so that the feeling of adding more energy corresponds to squeezing harder. The 40mm range of movement is well matched to the range of finger motion.

In our system, the left hand usually sets the parameters that control the characteristics of the energy flows that are being emitted, somewhat analogously to the way the left hand of a guitarist shapes the frequency that will be emitted from each string. With two independent vectors per finger, the Linear FSRs make it possible to implement advanced control architectures.