Posted by on Sep 5, 2017 in Capstone Blog | Comments Off on Text Based Proposal


Music, specifically performing music, is a fun and engaging creative pastime around the world. Regardless of the person, music has universally positive effects on one’s well-being. Unfortunately, many people lack the confidence, resources, or physical or mental ability to participate in the creation of music. That said, I would like to create an application that allows a user(s) to virtually “play” a variety of instruments using only physical gestures. By this method in tandem with a variety of parameters a user can define through the application’s interface, I want to make the process of “playing” an instrument (and playing it well) more accessible and immediately engaging for any given person.


The Problem

I believe that music is a powerful tool. It’s a medium that is not bound by any cultural norms, language barriers, age or origins. Both the acts of performing and perceiving music have a myriad of effects on any able-eared listener, including entertainment but expanding well beyond that. The performance of music in particular – the act of playing an instrument in a rhythmic and pleasing way – has been shown to have unique therapeutic and social benefits. But despite advancements in instrumentation technology, namely with the development of MIDI (Musical Instrument Digital Interface) devices, the idea of learning and playing a conventional instrument is still largely too daunting to the average person. In fact, if anything, musical developments in recent history have served to just further complicate the field and make the task of learning an instrument seem even more obtuse. In any case, the average non-musician often cites a lack of time, resources, interest, ability, and/or confidence with which to approach an instrument. Given this kind of restriction, many people wall themselves off from the many benefits music has been shown to have in many studies such as the following…

Note that each of these outline the benefits of musical education and participation, particularly at a young age. There are a variety of reasons why people don’t have the willpower or ability to learn an instrument but they are less-documented than reasons why people *should*. Here’s one article outlining some common reasons why younger people may choose to quit musical education.


The Solution

Other Solutions

The mobile application AUMI, created in part by UMaine New Media alumn John Sullivan, was the first point of reference to come to mind and will remain a focus throughout most of my project for sure. It exists primarily with the same goal in mind; as an alternative and digital means of music performance for the sake of easier accessibility of any given user. The same can be said of the following alternative solutions.

Not quite as close to the concept of my project as AUMI, but this motion-controlled application presents another more tangible method of controlling music in an unconventional but simpler manner.

A relatively famous but not terribly successful example of an attempt to simplify the act of music performance. Wii Music most notably put an emphasis on the entertainment value of music performance – particularly in a group environment – over its application as a legitimate musical platform.

This is probably the project that’s closest to how I envision the use of my application in the end. From what I can gather given the limited information publicly available on this, it wasn’t made with accessibility or the common user in mind. But it utilizes a Kinect and Max, so I may possibly contact the creator in the future with an outline of my intentions and some questions on how he went about achieving his version.

My Solution

In my experience as a musician and a new media undergraduate with a personal attachment to audio, I believe I have the capacity to develop a new type of musical interface through which anybody can perform and even produce music, be it on their own or in a group environment. Using Max/MSP and some form of camera input to detect motion, I can create a program that changes the way people think about music by essentially constructing a new kind of MIDI device that satisfies the many-to-many approach that is characteristic of new media. To name a few specific ways, my approach will share what was inaccessible first and foremost; the ability to perform and produce music solo or in a group in a way that is easy to learn and fun to do. In this way, I also hope to mobilize people to take up an interest in music performance and by extension share the results and the joy of that experience with others. It’s a DIY approach to a medium that most people perceive a skill-wall holding them back from, and in that way it also may serve to upstage the man by putting a more streamlined and accessible method of music production in the hands of anyone willing to spare the time to do so.



As a rough outlook on the development of this sort of project, I can foresee these tasks (subject to change as I continue to conduct research and build the actual program):

  • Research similar projects/motion-controlled musical interfaces
  • Try these interfaces if possible and assemble a list of pros and cons for each
  • Compile the pros and cons for reference in my own work and determine which qualities can and can’t be conceded
  • Outline the parameters that will be available through the application interface
  • List what factors, if any, should not be changed by user input
  • Mockup a visual demo of the application as intended in its ideal final iteration
  • Present mockup for class feedback
  • Sketch out pseudocode for the application itself with one instrument implemented
  • Research jitter modules in Max/MSP and depending on prior research, Kinect integration
  • Prototype a simplified version of a motion controlled instrument in Max/MSP
  • Test with peers and get feedback on rudimentary motion controls
  • Refine controller and begin defining parameters within the program
  • Prototype a UI enabling users to define parameters
  • Test with peers and receive feedback
  • Implement other instrumentation, refine gestures as needed and define them by instrument selected
  • Implement multi-point motion tracking
  • Refine UI for multiple users
  • Test multiple-user scenarios and document feedback
  • Integrate production features, namely recording and mixing
  • Finalize UI, malleable parameters, and functioning instrumentation
  • Prepare a setup and poster to share on New Media Night
  • Final presentation



  • I’m already in possession of most of the necessary components for this kind of project. They are as follows (unless noted otherwise, I already own each of these):
  • Laptop
  • Integrated/external webcam
  • Max/MSP subscription
  • Speaker setup for presentation purposes
  • MIDI controller and sound libraries (for production and use in the application)
  • Kinect (possible use depending on camera developments)