MUSEUM MUSIC MAP
APP DEMO ON TRANSLATION BETWEEN VISUAL, SOUND AND HAND GESTURE
Director: Anna Borou Yu Advisor: Professor Dan Michaelson Year: 2019
Final Project of Mobile Computing, Yale School of Art
Museum Music Map is an AR app designed to translate hand gesture in a time period into music edition. While going through a museum, visitors are provided with a piece of music selected by the curator. Generally visitors tend to remember some of the exhibits for different reasons, from “like it” to “look for more information later”. Instead of writing down notes in a traditional way (or even forget all the thoughts after the visit), now the users could make marks to the exhibits in the app with VR gestures, and the gestures are translated into musical adjustment to the music piece they are listening in real time, from rhythm to pitch. After visit, an individual sound map is created with selected scenes according to the user’s experience, and the scenes are in sequence when the user make the VR gestures marks. And the users could always come back to make more adjustments. When several users who visited a same exhibition associate their sound track together, a symphony based on various edition of the same music piece is composed.
With this multimedia app, users are encouraged to positively engage with exhibition and exhibits. At the same time, human motion is unconsciously revealed in the process of personally paying attention to specific objects and editing the music piece.
