TRANSLATION BETWEEN DANCE AND MUSIC 舞蹈与音乐的转译
RESEARCH Between Music Notation and Dance Notation
Authors: Anna Borou Yu, Jiajian Min Advisor: Professor Konrad Kaczmarek Year:2019 - 2020
Published by ACM SIGGRAPH Asia 2020
Translation between Dance and Music is an exploration on the translation between different art types, based on the Ballet Giselle. Here physical movement is adopted into digital signal and sound.
《舞蹈与音乐的转译》是对于不同艺术媒介之间相互转译的研究项目，基于芭蕾舞吉赛尔选段。利用动作捕捉，数字信号处理等技术手段，以及贝尼什舞谱、交响乐谱的比较研究，我们试图探索舞蹈与音乐深层联系。芭蕾舞蹈动作被转化为数字信号形成新的乐曲，新乐曲与原钢琴曲相互叠加形成全新的吉赛尔交响乐。
Always performed together, music and dance share similarities and correspondences. However, the relationship between these two art types have remained mysterious. This project is exploring an innovative strategy of deeper engagement between music and dance.In this project, a 1-minute clip from the ballet Giselle Act I (Scene 1.7 Peasant Pas de Deux, Male 1st Variation) is analyzed, and new music clips are generated and developed.
Historically, music notation and dance/movement notations are developed into various branches, from graphical and gurative to abstract and numerical, which seems incomparable. Luckily, the ballet Giselle is well archived, for which the music notation and the Benesh Dance Notation are both documented on the systematic structure of five-line stave, providing a similar visual image of ergonomically ideal matrix and securing an identical rhythm framework. In this project, a 1-minute clip from the ballet Giselle Act I (Scene 1.7 Peasant Pas de Deux, Male 1st Variation) is analyzed, and new music clips are generated and developed.
The expected outcome of the project is a New Symphony, laying all the music clips together based on the timeline of performance. The various music clips include Music 1: original music (played from music notation); Music 2: generated from the dancer motion (processed by software); and Music 3: generated from dance notation (visually translated to music notation and played).
Music 2 (motion captured and processed by algorithm): Several dancers are invited to perform, based on the dance notation and video documentations. The movement is motion captured by a Kinect XBox 360 and processed into digital signals by the music software MAX/MSP. Each time the movement is documented into two sets of waves according to the Benesh Notation principles: Music 2.1 a 5-buffer head/shoulders/waist/knees/feet body position wave, and Music 2.2 a 4-bu er hands/feet movement wave.
Music 3 (visually mapped from dance notation to music notation): The dance notation corresponds to the original music by respecting the rhythm and phrasing signs in the dance notation, and matches with original timing. At the same time, the symbols for body parts in dance notation are visually mapped to the music stave in corresponding place, with a similar thought in Music 2 processing. Also, the Music 3 is further detached as Music 3.1 (position frame) and Music 3.2 (hands/feet move Lines).
The Final result is a New Symphony, laying all the music channels together: Music 1.0 (original music), Music 2.1+Music 2.2 (from dance), and Music 3.1+Music 3.2 (from dance notation).
