Online Lyrics Tracking in Opera Performances

Charles Brazier

Lien visio : https://bordeaux-inp-fr.zoom.us/j/96931742161?pwd=OXNsUEFTeHNvRDNxUXlVWXBuNHlBdz09

*Abstract*: This presentation is concerned with the challenging task of tracking complete opera performances in real time and reliably following along in the corresponding musical score. Our initial approach aligns audio and corresponding score via common intermediate features that represent the spectral content present in the audio, as well as in a pre-recorded opera performance which serves as a reference and proxy to the score. In this work, we propose to exploit an additional source of information: the lyrics sung or spoken in the audio recordings. We do not assume the written lyrics to be available in textual form. Rather, the idea is to train an acoustic phoneme recognition model that extracts phoneme sequence estimates both from the reference (off-line) and the live performance (on-line), and to align these in real time, giving us a real-time lyrics tracking algorithm (without any explicit lyrics representation). For this purpose, we will describe our real-time-capable audio-to-lyrics alignment pipeline that is able to robustly track the lyrics of different languages, without additional language information. Then, we will propose a new pipeline for complete opera tracking, combining our baseline music tracking with our newly developed lyrics tracking.