Apprentissage de representation latentes pour l'audio spatialisé

Keywords

Signal Processing, AI, Machine Learning, Audio signal

Subject details

Motivations Humans rely on binaural hearing to process complex sound scenes. One example is speech communication in noisy environments with many competing sources, a challenge known as the cocktail party problem. Despite numerous studies on the topic, this ability is not fully understood [1]. Nevertheless, researchers have drawn inspiration from this physiological phenomenon for decades to develop multi-channel audio processing techniques, such as speech enhancement algorithms and computational auditory scene analysis. While some early work in CASA focused on how humans perceive complex audio scenes, most efforts have concentrated on the acoustic properties of the audio signal to achieve specific goals, such as extracting a source of interest [2] or localizing sound sources in space [3]. These studies rarely address how this information could be coded and processed together. Recent work in deep learning-based audio representations has demonstrated the strong ability of models to provide a well-structured latent space for audio signals [4], emphasizing aspects such as phonetic content. Concurrently, there has been an increasing effort to connect audio processing models (such as speech recognition or sound source localization models) to auditory perception, either at the brain level [4] or at the psychophysical level [5]. However, these works primarily focus on one aspect of the signal, either its content or the spatial localization of sound sources but not both at the same time which is an important aspect to understand how humans tackle the cocktail party problem. The aim of the project is to advance research in computer-based audio modeling and auditory modeling by proposing new multi-channel audio representation models with structured latent spaces. Goals and Objectives This work during the PhD aims to explore the representation of a sound source's position independently of its content, as well as the interplay between localization and source signal representation. Most existing models focus on representing the signal content while ignoring acoustic aspects such as sound source localization. This is partly due to the fact that these models are mainly single-channel models that can hardly access any spatial information. In a first step we will extend existing single-channel audio representation models to leverage multi-channel inputs that can account for spatial information. Then we will shape the latent space of the multi-channel audio representation model to explicitly encode the spatial localization of the sound source. Shape the latent space of the multichannel audio representation model to explicitly encode the spatial localization of the sound source. We will investigate two approaches: with or without explicit knowledge of the source position during training. When the source position is explicitly known at training time, we will leverage this information in a multitask framework (where the decoder is decomposed into one branch to reconstruct the signal and another to localize the signal) or as a constraint applied directly on the latent space (by performing an additional source localization task on the latent representation). This has been shown to be efficient in reproducing, for example, natural auditory cortex representation structures for sound identity in the latent space. To leverage multi-channel data without explicit information, we will investigate student-teacher approaches where the localization target is obtained from an auxiliary localization network [6] and self-supervised approaches [7] for example based on contrastive learning [8], where two versions of the same scene (subject to transforms that do not impact localization) are fed to the model.