Conversational AI Reading Group

What does it take for AI to understand the world through audio? In this talk, I will present our work on fully open large audio-language models, highlighting advances in architectures, training curricula, and internet-scale data curation. I will explore how unified encoders connect speech, sound, and music, and how new data and training infrastructure extend audio understanding from short clips to 30-minute recordings. Along the way, expert-level benchmarks reveal persistent gaps between current models and human audio reasoning. Finally, I will show how audio enriches real-world video comprehension, and discuss our work integrating audio into world models. Together, these directions highlight why listening is essential to AI systems that understand and model the world.

In the coming years, AI-generated music will outnumber all human-recorded music in history. Yet, this explosion of AI music faces a critical legal and philosophical issue: the requirement of human authorship for copyright, which limits the commercial value and legal protection of such works, and impacts the broader music industry. Thus, in the first part of this talk, I argue that the most important challenge in Music GenAI is designing the "co-creation loop" — building AI models as highly controllable tools rather than one-button algorithms. In the remainder of the talk, I will overview a selection of recent works where key areas of focus will include: (1) Control: Time-varying control over melody, dynamics, and rhythm (Music ControlNet), music generation with inference-time control (DITTO), and/or multi-track generation in a single pass with iterative accompaniment generation (Stemphonic). (2) Efficiency: Building faster, more efficient generative engines through Generative-First Neural Audio Autoencoders (GenAE) and step/layer distillation (Presto!). (3) Multi-Modal Alignment: Bridging modalities via metadata captioning (Music Metadata LLMs) and time-aligned video-to-music generation with zero-paired data (V2M-Zero). Such work demonstrates how we can build the technical controls needed to invite human iteration back into the creation process, and serves to build AI that enables verifiable proof of human creativity.

Large Language Models (LLMs) are increasingly being applied to audio processing, where they help interpret and generate meaningful patterns from complex sound inputs such as speech, music, and sound effects. When combined with acoustic models, LLMs offer significant potential for solving a wide range of challenges in audio processing, understanding and generation. This talk will highlight several recent developments in large audio-language models (LALMs), focusing on new algorithms and their applications to audio-centric tasks. Topics will include audio-text fusion and alignment, cross-modality audio applications, the construction of audio-language datasets, and emerging research directions in audio-language learning. We will showcase our recent work in areas such as audio generation and storytelling (e.g., AudioLDM, AudioLDM2, WavJourney), audio source separation (e.g., AudioSep), audio captioning and reasoning/question answering (e.g., ACTUAL and APT-LLMs), neural audio coding (e.g., SemantiCodec), audio editing (e.g., WavCraft), and the datasets (e.g., WavCaps, Sound-VECaps, AudioSetCaps) used to train and evaluate large audio- language models.

In this talk, I will present recent work on making spoken language recognition more robust across accented, multilingual, and previously unseen-language settings. I will begin by showing that standard language identification systems degrade dramatically on non-native speech, and that combining acoustic models with ASR-derived features substantially improves robustness to accent. I will then discuss hybrid systems for large-scale multilingual language identification and speech recognition, where combining embedding-based and generative methods improves performance across languages and dialects and helped to produce a top-ranked system in the Interspeech 2025 ML-SUPERB 2.0 challenge. Finally, I will cover our recent work on hybrid and LLM-based language verification, including systems that reason over multiple transcriptions and fuse acoustic and language-model signals for improved spoken language recognition and few-shot language verification. This approach resulted in top scores in the recent Speaker Odyssey TidyLang 2026 Challenge.

As spoken interface shifts from pipeline-style speech systems to interactive voice agents, text-centric evaluation becomes increasingly inadequate. In this talk, I will argue that trustworthy spoken agents require speech-native thinking along three axes: evaluation, safety, and security. I will focus on recent work on LALM-as-a-Judge for spoken dialogue, showing that safety evaluation decomposes into distinct capabilities - such as detection sensitivity, severity ranking, and temporal stability - and that these behave differently under audio degradation and transcription errors. I will then connect this perspective to earlier work on perceptual evaluation in speech processing (MAPSS, ICLR 26'), where misaligned metrics can obscure meaningful trade-offs, and conclude with emerging directions on speaker-representation security and voice privacy. Overall, the talk argues that spoken agents should not be treated as text systems with audio attached, but as audio-native systems with their own risks, failure modes, and evaluation requirements.

Large Audio Language Models has been a recent research focus in the audio community. These models exhibit impressive capabilities for audio understanding tasks such as question answering. Moreover they reveal a reasoning chain for their decisions, which is promising for explainability of these models. It is however unclear whether these reasoning chains truly reveal the inner decision mechanisms. I will talk talk about measuring the faithfulness of these reasoning chains, to assess if they can be used as trustworthy explanations. I will also later talk about our recent work grounding the the reasoning chains in the audio input, in order to make the model more closely listen to the audio.

We present a multimodal Bayesian network for symptom-level prediction of depression and anxiety from speech and voice data, developed and deployed as a real-world clinical product and evaluated on over 30,000 unique speakers. The model integrates paralinguistic and linguistic features through symptom-specific surrogate models into a joint Bayesian network that learns posterior distributions over individual symptom severities, inter-symptom relationships, and overall condition probabilities (depression ROC-AUC = 0.842, anxiety = 0.831; ECE ≤ 0.018). We examine multimodal integration, cross-channel redundancy, demographic fairness across multiple axes, direct clinician intervention in model predictions, and service user acceptability — arguing that Bayesian networks offer a principled, explainable, and intervenable framework for clinical translation of digital phenotyping. Real-world deployment also demands efficient inference. The acoustic embeddings underpinning our model inherit the standard mel-scaled, phase-discarded speech frontend — but is this optimal for the task? As a concrete example of challenging inherited assumptions, we show that downstream performance is preserved under significant bandwidth reduction, and that the resulting redundancy can be surgically removed — yielding up to 2× inference speedup with no retraining. This opens a wider question: perceptual frontends are designed to model what a listener hears, but for biomarker tasks where the clinical signal is imperceptible to human listeners, this may be the wrong prior. We outline what production-aligned alternatives could offer instead.

Audio source separation has traditionally been approached as a collection of specialized tasks, each requiring dedicated models and datasets. This talk presents a unified, prompt-conditioned framework that reformulates source separation as a task-aware problem, enabling a single model to flexibly handle multiple, even contradictory, objectives. Starting from Task-Aware Unified Source Separation (TUSS), I will show how learnable prompts enable explicit control over both the number and type of extracted sources, addressing the ill-posed nature of separation by incorporating task information directly into the model. I will then discuss FasTUSS, which significantly reduces computational cost while maintaining strong performance, and FlexIO, which extends prompt-based conditioning to jointly support variable input (microphones) and output (sources) configurations. Finally, I will introduce SUNAC, a source-aware neural audio codec that unifies separation and representation learning, enabling direct encoding of selected sources from mixtures with improved efficiency. These contributions highlight a general paradigm for controllable audio modeling and open new directions for integrating audio processing with downstream systems such as large language models and conversational agents.

General audio source separation is a key capability for multimodal AI systems that can perceive and reason about sound. Despite substantial progress in recent years, existing separation models are either domain-specific, designed for fixed categories such as speech or music, or limited in controllability, supporting only a single prompting modality such as text. In this work, we present SAM Audio, a foundation model for general audio separation that unifies text, visual, and temporal span prompting within a single framework. Built on a diffusion transformer architecture, SAM Audio is trained with flow matching on large-scale audio data spanning speech, music, and general sounds, and can flexibly separate target sources described by language, visual masks, or temporal spans. The model achieves state-of-the-art performance across a diverse suite of benchmarks, including general sound, speech, music, and musical instrument separation in both in-the-wild and professionally produced audios, substantially outperforming prior general-purpose and specialized systems. Furthermore, we introduce a new real-world separation benchmark with human-labeled multimodal prompts and a reference-free evaluation model that correlates strongly with human judgment.

Over the past few years, our work has focused on assessing and improving the linguistic capabilities of speech language models (LMs). In this talk, I will present our progress in this area, beginning with the derivation of the first scaling laws for speech language models and continuing with recent efforts to align speech (or image) and text modalities in order to leverage the knowledge captured by text-based large language models.I will also review recent work, including methods for generating and processing conversational speech, with a particular focus on medical settings. Finally, I will discuss our ongoing efforts to develop a speech-based large language model that can be used for various applications, including conversational agents and assistive technologies.

This talk takes a deep dive into the evolution of audio foundation models—from large-scale audio encoder pretraining to their integration with large audio language models (LALMs). Through the Auden project, we systematically decompose audio representation learning into five domains: speech semantics, paralinguistics, general audio, music, and spatial acoustics. We conduct comprehensive evaluations of audio encoders across this spectrum, from linear probing and downstream task transfer to full integration with LALMs. Our findings consistently highlight the central role of audio encoders as the foundation of LALM performance. Building on this insight, we further develop improved alignment methods to more effectively connect pretrained audio encoders with language models, enabling stronger multimodal reasoning and generation.

State-of-the-art ASR systems excel on close-talk benchmarks but struggle with far-field conversational speech, where error rates remain above 20%. Current benchmark datasets inadequately assess generalization across domains and real-world conditions, often relying on oracle segmentation that yields overly optimistic results. Distant ASR (DASR) faces unique challenges including overlapping speech, long-form processing and varied recording setups, and dynamic speaker interactions that significantly complicate system development. Despite these difficulties, spontaneous conversational speech represents the next frontier for developing more human-like AI agents capable of natural multi-party communication. This presentation examines the challenges of conversational speech processing and outlines two promising research directions. The first is end-to-end integration, which can mitigate the cascading errors that plague modular approaches. The second tackles data scarcity—a persistent bottleneck given the privacy concerns surrounding conversational recordings and the substantial cost of annotation. Here, the talk explores how large language models and text-to-speech synthesis can generate effective training data, alongside self-supervised learning techniques which can further dramatically reduce reliance on labeled corpora.

Separating useful from non-useful components in a noisy signal is of extreme importance in everyday applications where data are often contaminated by artifacts. For audio applications, these non-useful components can refer to e.g., additive environmental noises and/or convolutive room acoustic effects, such as reverberation.To this end, the amplitude modulation spectral signal representation, which measures the rate-of-change of spectral magnitude components, has been proposed. The motivation lies in the fact that, although many artifacts and environmental noise sources overlap in both time and frequency, the rate at which their amplitude spectra vary over time differs, hence becoming more separable in the new frequency–frequency modulation spectral domain. In this talk, I will showcase the numerous areas in which we have used the modulation spectrum to develop more robust and generalizable audio processing systems. Representative examples include, but are not limited to: speech enhancement, room acoustics characterization, robust speaker identification, bioacoustics, and pathological speech analysis. More recently, we started exploring the use of the modulation spectrum pipeline to characterize the temporal dynamics of universal speech representations, with applications in deepfake detection.

Task-oriented dialogue systems are designed to assist users in achieving specific, well-defined goals or tasks through natural language interactions. These systems act as a conversational bridge, connecting users to task-specific APIs or tools. Recent advancements have resulted in a significant paradigm shift with the integration of Large Language Models (LLMs) augmented with tool-calling into task-oriented dialogue systems and user simulators that are used for model evaluation and training. However, several issues remain that ponder the use of these systems in real applications. In this talk, I will share our latest research towards accountable LLM-based conversational agents that excel both in interacting with users and tool calling, showcasing its performance on dialogue system and agentic LLM benchmarks. We introduce novel reward mechanisms to train tool-based reasoning with reinforcement learning. Finally, I will discuss user simulation for model evaluation and training. We observed that LLM-based user simulators can deviate from user goals over multi-turn interactions. To address this, we propose a novel framework that tracks user goal progression throughout conversations, enabling the creation of user simulators that can autonomously monitor goal progression and reason to generate goal-aligned responses.

We will describe and illustrate the concept of hybrid (or model-based) deep learning for audio signal processing. This paradigm refers here to models that associates data-driven and model-based approaches in a joint framework by integrating our prior knowledge about the data in more controllable deep models. Prior knowledge can relate for instance to the production or propagation of sound (using an acoustic or physical model). In this presentation, we will first illustrate the concept and potential of such model-based deep learning approaches and then describe in more details its application to unsupervised music separation with source production models and speech dereverberation with physics-inspired models.

Editing complex real-world sound scenes is difficult because individual sound sources overlap in time. Generative models can fill-in missing or corrupted details based on their strong prior understanding of the data domain. We present a system for editing individual sound events within complex scenes able to delete, insert, and enhance individual sound events based on textual edit descriptions (e.g., "enhance Door'") and a graphical representation of the event timing derived from an "event roll" transcription. We present an encoder-decoder transformer working on SoundStream representations, trained on synthetic (input, desired output) audio example pairs formed by adding isolated sound events to dense, real-world backgrounds. Evaluation reveals the importance of each part of the edit descriptions -- action, class, timing. Our work demonstrates "recomposition" is an important and practical application.

Scaling laws have pushed foundation models to unprecedented capabilities, yet even the largest systems still struggle on the long tail: underrepresented domains, low-resource languages, and user-specific constraints. These failures don’t arise from a lack of parameters, but from how models are trained and how data is used. In this talk, I present two recent lines of work that treat data not merely as training fuel, but as a force multiplier for reliability, steerability, and equity in large models. First, I introduce Multilingual Arbitrage, a synthetic data routing strategy that exploits performance differences across teacher models to dramatically improve outcomes in multilingual systems. Second, I present TreasureMarkers, a training-time conditioning approach that injects structured control signals into the data itself, allowing models to respond to optional inference-time markers and yielding substantial improvements on long-tail and instruction-following benchmarks. Together, these methods illustrate how reframing data as leverage—not as a static resource—enables meaningful improvements in foundation model behavior where scaling alone fails.

The emergence of large language models (LLMs) has transformed spoken dialog systems, yet the optimal architecture for real-time on-device conversational agents remains an open question. While end-to-end approaches promise theoretical advantages, cascaded systems continue to outperform them in language understanding tasks, despite being constrained by sequential processing latency. In this talk, I will revise cascaded system and show a novel low-latency variant that overcomes traditional bottlenecks through architectural innovations and streaming optimizations. This cascaded system integrates streaming (a) conversational speech recognition with mixture-of-experts, (b) state-action augmented LLM, (c) text-to-speech synthesis, (d) neural vocoder, and (e) speaker modeling. Proposed cascaded system achieves sub-second response latency with complete on-device processing.

Movement disorders (MD) are produced due to the impaired functioning of certain areas of the brain. Several millions of people suffer from MD such as Parkinson’s, Huntington’s, and others, however diagnosis and monitoring are still highly subjective, time-consuming, and expensive. PD is the second most prevalent MD in the world and medical evaluations used to assess the neurological state of patients cover different aspects, including activities of daily living, motor tasks, speech production, and mood. Its multi symptomatic nature make the diagnosis and monitoring of PD a very challenging task. Several bio-signals need to be modeled and the impact of the disease differs among patients which produces another level of complexity especially when thinking of producing clinically acceptable/practical results. This talk tries to show different approaches that consider several bio-signals (e.g., speech, language, gait, facial expression, and handwriting) and methods of Pattern Recognition with the aim to find suitable models for PD diagnosis and monitoring. Results with classical feature extraction and classification methods will be presented together with experiments with CNN, LSTM, and attention-based architectures.

Humans possess the remarkable ability to listen to a desired sound in a complex acoustic scene composed of numerous competing sounds. This phenomenon, known as the cocktail party effect or selective hearing, enables us to understand an interlocutor in a noisy cafe, focus on a particular instrument in a song, or notice a siren on the road. In this talk, I will discuss target speech/sound extraction (TSE), a computational approach that aims to functionally emulate selective hearing. TSE isolates the signal of a target speaker or a specific target sound from a mixture of several speakers or sounds, using clues that identify the target within the mixture. Such clues might include a prerecorded enrollment audio from which the speaker’s vocal characteristics or the target sound's properties can be derived. I will introduce the foundations of neural TSE and present recent research on neural-based approaches for extracting both speech and arbitrary sounds.

In many scenarios, AI models have to satisfy external constraints on, e.g., energy consumption, communication, allocated GPU hours, or time. In addition, many of these constraints can vary over time, due to variable network conditions, equipment failure, or peak energy use. In this talk we overview recent work on "dynamic" architecture that are able to satisfy these constraints by reducing their internal computations (e.g., number of layers) dynamically. This is in contrast with standard efficient techniques for AI, which require to selectively retrain or quantize larger models and target only static constraint requirements. We overview a number of applications including audio modelling, and we show how dynamicity allows for more explainability and modularity.

The area of computational audio analysis -also called machine listening- continues to evolve. Starting from methods grounded in digital signal processing and acoustics, followed by supervised machine learning methods that require large amounts of labelled data, recent approaches for learning music audio representations are fueled by advances in the broader field of artificial intelligence. The talk will outline recent research carried out at the Centre for Digital Music of Queen Mary University of London focusing on emerging learning paradigms for making sense of music and audio data. Topics covered will include learning in the presence of limited audio data, the inclusion of other modalities such as natural language to aid learning music representations, and finally methods for learning from unlabelled audio data - with the latter being used as a first step towards the creation of music foundation models.

This talk will introduce Gemini 2.5, a natively multimodal audio model developed over the past year. We will detail the model's ability to generate human-like, responsive dialogue while leveraging industry-leading intelligence. The presentation will cover the key research and engineering efforts that enable Gemini 2.5 to understand, reason, and interact through voice, establishing a new standard for intelligent, conversational agents.

Detecting musical versions (different renditions of the same piece) is a challenging task with important applications. Because of the ground truth nature, existing approaches match musical versions at the track level (e.g., whole song). However, most applications require to match them at the segment level (e.g., 20s chunks). In addition, existing approaches resort to classification and triplet losses, disregarding more recent losses that could bring meaningful improvements. In this paper, we propose a method to learn from weakly annotated segments, together with a contrastive loss variant that outperforms well-studied alternatives. The former is based on pairwise segment distance reductions, while the latter modifies an existing loss following decoupling, hyper-parameter, and geometric considerations. With these two elements, we do not only achieve state-of-the-art results in the standard track-level evaluation, but we also obtain a breakthrough performance in a segment-level evaluation. We believe that, due to the generality of the challenges addressed here, the proposed methods may find utility in domains beyond audio or musical version matching.

Large Language Models (LLMs) have significantly influenced natural language processing (NLP) and are increasingly affecting speech processing. This talk examines the effects of LLMs on speech processing and multimodal intelligence. It reviews the progression from spoken language models (SLMs) to multimodal LLMs (MLLMs), referencing the model development process at Microsoft. The discussion includes comparisons of model architectures, advances in speech generation and understanding, and the integration of speech and text modalities. The talk concludes by presenting the recent Phi-4-multimodal model as an example.

In the recent years, AI has been adopted for several problems in spatial audio. In this talk an overview of some of the recent advances are presented, with a focus on the reconstruction of acoustic room responses from sparse data, nearfield acoustic holography and upsampling of microphone array measurements.

Deep learning systems can acquire uncontrolled goals, e.g., instrumental subgoals of human-specified objectives or imitated human aims, that conflict with human norms. We propose a safety case for theScientist AI, a purely non-agentic predictor whose sole function is to estimate the conditional probability of properties of the world, not what action to take to achieve a goal nor acting like a ChatBot which answers questions. Our safety case rests on two design claims. (1) The training setup prevents any learning signal that depends on changes the model could induce in the external world, eliminating incentives to select outputs that steer the world toward preferred states, as would happen with typical reinforcement learning. (2) A "truthification" pipeline converts data into atomic claims labelled by trust tiers, with the highest tier corresponding to claims verified by human science, to avoid uncontrolled agency arising from human imitation. While total honesty cannot be guaranteed in general, we argue that a sufficiently capable Scientist AI trained and calibrated using this pipeline is epistemically correct: When it issues a high-confidence claim, it is trustworthy. This differs from total honesty and solving the challenge of eliciting latent knowledge, as it leaves open the possibility that the AI expresses uncertainty despite knowing the answer. However, this asymmetric guarantee supports conservative decision-making and turns increased capability from a misalignment risk into higher reliability of confident claims. As an application, a Scientist AI could act as a guardrail for untrusted agents by predicting the probability of harm from candidate actions and vetoing any action whose predicted harm exceeds a threshold. To support this safety case, a set of Scientist AI design properties is proposed, including a latent variable model of the truthified claims. Every statement, latent or observed, that makes a claim in natural language about a property of the world, corresponds to a random variable. The anticipated improvement in generalization and explainability would ride on the language-understanding abilities of modern deep learning. Latent variable modeling would both increase the quantity of training signals for rarely or never observed claims and allow the generation and validation of interpretable explanations for any target statement.

This presentation will highlight some of our recent advancements in speaker recognition. We will begin by exploring techniques for finetuning self-supervised models, such as WavLM, for the dual tasks of speaker recognition and deepfake detection. To address the challenge of real-world application, we will then discuss essential model pruning methods for deploying these systems efficiently. The final section will introduce a weakly-supervised method for training speaker embedding extractors. We will demonstrate how this approach allows a model to be trained on raw, uncut VoxCeleb recordings, using only the audio and associated celebrity names, thereby eliminating the need for precise temporal annotations.

Discrete audio tokens are compact representations that aim to preserve perceptual quality, phonetic content, and speaker characteristics while enabling efficient storage and inference, as well as competitive performance across diverse downstream tasks. They provide a practical alternative to continuous features, enabling the integration of speech and audio into modern large language models (LLMs). As interest in token-based audio processing grows, various tokenization methods have emerged, and several surveys have reviewed the latest progress in the field. However, existing studies often focus on specific domains or tasks and lack a unified comparison across various benchmarks. This paper presents a systematic review and benchmark of discrete audio tokenizers, covering three domains: speech, music, and general audio. We propose a taxonomy of tokenization approaches based on encoder-decoder, quantization techniques, training paradigm, streamability, and application domains. We evaluate tokenizers on multiple benchmarks for reconstruction, downstream performance, and acoustic language modeling, and analyze trade-offs through controlled ablation studies. Our findings highlight key limitations, practical considerations, and open challenges, providing insight and guidance for future research in this rapidly evolving area.

Audio comprehension—including speech, non-speech sounds, and music—is essential for AI agents to interact effectively with the world. Yet,bresearch in audio processing has lagged behind other areas likeblanguage and vision, hindered by limited datasets, the need for advanced architectures, and training methods suited to the inherent complexities of audio. Our group is trying to bridge this gap with innovative solutions, starting with GAMA, our large audio-language model designed for advanced audio perception and complex reasoning. GAMA is built with a specialized architecture, optimized audio encoding, and a novel alignment dataset, that is used for audio understanding, reasoning, and hallucination reduction. GAMA’s development builds on our past research, such as MAST and SLICER and EH-MAM, which are novel approaches for learning strong audio representations from unlabeled data. Complementing this, we introduced ReCLAP, a state-of-the-art audio-language encoder, and CompA, one of the first projects to tackle compositional reasoning in audio-language models—a critical challenge given audio’s inherently compositional nature. We recently developed Audio Flamingo 2, an audio-language model with advanced long-audio understanding and reasoning capabilities. Audio Flamingo 2 achieves the state-of-the-art performance across over 20 benchmarks, with only a 3B parameter small language model Looking forward, we envision LALMs becoming integral to daily life, capable of conversational speech QA, information-extraction-based QA, and addressing knowledge-driven questions about diverse audio inputs. Achieving these ambitious goals requires both advanced data and architectures. Synthio, our latest synthetic data generation framework, supports this mission by generating data for complex audio understanding. Progress must also be measurable, so we’re dedicated to establishing comprehensive benchmarks. Our recent work, MMAU, rigorously tests LALMs on real-world tasks.

The Music Foundation Model Team at Sony AI is responsible for the building blocks of foundation models (deep generative modeling & multimodal pretraining) and the development of technologies for the creation, search, and protection of music, cinematic, and gaming media. I will introduce our recent works, which have been accepted at top venues including ICLR, NeurIPS, ICML, CVPR, and ICASSP, and showcase several demos that were made available with the release of commercial products powered by our technologies.

The quantification of audio aesthetics remains a complex challenge in audio processing, primarily due to its subjective nature, which is influenced by human perception and cultural context. Traditional methods often depend on human listeners for evaluation, leading to inconsistencies and high resource demands. This paper addresses the growing need for automated systems capable of predicting audio aesthetics without human intervention. Such systems are crucial for applications like data filtering, pseudo-labeling large datasets, and evaluating generative audio models, especially as these models become more sophisticated. In this work, we introduce a novel approach to audio aesthetic evaluation by proposing new annotation guidelines that decompose human listening perspectives into four distinct axes. We develop and train no-reference, per-item prediction models that offer a more nuanced assessment of audio quality. Our models are evaluated against human mean opinion scores (MOS) and existing methods, demonstrating comparable or superior performance. This research not only advances the field of audio aesthetics but also provides open-source models and datasets to facilitate future work and benchmarking.

In the first part of the talk, I will describe two very simple voice conversion approaches that operate directly on self-supervised (SSL) speech features. The first method, kNN-VC, replaces each SSL frame of a source utterance with its k-nearest neighbours from target speech. The second method, LinearVC, learns a linear projection between source and target frames. Despite their simplicity -- there is no need for expensive model training -- these methods perform competitively with state-of-the-art techniques. This should surprise us! In the second part of the talk, I will explore what these results reveal about the geometry of the SSL space and how LinearVC can be used to probe the space. I aim to show that voice manipulation serves as a useful means to investigate and understand the structure of SSL features.

Spoken language models has emerged as an interesting and promising research direction. Yet, the term ‘spoken language model’ is interpreted differently across research groups. In this talk, I will present my perspective on the current landscape of SLMs, with particular emphasis on their scaling behavior and evaluation methods. We will explore prominent models, including both textless SLMs and speech-text interleaving methods, and examine their scaling behavior. Building on this foundation, we will discuss current evaluation benchmarks and how can we go beyond spoken content, specifically, acoustic elements and prosodic information. I’ll conclude the talk with an overview of a new data generation pipeline, supported by empirical results showing that fine-tuning SLMs on this data significantly enhances prosodic understanding, while maintaining performance across other downstream tasks.

This talk traces the evolution of speaker diarization systems, from traditional multi-stage approaches to modern end-to-end architectures. I will begin with a historical overview, highlighting the shift toward end-to-end modeling and the subsequent partial return to hybrid approaches. The core of the talk focuses on loss functions designed to address key challenges in diarization: permutation invariance through the powerset loss, streaming diarization via a look-ahead loss, and the extension to joint speaker segmentation and separation using the PixIT loss.

As generative AI has revolutionized the field of speech and audio generation, there is an increasing need for automatic quality assessment methods. In recent years, data-driven speech quality assessment (SQA) methods based on deep neural networks (DNNs) have improved greatly, and have been used by both academic and industrial researchers to evaluate speech generation models. Nonetheless, there are still unsolved problems, and the application of such methods to other audio types remains underexplored. This talk will be divided in to three part. (1) I will first give an overview of SQA, with a special focus on recent developments of automatic SQA methods in the era of DNN. (2) I will talk about my experience in organizing the VoiceMOS challenge, a scientific event that aims to establish unified tasks, datasets, and evaluation processes, in order to compare different SQA approaches and shed light on insights and unsolved problems. (3) Finally, I will discuss the current challenges and future directions in the quality assessment of speech and beyond

Voicebox is a non-autoregressive generative speech model based on flow-matching and is trained to perform speech infilling given audio context and the corresponding text. The Voicebox model can be used for zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In this talk, we will first review the Voicebox model. We will then focus on the synthetic speech generation capability of the model and present several use cases of these synthetic signals in various applications including automatic speech recognition and spoken language understanding. Through a few early studies on using Voicebox generated speech signals, we will discuss the cost saving benefits of the approach in terms of speech data collection and potential shortcomings of using synthetic speech in these applications.

Connectionist Temporal Classification (CTC) is a widely used method for automatic speech recognition (ASR), renowned for its simplicity and computational efficiency. However, it often falls short in recognition performance. In this work, we propose the Consistency-Regularized CTC (CR-CTC), which enforces consistency between two CTC distributions obtained from different augmented views of the input speech mel-spectrogram. We provide in-depth insights into its essential behaviors from three perspectives: 1) it conducts self-distillation between random pairs of sub-models that process different augmented views; 2) it learns contextual representation through masked prediction for positions within time-masked regions, especially when we increase the amount of time masking; 3) it suppresses the extremely peaky CTC distributions, thereby reducing overfitting and improving the generalization ability. Extensive experiments on LibriSpeech, Aishell-1, and GigaSpeech datasets demonstrate the effectiveness of our CR-CTC. It significantly improves the CTC performance, achieving state-of-the-art results comparable to those attained by transducer or systems combining CTC and attention-based encoder-decoder (CTC/AED).

This presentation explores the forefront of generative AI research for sound design at Adobe Research. I will provide an overview of Latent Diffusion Models for audio, which form the foundation of our work, and introduce several recent advancements focused on controllability and multimodality. I will begin with SILA, a technique designed to enhance the control of sound effects generated through text prompts. Following this, I will present Sketch2Sound, a model that generates sound effects conditioned on both audio recordings and text. Lastly, I will examine MultiFoley, a model capable of generating sound effects from both silent videos and text. Throughout the talk, I will showcase a series of examples and demos to illustrate the practical applications and potential of these models, making the case that we are only beginning to unveil a completely new paradigm in how to approach sound design.

Deploying Automatic Speech Recognition (ASR) systems on-device or in the cloud presents distinct challenges and opportunities. On-device models can theoretically benefit from proximity to the user's speech, enabling personalization. However, methods for achieving this without labels often yield suboptimal performance. This talk will initially present Samsung AI Center Cambridge's latest research efforts in robust unsupervised adaptation of ASR models using as little as 1 minute of unlabeled speech. In the cloud, the increased availability of computational resources has driven the development of large language model-powered ASR models (SpeechLLMs), which are expected to offer higher accuracy. However, and as the second part of this talk will illustrate, caution is warranted when evaluating SpeechLLMs. Indeed, ASR benchmarks for SpeechLLMs are often flawed and may suffer from test sets contamination. The pretraining phase of most LLMs involves exposure to text content that is identical to the test sets of common ASR datasets, resulting in a clearly biased evaluation and potential performance impacts.

Sign languages are unwritten, low-resource languages that convey meaning through gestures of the hands, upper body, and face. They are used by millions of deaf and hard of hearing individuals around the world, but sign language technologies are far behind those for spoken and written languages. There has been some encouraging progress, including on tasks like isolated sign recognition and sign-to-written language translation. However, the current state of the art is far from handling arbitrary linguistic domains and visual environments. This talk will describe work in my group aimed at sign language understanding for real-world domains and conditions, including recent work on self-supervised representation learning for American Sign Language.

This talk presents advances in automatic speech recognition and spoken language identification, focusing on addressing key challenges in multilingual environments, low-resource scenarios, and robust representation learning. We explore innovative approaches to improve ASR performance across diverse languages, incorporate metadata for enhanced speech representations, and develop effective evaluation methodologies.

Neural audio codecs have significantly advanced audio compression by efficiently converting continuous audio signals into discrete tokens. These codecs preserve high-quality sound and enable sophisticated sound generation through generative models trained on these tokens. However, existing neural codec models are typically trained on large, undifferentiated audio datasets, neglecting the essential discrepancies between sound domains like speech, music, and environmental sound effects. This oversight complicates data modeling and poses additional challenges to the controllability of sound generation. To tackle these issues, we introduce the Source-Disentangled Neural Audio Codec (SD-Codec), a novel approach that combines audio coding and source separation. By jointly learning audio resynthesis and separation, SD-Codec explicitly assigns audio signals from different domains to distinct codebooks, sets of discrete representations. Experimental results indicate that SD-Codec not only maintains competitive resynthesis quality but also, supported by the separation results, demonstrates successful disentanglement of different sources in the latent space, thereby enhancing interpretability in audio codec and providing potential finer control over the audio generation process.

Neural audio codec models are crucial to modern generative AI pipelines, but until recently have largely relied on convolutional backbones. In this talk I share some recent work on building and scaling a transformer-based neural audio codec for speech. I’ll discuss my motivations for taking this approach, the challenges encountered along the way, and the successes and limitations of the work.

We will discuss Moshi, our recently released model. Moshi is capable of full-duplex dialogue, e.g. it can both speak and listen at any time, offering the most natural speech interaction to date. Besides, Moshi is also multimodal, in particular it is able to leverage its inner text monologue to improve the quality of its generation. We will cover the design choices behind Moshi in particular the efficient joint sequence modeling permitted by RQ-Transformer, and the use of large scale synthetic instruct data. Finally, I will discuss how this approach can be extended to new use cases like simultaneous speech translation.

Speech foundation models are transforming research by unifying speech-processing tasks through scaling data, model size, and task diversity. This shift has divided research roles, with large tech companies building foundational models and smaller entities focusing on refinement and analysis, raising concerns about explainability due to limited transparency. To address this, our group has developed Open Whisper-style Speech Models (OWSM) at Carnegie Mellon University, replicating OpenAI Whisper-style training using public data and our open-source toolkit ESPnet. Our models exhibit explainable behaviors due to their transparent development. We also investigate scaling laws and emergent capabilities in speech foundation models by studying model and data size impacts within the OWSM suite. This presentation will discuss these advancements and the research challenges they present to the speech and audio community, emphasizing open collaboration and transparency to enhance accessibility and interpretability in speech processing technologies.

Singing voice synthesis (SVS) has emerged as a rapidly evolving research area. However, achieving high-quality and expressive singing synthesis remains a challenging task, requiring large-scale curated datasets, effective modeling strategies, and robust evaluation frameworks. This talk will provide a comprehensive overview of the key components of SVS research, covering data curation, model development, and evaluation methodologies. We will introduce ACE-Opencpop and ACE-KiSing, two large-scale singing voice datasets designed to support diverse SVS applications. On the modeling side, we will explore TokSing, a discrete token-based SVS approach, and SingOMD, which leverages multi-resolution discrete representations to enhance synthesis quality. In terms of evaluation, we will discuss the Interspeech 2024 Challenge on Speech Processing Using Discrete Units, the SingMOS dataset for MOS prediction, and VERSA, a versatile evaluation toolkit for speech, audio, and music processing. By bridging data, modeling, and evaluation, this talk aims to provide insights into the current advancements and challenges in SVS research, highlighting emerging directions for improving naturalness, expressiveness, and overall synthesis quality.

Recent advances in single-channel speech enhancement have yielded substantial performance gains but often at the cost of prohibitive model sizes or inference complexity. In this talk, I present a three-part framework addressing these challenges by unifying speaker-agnostic compression, personalized modeling, and scalable architectures. First, we discuss speaker-agnostic model compression via low-bit quantization. Drawing on Bitwise Neural Networks (BNN) and Incremental Binarization on RNNs, we show how feedforward and recurrent networks can be quantized to binary parameters, thereby drastically reducing computational complexity with only a minor degradation in enhancement quality. We also explore discriminative hashing approaches (e.g., Boosted Locality Sensitive Hashing) to represent audio spectra as highly compact binary codes—enabling efficient lookups for source separation in resource-constrained devices. Next, we shift toward personalized speech enhancement—a speaker-aware model compression paradigm. Here, zero-shot approaches that exploit speaker embeddings or knowledge distillation serve to adapt compact models on-the-fly without requiring any clean speech data from the user. By selecting or distilling knowledge from large teacher networks, these personalized systems can adapt to new speakers or recurring noise conditions at test time, thus bridging the performance gap between large, general-purpose models and lightweight, specialized models. Finally, we introduce scalable and efficient enhancement architectures. Building on blockwise optimization (BLOOM-Net) and modified cold diffusion, we design flexible models whose internal blocks or iterative steps can be selectively engaged based on real-time resource constraints. This scalability lets the same network accommodate diverse deployment scenarios—ranging from low-power embedded devices to higher-capacity servers—while maintaining strong enhancement performance.

In today's landscape of natural language processing (NLP) and speech processing, developing applications often begins with fine-tuning a foundation model. However, teaching a foundation model new skills is not as straightforward as it seems. Despite the sophistication of current models, introducing new capabilities can often impair their original functions, a phenomenon known as catastrophic forgetting. While experience replay is a common solution, the lack of open-source training data for models like LLaMA poses challenges for continuous training. This talk will delve into recent research on fine-tuning language models, including their spoken counterparts, focusing on preserving their initial capabilities. This talk will also share some benchmarks related to the ongoing fine-tuning of foundation models.

This talk gives an overview of recent developments by NVIDIA NeMo team. We introduce Canary-1B, an open state-of-the-art speech recognition and translation model, and discuss the details of its training: synthetic data generation and efficient dataloading approach that scales to arbitrarily sized datasets. We demonstrate how Canary-1B training was further optimized to decrease the required number of GPUs by 4x with 2D bucketing and batch size optimizer techniques. Finally, we provide a brief overview of SALM and BESTOW architectures for SpeechLLMs and highlight our progress on efficient multimodal SpeechLLM training (EMMETT).

State-of-the-art speech recognition systems do not work well for many languages, limiting the digital participation of many speakers worldwide. To address this challenge, we need both better ways to reliably measure speech model performance, and new algorithms for bridging this performance gap. In this talk, I propose solutions to both these problems, beginning with ML-SUPERB 2.0, a new benchmark to evaluate multilingual speech models on language identification and automatic speech recognition (ASR) across languages and datasets. Indeed, our benchmark reveals large differences in ASR performance between languages, regardless of the modeling approach used. To mitigate this, I introduce a new model training objective based on distributionally robust optimization. Our new method reduces ASR performance differences between languages by minimizing the training loss of the worst-performing language. This work paves the way for more equal access to speech technology for speakers of all languages.

Neural audio codecs (NACs) have gained significant attention as essential technologies for audio compression and as foundational components for speech language models. In the era of speech LMs, there are both challenges and opportunities in the codec domain. This talk presents three topics of NACs, including modelling, evaluation and security. This talk introduces TS3-Codec, a Transformer-Based Simple Streaming Single Codec. TS3-Codec provides key benefits, including streaming capability, low computational demands, low bitrate, and a single codebook design, all while delivering high audio quality. Additionally, this talk presents Codec-SUPERB, the first benchmark designed to evaluate codec models in terms of reconstruction quality from both signal-level and application-level perspectives. Finally, this talk presents CodecFake, the first deepfake audio dataset based on codecs. The CodecFake dataset equips models to effectively counter codec-based speech generation systems.

Discrete audio tokens have recently gained considerable attention for their potential to connect audio and language processing, enabling the creation of modern multimodal large language models. Ideal audio tokens must effectively preserve phonetic and semantic content along with paralinguistic information, speaker identity, and other details. While several types of audio tokens have been recently proposed, identifying the optimal tokenizer for various tasks is challenging due to the inconsistent evaluation settings in existing studies. To address this gap, we release the Discrete Audio and Speech Benchmark (DASB), a comprehensive leaderboard for benchmarking discrete audio tokens across a wide range of discriminative tasks, including speech recognition, speaker identification and verification, emotion recognition, keyword spotting, and intent classification, as well as generative tasks such as speech enhancement, separation, and text-to-speech. Our results show that, on average, semantic tokens outperform compression tokens across most discriminative and generative tasks. However, the performance gap between semantic tokens and standard continuous representations remains substantial, highlighting the need for further research in this field.

He will discuss his recent work on generating explanations for audio models. While deep learning models excel at achieving high performance, they often function as black boxes, offering little transparency into their decision-making processes. His aim in this line of work is to develop methods that produce listenable explanations for these black-box audio models without compromising their original performance. Through several metrics, he demonstrates that the explanations generated by his approach remain faithful to the original model and are both listenable and understandable.

Continual learning in end-to-end automatic speech recognition (E2E-ASR) often suffers from catastrophic forgetting, where fine-tuning leads to significant performance degradation on previously seen data. While adapters offer a way to switch between fine-tuned models, they still underperform in unseen domains—a challenge when the input domain is unknown. We propose a method that reduces forgetting to just 3.4%, significantly outperforming fine-tuning strategies like LoRA, which exhibits a 49% forgetting rate. By linearly interpolating the parameters of multiple models fine-tuned from the same generalist model, we achieve a unified model that excels across diverse datasets. Moreover, this model can be iteratively fine-tuned and averaged while maintaining low forgetting rates. Our experiments demonstrate the robustness of this approach across various datasets and models, presenting a promising solution for continual learning in E2E-ASR.