AnchorPrompt

Abstract

Large audio-language models (LALMs) are sensitive to input perturbations, such as noise, waveform corruption, and adversarial injections. We propose AnchorPrompt, an efficient adaptation method that keeps the model frozen and learns a single block of eight prompt vectors inserted at the decoder input, between the audio and question embeddings. We train these vectors through self-distillation over diverse audio and text perturbations. To improve answer consistency, we use the model's prediction on the clean recording as the training target for answerable inputs. When the audio lacks sufficient clues to answer, the target is a refusal, which discourages hallucination. Consequently, the method requires no ground-truth labels. Furthermore, AnchorPrompt is perturbation-agnostic at inference, requiring no prior detection of perturbations and making it well-suited for zero-shot transfer. We evaluate three LALMs across three benchmarks and show that AnchorPrompt improves answer consistency while preserving or improving accuracy on clean audio. It also reduces hallucinations on unanswerable audio while reducing refusals on clean recordings. Finally, our evaluations confirm that the learned soft prompts generalize effectively to unseen perturbations.

Method

• Frozen model, eight vectors. One prompt of L = 8 vectors per model, shared across every perturbation.
• Label-free. Targets come from the model's own answer on the clean recording. Ground truth is only used to build injections and to evaluate.
• No perturbation detector. The same prompt is applied to every input, clean or not.
• Training perturbations. White noise at 20 to −20 dB SNR, masking of 20 to 100%, adversarial audio injection (a synthesized voice speaking a wrong option), and adversarial text injection (a sentence asserting a wrong option).

Audio examples

SAKURA test-split questions that the model answers correctly on clean audio. Each card plays the clean and the perturbed input and shows the answer of the frozen model alone (Base) and with the learned prompt (+ AnchorPrompt). The three models are shown one after the other; where possible the same recording is used for all of them.

Audio clips are from the SAKURA benchmark (Yang et al., Interspeech 2025) and are reproduced here in small numbers for illustration only; all rights remain with the original dataset authors.

Code and trained prompts

Each checkpoint is a single block of 8 vectors (under 0.5 MB) for one frozen model.

git clone https://github.com/poonehmousavi/anchorprompt && cd anchorprompt
pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu130

# evaluate a released prompt (test split)
bash scripts/eval.sh qwen2.5-omni checkpoints/qwen2.5-omni/pool.pt sakura

# train your own
bash scripts/train.sh qwen2.5-omni

See the README for data preparation and all options.

Citation

@misc{mousavi2026anchorprompt,
  title  = {AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models},
  author = {Mousavi, Pooneh and Ivry, Amir and Ravanelli, Mirco and Subakan, Cem},
  year   = {2026},
  url    = {https://poonehmousavi.github.io/anchorprompt}
}