BiMTokenizer: Preserving Semantic-Acoustic Balance in Low-Bitrate Speech Tokenization via Bidirectional State-Space Modeling

01Architecture

A single-tower acoustic codec trained end-to-end with discriminators, supervised additionally by a frozen semantic encoder (Whisper / SenseVoice) via reconstruction alignment and cosine-similarity objectives.

Figure 1. Overview of BiMTokenizer. The acoustic codec encodes a mel-spectrogram via stacked Mamba blocks into latent z_c, quantizes it through Residual Spherical Leech Quantization, and decodes through a Vocos head. During training, a frozen semantic encoder enforces alignment between the original and reconstructed speech in the semantic space (dashed paths are training-only).

02Reconstruction · LibriSpeech test-clean

Each column is the same utterance reconstructed by a different codec. Bitrates are reported next to each method. Our variants are highlighted in red.

All baselines reproduced using official pre-trained checkpoints at their respective bitrates.

03Reconstruction · LibriSpeech test-other

The noisier test-other set, drawn from speakers and recording conditions held out from training.