nvidia

/

nemotron-speech-streaming-en-0.6b

like 297

Follow

NVIDIA 47.4k

Hi NVIDIA team,

We're using nemotron-speech-streaming-en-0.6b for real-time streaming ASR and have observed that the RNNT decoder frequently stalls (emits only blank tokens) for 2-8+ seconds after encountering sentence-ending punctuation.

Environment

• NeMo Toolkit: 2.6.0
• Model: nvidia/nemotron-speech-streaming-en-0.6b
• GPU: NVIDIA L4 (24GB)
• CUDA: 12.1

Problem Description

During streaming inference, the model processes audio normally until it outputs a sentence-ending period. After the period, conformer_stream_step() continues to return the same transcript for many consecutive chunks (40+ chunks, representing 3-8 seconds of audio), even though new speech is being spoken.

Example from live radio transcription:

[  3.06s] Miller is running restaurant. Louis Miller Barbecue.
[  3.16s] Miller is running restaurant. Louis Miller Barbecue. There's a
...
[  3.72s] Miller is running restaurant. Louis Miller Barbecue. There's a ample tables and
[  3.72s - 6.40s] (STALLED - same transcript returned for 2.7 seconds)
[  6.40s] Miller is running restaurant. Louis Miller Barbecue. There's a ample tables and

The transcript remains frozen at "There's a ample tables and" while the speaker continues talking. Content spoken during the stall is lost.

Reproduction

We're using the exact approach from your Cache-Aware Streaming tutorial:

# Initialize cache state
cache_last_channel, cache_last_time, cache_last_channel_len = \
    model.encoder.get_initial_cache_state(batch_size=1)
previous_hypotheses = None
pred_out_stream = None

# Pre-encode cache
pre_encode_size = model.encoder.streaming_cfg.pre_encode_cache_size[1]
cache_pre_encode = torch.zeros((1, num_features, pre_encode_size), device=device)

# Process each chunk
for audio_chunk in audio_stream:
    processed_signal, processed_signal_length = model.preprocessor(
        input_signal=audio_tensor, length=audio_len
    )

    # Concatenate pre-encode cache
    processed_signal = torch.cat([cache_pre_encode, processed_signal], dim=-1)
    processed_signal_length += cache_pre_encode.shape[-1]
    cache_pre_encode = processed_signal[:, :, -pre_encode_size:].clone()

    (pred_out_stream, transcribed_texts, cache_last_channel,
     cache_last_time, cache_last_channel_len, previous_hypotheses
    ) = model.conformer_stream_step(
        processed_signal=processed_signal,
        processed_signal_length=processed_signal_length,
        cache_last_channel=cache_last_channel,
        cache_last_time=cache_last_time,
        cache_last_channel_len=cache_last_channel_len,
        keep_all_outputs=False,
        previous_hypotheses=previous_hypotheses,
        previous_pred_out=pred_out_stream,
        return_transcription=True,
    )

Model config shows:

• chunk_size: [105, 112] (1.12s native chunk)
• shift_size: [105, 112]
• pre_encode_cache_size: [0, 9]

What We've Tried

1. keep_all_outputs=True - No improvement
2. Resetting previous_hypotheses after stall detection - Partially helps but loses context
3. Resetting previous_hypotheses on period detection - Makes it worse
4. Not carrying over previous_hypotheses - Breaks transcription entirely

Questions

1. Is this stalling behavior expected for the RNNT decoder after sentence boundaries?
2. Is there a recommended way to handle long-form streaming transcription with multiple sentences?
3. Should we be resetting decoder state at certain points, and if so, how?
4. Are there any parameters we might be missing that could help?

We're building a real-time voice agent and need continuous transcription without these multi-second gaps. Any guidance would be greatly appreciated.

Thank you!