I wanted better token timestamps without adding a separate forced-alignment model. It turns out Whisper’s decoder cross-attention already contains a fairly precise alignment signal, straight from the model internals. Whisper uses an encoder-decoder architecture: the encoder compresses the audio into features, and the decoder generates text token by token from those features. Cross-attention is how the decoder decides which encoder time regions to focus on for each generated token.

Key Takeaways

  • Cross-attention weights provide usable token-to-time alignment.
  • Alignment remained stable even with long inserted silence segments.
  • This creates a practical internal diagnostic tool for Whisper pipelines.

I was curious whether we could improve Whisper’s timestamps by looking at its decoder cross-attention weights. It turns out we can.

Where is the Model Listening?

Whisper is an encoder-decoder Transformer. The encoder processes the whole audio chunk, and the decoder generates text one token at a time. For every token, it looks back at the encoder’s output to find the relevant audio features, and that look-back lives in the cross-attention matrices.

If you extract those matrices, you can see which parts of the input spectrogram the model focuses on for each token.

In the example below, I used audio with two sentences: “Is there a McDonalds around here?” and “I’m here often.” The generated tokens are on the left (Y-axis), and you can watch the model’s focus move across the encoder’s output (X-axis) as it decodes.

Cross-attention alignment heatmap with 7-second pauses

Token-to-time alignment heatmap with a 7-second pause.

Testing with Silence

To check that this holds up, I fed the model audio with long, deliberate pauses (7 and 13 seconds) between sentences. In the heatmaps, the attention shifts to match the actual timing of the speech, even across the gaps.

Cross-attention alignment heatmap with 13-second pauses

Token-to-time alignment heatmap with a 13-second pause.

In these visualizations:

  • Y-axis: The generated tokens.
  • X-axis: The audio timeline (spectrogram).
  • Brightness: Strength of attention.

Why This Matters

The nice part is that this alignment is emergent. We never trained the model to line up tokens with time; it just has to do this to transcribe well.

Even with 1D convolutions and transpositions happening under the hood, the cross-attention patterns map back onto the original audio in a roughly linear way. The model listens in chronological order, token by token, because that’s the simplest way to solve the task.

In practice this gives you a free alignment tool that doesn’t need an external forced aligner (like Wav2Vec2 or the Montreal Forced Aligner), which makes it a handy diagnostic for any Whisper-based pipeline.

Final Thoughts

It’s satisfying to see the internal weights line up so cleanly with the physical timing of the audio. The cross-attention here isn’t just a mathematical convenience — it’s a direct map of where the model is looking in time.