Our “best loss” checkpoint was not our best ASR model. This run made it clear that for ASR, WER and human review should drive model selection, not loss alone.

Key Takeaways

  • Loss curves can look unstable while WER still improves.
  • Epoch-boundary loss drops are not automatically a failure signal.
  • Checkpoint selection should be tied to task metrics, then human verification.

While fine-tuning Whisper, we observed a peculiar loss behavior that challenges the standard intuition of deep learning: the model’s training loss and its actual performance metric (WER) started to drift apart.

Divergence of Loss and WER and sharp drop of Loss when Epoch restarts

Training loss and validation WER diverging across epochs.

The “Epoch Collapse” Pattern

In the chart above you can see sharp, vertical drops in the training loss (green). They line up exactly with the start of each new epoch: as the training loop returns to the first batch, the loss collapses for a moment.

It looks alarming, but it isn’t necessarily a failure. While the loss jumps around, the validation WER — the metric that actually matters for transcription quality — keeps declining steadily.

The Practical Lesson: Don’t Blindly Trust the Loss

The main takeaway from this run: the checkpoint with the lowest validation loss was not the one with the lowest validation WER.

Loss and WER measure different things, so they do not always improve together. If we had picked the checkpoint by best loss alone, we’d have shipped a worse transcriber.

Final Thoughts

The lesson I took from this: automated metrics guide the training, but they don’t make the final call.

WER is a better signal than raw loss, but it has blind spots too — especially with Swiss-German dialects, where spelling isn’t standardized. In the end we still relied on human review to pick the final model. The metrics got us close; human ears made the decision.