Skip to content

Fix stale gradient accumulation in non-AMP training - #217

Open
sopenlaz0 wants to merge 1 commit into
Topdu:mainfrom
sopenlaz0:fix/non-amp-zero-grad
Open

Fix stale gradient accumulation in non-AMP training#217
sopenlaz0 wants to merge 1 commit into
Topdu:mainfrom
sopenlaz0:fix/non-amp-zero-grad

Conversation

@sopenlaz0

Copy link
Copy Markdown

Summary

  • Clear optimizer gradients after each non-AMP optimizer step.
  • Add a regression test exercising two batches through Trainer.train().

Problem

The AMP path calls optimizer.zero_grad(set_to_none=True) after stepping, but
the non-AMP path did not. Consequently, each batch's backward pass accumulated
onto gradients retained from every preceding batch.

With two identical batches and a zero-learning-rate optimizer, the regression
test observes gradients of 1.0 then 2.0 before this fix. After the fix, both
steps observe 1.0, and gradients are cleared after the final step.

Verification

python -m pytest -q tests/test_trainer.py
1 passed

@sopenlaz0
sopenlaz0 marked this pull request as ready for review August 7, 2026 03:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant