Adaptive Gradient Clipping Python Packages

zclip

Official implementation of the paper: "ZClip: Adaptive Spike Mitigation for LLM Pre-Training".

gradient-accumulator

:dart: Gradient Accumulation for TensorFlow 2

nfnets-pytorch

NFNets and Adaptive Gradient Clipping for SGD implemented in PyTorch. Find explanation at tourdeml.github.io/blog/