Pytorch: DeepSpeed
https://www.deepspeed.ai/
https://github.com/microsoft/DeepSpeed
Microsoft
,
DeepSpeed
,
ZeRO
,
ZeRO-Offload
,
distributed computing
,
1-bit adam