Live page · Day archive

Factored One-sided Adam-Moment for Practical and Scalable SOAP

Read the original at HF Daily Papers

Summary

FOAM is a lightweight SOAP variant that uses input-only preconditioning and a factored Adam-moment estimator to reduce memory and eigendecomposition overhead.

Carried by: HF Daily Papers. First seen: .