Robust ADAM optimizer based on averaging aggregation functions

Training on contaminated data (outliers, heavy tails, label noise, preprocessing artifacts) makes arithmetic averaging in empirical risk unstable: multiple anomalies bias estimates, destabilize optimization steps, and degrade generalization ability. There is a need for a way to improve the robustness without changing the loss function or model architecture.

Aim. The paper aims to develop and demonstrate an alternative approach to batch averaging in ADAM, replacing it with a robust penalty-based averaging aggregation function, which mitigates the influence of outliers, while still maintaining the benefits of moment-based and coordinate-wise step adaptation.

Methods. Penalized, averaging aggregation means are used. The Huber dissimilarity function is used. Newton's method is used to find the optimal center and weights for batch elements. Performance is evaluated in a controlled experiment with synthetic outliers, by comparing it to the standard ADAM algorithm for training stability.

Results. Robust ADAM showed more robust training for synthetic linear regression, with the resulting model remaining stable even with up to 20% of outliers. The method keeps providing computational efficiency and compatibility by adding only a small number of iterations of the robust center search to each batch, while sustaining the same asymptotic behavior. With a quadratic penalty function, it degenerates into standard Adam, confirming the validity of the generalization.

Conclusion. A modification of the Adam optimization algorithm has been made using M-means. This method ensures the stability of linear regression, with outliers even up to 20%. The exact limitations are still to be determined. Computational overhead is associated with calculating the optimal value for each batch. However, due to the rapid convergence (approximately three iterations using Newton's method), the algorithm slowdown is not significant.