AI Model Robustness Appears Early, Fades During Training

A significant observation in deep neural network training has been identified: a "robustness fading" phenomenon. This refers to the tendency of shallow layers in a network to spontaneously develop robust representations and exhibit flat loss landscapes during the initial phases of training. However, these beneficial properties are not maintained as the training progresses towards standard convergence, leading to a decline in the model's overall robustness to natural corruptions. To counteract this fading, the researchers propose a framework that strategically intervenes in the training dynamics to stabilize these early-emergent robust priors. The framework includes two parameter-free strategies: Early-Phase Stabilization (EPS) and Asymmetric Weight Reversion (AWR). These methods aim to either stabilize or recover the robust configurations of shallow layers without requiring modifications to the model architecture or the introduction of new learnable parameters. Extensive experiments across various benchmarks and architectures demonstrate that this framework significantly improves downstream transfer, dynamic adaptation, and performance in diverse computer vision applications.