Diametrical Risk Minimization: Theory and Computations
Loading...
Authors
Norton, Matthew
Royset, Johannes O.
Subjects
empirical risk minimization
generalization error
solution stability
generalization error
solution stability
Advisors
Date of Issue
2019-10-31
Date
Publisher
ArXiv
Language
Abstract
The theoretical and empirical performance of Empirical Risk Minimization (ERM) often suffers when loss functions are poorly behaved with large Lipschitz moduli and spurious sharp minimizers. We propose and analyze a counterpart to ERM called Diametrical Risk Minimization (DRM), which accounts for worst-case empirical risks within neighborhoods in parameter space. DRM has generalization bounds that are independent of Lipschitz moduli for convex as well as nonconvex problems and it can be implemented using a practical algorithm based on stochastic gradient descent. Numerical results illustrate the ability of DRM to find quality solutions with low generalization error in chaotic landscapes from benchmark neural network classification problems with corrupted labels.
Type
Preprint
Description
Series/Report No
Department
Operations Research (OR)
Organization
Identifiers
NPS Report Number
Sponsors
This work is supported in part by AFOSR under F4FGA08272G001.
Funder
This work is supported in part by AFOSR under F4FGA08272G001.
Format
20 p.
Citation
Norton, Matthew, and Johannes O. Royset. "Diametrical Risk Minimization: Theory and Computations." arXiv preprint arXiv:1910.10844 (2019).
Distribution Statement
Rights
This publication is a work of the U.S. Government as defined in Title 17, United States Code, Section 101. Copyright protection is not available for this work in the United States.