Evolution and Role of Optimizers in Training Deep Learning Models

Document Type

Article

Publication Date

1-1-2024

Abstract

To perform well, deep learning (DL) models have to be trained well. Which optimizer should be adopted? We answer this question by discussing how optimizers have evolved from traditional methods like gradient descent to more advanced techniques to address challenges posed by high-dimensional and non-convex problem space. Ongoing challenges include their hyperparameter sensitivity, balancing between convergence and generalization performance, and improving interpretability of optimization processes. Researchers continue to seek robust, efficient, and universally applicable optimizers to advance the field of DL across various domains.

Identifier

85203537468 (Scopus)

Publication Title

IEEE/CAA Journal of Automatica Sinica

External Full Text Location

https://doi.org/10.1109/JAS.2024.124806

e-ISSN

23299274

ISSN

23299266

First Page

2039

Last Page

2042

Issue

10

Volume

11

Grant

2023KY0055

This document is currently not available here.

Share

COinS