Automated Deep Learning: Neural Architecture Search Is Not the End
By Xuanyi Dong, Complex Adaptive Systems Lab, University of Technology Sydney, Australia and Brain Team, Google Research, USA, xuanyi.dxy@gmail.com | David Jacob Kedziora, Complex Adaptive Systems Lab, University of Technology Sydney, Australia, david.kedziora@uts.edu.au | Katarzyna Musial, Complex Adaptive Systems Lab, University of Technology Sydney, Australia, katarzyna.musial-gabrys@uts.edu.au | Bogdan Gabrys, Complex Adaptive Systems Lab, University of Technology Sydney, Australia, bogdan.gabrys@uts.edu.au
Abstract
Deep learning (DL) has proven to be a highly effective approach for developing models in diverse contexts, including visual perception, speech recognition, and machine translation. However, the end-to-end process for applying DL is not trivial. It requires grappling with problem formulation and context understanding, data engineering, model development, deployment, continuous monitoring and maintenance, and so on. Moreover, each of these steps typically relies heavily on humans, in terms of both knowledge and interactions, which impedes the further advancement and democratization of DL. Consequently, in response to these issues, a new field has emerged over the last few years: automated deep learning (AutoDL). This endeavor seeks to minimize the need for human involvement and is best known for its achievements in neural architecture search (NAS), a topic that has been the focus of several surveys. That stated, NAS is not the be-all and end-all of AutoDL. Accordingly, this review adopts an overarching perspective, examining research efforts into automation across the entirety of an archetypal DL workflow. In so doing, this work also proposes a comprehensive set of ten criteria by which to assess existing work in both individual publications and broader research areas. These criteria are: novelty, solution quality, efficiency, stability, interpretability, reproducibility, engineering quality, scalability, generalizability, and eco-friendliness. Thus, ultimately, this review provides an evaluative overview of AutoDL in the early 2020s, identifying where future opportunities for progress may exist.
Automated Deep Learning: Neural Architecture Search Is Not the End
Deep learning (DL) has proven to be a highly effective approach for developing models in diverse contexts, including visual perception, speech recognition, and machine translation. Automated deep learning (AutoDL) endeavors to minimize the need for human involvement and is best known for its achievements in neural architecture search (NAS).
In this monograph, the authors examine research efforts into automation across the entirety of an archetypal DL workflow. In so doing, they propose a comprehensive set of ten criteria by which to assess existing work in both individual publications and broader research areas, namely novelty, solution quality, efficiency, stability, interpretability, reproducibility, engineering quality, scalability, generalizability, and eco-friendliness.
Aimed at students and researchers, this monograph provides an evaluative overview of AutoDL in the early 2020s, identifying where future opportunities for progress may exist.
