Components of Neural Architecture Search

Within deep learning research, Neural Architecture Search (NAS) is a developing field that aims to improve model performance and applicability. Even with all of its potential, NAS implementation can be difficult. Upon closer inspection, NAS may be broken down into three basic components: search space, search strategy/algorithm, and Evaluation strategy. These elements can be treated in a variety of ways to maximize the search for efficient neural network designs. Gaining insight into how these components interact is essential to utilizing NAS to its maximum potential in improving the performance and capacities of deep learning models and applications.

These components are:

• Search Space: The search space in Neural Architecture Search defines the set of possible neural network architectures that the algorithm explores to find an optimal model.
• Search Strategy: The search strategy in Neural Architecture Search dictates the method or algorithm used to navigate and explore the defined search space to find optimal neural network architectures.
• Evaluation Strategy: The evaluation strategy in Neural Architecture Search involves assessing the performance of candidate neural network architectures within the search space to determine their effectiveness based on predefined criteria or objectives.

Establishing the Search Space

The NAS search space is pivotal in dictating the potential architectures discoverable by the NAS algorithm. It comprises a set of operations that delineate the fundamental structure (skeleton) of the network, the nature of units or blocks defining the layers, and the permissible connectivity between layers to formulate architectures.

The richness of the search space corresponds to its complexity and versatility. However, an expansive search space also escalates the challenges and computational costs associated with identifying the optimal architecture. Various types of operations contribute to defining the search space, including sequential layer-wise operations, cell-based representations, hierarchical structures, and more.

Formulating the Search Strategy

The search strategy plays a crucial role in determining how the NAS algorithm explores various neural networks. In essence, the algorithm optimizes the performance metrics of child models, treating them as rewards, to generate high-performance architecture candidates from a sample population of network candidates.

Diverse methods enhance the efficiency of search strategies for quicker and more consistent results. These include random search, evolutionary algorithms, Bayesian approaches, and reinforcement learning. Recent findings indicate that evolutionary methods perform comparably to reinforcement learning, exhibiting better “anytime performance” and converging on smaller models.

As an evolution from discrete search spaces, the introduction of a continuous architecture search space has given rise to differentiable search methods, allowing for gradient-based optimization.

• Reinforcement Learning: Reinforcement learning is a search technique that guides machine learning models to make a series of decisions, aiming to find a problem-solving approach that maximizes rewards. This method involves trial and error, where the model receives rewards and penalties, allowing it to learn and make decisions in complex environments. Reinforcement learning is integral to Neural Architecture Search (NAS), addressing significant challenges in computational and memory resources.
  In 2018, Hsu et al. introduced a noteworthy contribution to NAS using reinforcement learning known as Multi-Objective Neural Architecture Search (MONAS). This approach focuses on optimizing scalability while ensuring high accuracy and minimizing power consumption, providing a solution to critical bottlenecks in NAS.

• Random Search: Random search in Neural Architecture Search (NAS) involves selecting a neural network architecture from the search space through a random process. This method is resource-intensive, opting for a brute-force approach rather than an efficient strategy. The randomness in selecting architectures makes it an expensive process, often requiring a substantial amount of GPU time, ranging from hundreds to thousands of GPU days for a single search. The duration of the search is contingent on the complexity of the search space, contributing to the computational demands associated with random search in NAS.

• Bayesian Optimization: In neural architecture search (NAS), Bayesian optimization refers to the process of utilizing probabilistic models to direct the search for the best possible neural network configurations. It simulates the intricate and costly-to-evaluate neural network performance landscape using surrogate models, usually Gaussian processes. Bayesian optimization effectively traverses the search space, reducing the number of experiments necessary by iteratively choosing configurations that are likely to improve performance. This method works well for finding high-performing neural network topologies in NAS because it allows for the efficient use of limited computational resources.
• Evolutionary algorithms: Aside from choosing the appropriate genetic evolution parameters, such as growth rate and death rate, there’s also the requirement to assess how well neural network topologies are represented in the genotypes we employ for digital evolution. Compositional Pattern Producing Networks (CPPNs), on the other hand, offer a powerful indirect encoding that can be improved with NEAT.
  HyperNEAT is a variant of NEAT that uses CPPNs for encoding and evolves with the NEAT algorithm. Also an indirect encoding method for evolution: algorithms employs directed acyclic graphs to encode various neural network designs.

Determining the Evaluation Strategy

In the process of neural architecture search (NAS), the algorithm engages in extensive training, evaluation, validation, and performance comparison to identify the optimal neural network. However, full training for each neural network is resource-intensive, demanding substantial time and compute resources, often in the order of thousands of GPU days. To mitigate these evaluation costs, various strategies are employed:

• Proxy task performance: using a substitute task that shares traits with the primary objective to accelerate architectural search.
• Low-fidelity Performance Estimation: Employing techniques like early exit, training on a data subset, or downscaled models to estimate performance quickly.
• Weight Inheritance: Inheriting weights from previously trained models to kickstart training and reduce resource-intensive retraining.
• Weight Sharing: Sharing weights among architectures during training to promote information exchange and accelerate convergence.
• Learning-Curve Extrapolation: Predicting the learning curve of a model based on early training epochs, allowing early termination if performance is unsatisfactory.
• Network Morphism: Adapting model structures during training to explore a diverse set of architectures efficiently.

One-Shot Approach: Integrating Search and Evaluation

The goal of the One-Shot Neural Architecture Search (NAS) method is to complete the search and assessment in a single iteration. In order to quickly assess several architectures, it offers a proxy job or dataset during the search phase. This entails training a super network that includes multiple candidate architectures; just a portion of the network is turned on for assessment. The One-Shot NAS is more effective in finding high-performing neural network designs because it simultaneously optimizes and evaluates architectures, which lessens the computational load associated with standard NAS approaches.

Neural Architecture Search (NAS) falls within the realm of automated machine learning (AutoML). AutoML is a comprehensive term encompassing the automation of diverse tasks in the application of machine learning to real-world challenges. The article explores the fundamentals, and applications of the NAS algorithm.