IZeroDNAS: innovative differentiable NAS using zero-cost metrics and SuperNet initialization

With the rapid advancement of deep learning, an increasing number of neural network architectures have been developed to meet the growing demand for both high performance and computational efficiency. In many real-world applications, such as autonomous driving and real-time object tracking, models are required not only to achieve high accuracy but also to satisfy strict hardware constraints, including low latency, limited memory footprint and reduced energy consumption. To address these challenges, neural architecture search (NAS) has emerged as a promising paradigm for automatically discovering network architectures that are tailored to specific tasks and deployment conditions.

By systematically exploring the design space, NAS relieves researchers from the burden of manual architecture engineering. In practice, particularly for edge computing and embedded AI systems, the discovered architectures must be compact and hardware-aware – optimized not only for accuracy but also for resource metrics such as FLOPs, latency and model size. Despite its potential, NAS remains a highly computationally expensive process. Traditional sampling-based NAS methods – including reinforcement learning-based, evolutionary and differentiable approaches (Liu et al., 2019; Real et al., 2019; Zoph and Le, 2016) – often require hundreds or even thousands of GPU days to converge to a final architecture. These methods typically treat NAS as a black-box optimization over a discrete and combinatorial space, necessitating the evaluation of a large number of sampled architectures and leading to significant inefficiency.

SuperNet-based Differentiable NAS (DNAS) methods (Guo et al., 2020a; Liu et al., 2019; Yu et al., 2020) have been introduced to alleviate the computational burden of conventional NAS. By consolidating all candidate architectures into a unified SuperNet framework and relaxing the discrete search space into a continuous one, these approaches enable gradient-based optimization. Consequently, they substantially accelerate the search process compared to purely sampling-based methods. Nevertheless, even with DNAS, training a large SuperNet remains time-consuming, particularly for large-scale tasks such as object detection.

Recent works have explored zero-cost proxies (Abdelfattah et al., 2021; Li et al., 2023; Mellor et al., 2021) as a complementary approach to further reduce search time. These proxies estimate a model’s performance without requiring training and, when combined with sampling-based strategies like evolutionary algorithms, can efficiently rank candidate architectures. However, most existing metrics were originally designed for simpler classification models or relatively small search spaces (e.g. networks with fewer than 10 M parameters) and do not generalize well to the complex architectures typical of object detection tasks.

To overcome these limitations, we present an efficient search framework for large-scale object detection models, namely, ZeroDNAS and its improved variant IZeroDNAS. ZeroDNAS integrates zero-cost metrics directly into the differentiable NAS optimization process, eliminating the need to train candidate models during the search. This enables differentiable NAS to be applied to macro-architecture search spaces while significantly reducing computational cost. Despite these advantages, Table 5 shows that zero-cost rankings are highly sensitive to SuperNet initialization. Variations in initial weights produce inconsistent rankings, leading to unstable search behavior and degraded performance. Moreover, the original ZeroDNAS does not account for deployment constraints such as inference latency and model size, limiting its suitability for edge devices. These issues are summarized in the upper section of Figure 1.

We address these challenges with IZeroDNAS, an enhanced version of ZeroDNAS that incorporates SuperNet warm-up initialization and hardware-aware loss functions, as illustrated in the lower section of Figure 1. Before architecture search, the SuperNet is trained for a few epochs to better approximate realistic performance behavior. This warm-up step substantially improves the consistency of zero-cost rankings across different random initializations, yielding more stable and robust search outcomes. Crucially, this phase does not aim to improve model accuracy but to stabilize ranking, which differentiates it from conventional pretraining and requires minimal additional computation. Once warm-up is completed, the search proceeds using the ZeroDNAS algorithm. Finally, hardware-aware loss terms – including depth-aware and latency-aware objectives – guide the search toward compact, fast and device-optimized architectures suitable for real-world deployment. Our contributions can be summarized as follows:

• We propose IZeroDNAS, a fast and robust NAS framework that integrates zero-cost proxies into differentiable NAS. This substantially reduces search time and memory usage, while enabling architectures tailored to device-specific constraints.

• We introduce SuperNet warm-up initialization to enhance the stability of zero-cost rankings in a resource-efficient manner. To balance stability and efficiency, we further provide two variants: a standard warm-up for robust accuracy and a depthwise-accelerated warm-up that significantly reduces initialization time.

• We incorporate hardware-aware loss terms to guide the search toward architectures with fewer parameters and faster inference on the target device.

• Our framework is evaluated across diverse search spaces, including ScaledYOLOv4 (Wang et al., 2021a) and YOLOv7 (Wang et al., 2023), demonstrating superior performance compared to manually designed object detection models.

Relationship to prior work: This work extends our previous conference paper, ZeroDNAS (Poliakov et al., 2023), which first explored the combination of differentiable NAS and zero-cost proxies for efficient architecture search. While ZeroDNAS significantly accelerated the search process, it did not address issues of ranking instability or deployment constraints. In this work, we address these limitations by proposing IZeroDNAS, which incorporates warm-up initialization and hardware-aware objectives. Furthermore, we provide new experimental analyses to demonstrate the robustness and generalization of the proposed method across search spaces and random seeds.

In this section, we comprehensively review prior works related to NAS, with an emphasis on its application to object detection. We begin with a discussion of the evolution of NAS techniques, from sampling-based and differentiable approaches to recent advancements in zero-cost proxies. Next, we explore existing NAS strategies for object detection, highlighting their limitations in practical deployment scenarios. Finally, we examine manually designed object detectors that offer valuable insights into efficient and accurate architectural design, which we leverage to construct a more effective NAS search space.

NAS has emerged as a powerful approach to automate neural network design. Early methods, including NAS-RL (Zoph and Le, 2016) and Aging Evolution (Real et al., 2019), primarily relied on sampling-based strategies that repeatedly train and evaluate candidate architectures. While effective in discovering competitive models, these approaches require enormous computational resources, often reaching thousands of GPU days, which limits their practical applicability.

To alleviate this burden, weight-sharing techniques such as ENAS (Pham et al., 2018), SPOS (Guo et al., 2020a) and BigNAS (Yu et al., 2020) introduced the concept of a SuperNet, where multiple architectures are embedded and share parameters during training. These methods significantly reduce the training cost by avoiding redundant training of individual models. Nevertheless, due to their reliance on discrete sampling, they often struggle to capture fine-grained architectural trade-offs and may converge to suboptimal solutions.

To further improve efficiency and search quality, differentiable NAS (DNAS) approaches such as DARTS (Liu et al., 2019) and FBNet (Wu et al., 2019) relaxed the search space from discrete to continuous domains. This enabled the use of gradient-based optimization to simultaneously tune both architecture and operation weights. While these methods offer improved convergence speed and search quality, they demand high GPU memory and often suffer from overfitting or performance collapse when scaled to more complex tasks beyond image classification.

Recently, zero-cost proxies have emerged as a compelling alternative to reduce the computational demands of NAS. These methods aim to estimate the relative performance of architectures without any training or with minimal computation. For instance, NASWOT (Mellor et al., 2021), SynFlow (Abdelfattah et al., 2021), ZiCo (Li et al., 2023) and SpeedyNAS (Ru et al., 2021) introduced analytical or gradient-based metrics to quickly rank candidate architectures. While these proxies have shown promise on classification benchmarks, they often lack generalization when applied to large-scale or complex tasks such as object detection. Furthermore, they are known to be highly sensitive to model initialization and network depth, which undermines their stability and robustness in real-world NAS pipelines.

Designing NAS pipelines for object detection presents a unique set of challenges, primarily due to the increased complexity and computational requirements of detection networks. Unlike classification tasks that involve a single prediction head, object detectors consist of multi-stage pipelines including backbone, feature pyramid networks (FPN) and detection heads. This makes the search space significantly larger and more entangled.

Early efforts in NAS for object detection, such as Auto-FPN (Xu et al., 2019), NAS-FPN (Ghiasi et al., 2019) and NAS-FCOS (Wang et al., 2020a), concentrated on searching the FPN structure. These works demonstrated performance gains but often resulted in irregular and overly complex topologies that hinder real-time deployment, particularly on resource-constrained devices.

To address the fragmented nature of these search results, subsequent methods like SM-NAS (Yao et al., 2020) and Hit-Detector (Guo et al., 2020b) proposed a joint search of backbone and FPN components. This holistic view led to more coherent architectures and improved accuracy, but at the expense of increased GPU memory usage and search time, which remain prohibitive for many practitioners.

EAutoDet (Wang et al., 2022b) improved upon this by constraining the SuperNet size and optimizing differentiable parameters under a tighter computational budget. While this approach achieved a better trade-off between performance and efficiency, it still required expensive hardware (e.g. NVIDIA V100 GPUs) and multi-day search periods.

Recent approaches have explored integrating zero-cost proxies into object detection NAS frameworks. For instance, MAEDet (Sun et al., 2022) proposed a memory-aware proxy to guide early-stage pruning. Zero-DNAS (Poliakov et al., 2023) introduced a constraint-aware NAS framework that leverages zero-cost metrics and latency prediction to efficiently search for deployment-friendly object detectors. The authors demonstrated significant speedups in the search process while maintaining competitive accuracy (Poliakov et al., 2023). However, these proxy-based NAS methods often inherit the instability issues discussed earlier and may fail to generalize across different detection frameworks or deployment platforms. Moreover, many of these works ignore hardware-awareness during optimization, leading to architectures that are theoretically efficient but practically unoptimized for real devices.

While NAS methods strive to automate architecture design, many of the most effective object detectors in recent years – such as YOLOv7, PP-YOLOE and RT-DETR – have been manually crafted through expert knowledge and extensive empirical validation. These models illustrate several valuable design principles, including scale-aware representations, early fusion strategies and re-parameterized convolution blocks, which have proven effective across diverse tasks and platforms.

Despite their handcrafted nature, these models consistently outperform many NAS-based detectors in terms of accuracy-efficiency trade-offs, particularly on edge devices. Thus, they offer a strong prior for constructing NAS search spaces that are both expressive and hardware-friendly. Unfortunately, many prior NAS methods neglect these insights, resulting in either overly large or inefficient architectures.

In our work, we address these limitations by incorporating inductive biases from manually designed models into our NAS search space. Moreover, we propose a robust NAS framework that includes SuperNet warm-up initialization to stabilize zero-cost metrics and a hardware-aware loss function that aligns the search process with deployment requirements. Details of our proposed approach are elaborated in Section 5.

The classical NAS problem is defined as:

where A is the search space and $α$ is an architecture sampled from A. The goal is to identify an architecture $α$ that achieves the best performance when trained with weights $wα$⁠.

Differentiable NAS (DNAS) (Wu et al., 2019) reformulates NAS as a continuous optimization problem by introducing trainable importance parameters $θ$⁠. Once optimized, these parameters are used to compute architecture probabilities p via the Gumbel-Softmax (Jang et al., 2017), which indicates the selection probability of each candidate. Specifically, the architecture probability $pi,j$ for the j-th candidate in the i-th stage is given by:

where $θi$ denotes the importance array of i-th searchable stage and $θi,j$ is the importance parameter of j-th choice in $θi$⁠. $gi,j$ represents the noise from a Gumbel distribution and the temperature t adjusts the softness of the output probability.

An architecture $α$ can then be sampled according to these probabilities, with the sampling probability expressed as:

where $bi,j$ is j-th candidate block at i-th searchable stage and $bi$ is a block of i-th stage that sampled from a candidate set ${bi,j}$⁠.

The objective of DNAS is to simultaneously optimize the importance parameters $θ$ and the SuperNet weights w to sample high-performing architectures, formalized as:

Thus, the primary bottleneck in DNAS lies in training both the SuperNet weights w and the architecture parameters p, leading to a highly time-consuming process. Moreover, training all candidate blocks in parallel requires substantial memory, especially as the search space grows.

To address the computational challenges in DNAS, including high training time and memory consumption, ZeroDNAS (Poliakov et al., 2023) decouples the training of SuperNet weights and architecture parameters by incorporating zero-cost proxies. It optimizes only the importance parameters, leveraging the NASWOT (Mellor et al., 2021) metric for architecture performance estimation.

The key idea of NASWOT is that better architectures tend to produce more diverse feature maps for different input images. It evaluates the similarity between activation outputs of a randomly initialized network across different inputs within a mini-batch. First, activation patterns of k input samples are encoded as binary codes. Next, a correlation matrix K is formed based on the Hamming distances among these codes. Finally, the WOT score is computed as the logarithm of the determinant of K:

where a higher WOT score indicates better expected architecture performance.

Unlike the original NASWOT, which computes scores for the entire architecture, ZeroDNAS evaluates the WOT metric individually for each candidate to align with the architecture parameters. A WOT score map $MWOT∈R|Oi|×S$ is constructed for the search, where $|Oi|$ is the number of candidates in the i-th stage and S is the total number of searchable stages.

To discover an optimal architecture that balances accuracy and model complexity, ZeroDNAS measures the FLOPs of each candidate block, storing them in an FLOPs map $Mflops∈R|Oi|×S$⁠. The importance parameters $θ$ are then updated based on both $MWOT$ and $Mflops$⁠, reflecting the trade-off between candidate quality and computational cost. The expected zero-cost score and FLOPs of the SuperNet are computed using the architecture probabilities p from equation (2):

where $MWOTi,j$ and $Mflopsi,j$ correspond to the zero-cost score and FLOPs of the j-th candidate block at the i-th stage.

Finally, to define the objective for optimizing the importance parameter $θ$ under the constraint of target FLOPs, ZeroDNAS incorporates the zero-cost loss $Lzero_cost$ and FLOPs loss $Lflops$ for the search. The FLOPs loss is measured by the mean square error between the achieved and target FLOPs:

where $λ$ and $ϕ$ control the relative importance of zero-cost performance and computational cost.

Although ZeroDNAS (Poliakov et al., 2023) addressed the search time challenges by using a zero-cost proxy, we found that its results are sensitive to different SuperNet initializations and are not well-suited for real-world device applications, as illustrated in Figure 1. To overcome these limitations, we propose IZeroDNAS, which builds upon ZeroDNAS (Poliakov et al., 2023), incorporating SuperNet warm-up initialization and hardware-aware loss. The whole IZeroDNAS process is illustrated in Figure 2. For a fair comparison of our improvements, we use the same ScaledYOLOv4 search space as ZeroDNAS. In addition, we designed a YOLOv7 search space to accommodate future applications.

We consider two primary search spaces in our experiments: the ScaledYOLOv4 search space and a newly designed YOLOv7 search space, reflecting recent architectural advancements and practical application scenarios. The detailed configurations of both search spaces are described below.

Our SuperNet integrates all candidate architectures into a single structure, accelerating the search process and increasing the likelihood of discovering optimal designs. Unlike early cell-based NAS methods such as DARTS (Liu et al., 2019) and ENAS (Pham et al., 2018), which often produce fragmented architectures, we adopt a block-wise search approach inspired by FBNet (Wu et al., 2019), which generalizes better to practical applications.

The ScaledYOLOv4 search space builds on the manually designed model (Wang et al., 2021a), which leverages cross stage partial (CSP) blocks (Wang et al., 2020b) to improve gradient flow and reduce computational cost. Each stage corresponds to a CSP block, parameterized by d, the number of Darknet bottleneck modules and g, the channel reduction ratio. As shown in Figure 3(a), the backbone and neck each contain four searchable stages (green blocks). Backbone stages use $d∈2,4,6,8$ and $g∈0.25,0.50,0.75$⁠, while neck stages use $d∈0,2,4,6$ with the same g values, yielding 12 candidate options per stage and a total of $128 ≈ 4.3×108$ architectures.

This design is extended to the ScaledYOLOv4-P5 variant by including nine searchable blocks (green and blue in Figure 3(a). Backbone stages use $d∈1,3,7,15$⁠, neck stages $d∈0,3,6,9$⁠, while g remains unchanged, resulting in $129 ≈ 5.1×109$ possible architectures. Figure 3(b) illustrates the CSP block structure, where d and g control both computation and feature representation.

To capture recent design advances, we define a YOLOv7-based search space using ELAN and E-ELAN blocks (Wang et al., 2022a, 2023). ELAN balances long and short gradient paths to improve feature reuse, while E-ELAN increases network cardinality for enhanced gradient diversity.

The YOLOv7 SuperNet contains eight searchable stages (green blocks in Figure 4(a), each defined by a channel scaling factor $g∈0.75,1.00,1.25$ and a binary connection pattern specifying which intermediate layers are concatenated to the final output, as shown in Figure 4(b). For full YOLOv7, the five-layer pattern gives $25×3=96$ options per stage, resulting in $(32×3)8 ≈ 7.2×1015$ total architectures. For YOLOv7-tiny, the four-layer pattern yields $24×3=48$ options per stage, totaling $(16×3)8 ≈ 2.8×1013$⁠, suitable for edge scenarios with strict latency and memory constraints.

Overall, this unified block-wise search space allows IZeroDNAS to efficiently explore diverse architectures across both ScaledYOLOv4 and YOLOv7, while accounting for feature representation, depth and computational requirements.

To improve the stability of zero-cost proxy evaluations, we propose two variants of SuperNet warm-up initialization, which are described in the following subsections.

Zero-cost proxies offer a time-efficient method for evaluating a model without the need for training. However, they are highly sensitive to the model’s initial state when applied to large models. This sensitivity can lead to significant variations in ranking results, resulting in searched architectures with inconsistent performance. As shown in Table 5, this sensitivity to initialization can produce unstable results. To mitigate this, we introduce a SuperNet warm-up initialization before the search stage. In this phase, the randomly initialized SuperNet weights are trained for a few epochs with fixed importance parameters (⁠$θ$⁠). This process stabilizes the network’s feature representations, improving the reliability of zero-cost proxies and producing more consistent architecture rankings. Importantly, this warm-up requires only a small number of epochs, as it is solely intended to stabilize the ranking rather than optimize full model accuracy. In contrast, traditional SuperNet training typically requires 50–150 epochs to achieve comparable accuracy, making our approach highly time-efficient.

We also observe that the Gumbel softmax temperature t in equation (2) significantly affects ranking robustness. Early differentiable NAS (DNAS) algorithms used a high temperature during burn-in to ensure sufficient gradient signals for all candidate blocks. However, such high temperature effectively transforms the SuperNet into a structure akin to InceptionNet (Szegedy et al., 2015), leading to feature confusion among candidate blocks and inaccurate zero-cost rankings. Through empirical analysis presented in Section 6.3, we find that using a lower temperature substantially enhances the robustness of zero-cost proxies for evaluating model performance.

While the conventional warm-up improves ranking stability, it still introduces a nonnegligible computational overhead. To reduce this cost, we propose a depthwise-accelerated warm-up initialization. The core idea is to train a lightweight approximation of the standard convolutional filters using depth-wise separable convolutions. Specifically, instead of training the standard convolutional filters $Wstd∈RCout×Cin×k×k$ directly, we decompose them into a depth-wise filter $Wdw∈RCin×1×k×k$ and a point-wise filter $Wpw∈RCout×Cin×1×1$⁠. The standard convolution can then be approximated by:

where i and j index the output and input channels and $(u,v)$ index the spatial kernel positions. This decomposition substantially reduces both the number of parameters and the computational cost during warm-up training, while the merged standard filters are restored during inference to maintain equivalent performance. An overview of this scheme is illustrated in Figure 5.

In summary, the depthwise-accelerated variant provides a computationally efficient alternative to conventional warm-up, preserving the benefits of stabilizing zero-cost rankings while significantly reducing initialization time.

In real-world applications, such as automated vehicles, fast inference on specific edge devices is crucial. While FLOPs are commonly used as a constraint to indicate efficiency, fewer FLOPs do not always translate to faster inference on a given device. Instead, measured inference time is the most accurate performance indicator. Moreover, for models with similar FLOPs, we observed that shallow, wide models tend to have faster inference times than deeper, narrower ones, as shown in Table 15 and discussed in Section 6.8. To ensure optimal performance on specific devices, we incorporate both inference time and model depth into our optimization process through latency-aware loss (⁠$Llatency$⁠) and depth-aware loss (⁠$Ldepth$⁠), collectively referred to as hardware-aware loss.

To obtain the real inference time on a specific device, a straightforward approach would be to measure the inference time of each architecture directly on the target device during the search process. However, this is both time-consuming and impractical. Instead, we use a prebuilt look-up table for each candidate block, which contains the real inference times measured on the target device. By leveraging this look-up table, we can estimate the overall inference time of different architectures on the target device based on the candidate blocks’ real inference time:

where $latencyi,j$ is the time of the j-th candidate block at the i-th stage in the look-up table. $β$ is the hyperparameter to balance the importance of the latency-aware loss $Llatency$⁠.

To find shallower models, which typically offer faster speeds, we integrate depth-aware loss to regulate the model depth:

where $depthi,j$ is the depth of the j-th candidate block at the i-th stage. $ω$ is the hyperparameter to balance the importance of the depth-aware loss $Ldepth$⁠.

Finally, combined with equation (10), the overall loss function of IZeroDNAS is:

This section presents the results of our IZeroDNAS process applied to ScaledYOLOv4 and YOLOv7. We use mean average precision (mAP) as the accuracy metric, reporting both mAP (IOU ranging from 50$%$ to 90$%$⁠) and AP50 (IOU of 50$%$⁠) as evaluation measures. During the warm-up initialization, we trained the SuperNet weights using a batch size of eight and a resolution of 416x416 for 20 epochs on the Pascal VOC 2007 data set. We used the Stochastic gradient descent optimizer with an initial learning rate of 0.01, gradually reduced to 0.002 using a cosine annealing scheduler. For the search stage, we switched to the Adam optimizer, setting hyperparameters $λ=0.03$ for $LWOT$⁠, $ϕ=0.005$ for $LFLOPs$⁠, $β=0.002$ for $Llatency$ and $ω=0.001$ for $Ldepth$ in the overall loss function, based on our experience. The Gumbel temperature was gradually decreased from 5 to 0.1 during this phase. For evaluating the model’s real-world performance, we used the Qualcomm Robotics RB5 platform as our target device and measured the inference time (ms) for one input image to assess the model’s inference speed. Before applying our IZeroDNAS, we prebuilt a look-up table of real inference speeds for the RB5 device, which was used during the search process. To further validate the hardware-aware loss under different platforms, the resulting architectures were also tested on NVIDIA Jetson TX2 and Jetson AGX Orin devices.

To evaluate our IZeroDNAS, we assess three key aspects: performance compared to manually designed models, performance compared to NAS-based methods within the same search space and performance compared to NAS-based methods in different search spaces.

Manually designed models: Within the ScaledYOLOv4 search space, we compare models searched by IZeroDNAS with manually designed models (Wang et al., 2021a) and our base method, ZeroDNAS (Poliakov et al., 2023), on the Pascal VOC data set. As shown in Table 1, IZeroDNAS not only outperforms the manually designed models in metrics such as mAP and FLOPs – demonstrating its ability to discover high-performance models with fewer parameters but also achieves the fastest inference time on edge devices, which is highly meaningful for real-world applications. Further discussion can be found in 6.2.

NAS-based methods in the same search space: We compare our searched models with other NAS methods within the same ScaledYOLOv4 search space on the Pascal VOC data set. As shown in Table 2, IZeroDNAS identifies models with superior performance while maintaining fast search times. Although our search time is slightly longer than ZeroDNAS, we achieve higher accuracy (47.1% mAP) and fewer parameters, demonstrating improved search stability and efficiency. To ensure fairness, we reevaluate the previous NAS methods – Cream (Peng et al., 2020), AttentiveNAS (Wang et al., 2021b) and our earlier ZeroDNAS – under an additional efficiency loss constraint. These methods were originally optimized only for FLOPs, which correlate with computational complexity but do not directly reflect inference latency. Incorporating efficiency loss changes the optimization objective from accuracy-only to a dual objective that simultaneously encourages shallower, faster architectures. Consequently, a slight drop in mAP is expected, as reducing network depth improves latency at the expense of representational capacity. In contrast, our IZeroDNAS variants explicitly optimize with both depth loss and latency loss, yielding a more faithful and effective search objective. This design enables a superior accuracy–latency trade-off, as evidenced in Table 2, where IZeroDNAS consistently outperforms prior NAS approaches in both metrics.

NAS-based methods in different design spaces: To evaluate generalization across search spaces, we compare our models with recent NAS methods designed on different search spaces using the COCO data set, a standard benchmark for object detection. Table 3 shows that IZeroDNAS can discover competitive architectures in under 5 h on a single RTX 2080 GPU, which is three times faster than MAE-DET (Sun et al., 2022), a state-of-the-art zero-cost NAS method. While the warm-up initialization slightly increases search time compared to ZeroDNAS (Poliakov et al., 2023), IZeroDNAS maintains excellent computational efficiency with lower FLOPs and demonstrates strong generalization across data sets.

YOLOv7 search space: Finally, we extend our evaluation to the YOLOv7 search space, comparing our searched models against manually designed models and ZeroDNAS. As summarized in Table 4, the warm-up initialization facilitates highly effective searches, achieving 50.1% mAP for YOLOv7 and 33.8% mAP for YOLOv7-tiny. In terms of AP50, we obtain 74.3% for YOLOv7 and 58.7% for YOLOv7-tiny. Across both the ScaledYOLOv4 and YOLOv7 search spaces, these results underscore the robustness and stability of IZeroDNAS in discovering high-performing models across different YOLO architectures, highlighting its versatility for a wide range of object detection tasks.

Impact of proposed methods: To validate the key contributions of our algorithm – stability improvement and real-world device applicability – we compare the average mAP and variance across four SuperNet initializations (random seeds), along with the actual inference time on the target device. As shown in Table 5, the previous work, ZeroDNAS (Poliakov et al., 2023), is highly sensitive to different SuperNet initializations, resulting in a large mAP variance of 5.687 and an overall mAP of only 44.7%. In addition, its inference time on the target edge device is slower than that of the manually designed model. In contrast, with our proposed warm-up initialization, IZeroDNAS achieves much more consistent performance, with a significantly reduced mAP variance of 0.0025, enhancing the stability of the search algorithm while also improving overall accuracy to 48% mAP. Furthermore, the hardware-aware loss, which accounts for real-world inference, directs the search toward smaller models that typically achieve faster inference on the target device. By incorporating these two methods, IZeroDNAS consistently discovers the best candidate architecture, achieving both superior accuracy and a compact model with optimized inference speed, making it well-suited for real-world applications.

Influence of each loss: To further investigate the impact of each loss component on the search process, we conducted an ablation study of the IZeroDNAS loss function, as shown in Table 6. In this table, the target constraint is set at 70 G FLOPs, similar to the manually designed YOLOv4-P5 model. As previously mentioned, the zero-cost loss serves as a critical metric for IZeroDNAS in guiding the model search. However, without additional loss constraints, the search process becomes uncontrolled, potentially leading to suboptimal models that waste computational resources. By incorporating FLOPs loss, the search can better align with the target constraint.

Next, we focus on the proposed hardware-aware losses, including depth-aware loss and latency-aware loss. Depth-aware loss helps identify models that outperform manually designed ones while also providing slightly faster inference on the target device. On the other hand, latency-aware loss focuses on real-world inference time by using the prebuilt look-up table. This approach leads to the discovery of smaller models with faster inference times on the target device, though at the expense of model accuracy. Fast inference but low accuracy is not ideal. Therefore, IZeroDNAS combines all these losses to achieve an optimal balance between accuracy and device performance under the target constraint.

Warm-up NAS: Our NAS-specific warm-up goes beyond the traditional goal of stabilizing learning rates. It is designed to improve architecture search stability by reducing fluctuations in SuperNet parameter initialization, Gumbel sampling probabilities and zero-cost proxy rankings – factors critical for zero-cost NAS. While this increases the nominal runtime to 6.2 h, it markedly enhances search stability and the reliability of sampled architectures. As shown in Table 7, IZeroDNAS consistently achieves higher mAP across different random seeds compared to ZeroDNAS. Although ZeroDNAS has a lower nominal search time (1.8 h), its instability often requires multiple trials (effectively 1.8 h × N trials) to find high-performing architectures. In contrast, the greater stability of IZeroDNAS allows a single search to reliably identify top architectures, highlighting the essential role of NAS-specific warm-up in improving both search efficiency and performance.

WOT stability during warm-up initialization: We analyze the rankings of candidate blocks to assess the stability of WOT throughout the SuperNet warm-up initialization phase. We measure the ranking correlation between any two WOT maps (⁠$MWOT$⁠) across different random seeds using Kendall’s Tau (Kendall, 1938). Kendall’s Tau values close to 1 indicate strong positive correlation, while values near −1 reflect strong negative correlation. To reduce the sensitivity of zero-cost rankings to SuperNet initialization, we aim for higher positive correlations between rankings across different random initializations. As shown in Figure 6, the warm-up phase gradually increases this ranking correlation, leading to a more stable WOT metric. In addition, Table 5 demonstrates that the improved stability during warm-up benefits the performance of searched results in IZeroDNAS compared to the base ZeroDNAS method.

Parameter convergence effects: We observe that parameter variance consistently decreases across seeds (Figure 7), suggesting that more stable parameters produce more consistent features, which reduces WOT variance and improves ranking correlation. Unlike prior zero-cost proxies (Abdelfattah et al., 2021; Li et al., 2023; Mellor et al., 2021), which rely solely on random initialization, our NAS-specific warm-up provides a simple yet effective guideline to stabilize zero-cost metrics in larger models. Moreover, IZeroDNAS can serve as a modular replacement for zero-cost proxies, facilitating seamless integration with future improvements.

Practical impact on detection performance: To further assess the practical impact of warm-up, we examined how additional warm-up epochs affect actual detection performance. Figure 8 shows that as warm-up progresses, AP50 not only becomes more stable but also improves slightly, converging around 20 epochs. The variance of AP50 across the four random seeds decreases as well, indicating that sufficient warm-up enhances both WOT stability and detection reliability.

Consistency across initializations: Beyond correlation analysis, we further investigate the consistency of architecture selection under different SuperNet initializations. As shown in Figure 9, each SuperNet i represents the i-th randomly initialized model. Without warm-up (left), different initializations favor disjoint candidate groups (e.g. SuperNet 3 favors candidates 9–12, while SuperNet 2 prefers 5–8). After warm-up (right), all SuperNet i consistently converge to selecting candidates within the range 5–8, indicating significantly improved search stability. While a formal mathematical proof of this property remains open, our empirical results suggest that warm-up serves as an effective and practical means of stabilizing zero-cost metrics in large-scale SuperNets.

Effect of Gumbel temperature: Besides warm-up effects, the choice of the Gumbel temperature hyperparameter (t in 2) also significantly influences ranking correlation during initialization. As mentioned in 3, temperature t controls the softness of the output probabilities, thereby affecting architecture selection. As seen in Figure 6 and Table 8, both the searched results and the effect of warm-up vary with different temperature settings. A lower temperature (⁠$t=1$⁠) leads to overall instability, suggesting difficulties in correctly ranking candidate blocks and requiring longer training to improve ranking correlation. In contrast, higher temperatures (⁠$t=3$ and $t=5$⁠) enable faster convergence of ranking correlation by providing stronger supervisory signals for candidate blocks. However, as shown in Table 8, an excessively high temperature (⁠$t=5$⁠) may result in inaccurate ranking and degraded performance. Therefore, in our experiments, setting $t=3$ during the SuperNet warm-up initialization yields the most stable and optimal results.

In this section, we explain why NASWOT was chosen as our zero-cost proxy for the experiments. Each zero-cost proxy is grounded in a different underlying principle: SNIP (Lee et al., 2019) focuses on parameter sensitivity, NASWOT (Mellor et al., 2021) emphasizes feature similarity and ZiCo (Li et al., 2023) considers the influence of training gradient properties on convergence. These distinct design philosophies lead to varying estimation capabilities across different task models. To identify the most suitable proxy for YOLO-series object detection, we applied several zero-cost proxies in combination with an evolutionary algorithm, following the approaches outlined in their respective papers.

As shown in Table 9, NASWOT achieves overall competitive performance compared to manually designed models and outperforms the other zero-cost proxies. Based on these results, we selected NASWOT as the zero-cost proxy for our experiments. Nevertheless, it is important to note that while NASWOT can identify models with competitive accuracy, it tends to produce larger models with approximately 58 M parameters, which is less suitable for edge device deployment. In contrast, our IZeroDNAS approach not only discovers smaller models with only 47 M parameters but also achieves a higher mAP of 47.1%, surpassing the baseline.

To evaluate the generalizability of our hardware-aware loss, we extended experiments to two embedded platforms, Jetson TX2 and Jetson AGX Orin, with results summarized in Tables 10 and 11.

Across both devices, the IZeroDNAS variants consistently achieve a favorable balance between detection accuracy and inference efficiency. In particular, IZeroDNAS-P5 and IZeroDNAS-CSP exhibit substantially lower latency compared to the ScaledYOLOv4 baselines and the original ZeroDNAS models. On the Jetson TX2, IZeroDNAS reduces inference time while maintaining or slightly improving mAP, whereas on the Jetson AGX Orin, it attains the lowest latency among all evaluated architectures. These findings demonstrate that the proposed hardware-aware loss effectively generalizes across heterogeneous platforms, enabling architectures that are simultaneously high-performing and low-latency, which is critical for real-time deployment scenarios.

To ensure a fair analysis, we analyze the results obtained from IZeroDNAS under complex constraints. We observe that IZeroDNAS tends to favor shallower models compared to manually designed ones, achieving better accuracy. The following discussion will explore IZeroDNAS tendencies within the search spaces of ScaledYOLOv4 (Wang et al., 2021a) and YOLOv7 (Wang et al., 2023).

ScaledYOLOv4 search space: To achieve models of varying complexity, manually designed ScaledYOLOv4 models (Wang et al., 2021a) use a scaling approach that linearly increases the network’s depth and channel count. However, we have observed that IZeroDNAS tends to find more accurate models with shallow and wider architectures, characterized by lower depth and more channels. As shown in Table 12, manually designed YOLOv4-CSP has 30 Darknet bottleneck modules, which indicates the model depth within our searchable blocks in total, whereas our IZeroDNAS-CSP has just 20 Darknet bottleneck modules. Similarly, YOLOv4-P5 has 52 bottleneck modules, which is substantially deeper than the 27 modules in our IZeroDNAS-P5.

ScaledYOLOv4 search space: As shown in Table 13, the v7-IZeroDNAS model without hardware-aware loss tends to select connection paths from shallower layers, as indicated by the lower values of connection choices compared to the manually designed model. This suggests that v7-IZeroDNAS reduces the overall model depth, consistent with the patterns observed in the ScaledYOLOv4 search space. In contrast, the v7-IZeroDNAS-tiny model without hardware-aware loss maintains a depth similar to YOLOv7-tiny, likely because the tiny version is already underfitting and does not benefit from further depth reduction. We also analyze the shortest and longest gradient paths. In our search space, these paths are calculated by summing the smallest and largest connection choices, respectively. While Wang et al. (2022a) highlighted the importance of tuning both shortest and longest gradient paths, they did not provide precise guidelines for models of different scales. In our experiments, the searched models tend to increase the length of the shortest gradient path compared to manually designed models, which may provide useful insights for gradient path selection.

To provide a more comprehensive evaluation, we include a comparison with other YOLO-based NAS methods, such as YOLO-NAS. Because the details of YOLO-NAS’s search space, strategy and implementation are not publicly available, we cannot exactly reproduce its search process for a fully fair comparison. Attempts to apply our IZeroDNAS framework to the YOLO-NAS search space were not feasible due to the lack of official documentation or detailed description of the search space.

Instead, we use the officially released YOLO-NAS models (YOLO-NAS-S, YOLO-NAS-M and YOLO-NAS-L) as a reference and fine-tune them on the VOC data set. The results are reported in Table 14 alongside our IZeroDNAS architectures. Although the original search spaces differ, making the comparison not strictly fair, these results serve as a practical performance reference. As shown, IZeroDNAS achieves slightly lower accuracy than the fine-tuned YOLO-NAS models, but it consistently provides substantially faster inference speeds. This demonstrates the effectiveness of our hardware-aware design in achieving a favorable trade-off between accuracy and efficiency, making IZeroDNAS particularly suitable for deployment scenarios with strict latency requirements.

As discussed in Section 6.6, IZeroDNAS often favors shallower architectures to achieve higher accuracy. Beyond accuracy, inference time is another crucial factor for edge deployment, where computational resources are limited. Thus, identifying architectures that can deliver faster inference under the same FLOPs constraint becomes an important objective. To illustrate this, we compare ResNet152 (He et al., 2015) and Wide ResNet50$_$2 (Zagoruyko and Komodakis, 2016), two architectures with similar FLOPs but markedly different depths. Table 15 reports their inference times on both an Nvidia 3090ti GPU and the Qualcomm Robotics RB5 platform, our experimental target device. Despite their comparable FLOPs, ResNet152, with its 152 layers, exhibits substantially slower inference than Wide ResNet50$_$2, which has only 50 layers but a wider design. The results highlight that model depth plays a critical role in practical efficiency, with shallower-and-wider networks offering significant speed advantages without compromising FLOPs. Motivated by this observation, we incorporate a depth-aware loss into our hardware-aware loss formulation. This additional term guides the search toward architectures that not only achieve accuracy improvements, as seen in Section 6.6, but also exhibit faster inference on real devices. As shown in Table 15, this strategy is consistent with empirical trends, enabling IZeroDNAS to better balance accuracy, efficiency and real-world deployment performance.

We introduce IZeroDNAS to improve the search stability of differentiable NAS with zero-cost proxies and to optimize searched models for edge devices in terms of size and inference time. To mitigate the sensitivity of initialization, we propose a SuperNet warm-up scheme that enhances ranking correlation and stability, making the search more robust. To further accelerate this process, we present a depthwise-accelerated initialization, which reduces training cost by leveraging depthwise separable convolutions during warm-up while maintaining equivalent performance at inference. In addition, hardware-aware loss guides the search toward compact, fast and accurate models. Experiments demonstrate that IZeroDNAS surpasses state-of-the-art NAS methods and manually designed models in both accuracy and real-device efficiency. Future work will explore broader zero-cost proxies for stability and more accurate predictors of device performance.

Van-Tin Luu and Tao-Ting Chen contributed equally to this work.