Generalized Differentiable Neural Architecture Search with Performance and Stability Improvements

This work introduces improvements to the stability and generalizability of Cyclic DARTS (CDARTS). CDARTS is a Differentiable Architecture Search (DARTS)-based approach to neural architecture search (NAS) that uses a cyclic feedback mechanism to train search and evaluation networks concurrently, thereby optimizing the search process by enforcing that the networks produce similar outputs. However, the dissimilarity between the loss functions used by the evaluation networks during the search and retraining phases results in a search-phase evaluation network, a sub-optimal proxy for the final evaluation network utilized during retraining. ICDARTS, a revised algorithm that reformulates the search phase loss functions to ensure the criteria for training the networks is consistent across both phases, is presented along with a modified process for discretizing the search network's zero operations that allows the retention of these operations in the final evaluation networks. We pair the results of these changes with ablation studies of ICDARTS' algorithm and network template. Multiple methods were then explored for expanding the search space of ICDARTS, including extending its operation set and implementing methods for discretizing its continuous search cells, further improving its discovered networks' performance. In order to balance the flexibility of expanded search spaces with minimal compute costs, both a novel algorithm for incorporating efficient dynamic search spaces into ICDARTS and a multi-objective version of ICDARTS that incorporates an expected latency penalty term into its loss function are introduced. All enhancements to the original search algorithm are verified on two challenging scientific datasets. This work concludes by proposing and examining the preliminary results of a preliminary hierarchical version of ICDARTS that optimizes cell structures and network templates.