Selective Predictive Training and Structured Predictability in Neural Training Dynamics
Date Issued
August 2026
Author(s)
Wang, Fanqi
The University of Tennessee
Advisor(s)
Hairong Qi
Additional Advisor(s)
Dongsheng Ding
Dan Wilson
Zachary G. Nicolaou
Abstract
The training of modern deep neural networks is still governed by long sequences of iterative gradient-based updates, and most acceleration techniques reduce the cost per step rather than the step-by-step nature of optimization. This dissertation studies predictive training as an alternative paradigm: when recent training dynamics contain usable short-horizon structure, selected parts of the optimization trajectory may be advanced by prediction rather than traversed step by step.
The first part develops Predictive Differential Training (PDT), which combines Koopman/DMD-based weight forecasting with standard gradient-based optimization. Rather than applying predictions globally, PDT uses a coordinate-wise reliability mask based on scale control and dynamic consistency: coordinates whose predicted motion is sufficiently large, bounded, and directionally consistent with the local optimizer trajectory are advanced by prediction, while the rest fall back to the baseline optimizer. This hybrid update keeps gradient-based correction as a stabilizing reference while exploiting predictability where available. Across CIFAR-10 and ImageNet-1K classification, several architectures and optimizers, and a representative self-supervised setting, PDT reduces time-to-baseline training targets while maintaining comparable final performance, and ablations show that neither criterion alone is sufficient.
The second part develops a measurement framework for structured predictability in training dynamics. Rather than equating predictability with the performance of a single forecaster, it uses complementary probes of directional persistence, recent-update subspace alignment, and predictor-based selectivity to measure where short-horizon temporal redundancy appears across parameter roles, training stages, architectures, and optimizers. Predictability is found to be localized, stage-dependent, architecture-shaped, and optimizer-conditioned: predictable regions often form a minority of the parameter space, and their location and persistence vary substantially across models and training conditions.
Together, these contributions support a conditional view of predictive acceleration: training trajectories can contain exploitable temporal redundancy, but this redundancy is neither global nor uniform, so prediction is justified only where short-horizon regularity can be detected and used with appropriate safeguards. By pairing an online selective training method with an offline diagnostic framework for temporal predictability, the dissertation lays groundwork for measurement-guided predictive training, mechanism-aware online masking, optimizer-state-aware prediction control, and overhead-aware acceleration of large-scale training.
Degree
Doctor of Philosophy
Major
Computer Engineering
File(s)![Thumbnail Image]()
Name
WANG-PRIMARY-2026.pdf
Size
18.01 MB
Format
Adobe PDF
Checksum (MD5)
71633b3ca5c0841354789d2f23736db3