The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop: Workshop on Software and Hardware Co-Design of Deep Learning Systems in Accelerators (SHDA)
Authors: Marcel Aach (Forschungzentrum Juelich, University of Iceland); Rakesh Sarma and Eray Inanc (Forschungzentrum Juelich); Morris Riedel (University of Iceland, Juelich Supercomputing Centre (JSC)); and Andreas Lintermann (Forschungzentrum Juelich)
Abstract: Hyperparameter Optimization (HPO) of Neural Networks is a computationally expensive procedure, that has the potential to benefit from the use of novel accelerator capabilities. This paper investigates the performance of three popular HPO algorithms in terms of the achieved speed-up and model accuracy, utilizing early stopping, Bayesian, and genetic optimization approaches, in combination with mixed precision functionalities on NVIDIA A100 GPUs with Tensor Cores. The benchmarks are performed on 64 GPUs in parallel on three datasets: two from the vision and one from the CFD domain. The results show that, depending on the algorithm, larger speed-ups can be achieved for mixed precision compared to full precision HPO if the checkpoint frequency is kept low. In addition to the reduced runtime, also small gains in generalization performance on the test set are observed.