Expanding an adaptive learning-rate algorithm to handle mini-batch training

Sammanfattning

Resilient backpropagation, Rprop, is a robust and accurate optimization method used in neural network training with batch-learning. As a result of its adaptive step sizes, Rprop requires copious amounts of data at each iteration which slows it down when dealing with large datasets, compared with mini-batch methods. We create and empirically evaluate a version of Rprop, S-Rprop, which can handle mini-batch learning. S-Rprop with optimized hyper-paramaters matches the Stochastic Gradient Descent (SGD) benchmark performance with optimized hyper-parameters using the same convolutional neural network (CNN) architecture. In a deep-learning setting designed to generate vanishing gradient problems, we show that S-Rprop outperforms both Rprop and SGD when re-using the optimal parameters from the CNN.