An Empirical Study on the Generation of Linear Regions in ReLU Networks : Exploring the Relationship Between Data Topology and Network Complexity in Discriminative Modeling

Sammanfattning

The far-reaching successes of deep neural networks in a wide variety of learning tasks have prompted research on how model properties account for high network performance. For a specific class of models whose activation functions are piecewise linear, one such property of interest is the number of linear regions that the network generates. Such models themselves define piecewise linear functions by partitioning input space into disjoint regions and fitting a different linear function on each such piece. It would be expected that the number or configuration of such regions would describe the model’s ability to fit complicated functions. However, previous works have shown difficulty in identifying linear regions as satisfactory predictors of model success. In this thesis, the question of whether the generation of linear regions due to training encode the properties of the learning problem is explored. More specifically, it is investigated whether change in linear region density due to model fitting is related to the geometric properties of the training data. In this work, data geometry is characterized in terms of the curvature of the underlying manifold. Models with ReLU activation functions are trained on a variety of regression problems defined on artificial manifolds and the change in linear region density is recorded along trajectories in input space. Learning is performed on problems defined on curves, surfaces and for image data. Experiments are repeated as the data geometry is varied and the change in density is compared with the manifold curvature measure used. In no experimental setting, was the observed change in density found to be clearly linked with curvature. However, density was observed to increase at points of discontinuity. This suggests that linear regions can in some instances model data complexities, however, the findings presented here do not support that data curvature is encoded by the formation of linear regions. Thus, the role that linear regions play in controlling the capacity of piecewise linear networks remains open. Future research is needed to gain further insights into how data geometry and linear regions are connected.