US patent US12801917
Method and system for optimizing activation functions in neural networks
Abstract
A method and system for optimizing activation functions to improve efficiency of neural networks is provided. Activation function benchmark datasets are created by training neural network architectures with a plurality of activation functions. The activation function benchmark datasets comprise results obtained through the training for various tasks. Features indicative of performance of the activation functions in the activation benchmark datasets are identified. A metric space is created by analyzing identified features of activation functions, which is a low-dimensional representation of the activation functions. Activation functions are optimized based on metric space by employing dimensionality reduction techniques to visualize the metric space. Subsequently, characteristics of optimized activation functions which are predictive of performance are determined. Activation functions are selected from optimized activation, which are applied to improve real-world tasks carried out by neural networks.