Beyond Weighted Summation: Learnable Nonlinear Aggregation Functions for Robust Artificial Neurons
arXiv:2603.19344v1 Announce Type: new Abstract: Weighted summation has remained the default input aggregation mechanism in artificial neurons since the earliest neural network models. While computationally efficient, this design implicitly behaves like a mean-based estimator and is therefore sensitive to noisy or extreme inputs. This paper investigates whether replacing fixed linear aggregation with learnable nonlinear alternatives can improve neural network robustness without sacrificing trainability. Two differentiable aggregation mechanisms are introduced: an F-Mean neuron based on a learnable power-weighted aggregation rule, and a Gaussian Support neuron based on distance-aware affinity weighting. To preserve the optimisation stability of standard neurons, hybrid neurons are proposed that interpolate between linear and nonlinear aggregation through a learnable blending parameter. Evaluated in multilayer perceptrons and convolutional neural networks on CIFAR-10 and a noisy CIFAR-10
arXiv:2603.19344v1 Announce Type: new Abstract: Weighted summation has remained the default input aggregation mechanism in artificial neurons since the earliest neural network models. While computationally efficient, this design implicitly behaves like a mean-based estimator and is therefore sensitive to noisy or extreme inputs. This paper investigates whether replacing fixed linear aggregation with learnable nonlinear alternatives can improve neural network robustness without sacrificing trainability. Two differentiable aggregation mechanisms are introduced: an F-Mean neuron based on a learnable power-weighted aggregation rule, and a Gaussian Support neuron based on distance-aware affinity weighting. To preserve the optimisation stability of standard neurons, hybrid neurons are proposed that interpolate between linear and nonlinear aggregation through a learnable blending parameter. Evaluated in multilayer perceptrons and convolutional neural networks on CIFAR-10 and a noisy CIFAR-10 variant with additive Gaussian corruption, hybrid neurons consistently improve robustness under noise while F-Mean hybrids also yield modest gains on clean data. The three-way hybrid achieves robustness scores of up to 0.991 compared to 0.890 for the standard baseline, and learned parameters converge consistently to sub-linear aggregation (p $\approx$ 0.43--0.50) and high novelty utilisation ($\alpha$ $\approx$ 0.69--0.79). These findings suggest that neuron-level aggregation is a meaningful and underexplored design dimension for building more noise-tolerant neural networks.
Executive Summary
This article presents a novel approach to improving the robustness of artificial neural networks by introducing learnable nonlinear aggregation functions. The authors propose two differentiable aggregation mechanisms: the F-Mean neuron and the Gaussian Support neuron. To balance robustness and trainability, hybrid neurons are introduced that interpolate between linear and nonlinear aggregation. The evaluation on CIFAR-10 and its noisy variant demonstrates significant improvements in robustness. The findings suggest that learning nonlinear aggregation rules can lead to more noise-tolerant neural networks. Furthermore, the learned parameters converge to sub-linear aggregation and high novelty utilisation, indicating that neuron-level aggregation is a meaningful design dimension. The study's results and methods have the potential to advance the field of neural networks and inspire further research.
Key Points
- ▸ Introduction of learnable nonlinear aggregation functions for artificial neural networks
- ▸ Proposal of two differentiable aggregation mechanisms: F-Mean neuron and Gaussian Support neuron
- ▸ Development of hybrid neurons that interpolate between linear and nonlinear aggregation
- ▸ Evaluation on CIFAR-10 and its noisy variant demonstrates significant improvements in robustness
Merits
Strength
The study introduces a novel approach to improving robustness in neural networks, which is a significant contribution to the field. The experimental evaluation demonstrates the effectiveness of the proposed method, and the results are well-justified by mathematical analysis.
Strength
The study's focus on neuron-level aggregation is a meaningful design dimension, which is an underexplored area in neural network research. The results of this study can inspire further research in this area.
Demerits
Limitation
The study is limited to evaluating the proposed method on a specific dataset, and it would be beneficial to evaluate it on other datasets as well to generalise the results.
Limitation
The study assumes that the noise in the input data follows a Gaussian distribution, which may not be the case in many real-world applications. Further research is needed to investigate the robustness of the proposed method under different types of noise.
Expert Commentary
The study presents a novel approach to improving the robustness of artificial neural networks by introducing learnable nonlinear aggregation functions. The authors propose two differentiable aggregation mechanisms: the F-Mean neuron and the Gaussian Support neuron. The evaluation on CIFAR-10 and its noisy variant demonstrates significant improvements in robustness. The findings suggest that learning nonlinear aggregation rules can lead to more noise-tolerant neural networks. Furthermore, the learned parameters converge to sub-linear aggregation and high novelty utilisation, indicating that neuron-level aggregation is a meaningful design dimension. The study's results and methods have the potential to advance the field of neural networks and inspire further research.
Recommendations
- ✓ Further research is needed to evaluate the proposed method on other datasets and to investigate its robustness under different types of noise.
- ✓ The proposed method can be used to improve the robustness of neural networks in various applications, and its implementation can be explored in fields such as image classification, natural language processing, and speech recognition.
Sources
Original: arXiv - cs.LG
