A Family of Adaptive Activation Functions for Mitigating Failure Modes in Physics-Informed Neural Networks
arXiv:2603.18328v1 Announce Type: new Abstract: Physics-Informed Neural Networks(PINNs) are a powerful and flexible learning framework that has gained significant attention in recent years. It has demonstrated strong performance across a wide range of scientific and engineering problems. In parallel, wavelets have been extensively used as efficient computational tools due to their strong approximation capabilities. Motivated by the common failure modes observed in standard PINNs, this work introduces a novel family of adaptive wavelet-based activation functions. The proposed activation functions significantly improve training stability and expressive power by combining trainable wavelet functions with either trainable or fixed hyperbolic tangent and softplus functions. Five distinct activation functions are developed within the PINN framework and systematically evaluated across four representative classes of partial differential equations (PDEs). Comprehensive comparisons using bar pl
arXiv:2603.18328v1 Announce Type: new Abstract: Physics-Informed Neural Networks(PINNs) are a powerful and flexible learning framework that has gained significant attention in recent years. It has demonstrated strong performance across a wide range of scientific and engineering problems. In parallel, wavelets have been extensively used as efficient computational tools due to their strong approximation capabilities. Motivated by the common failure modes observed in standard PINNs, this work introduces a novel family of adaptive wavelet-based activation functions. The proposed activation functions significantly improve training stability and expressive power by combining trainable wavelet functions with either trainable or fixed hyperbolic tangent and softplus functions. Five distinct activation functions are developed within the PINN framework and systematically evaluated across four representative classes of partial differential equations (PDEs). Comprehensive comparisons using bar plots demonstrate improved robustness and accuracy compared to traditional activation functions. Furthermore, the proposed approach is validated through direct comparisons with baseline PINNs, transformer-based architectures such as PINNsFormer, and other deep learning models, highlighting its effectiveness and generality.
Executive Summary
This article introduces a novel family of adaptive wavelet-based activation functions designed to mitigate common failure modes in Physics-Informed Neural Networks (PINNs). The proposed activation functions combine trainable wavelet functions with either trainable or fixed hyperbolic tangent and softplus functions, significantly improving training stability and expressive power. The authors develop five distinct activation functions within the PINN framework and systematically evaluate their performance across four representative classes of partial differential equations (PDEs). The results demonstrate improved robustness and accuracy compared to traditional activation functions, making the proposed approach a valuable addition to the PINN framework. The implications of this work are significant, as PINNs have shown strong performance across various scientific and engineering problems.
Key Points
- ▸ Introduces a novel family of adaptive wavelet-based activation functions for PINNs
- ▸ Combines trainable wavelet functions with hyperbolic tangent and softplus functions
- ▸ Significantly improves training stability and expressive power
- ▸ Demonstrates improved robustness and accuracy compared to traditional activation functions
- ▸ Extensively evaluated across four representative classes of PDEs
Merits
Strength
The proposed activation functions demonstrate improved performance and robustness compared to traditional activation functions, making them a valuable addition to the PINN framework.
Adaptability
The use of trainable wavelet functions allows for adaptability to different problem domains and datasets.
Flexibility
The combination of trainable and fixed hyperbolic tangent and softplus functions provides flexibility in selecting the optimal activation function for a given problem.
Comprehensive Evaluation
The authors systematically evaluate the performance of the proposed activation functions across four representative classes of PDEs, providing a comprehensive understanding of their strengths and limitations.
Demerits
Limitation
The proposed activation functions may not be universally applicable, and their performance may degrade in certain problem domains or datasets.
Computational Cost
The use of trainable wavelet functions may increase the computational cost of training PINNs, potentially limiting their adoption in certain applications.
Expert Commentary
The proposed activation functions demonstrate a significant improvement in the performance and robustness of PINNs, making them a valuable contribution to the field of deep learning. The use of trainable wavelet functions allows for adaptability to different problem domains and datasets, and the combination of trainable and fixed hyperbolic tangent and softplus functions provides flexibility in selecting the optimal activation function for a given problem. However, the proposed activation functions may not be universally applicable, and their performance may degrade in certain problem domains or datasets. Additionally, the use of trainable wavelet functions may increase the computational cost of training PINNs, potentially limiting their adoption in certain applications. Overall, the proposed activation functions have the potential to revolutionize the field of deep learning and PINNs, enabling more accurate and robust predictions in various scientific and engineering applications.
Recommendations
- ✓ The authors should explore the application of the proposed activation functions in various real-world problems, such as climate change, energy, and transportation.
- ✓ Future research should focus on developing more robust and accurate deep learning models, such as the proposed activation functions, to address the limitations of traditional deep learning models.
