PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks
arXiv:2603.13347v1 Announce Type: new Abstract: Biological neural systems employ diverse neurotransmitters -- glutamate, GABA, dopamine, acetylcholine -- to implement distinct signal-processing modalities within shared neural circuits. In contrast, modern transformers apply a single fixed activation function across all feed-forward neurons. We introduce PolyGLU (Polychromatic Gated Linear Unit), a drop-in replacement for SwiGLU that enables each FFN neuron to dynamically route among K=4 activation functions via a differentiable mechanism combining learned static preferences with input-conditioned gating, trained end-to-end with Gumbel-Softmax. We train PolychromaticLM, a 597M-parameter transformer, on ~10B tokens using a single NVIDIA A100 GPU. Our key finding is emergent routing behavior: without any explicit sparsity loss or entropy regularization, the routing mechanism converges to near-deterministic activation selections (mean dynamic entropy = 0.030% of maximum), with a striking
arXiv:2603.13347v1 Announce Type: new Abstract: Biological neural systems employ diverse neurotransmitters -- glutamate, GABA, dopamine, acetylcholine -- to implement distinct signal-processing modalities within shared neural circuits. In contrast, modern transformers apply a single fixed activation function across all feed-forward neurons. We introduce PolyGLU (Polychromatic Gated Linear Unit), a drop-in replacement for SwiGLU that enables each FFN neuron to dynamically route among K=4 activation functions via a differentiable mechanism combining learned static preferences with input-conditioned gating, trained end-to-end with Gumbel-Softmax. We train PolychromaticLM, a 597M-parameter transformer, on ~10B tokens using a single NVIDIA A100 GPU. Our key finding is emergent routing behavior: without any explicit sparsity loss or entropy regularization, the routing mechanism converges to near-deterministic activation selections (mean dynamic entropy = 0.030% of maximum), with a striking depth-dependent specialization pattern -- early layers prefer GELU while deep layers strongly favor Tanh. Three layers maintain elevated routing entropy, suggesting computational flexibility points. The routing architecture adds only 0.23% parameter overhead (~1.4M parameters) and proves fully robust to supervised fine-tuning: routing entropy remains constant at ln(4) throughout 13,067 SFT steps. On standard benchmarks, PolychromaticLM achieves 62-89% of Qwen3-0.6B-Base performance despite training on 3,600x fewer tokens. All code, weights, and training infrastructure are released under Apache 2.0.
Executive Summary
The article introduces PolyGLU, a state-conditional activation routing mechanism for transformer feed-forward networks. This innovative approach enables each neuron to dynamically route among multiple activation functions, inspired by the diverse neurotransmitters employed in biological neural systems. Key findings include emergent routing behavior, depth-dependent specialization patterns, and computational flexibility points. PolyGLU is a drop-in replacement for SwiGLU, adding minimal parameter overhead and proving robust to supervised fine-tuning. The authors demonstrate state-of-the-art performance on standard benchmarks, despite training on significantly fewer tokens. This breakthrough has significant implications for the development of more efficient and adaptive transformer architectures, offering a promising direction for future research in natural language processing and deep learning.
Key Points
- ▸ Introduction of PolyGLU, a state-conditional activation routing mechanism for transformers
- ▸ Emergent routing behavior and depth-dependent specialization patterns in PolyGLU
- ▸ Computational flexibility points and robustness to supervised fine-tuning
Merits
Advancements in Transformer Architecture
PolyGLU's ability to dynamically route among multiple activation functions enables more efficient and adaptive transformer architectures, pushing the boundaries of current transformer designs.
Scalability and Robustness
The proposed mechanism is robust to supervised fine-tuning and demonstrates state-of-the-art performance on standard benchmarks, even with significantly fewer training tokens.
Demerits
Computational Overhead
While the added parameter overhead is minimal, the computational requirements for training and inference with PolyGLU are not explicitly addressed, which may impact its practical applicability.
Limited Exploration of Activation Functions
The study focuses on a limited set of activation functions, and further investigation into the effects of different activation functions and their interactions is warranted.
Expert Commentary
The introduction of PolyGLU represents a significant departure from traditional transformer architectures, offering a more adaptive and efficient approach to signal processing. The emergent routing behavior and depth-dependent specialization patterns observed in this study highlight the complex interactions between activation functions and neural architectures. While the computational overhead is minimal, further investigation into the scalability and robustness of PolyGLU is warranted. The study's findings have significant implications for the development of more efficient and adaptive transformer architectures, and its connections to sparsity, entropy regularization, and neural architecture search make it a promising direction for future research.
Recommendations
- ✓ Further investigation into the effects of different activation functions and their interactions on the emergent routing behavior and depth-dependent specialization patterns.
- ✓ Exploration of the computational requirements for training and inference with PolyGLU, and the development of more efficient implementations.
