Structured Multidimensional Representation Learning for Large Language Models
arXiv:2603.05727v1 Announce Type: new Abstract: Transformer architectures achieve state-of-the-art performance across a wide range of pattern recognition and natural language processing tasks, but their scaling is accompanied by substantial parameter growth and redundancy in the embedding dimension. In this work, we introduce a structured spectral factorization of the embedding space based on the L-product for third-order tensors. By reshaping token representations into spectral tensor slices and performing attention and feed-forward operations in the transform domain, we obtain a Tensor Transformer architecture that decomposes the encoder into p independent spectral sub-transformers while preserving standard Transformer semantics. We prove that the proposed L-Transformer is spectrally equivalent to p parallel Transformers operating on reduceddimensional embeddings, which yields approximately 1/p reduction (up to lower-order terms such as biases and normalization parameters) in encode
arXiv:2603.05727v1 Announce Type: new Abstract: Transformer architectures achieve state-of-the-art performance across a wide range of pattern recognition and natural language processing tasks, but their scaling is accompanied by substantial parameter growth and redundancy in the embedding dimension. In this work, we introduce a structured spectral factorization of the embedding space based on the L-product for third-order tensors. By reshaping token representations into spectral tensor slices and performing attention and feed-forward operations in the transform domain, we obtain a Tensor Transformer architecture that decomposes the encoder into p independent spectral sub-transformers while preserving standard Transformer semantics. We prove that the proposed L-Transformer is spectrally equivalent to p parallel Transformers operating on reduceddimensional embeddings, which yields approximately 1/p reduction (up to lower-order terms such as biases and normalization parameters) in encoder parameters under fixed total embedding size. When instantiated with a real-valued Discrete Cosine Transform (DCT), the method remains fully differentiable and compatible with existing training pipelines. Beyond compression, the spectral decomposition introduces an inductive bias over embedding frequencies, enabling slice-dependent frequency scaling that improves generalization. Experiments on IMDB and AG~News show that the proposed model can substantially reduce encoder parameters (up to 75\% for p=4) while maintaining competitive accuracy. On IMDB, the tensorized encoder matches or improves upon the standard baseline under compression, whereas on AG~News at moderate width we observe a small accuracy decrease in exchange for a 4 times encoder reduction; at BERT-base width (d=768), performance returns to parity.
Executive Summary
This article introduces a novel spectral factorization approach to mitigate parameter redundancy in large language models by restructuring Transformer embeddings via L-product tensor decomposition. By transforming token representations into spectral tensor slices and conducting attention and feed-forward operations in the transform domain, the authors present a Tensor Transformer architecture that decomposes the encoder into independent spectral sub-transformers while retaining semantic equivalence. They demonstrate spectral equivalence to parallel Transformers on reduced embeddings, achieving approximately 1/p parameter reduction without compromising accuracy. Experiments on IMDB and AG~News validate efficacy: up to 75% parameter reduction with maintained or improved performance, particularly at BERT-base width. The method leverages a real-valued DCT for differentiability and compatibility with existing pipelines, offering both computational efficiency and inductive bias improvement.
Key Points
- ▸ Spectral tensor decomposition reduces encoder parameters via L-product factorization
- ▸ Preservation of Transformer semantics via spectral equivalence
- ▸ Experimental validation shows parameter reduction without significant loss in accuracy
Merits
Parameter Efficiency
Significant reduction in encoder parameters (up to 75%) without compromising model performance, offering scalable solutions for large-scale LLMs.
Demerits
Accuracy Trade-off
At moderate widths (e.g., AG~News at moderate compression), a small accuracy decrease is observed, indicating potential limitations in certain architectural configurations.
Expert Commentary
The Tensor Transformer represents a substantive advancement in transformer architecture by introducing a mathematically rigorous, spectrally equivalent decomposition that aligns with both computational efficiency and functional equivalence. The use of spectral tensor slices introduces a novel inductive bias that may have broader implications beyond parameter reduction—potentially influencing signal processing and frequency-aware learning paradigms. While the experiments validate efficacy on benchmark datasets, the authors’ careful control for biases via DCT differentiability and lower-order term consideration enhances credibility. Notably, the scalability of this approach is particularly compelling: the reduction factor is linear in p, suggesting potential extensions to higher-order tensor decompositions or hybrid architectures. One potential extension is the integration with quantization-aware training or mixed-precision architectures to amplify the efficiency gains. Overall, this work bridges theoretical elegance with empirical validation, positioning itself as a foundational contribution to the field of efficient transformer design.
Recommendations
- ✓ Integrate Tensor Transformer variants into mainstream LLM training pipelines as default compression options
- ✓ Explore synergies with mixed-precision quantization or hybrid frequency-domain training to amplify efficiency gains
