Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression
arXiv:2603.23527v1 Announce Type: new Abstract: Prompt compression is often evaluated by input-token reduction, but its real deployment impact depends on how compression changes output length and total inference cost. We present a controlled replication and extension study of benchmark-dependent output dynamics under aggressive compression, covering 5,400 API calls across three benchmarks and multiple providers. To explain conflicting prior observations, we formalize instruction survival probability (Psi), a structural metric that captures whether task-critical prompt segments remain after truncation. Results show a strong benchmark effect: under r=0.3, DeepSeek exhibits severe output expansion on MBPP (56x, Psi approx 0.15) but substantially lower expansion on HumanEval (5x, Psi approx 0.72), while GPT-4o-mini is comparatively stable across benchmarks. This reconciles the apparent discrepancy between previously reported extreme explosion and lower replication effects by identifying p
arXiv:2603.23527v1 Announce Type: new Abstract: Prompt compression is often evaluated by input-token reduction, but its real deployment impact depends on how compression changes output length and total inference cost. We present a controlled replication and extension study of benchmark-dependent output dynamics under aggressive compression, covering 5,400 API calls across three benchmarks and multiple providers. To explain conflicting prior observations, we formalize instruction survival probability (Psi), a structural metric that captures whether task-critical prompt segments remain after truncation. Results show a strong benchmark effect: under r=0.3, DeepSeek exhibits severe output expansion on MBPP (56x, Psi approx 0.15) but substantially lower expansion on HumanEval (5x, Psi approx 0.72), while GPT-4o-mini is comparatively stable across benchmarks. This reconciles the apparent discrepancy between previously reported extreme explosion and lower replication effects by identifying prompt structure, not provider identity alone, as the primary moderator. We introduce the Compression Robustness Index (CRI) for cross-benchmark evaluation and show that single-benchmark assessments can produce misleading conclusions about compression safety and efficiency. To contextualize energy claims, we incorporate companion direct NVML measurements from rented RunPod GPUs and show that token savings can overstate joule savings. These findings motivate benchmark-diverse testing and structure-aware compression policies for reliable, energy-conscious LLM deployment.
Executive Summary
This study examines the effects of aggressive prompt compression on large language models' (LLMs) output dynamics, revealing a strong benchmark effect where the same compression rate yields vastly different results across different benchmarks and models. The authors introduce a new metric, Compression Robustness Index (CRI), to assess cross-benchmark evaluation and highlight the limitations of single-benchmark assessments. The study also explores the energy efficiency of LLM deployment, suggesting that token savings can overstate joule savings. The findings emphasize the importance of benchmark-diverse testing and structure-aware compression policies for reliable and energy-conscious LLM deployment.
Key Points
- ▸ Aggressive prompt compression results in benchmark-dependent output dynamics.
- ▸ The introduction of Compression Robustness Index (CRI) enables cross-benchmark evaluation.
- ▸ Single-benchmark assessments can produce misleading conclusions about compression safety and efficiency.
Merits
Strength in Methodology
The study employs a controlled replication and extension approach, covering 5,400 API calls across three benchmarks and multiple providers, providing robust evidence to support the findings.
Insightful Findings
The study reveals the importance of benchmark structure in determining the effects of compression, rather than provider identity alone, which improves our understanding of LLM deployment.
Demerits
Limited Generalizability
The study focuses on a specific set of benchmarks and models, which may not be representative of all LLMs, limiting the generalizability of the findings.
Energy Efficiency Measurement
The study's method for measuring energy efficiency may not be comprehensive, relying on direct NVML measurements from rented RunPod GPUs, which might not capture the full scope of energy consumption.
Expert Commentary
This study makes a significant contribution to the field of natural language processing by highlighting the importance of benchmark structure in determining the effects of compression on LLMs' output dynamics. The introduction of CRI is a valuable addition to the field, enabling cross-benchmark evaluation and assessment of compression safety and efficiency. However, the study's limited generalizability and the potential for energy efficiency measurement limitations warrant further investigation. Nonetheless, the findings have significant practical and policy implications for the reliable and energy-conscious deployment of LLMs.
Recommendations
- ✓ Future studies should investigate the effects of compression on LLMs' output dynamics across a broader range of benchmarks and models.
- ✓ Developers and deployers should prioritize benchmark-diverse testing and structure-aware compression policies to ensure reliable and energy-efficient LLM deployment.
Sources
Original: arXiv - cs.CL
