DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation
Summary: arXiv:2604.09089v1 Announce Type: cross
Abstract
Large Language Models (LLMs) for code generation can replicate insecure patterns from their training data. To mitigate this, a common strategy for security hardening is to fine-tune models using supervision derived from the final transformer layer. However, this design may suffer from a final-layer bottleneck: vulnerability-discriminative cues can be distributed across layers and become less detectable near the output representations optimized for next-token prediction.
Introduction
In recent years, the rise of Large Language Models has transformed various fields, including software development. However, the ability of these models to generate code has raised significant security concerns. A critical issue is that these models can inadvertently replicate insecure coding patterns learned from their training datasets. As a response, researchers have sought methods to enhance the security of code generated by LLMs.
Identifying the Bottleneck
One prevalent strategy is to fine-tune the models based on supervision from the final transformer layer. This approach, while common, has been identified as potentially problematic due to a phenomenon known as the final-layer bottleneck. Vulnerability-related signals are often found distributed across the model’s layers. As the signal moves closer to the output layer, the ability to detect these security cues diminishes, leading to less secure code generation.
Introducing DeepGuard
To address this issue, researchers have developed DeepGuard, a novel framework designed to leverage the distributed security-relevant cues present in multiple upper layers. DeepGuard employs an attention-based module to aggregate these representations, enhancing the model’s ability to detect vulnerabilities throughout its architecture.
Key Features of DeepGuard
- Multi-Layer Representation Aggregation: DeepGuard collects and combines information from various upper layers to capture a holistic view of security cues.
- Security Analyzer: The aggregated signal is utilized by a dedicated security analyzer, ensuring that the generated code adheres to security standards.
- Multi-Objective Training: The framework balances security enhancement with functional correctness, enabling the generation of code that is both secure and operationally sound.
- Lightweight Inference-Time Strategy: DeepGuard supports a streamlined inference process that allows for efficient deployment in real-world applications.
Results and Performance
Extensive experiments conducted across five different code LLMs demonstrate that DeepGuard significantly improves the secure-and-correct generation rate. On average, DeepGuard enhances this rate by 11.9% when compared to strong baseline models such as SVEN. Furthermore, the framework preserves functional correctness while demonstrating the ability to generalize across previously unseen vulnerability types.
Conclusion
DeepGuard represents a significant advancement in the field of secure code generation. By overcoming the limitations of final-layer bottlenecks and effectively aggregating multi-layer signals, it sets a new standard for security in LLM-generated code. Researchers and practitioners interested in exploring DeepGuard can access the code publicly available at https://github.com/unknownhl/DeepGuard.