EffiPair: Improving the Efficiency of LLM-generated Code with Relative Contrastive Feedback
Summary: arXiv:2604.05137v1 Announce Type: cross
Abstract
Large language models (LLMs) often generate code that is functionally correct but inefficient in terms of runtime and memory usage. Traditional methods for enhancing code efficiency generally depend on absolute execution feedback, such as profiling a single program’s runtime or memory consumption. However, these methods can be costly and offer limited guidance for code refinement. In this article, we introduce Relative Contrastive Feedback (RCF), a novel inference-time feedback mechanism that does not require any model fine-tuning or parameter updates.
Introducing Relative Contrastive Feedback
RCF works by comparing two structurally similar programs that are designed to perform the same task, highlighting the differences that contribute to better efficiency. This approach allows for a more nuanced understanding of performance discrepancies between candidate solutions, aiming to provide targeted feedback that drives improvements without the need for extensive profiling.
EffiPair Framework
Building on the principles of RCF, we present EffiPair, an innovative inference-time iterative refinement framework. EffiPair operates entirely at test time by following a systematic process outlined below:
- Generating Multiple Candidate Solutions: EffiPair creates various potential solutions for the given task.
- Identifying Informative Program Pairs: The framework selects pairs of programs that exhibit significant efficiency gaps, allowing for focused comparisons.
- Summarizing Execution Differences: By condensing the execution differences into lightweight feedback, EffiPair provides actionable insights to refine the generated code.
- Producing More Efficient Solutions: Utilizing the feedback from pairwise comparisons, EffiPair iteratively enhances the efficiency of the solutions.
Advantages of EffiPair
One of the key innovations of EffiPair is its ability to replace isolated scalar feedback with pairwise contrastive comparisons. This shift not only offers more direct guidance for code improvement but also significantly reduces the overhead associated with profiling and prompting. The outcome is a more efficient and effective refinement process.
Experimental Results
Our experiments on code-efficiency benchmarks demonstrate that EffiPair consistently enhances efficiency while maintaining correctness. For instance, when utilizing DeepSeek-Chat V3.2, EffiPair achieves an impressive speedup of up to 1.5 times compared to code generation without performance feedback. Moreover, it reduces token usage by over 90% in comparison to previous methodologies.
Conclusion
EffiPair represents a significant advancement in the field of code generation by leveraging Relative Contrastive Feedback. By focusing on comparative execution analysis rather than absolute profiling, it provides a more efficient pathway for refining LLM-generated code, enhancing both runtime performance and memory efficiency. The implications of this research are poised to influence future developments in AI-driven code generation.