LLM-Rosetta: A Hub-and-Spoke Intermediate Representation for Cross-Provider LLM API Translation
Summary: arXiv:2604.09360v1 Announce Type: cross
The landscape of Large Language Model (LLM) providers has expanded rapidly, leading to a fragmented ecosystem where applications are often tightly integrated with individual vendor APIs. This situation poses challenges for developers who wish to switch providers or create applications that operate across multiple LLM environments. The need for numerous bilateral adapters, which increase in complexity as the number of providers grows, has resulted in significant limitations on portability and the development of multi-provider architectures.
Despite the substantial differences in API syntax among leading LLM providers, it has been observed that they share a fundamental semantic core. The real difficulty lies in the vast array of syntactic variations rather than deep incompatibilities in semantics. Recognizing this, a novel solution has been proposed: LLM-Rosetta, an open-source translation framework designed to bridge the gaps between different LLM API formats.
Introducing LLM-Rosetta
LLM-Rosetta utilizes a hub-and-spoke Intermediate Representation (IR) that effectively captures the shared semantic elements among various LLM APIs. This framework is built upon a comprehensive content model comprising nine types and a stream event schema consisting of ten types. Key components identified within this structure include:
- Messages
- Content Parts
- Tool Calls
- Reasoning Traces
- Generation Controls
The architecture of LLM-Rosetta is modular, allowing for the addition of each API standard independently. This modularity is crucial for maintaining flexibility and ensuring that the framework can adapt to the evolving landscape of LLM providers.
Key Features and Benefits
LLM-Rosetta supports bidirectional conversion between various provider APIs and the Intermediate Representation for both request and response payloads. Notable features include:
- Chunk-level streaming capabilities
- Stateful context management
- Lossless round-trip fidelity
- Efficient conversion overhead of less than 100 microseconds
These features position LLM-Rosetta as a competitive alternative to existing solutions like LiteLLM, which, while effective, may lack the bidirectionality and provider neutrality that LLM-Rosetta offers.
Implementation and Deployment
Currently, LLM-Rosetta includes converters for four major API standards: OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Google GenAI. This coverage addresses a significant portion of the commercial LLM landscape, enabling developers to utilize a wider array of tools without the burdens of extensive reconfiguration.
Empirical evaluations of LLM-Rosetta have confirmed its effectiveness, demonstrating accurate streaming behavior and compliance with the Open Responses compliance suite. Additionally, it has been successfully deployed in production environments, such as at Argonne National Laboratory.
Accessing LLM-Rosetta
Developers interested in leveraging this innovative framework can access the code and additional resources at https://github.com/Oaklight/llm-rosetta. This open-source initiative is set to enhance the interoperability of LLM applications and reduce the friction associated with transitioning between different providers.