Internal APIs Are All You Need: Shadow APIs, Shared Discovery, and the Case Against Browser-First Agent Architectures
As the digital landscape evolves, the interaction between autonomous agents and the web is becoming increasingly prominent. However, a significant challenge persists: most websites are designed for human browsers rather than automated agents. This fundamental mismatch creates hurdles for the emerging concept known as the “Agentic Web,” which aims to facilitate seamless interactions between agents and web services.
In the quest for efficient agent interaction, a common approach involves agents browsing web pages, inspecting Document Object Models (DOMs), and reverse-engineering callable routes. Unfortunately, this process is often slow, brittle, and redundantly repeated across different agents. This inefficiency necessitates a new paradigm that leverages existing web structures.
Shadow APIs: A Hidden Resource
Our research highlights the presence of internal APIs, often referred to as shadow APIs, which are already integrated into the architecture of modern websites. These first-party endpoints power the core functionality of websites, yet they remain largely untapped for agent interactions. By recognizing and utilizing these shadow APIs, agents can significantly streamline their operations.
Introducing Unbrowse
To address the inefficiencies associated with browser-based route discovery, we introduce Unbrowse, a shared route graph that transforms this discovery process into a collectively maintained index of callable first-party interfaces. Unbrowse operates by passively learning routes from real browsing traffic, which allows it to serve cached routes through direct API calls.
Performance Benchmarking
In a live-web benchmark that tested equivalent information-retrieval tasks across 94 different domains, Unbrowse showcased remarkable performance improvements. The results indicated that fully warmed cached execution averaged just 950 milliseconds, in stark contrast to 3,404 milliseconds required for Playwright browser automation. This translates to a 3.6 times mean speedup and a 5.4 times median speedup, with well-cached routes completing in under 100 milliseconds.
A Self-Correcting System
Unbrowse operates with a three-path execution model that leverages local cache, a shared graph, or a browser fallback. This ensures that the system remains voluntary and self-correcting, adapting to the needs of users and the reliability of the cached data.
Micropayment Model
The adoption of Unbrowse is further supported by a three-tier micropayment model, facilitated via the x402 protocol. This model includes:
- Tier 1: A one-time install fee for access to discovery documentation.
- Tier 2: Optional per-execution fees for site owners who choose to participate.
- Tier 3: Per-query search fees for graph lookups.
Each tier is grounded in a rational adoption framework: agents utilize the shared graph only when the total fees incurred are less than the expected costs associated with browser rediscovery.
Conclusion
In conclusion, the evolution of the Agentic Web necessitates a shift towards internal APIs and away from traditional browser-first architectures. By leveraging shadow APIs through systems like Unbrowse, we can facilitate faster, more efficient interactions between autonomous agents and web services, paving the way for a more integrated digital ecosystem.