A proxy can work and still make every repeated prompt expensive

I won a Google DeepMind hackathon and received $10,000 in GCP credits. Since Google Cloud doesn't support direct Anthropic-style API keys, I had to build a local proxy to route my Hermes Agent. That is where a silent cost leak began.

Context & Setup: A $10,000 Vault from Google

In early 2026, I was lucky enough to win a hackathon hosted by Google DeepMind, receiving $10,000 in Google Cloud Platform (GCP) credits as my prize.

With this substantial budget, my goal was clear: configure Hermes Agent—my continuous AI assistant—to call the most advanced models available, like Claude Opus 4.8 and Claude Fable 5, to handle complex daily workloads.

However, Google Cloud doesn’t allow you to simply generate a direct API key to plug into an agent like other standard providers. Secure access is strictly routed through Google’s service accounts and OAuth credential workflows.

To bridge this, I wrote a lightweight local proxy running in the background on port :8765. Its job was simple: whenever Hermes Agent sent a standard API request, the proxy attached the necessary Google credentials, forwarded it to Vertex AI to ask Claude, and streamed the response back to me.

The system was functionally flawless. I chatted smoothly, tool use worked perfectly, and not a single error popped up in our logs for weeks.

The Struggle: A Silent 3.47 Million VND Daily Bill

The illusion shattered when I logged into my Google Cloud Billing console to check my balance. During a heavy testing day, GCP reported a daily run rate of ₫3,467,325 (approximately 3.47 Million VND):

• Claude Opus 4.8: ₫2,360,899
• Claude Sonnet 5: ₫1,096,174
• Claude Fable 5: ₫10,252

I checked the agent’s debug output. Every call had returned with an HTTP 200 OK, with zero retries or failed connections. Yet, every subsequent question in the same session cost just as much as the very first prompt.

For regular agent users, each continuous session carries a massive amount of stable, identical context: system guidelines, tool definitions, active skills, and earlier chat history. This background text easily exceeds 30,000 words.

Modern SOTA models support Prompt Caching. When enabled, after the first turn, the provider saves that massive block of stable context on their server, offering up to a 90% discount on subsequent turns. Instead of paying to reprocess 30,000 words on every small question, you only pay a fraction for the new query.

My proxy was functionally healthy, but it was completely missing this cost-saving feature.

What Didn’t Work: The OpenAI-to-Anthropic Blindspot

When writing the proxy’s translation code, I strictly mapped standard API parameters: model names, messages, system prompts, temperature, and streaming.

Because prompt caching is an Anthropic-specific API optimization, it doesn’t have an equivalent field in standard OpenAI chat schemas. Claude requires an explicit cache_control breakpoint tag embedded inside the outbound payload to trigger caching.

Because my local proxy translator omitted this Anthropic-specific tag when forwarding data to GCP, Google’s servers processed the payload as an entirely new stream. Vertex AI re-compiled the whole 30,000-word context on every single turn, billing us at 100% price.

This is a dangerous leak because it never triggers a technical warning. Your output remains clean and fast; only your budget is bleeding in the background.

The Shift: Inserting the Cost-Saving Valve

I added translation logic directly to the local proxy on my Mac Mini. Instead of altering Hermes or the caller payload, the proxy now injects the ephemeral breakpoint directly before calling Vertex AI:

# Enable Anthropic/Vertex ephemeral prompt caching.
# The provider places a breakpoint on the last cacheable block.
if os.environ.get("GCP_CLAUDE_DISABLE_CACHE", "") != "1":
    akw["cache_control"] = {"type": "ephemeral"}

I also exposed these token metrics in our terminal output to make sure cache hits are always visible:

# Log raw caching metrics on each response
print(f"Tokens read from cache: {r.usage.cache_read_input_tokens}")
print(f"Tokens written to cache: {r.usage.cache_creation_input_tokens}")

A subsequent live test verified that the change resolved the issue instantly:

Turn 1: Cache Created (write = 32,260 tokens, read = 0)
Turn 2: Cache Read Successful (write = 0, read = 32,260 tokens)
Turn 3: Cache Read Successful (write = 0, read = 32,260 tokens)

Instead of paying to pre-process 32,260 tokens over and over, the system read them directly from the cache at a 90% discount.

The Measured Outcomes

In a controlled 5-turn Sonnet 5 test running a stable 32,260-token context, this microscopic logic change delivered immense results:

Google Cloud Monitoring later recorded 17,622,536 cache-read input tokens in live traffic, successfully shielding my $10,000 DeepMind credits from rapid, silent exhaustion.

Let’s Figure Your Workflow Out

If you run your own local proxies, custom translators, or host agents on private servers, the lesson is clear: A successful functional response (receiving an answer) does not guarantee an economically optimal connection.

If you suspect your token usage is piling up unexpectedly, or want to verify if your custom API connectors are running with cost optimizations enabled, let’s connect:

• Discord: annguyen175 (Or send a direct message via this link)
• Email: [email protected]
• Facebook Discussion Group: vnhermesagent

Send over your proxy architecture and request log schemas. We’ll trace your data pathways together to seal any token leaks and optimize your connection.