Browser AI Just Killed My Edge Functions Bill: A Field Report on Gemini Nano in Chrome

A 71% drop in inference spend, with one caveat

Last quarter our edge inference bill at Vercel hit $11,400 a month, mostly from a single feature: a writing-assist tool that runs a Llama 3.1 8B model behind a Cloudflare Workers AI endpoint. In April 2026 we shipped a version that runs entirely in the browser using Chrome's built-in Gemini Nano model. Our bill on that feature fell to $3,300. The catch: it only works for 64% of our traffic, because not every user is on Chrome 134+ with the right hardware.

Even with that caveat, browser-side AI is the most consequential platform shift of the year. Here is the field report.

What Chrome actually shipped

Chrome 134, released March 2026, made the Prompt API generally available after two years of origin trials. The model behind it is Gemini Nano, a roughly 4B-parameter Google model that ships once per Chrome version and runs on-device using WebGPU on capable hardware.

The API is comically small:

const session = await ai.languageModel.create({
  systemPrompt: "You rewrite text to be clearer."
});
const result = await session.prompt("Make this concise: ...");

There is also ai.summarizer, ai.writer, ai.rewriter, and ai.translator, each with a higher-level interface tuned for the task. Edge in 2025 followed with its own variant, and Safari 19 has a "Foundation Model API" in preview. The shape is clearly converging on something the WebML Working Group will standardize.

What it is actually good at

After two months of shipping, on-device Gemini Nano is genuinely production-grade for:

• Short rewrites and tone shifts. Headline polishing, email subject lines, slack message rephrasing. Sub-300ms typical.
• Summarization of pasted text. Up to roughly 4K input tokens, output is coherent.
• Classification and intent detection. Sentiment, topic, language. Surprisingly accurate for the size.
• Form autofill and field validation. "Does this look like a real address?" type checks.

Where it falls over:

• Anything requiring world knowledge after the model snapshot. Nano knows nothing about events post-training, and there is no RAG out of the box.
• Long-form generation. Output coherence drops past 400 tokens.
• Code generation. It is not a coding model. Use a server-side one.
• Reasoning tasks. Math, multi-step logic, anything chain-of-thought-y. Move to Claude Opus 4.7 or Gemini 2.5 Pro.

The numbers

Comparing our writing-assist feature before and after the port:

| Metric | Llama 3.1 8B on Workers AI | Gemini Nano in browser | |---|---|---| | Median latency | 740ms | 290ms | | p95 latency | 2.1s | 680ms | | Monthly inference cost | $11,400 | $3,300 | | Coverage | 100% of users | 64% of users | | Privacy posture | Data leaves device | Data stays on device |

The latency win is the biggest user-facing improvement. With server inference, the spinner is the feature. With Nano, the rewrite appears effectively instantly, which changed how often users invoked it (uses per session went from 1.4 to 3.7).

The hardware reality

Gemini Nano needs roughly 4GB of free VRAM. In practice, it runs on:

• Most M1/M2/M3/M4 Macs
• Modern Windows laptops with discrete GPUs or recent integrated graphics
• Pixel 8/9 phones via Android Chrome
• Most desktop Chromebooks released after 2023

It does not run on:

• Older Intel integrated graphics
• Most low-end Android devices
• Anything in incognito mode by default

Detection is a single call: await ai.languageModel.capabilities(). We use that to feature-flag the on-device path and fall back to our server endpoint otherwise. That hybrid is the real pattern.

The pitfalls nobody warned us about

Three things cost us a week each:

1. The model downloads on first use. It is a roughly 1.8GB download. Users on metered connections see a one-time delay. We added a "warming" call on app load to make it happen quietly.
2. Output non-determinism is higher than expected. Same prompt, same seed, slightly different output across Chrome versions. Snapshot tests against Nano outputs are useless. Test the wrapper logic, not the model.
3. Quotas exist and are silent. Sustained high-volume use throttles. Documented limit is around 6 requests/second per origin. Build a queue.
4. Streaming and cancellation are not first-class. The Prompt API has a streaming variant, but cancellation mid-stream is awkward and leaks resources if you do it wrong. Always pair with an AbortController and explicitly call session.destroy() when the user navigates away. We had a memory leak in our first beta because we relied on garbage collection and got bitten on long-lived dashboard sessions.
5. The system prompt has a hard ceiling. Roughly 1024 tokens, smaller than I expected. Long elaborate persona prompts that work on Claude or GPT-5 will be silently truncated by Nano. Keep it terse.

Privacy and compliance: the part legal cared about

Our legal team's first reaction to "we are running AI in the browser" was suspicion, then delight, then more suspicion. The delight was real: for our European users, on-device inference removes a category of GDPR concern. No data leaves the device for those calls, full stop. We updated our privacy policy in two paragraphs instead of two pages.

The lingering suspicion was about model updates. Gemini Nano updates with Chrome, and the Chrome update channel is not under our control. A subtle change in model behavior could silently change product output. Our answer was monitoring: we sample 1% of on-device responses with user consent, ship them to a server-side eval queue, and compare against historical baselines. When Chrome 135 rolled out in early May 2026 with a refreshed Nano, we caught a measurable shift in summary length within 36 hours. Not a regression, but exactly the kind of thing you want to know before customer support tells you about it.

When to use on-device vs server-side

We landed on a clear decision matrix:

1. Latency-sensitive UX touches (rewrite, summarize on paste, validate on blur): on-device.
2. Anything privacy-sensitive (medical notes, internal docs): on-device.
3. Anything requiring fresh knowledge, RAG, or tool use: server-side.
4. Anything over 400 tokens of output or needing a frontier-grade model: server-side.

The hybrid means our app is faster and cheaper for the majority of users on the majority of features, and still works for the rest.

What about WebGPU and bring-your-own-model?

A separate strand of browser AI is running your own model via WebGPU and ONNX or transformers.js. This is more flexible — you pick the model — but considerably harder. You ship a multi-hundred-megabyte model yourself, pay the bandwidth, and own the compatibility matrix. We tried this path for a specialized classifier and switched to Nano's built-in classifier API for that workload. For most teams, the built-in Prompt API is the pragmatic choice today; bring-your-own-model is for cases where you genuinely need a fine-tuned or specialized model and have the engineering budget to maintain it.

For the curious, the toolchain is real and shipping: transformers.js v3 supports WebGPU acceleration, and small specialized models like Phi-3-mini and Gemma 2B run acceptably in the browser on capable machines. The ceiling keeps rising. But the integration cost is meaningfully higher than calling ai.languageModel.prompt().

A useful rule of thumb we landed on: if a built-in API exists for your task — summarizer, rewriter, translator, classifier — use it. The built-in APIs are tuned for the specific task and faster than asking the general Prompt API to do the same job. Only fall back to the raw Prompt API for tasks that have no built-in equivalent. We initially used Prompt API for everything and rewrote about a third of our calls to the specialized APIs after measuring quality. The translator API in particular outperformed the same translation request through Prompt API by a comfortable margin, and ran roughly 30% faster.

What this means for you

Browser AI is not a curiosity. It is a real architectural option in mid-2026, and ignoring it leaves money and latency on the table. The starting move is small:

• Audit your inference workload for short-form, low-stakes tasks.
• Pick one and ship it behind a capability check.
• Measure cost, latency, and engagement before and after.
• Decide whether the hybrid pattern earns its place permanently.

The standardization story is still messy. The Prompt API is a Google initiative until the WebML group finalizes a cross-vendor spec, which Apple's Safari 19 work suggests is coming. Until then, build with feature detection and a server fallback, and you get the wins now without painting yourself into a corner.

The era of "every AI request hits a GPU in Virginia" is ending. Quietly, in a Chrome update most users never noticed, the browser became an inference platform.

A final practical note: when you ship features that depend on on-device AI, communicate it to users. Not in a creepy way — just a small badge that says "Runs on your device" near the feature. We A/B tested this label and engagement went up 9% on the on-device version compared to an unlabeled control. Users in 2026 are increasingly aware of where their data goes, and a credible privacy claim is now a feature in itself. The competitive moat for browser AI is not just speed and cost. It is trust. That changes which products win in the next twenty-four months in ways that are still being figured out, but the early signal is unambiguous: if you can do it on-device, you should, and you should tell people you did.

FAQ