Running LLMs in the Browser: How We Use WebGPU and WebLLM for Privacy-First Resume Editing

Every AI-powered resume tool on the market today sends your data to a server. They’ll tell you it’s “secure.” They’ll point to SOC 2 compliance badges. But here’s the engineering reality: if your resume leaves your device, it is no longer exclusively yours. It lives in a database, in a log file, in a training pipeline, or in a data breach waiting to happen.

We refused to accept that trade-off. So we built something different: a fully client-side AI resume rewriter that runs a 500-million-parameter language model directly in your browser using WebGPU and Transformers.js. No API keys. No cloud inference. No server.

In this technical deep dive, you’ll learn:

Why serverless AI is the only truly private architecture for sensitive documents
How WebGPU unlocks GPU-accelerated inference in the browser
The exact stack we used (Astro, Transformers.js, Xenova/Qwen1.5-0.5B-Chat)
The engineering challenges of shipping a 300MB model to a browser tab
Performance benchmarks and real-world latency numbers

The Privacy Paradox of Modern AI Tools

AI resume rewriters have exploded in popularity. Tools like Teal, Resume.io, and Jobscan offer “AI-powered bullet rewriting” that sounds magical — until you read the fine print. Nearly all of them operate on a client-server architecture:

You paste your resume into a web form
Your data is encrypted in transit (TLS)
It’s decrypted on a cloud server
An LLM API (OpenAI, Anthropic, or a self-hosted model) processes it
The rewritten text is sent back

This looks secure on the surface. But the attack surface is massive:

Server logs may retain your full resume for debugging
Third-party LLM APIs can train on your data unless you pay for enterprise tiers
Data breaches expose resumes containing home addresses, phone numbers, employment history, and salary data
Compliance gaps — most tools are not GDPR-compliant for EU users because they can’t guarantee deletion

The bottom line: If your resume touches a server, you are trusting a third party to not read, log, leak, or monetize your most sensitive professional document. That’s not a bet we’re willing to ask our users to make.

Why Client-Side AI Changes Everything

Client-side AI flips the architecture entirely. Instead of sending your data to the model, we send the model to your data. The LLM downloads directly into your browser’s memory, runs on your local GPU or CPU, and never transmits your resume text off your device.

The privacy implications are absolute:

Zero network calls containing your resume data
Zero server logs with your personal information
Zero third-party LLM APIs with opaque data policies
Zero risk of data breach from our infrastructure — because we don’t have infrastructure that touches your data

But client-side AI isn’t just a privacy play. It also delivers tangible performance benefits:

No latency from round-trips to a distant data center
No rate limits or API quotas
No subscription gates for AI features
Offline capability once the model is cached

WebGPU: The Browser as a Compute Engine

Until recently, running LLMs in the browser was computationally infeasible. WebGL was the only GPU API available, and it was designed for graphics — not general-purpose compute. Then WebGPU arrived.

What Is WebGPU?

WebGPU is a next-generation web standard that exposes modern GPU capabilities (Vulkan, Metal, DirectX 12) directly to JavaScript. Unlike WebGL, it supports general-purpose GPU compute shaders, making it ideal for matrix-heavy operations like neural network inference.

Key capabilities that make browser LLMs possible:

Compute pipelines — Run massively parallel matrix multiplication kernels
Unified memory model — Direct GPU buffer access without expensive CPU-GPU copies
Shader languages — Write WGSL (WebGPU Shading Language) for custom inference kernels
Cross-platform — Works on Chrome, Edge, Firefox, and Safari with native performance

WebGPU vs. WebGL for LLM Inference

Feature	WebGL	WebGPU
Compute Shaders	❌ No	✅ Yes
Matrix Operations	Slow (simulated)	Native, parallel
Memory Efficiency	High CPU-GPU overhead	Unified, low overhead
LLM Inference Speed	~10-20 tokens/sec	~50-100 tokens/sec
Browser Support	Universal	Modern browsers (94%+)

For our use case — rewriting resume bullets with a 500M-parameter model — WebGPU delivers 4-5x faster inference than a CPU-only fallback and 2-3x faster than a WebGL-based approach.

Our Stack: Astro + Transformers.js + Xenova/Qwen1.5

Building a client-side AI rewriter required careful technology choices. We needed a framework that was fast, minimal, and compatible with heavy browser-side computation. Here’s the exact architecture we landed on.

Framework: Astro

We chose Astro for its zero-JS-by-default philosophy. Most resume tools are bloated React apps that ship 500KB+ of JavaScript before the user can even paste their resume. Astro lets us deliver static HTML for the landing page and blog, then hydrate only the interactive components (the editor, the AI panel) where needed.

Why Astro matters for client-side AI:

The initial page load is under 50KB of JavaScript
The heavy AI model loads on-demand when the user clicks “Rewrite with AI”
No hydration overhead for static content like our blog or marketing pages

AI Runtime: Transformers.js

Hugging Face’s Transformers.js is the bridge that makes this possible. It’s a port of the Python Transformers library to ONNX Runtime Web, allowing pre-trained models to run in the browser via WebGPU or WebGL fallbacks.

Key technical details:

Models are quantized to INT8 to reduce size and memory footprint
We use the ONNX Runtime Web backend with a WebGPU execution provider
The pipeline handles tokenization, inference, and decoding end-to-end in the browser

Model Choice: Xenova/Qwen1.5-0.5B-Chat

We evaluated multiple models before settling on Qwen1.5-0.5B-Chat via the Xenova ONNX ports:

Selection criteria:

Small enough to download (~300MB quantized) without destroying user experience
Strong instruction-following for “rewrite this bullet to be more impactful” prompts
Permissive license for commercial use
Fast inference on consumer GPUs (sub-3-second generation for short bullet rewrites)

We tested Llama-2-7B, Mistral-7B, and Phi-2. All were too large (1.5GB+) or too slow for real-time browser inference. Qwen1.5-0.5B strikes the sweet spot: small enough to be practical, capable enough to be useful.

The Engineering Challenges Nobody Talks About

Shipping a 300MB model to a browser tab sounds simple in theory. In practice, it was a months-long engineering effort. Here are the three hardest problems we solved.

Challenge 1: Progressive Model Downloads

A 300MB download on a 4G connection would take 2-3 minutes and likely trigger a browser timeout. We implemented a progressive chunked download system:

The model is split into 10MB shards
Each shard downloads with retry logic and exponential backoff
A custom progress bar UI floats in the top-right corner showing real-time MB downloaded
The model is cached in IndexedDB after the first download, so subsequent visits load instantly

Result: First-time users see a ~45-second download on average broadband. Return users get sub-2-second model loading from cache.

Challenge 2: WebGPU Context Lifecycle Management

WebGPU device contexts are finicky. If a user switches tabs, minimizes the browser, or puts their laptop to sleep, the GPU context can be lost. We had to build a robust context recovery system:

Detect gpu.lost events and gracefully re-initialize the inference pipeline
Fall back to CPU (WASM) inference if WebGPU becomes unavailable
Provide clear UI messaging: “Switching to CPU mode — rewriting may take slightly longer.”

Result: The AI rewriter works reliably across Chrome, Edge, and Firefox, even on systems with aggressive power management.

Challenge 3: Memory Pressure on 8GB RAM Machines

A 500M-parameter model in INT8 still consumes ~600MB of GPU/CPU memory. Combined with PDF parsing, the Quill rich text editor, and our Kanban board, we were pushing the limits of budget laptops.

Our optimizations:

Lazy loading — The model only downloads when the user explicitly clicks “Rewrite”
Model eviction — If the user hasn’t used the AI panel in 10 minutes, we release the model from memory (it stays in IndexedDB for fast re-loading)
Off-main-thread inference — All LLM operations run in a Web Worker to prevent UI freezing

Result: The scanner runs smoothly on 8GB RAM machines, including mid-range Chromebooks.

Performance Benchmarks: Real Numbers

We benchmarked our client-side AI rewriter against cloud-based alternatives under identical conditions. Here’s what we found:

Metric	Our Client-Side AI	Cloud API (GPT-4o-mini)	Cloud API (Claude Haiku)
First inference latency	45s (download) + 2.3s	1.2s	1.8s
Subsequent inference latency	2.3s	1.2s	1.8s
Privacy	Absolute (zero network)	Server logs + API retention	Server logs + API retention
Offline capable	Yes	No	No
Rate limits	None	Yes (RPM/TPM caps)	Yes (RPM/TPM caps)
Cost to user	Free	$5-20/mo subscription	$5-20/mo subscription
Data ownership	100% user	Platform-dependent	Platform-dependent

The trade-off is clear: First-time users sacrifice ~45 seconds for model download. After that, every rewrite is free, private, unlimited, and offline-capable. For a tool that handles sensitive career data, we believe that’s the right trade-off.

Why This Architecture Matters for the Future of Web AI

We’re not just building a resume scanner. We’re proving that the future of AI doesn’t require surrendering your data.

Every major AI company is pushing cloud inference because it’s profitable. But the technical capability for powerful, private, client-side AI is already here. WebGPU is mature. Transformers.js is production-ready. Quantized models are surprisingly capable.

What this means for users:

You can get AI-powered resume help without trusting a third party with your employment history
You can use AI tools on airplane Wi-Fi, corporate firewalls, or restrictive networks
You own your data, your model weights, and your inference outputs

What this means for developers:

The browser is a legitimate AI deployment target
You can build sophisticated NLP features without API bills
Privacy-first architecture is a competitive differentiator, not a limitation

How to Try Our Client-Side AI Rewriter

Our private ATS scanner includes the AI rewriter as a built-in feature. Here’s how it works:

Upload your resume (PDF or DOCX) — parsed entirely in the browser, never uploaded
Select any bullet in the rich text editor
Click “Rewrite with AI” — the model downloads (first time only) and generates a stronger, keyword-optimized version
Compare, edit, and accept the suggestion — all without a single network call carrying your data

→ Try the private AI resume rewriter in our client-side scanner

Key Takeaways

Client-side AI using WebGPU and Transformers.js is production-ready for resume rewriting and NLP tasks
WebGPU delivers 4-5x faster inference than CPU-only approaches and is supported by 94%+ of modern browsers
A 500M-parameter model (Qwen1.5-0.5B-Chat) is sufficient for high-quality bullet rewriting while staying under 300MB
Progressive downloads, IndexedDB caching, and Web Workers are essential for shipping large models to consumer devices
The privacy benefits are absolute — zero server logs, zero API retention, zero data breach risk

The web browser is no longer just a document viewer. It’s a secure, private AI compute environment. And we’re just getting started.

Want to see it in action? Launch our private ATS scanner with built-in AI rewriting →