Web AR API Architecture: How Face Tracking Runs in a Browser

A web AR API runs four moving parts in a fixed order: a camera stream, a WebAssembly runtime, a per-frame tracking and render loop, and an output sink. Production problems usually come from what the browser does to those parts, rather than from tracking accuracy. Hidden tabs, contended GPUs and objects the page never releases account for most of them.

This is an architecture walkthrough of that pipeline, written for the engineer who has been handed "put a virtual try-on in the product page" and wants to know what they are actually taking on. The examples use the Banuba Web AR SDK because we can show its real API surface and its published failure modes, but the stages, the browser constraints and the leak patterns hold whichever SDK sits in the middle.

The pipeline, in one line

Every browser AR SDK worth using reduces to the same flow:

Input -> Player + Effect -> Output

Input is a frame source. The Player owns the runtime and the render loop. The Effect is the thing being applied to a detected face. Output is where processed frames go. In the Banuba Web AR SDK those are literal class names, which makes the shape easy to read:

import { Webcam, Player, Module, Effect, Dom } from "https://cdn.jsdelivr.net/npm/@banuba/webar/dist/BanubaSDK.browser.esm.js"

const player = await Player.create({ clientToken: "xxx-xxx-xxx" })

await player.addModule(new Module("https://cdn.jsdelivr.net/npm/@banuba/webar/dist/modules/face_tracker.zip"))

await player.use(new Webcam())

player.applyEffect(new Effect("Glasses.zip"))

Dom.render(player, "#webar-app")

Six lines to a live AR camera is how short the happy path is. Everything below covers the four stages behind those six lines, and the browser behaviors that break them.

One change to make before this goes anywhere near production: pin the version. The paths above resolve to whatever is current, which means a minor release can change behaviour with no deploy on your side, and the module archive has to load from the same version as the bundle. Add the version to both URLs, or install from npm and bundle it.

Stage one: the camera, and the gate in front of it

Before any AR code runs, two things have to be true, and neither is negotiable.

The page has to be a secure context. Camera access is only exposed on HTTPS and on localhost, so a staging environment served over plain HTTP fails at the first call with no useful signal about why. Read the MDN MediaStream reference once properly, along with MediaTrackConstraints, because resolution and frame rate requests are negotiated there and a constraint the device cannot meet is a common cause of a stream that opens and then looks wrong.

The device also has to give you WebGL 2.0. Banuba's Web AR requirements state that the SDK runs on any browser with WebGL 2.0 or higher, and the browser matrix behind that is on caniuse, which is independent of any vendor.

Do not ship a browser support table, ship a runtime check

Support tables are the wrong tool here, and the reason is specific. WebGL 2.0 availability varies by device and by graphics driver inside a single browser version, so a version floor can be met by a browser that still cannot give you a context. A table tells you what is likely. Only the runtime answer is true:

function canRunWebAR() {

const gl = document.createElement("canvas").getContext("webgl2")

return Boolean(gl) && window.isSecureContext

}

Use caniuse to size the population you are about to exclude, and use the check above to decide what any individual user sees.

The practical consequence for a product page is that the AR entry point needs a real fallback branch, not an error toast. A user without WebGL 2.0 should get the static product image, and an analytics event should tell you how often that happens. That number is worth more than any published support matrix, because it is measured on your own traffic.

Ask about the client token before you launch

clientToken is a client-side value. It ships in the bundle and anyone can read it in devtools. That is normal for this class of SDK and it is not a defect, but it is the first thing a security reviewer will raise, and the answer cannot be that it is secret.

So make it a procurement question rather than a launch-day surprise. Ask the vendor how a token is scoped: which origins it will work from, how that allowlist is configured, and what happens when someone lifts it and calls it from their own domain. Get the answer in writing during evaluation. Any SDK in this category should have an origin allowlist.

Stage two: loading the runtime

This is the stage that surprises teams migrating from a native SDK. A web AR API ships as WebAssembly plus a set of neural network modules, and those modules arrive over the network as separate archives. In the snippet above, face_tracker.zip is a distinct fetch from the SDK bundle itself.

Two things follow from that.

The first frame is not free

There is a cold start covering the WASM instantiation and the module download, and it is paid on the user's connection, not yours. Preloading the module while the user is still reading the page moves that cost off the interaction, so the try-on button opens a camera that is already warm.

The effect fails to load: the archive is not a .zip, or it is empty

Two conditions throw at effect load. The Banuba SDK rejects an effect whose source is not a .zip archive, and it rejects a zero-length effect. Both are good behaviors and both are surprising the first time a build pipeline strips or rewrites the asset.

Catch this in CI rather than at runtime. If effects come out of a design tool and get committed by someone who is not an engineer, validate in the pipeline that every effect is a non-empty zip before it reaches a deploy. The failure is silent in review and loud in production.

Stage three: the per-frame loop

Once the runtime is up, the loop does detection, then effect evaluation, then render, once per frame. Frame budget is the whole game at this stage.

There are no published frame rate figures for the Web AR SDK. Ours included. What exists is a native Face AR SDK technical specification, measured on phones rather than in a browser, and a browser build behaves differently: WebAssembly and WebGL add overhead, and the page competes for the GPU with everything else open.

So the table below cannot tell you what your users will see. What it can do is rank features by cost, and that ranking does carry into a browser even when the numbers do not.

Native Face AR SDK, vendor lab measurements

Three things the ranking tells you.

Single-face tracking and a face filter cost about the same, because a filter that only needs the face mesh rides along on tracking that is already happening. Multi-face is where low-end Android starts to slide, and five faces is a documented ceiling for acceptable quality rather than a hard limit. Anything that adds a second segmentation network, for lips or hair or background, is a different cost class and needs measuring on its own.

Two things the table cannot tell you, and it is fair to say why. The specification page itself opens by stating the values came from fixed lab conditions and that developers should test in their own environment, which is true of every vendor's published numbers. And no device model, OS version, chipset or SDK build is published alongside them, so there is nothing to compare your own hardware against.

Which leaves one number that means anything for a browser build: the one you take yourself. Two habits to build early. Pull the oldest handset still visible in your own analytics and make that the device you sign off on. And keep per-device readings, because an average across a device mix hides the exact population that will churn. Banuba's comparison of tested AR SDKs for developers documents how that test setup is put together if you want a starting protocol.

Stage four: output, and the WebRTC case

The output sink is the part teams under-scope. A processed frame can go to a DOM element, to a Blob for a photo or a video, or to a MediaStream that a third party consumes. That last option is what makes browser AR work in a video call: the SDK's stream becomes the outgoing track on a WebRTC peer connection, and the remote participant sees the effect.

Scope the sink early, because a MediaStream output carries a browser constraint that a DOM output does not.

Where a browser AR pipeline degrades

Three failure modes have nothing to do with tracking quality. Two of them are browser behaviors rather than SDK behaviors, which matters for how you check them: browser bugs get fixed and vendor known-issues pages get updated, so read the current version of both before you design around either. The state described below is what the documentation says, not a permanent property of the web.

Safari pauses your stream when the tab is hidden

Safari pauses MediaStreams obtained from canvas.captureStream() when the tab is not visible, per the Page Visibility API definition of visibility. The behavior arrived in the 15.3 release. Banuba's web known-issues documentation lists it and states there is no known workaround.

For a WebRTC call, that means a participant who minimizes the browser freezes for everyone else. If your product is a video call, this belongs in the design review, not in the bug tracker later. Handle the visibility change explicitly and tell the other participants what happened:

document.addEventListener("visibilitychange", () => {

if (document.hidden) notifyPeers("camera paused by the browser")

})

notifyPeers is your own signalling call, whatever the app already uses to send state to the other side. Keep that handler even if Safari changes this behavior, because a paused outgoing track is a state your UI should be able to express regardless of what caused it.

A dangling Player instance leaks until the page crashes

The common pattern is a start button that creates a Player and a stop button that stops the webcam and unmounts the DOM node. That leaves the Player object alive and uncollectable, and a start-stop-repeat flow drains device RAM until the page dies.

The Player owns the WASM instance and the GPU resources, so it needs releasing explicitly. Stopping the webcam and unmounting the DOM does not do it:

let player, webcam

document.querySelector("#start").onclick = async () => {

  player = await Player.create({ clientToken: "xxx-xxx-xxx" })

  await player.addModule(new Module("https://cdn.jsdelivr.net/npm/@banuba/webar/dist/modules/face_tracker.zip"))

  await player.use((webcam = new Webcam()))

  player.play()

  Dom.render(player, "#webar-app")

}

document.querySelector("#stop").onclick = async () => {

  webcam.stop()

  Dom.unmount("#webar-app")

  await player.destroy()

  player = null

}

// the case people forget: the user leaves without pressing stop

window.addEventListener("pagehide", () => player?.destroy())

One Player per open, destroyed on close, is the pattern that cannot leak. It costs a cold start on every open, which is the argument for the module preload in stage two: warm the modules once on page load, and each subsequent Player creation is cheaper than the first.

Unlike the other two failure modes, this one is ours and it does not get fixed by a browser release. It is the first thing to check when a page running web AR crashes, and most such reports never reach the AR code at all.

Effect animations lag in Safari because of range requests

The cause is a WebKit defect, tracked as WebKit 232076, and the workaround is a service worker that proxies the video requests. Banuba ships a ready-made range-requests.sw.js for it. Check the tracker for the current status before you add the shim, because if it has been resolved you would be adding a service worker for nothing.

The general lesson matters more than this specific bug: a meaningful share of web AR defects turn out to be browser defects with vendor-supplied shims, so during triage check the browser bug tracker before the SDK's.

What a team actually inherits

The build-versus-buy question in browser AR is narrower than it looks, because the two options solve different problems.

The WebXR Device API is the W3C standard for immersive sessions, talking to device hardware directly. It is the right foundation for spatial work: placing a 3D object on a detected floor plane, architectural walkthroughs, room-scale experiences. Its scope stops short of a face mesh, lip segmentation or makeup shaders, and its camera-based AR support on mobile Safari has historically been the weak point. A face-tracking product built on WebXR alone therefore means owning the computer vision layer. Teams that want to assemble that layer from parts usually reach for Google's MediaPipe face landmarker, which is a real option when someone can own model selection and GPU delegate behavior.

A commercial SDK moves that work behind an API, and moves the cost from engineering months to license fees plus a page-weight budget. In exchange you inherit a dependency whose failure modes you did not choose. That is the reason to read a vendor's documented issues before signing rather than after.

Two rough guides. Spatial features start at WebXR. Face features on a timeline of a quarter or less are usually cheaper through an SDK. And the evaluation that predicts production behavior is the one built on a vendor's published failure modes plus per-device numbers you took yourself, because a demo reel is recorded on hardware you will never ship to.

Bottom line

The difficulty in browser AR sits in the runtime environment rather than in the tracking. A browser can hide the tab, contend for the GPU and hold objects the page thought it had released, and those behaviors are what turn a working demo into a support queue.

Five things belong in the build before launch:

1. A runtime WebGL 2.0 and secure context check, with a static fallback branch

2. The module preload, so the cold start is paid before the click

3. A destroy() path that actually runs, including on pagehide

4. An explicit response to a visibility change

5. A pinned SDK version, and a written answer on how the client token is scoped

Then measure frame rate per device on the oldest hardware your analytics actually show, because no published web figure will do that for you.

If you want the running code rather than the architecture, the Banuba web quickstart repositories cover plain JavaScript, React, Angular and Vue.

FAQ

What does a web AR API need from the browser?

A secure context for camera access and WebGL 2.0 for rendering. getUserMedia is only exposed over HTTPS and on localhost, and the SDK runtime needs WebGL 2.0 or higher. Feature-detect both at runtime with a static fallback rather than relying on a version table, because WebGL 2.0 availability varies by device and driver within a single browser version.

Why is the first frame slow?

Because a web AR SDK loads a WebAssembly runtime and downloads its neural network modules as separate archives before the loop can start. That cold start is paid on the user's connection. Preload the modules while the user is still reading the page instead of at the moment they open the camera.

Is the client token safe to expose?

It is a client-side value and it will be visible in devtools, which is expected for this class of SDK. Secrecy is not the protection, scoping is. Ask the vendor during evaluation which origins a token is valid from and how that is configured, and get the answer in writing before launch.

How fast is face tracking in a browser?

There is no published figure, ours included. The frame rates on Banuba's technical specification page are native SDK lab measurements, and WebAssembly plus WebGL add overhead on top while the browser competes for the GPU with the rest of the page. The only meaningful number is one measured on your own device mix, and the oldest handset in your analytics is the one to sign off on.

Does browser AR send face data to a server?

It depends on the SDK's architecture, so check the specific one. The Banuba Web AR SDK processes frames on the device through WebAssembly and WebGL, which keeps biometric data on the user's hardware and is usually the shorter conversation with a privacy reviewer.

Can browser AR run inside a video call?

Yes. The SDK outputs a MediaStream that a WebRTC peer connection can use as an outgoing track. Design around the Safari behavior first: MediaStreams from canvas.captureStream() pause when the tab is hidden, and Banuba's known-issues documentation lists no workaround. The app needs an explicit response to a visibility change either way.

How do I stop a web AR page from crashing after repeated use?

Call destroy() on the Player. Stopping the webcam and unmounting the DOM node leaves the Player alive holding the WASM instance and GPU resources, and a start-stop-repeat flow drains device RAM until the page dies. Create one Player per open, destroy it on close, and add a pagehide handler for the user who leaves without pressing stop.

How many faces can be tracked at once?

Banuba's documentation gives five as the maximum number trackable with acceptable quality on most mobile devices, with the real ceiling set by the device and screen proportions. On native, five faces cost about three frames per second against single-face tracking on Android and nothing measurable on iOS. Treat that as a starting hypothesis for a browser build rather than a figure, and measure it.