Research & Development

AI agents are becoming useful at the moment they can see your screen. That is also the moment they become dangerous. Almost every browser agent today captures the page and sends it to a model on a remote server, so your passwords, identity documents, faces and bank details leave your device before anything has decided whether they should.

MUDRA started from one research question:

Can an AI agent act on a web page without ever seeing its sensitive content?

Our answer is yes, provided perception is split at the trust boundary. Everything that has to look at the page (reading the layout, finding identifiers, detecting faces, reading text inside images) runs on the device. Only an anonymised description crosses the network: which fields exist, what kind of data they hold, and where the buttons are. The planner decides what to do. Your device does it, substituting the real values locally at the moment of execution.

The model plans on the shape of your data, never its content.

What we set out to prove

We framed the work as four hypotheses, each of which had to survive measurement rather than argument.

H1 — Perception can move to the edge. A lightweight browser extension, on ordinary hardware and with no GPU, can do the visual work that currently justifies sending screenshots to the cloud.

H2 — Indian identifiers can be detected precisely, not just greedily. A detector that knows the mathematics of Indian identifiers (Verhoeff for Aadhaar, Luhn for cards) can tell a real identifier from a look-alike, where pattern matching alone cannot.

H3 — Redaction can be verified, not assumed. A redacted image can be read back and proven clean before it is sent.

H4 — Privacy and usefulness are not a trade-off. An agent working on sealed references can still complete real tasks such as filling forms, navigating and submitting, without the planner ever holding a real value.

Research highlights

1. Checksum-aware detection for Indian identifiers

Most PII detectors are pattern matchers. A 12-digit number looks like an Aadhaar number, so it gets redacted. That fails in both directions: order numbers, invoice IDs and ticket references are sealed as identity documents, which breaks the agent's usefulness, while nothing distinguishes them from the real thing.

MUDRA validates identifiers with the arithmetic their issuers built in:

• Aadhaar — Verhoeff checksum, a dihedral-group check digit that catches every single-digit error and every adjacent transposition

• Payment cards — the Luhn algorithm

• PAN, IFSC, UPI VPA, passport, GSTIN, phone, email — structural validation with whole-token matching

Whole-token matching matters more than it looks. A GSTIN contains a complete, valid PAN as characters 3–12. A substring matcher flags every GSTIN as a PAN. MUDRA leaves it untouched because it matches identifiers as whole tokens.

2. The honest limit: identifiers with no checksum

PAN and IFSC codes carry no check digit. No amount of cleverness can tell a PAN-shaped warehouse SKU from a real PAN by the characters alone. We made a deliberate, documented choice: where a checksum exists, we use it and decoys pass through; where none exists, we over-seal on purpose. A sealed SKU costs one unreadable field. An unsealed PAN is the failure this project exists to prevent.

3. Verified redaction (re-OCR)

Redaction usually works like this: draw black boxes where you think the sensitive content is, then send the image. If a box is off by a pixel, lands in the wrong coordinate space, or misses a region, nobody finds out.

MUDRA reads its own output back. After masking, it runs OCR over the redacted image and checks two things: that none of the values captured in this observation are still readable, and that no newly read text passes the PII detector. If either check fails, the image is not sent. We turned "we redact" into "we verify."

4. Canary tokens — proof of non-leakage

Asserting that nothing leaked is easy. Proving it is harder. Every observation, MUDRA plants three synthetic, format-valid tracer identifiers into the page, and they travel through exactly the same code paths real data does. Immediately before any request leaves the device, the fully serialised body is scanned for them. If a single tracer is found, the request is aborted and the session stops. Planner responses are scanned the same way, so a model echoing a tracer back is caught too.

The result is a live, per-request answer to the question every privacy tool is asked: how do you know?

5. Adversarial research: attacking our own agent

We built the demonstration attacks before we trusted the defences, and the attacks found real flaws.

• A prompt injection hidden in the page told the agent to click "Transfer ₹50,000." This exposed a real hole: every click was classified as a plain "click", which every task is allowed, so an injection didn't need a new permission. It only needed one the task already had. We fixed it by reading each action's effect from the control itself.

• The same attack with a disguised label ("Continue" over a payment form) showed that labels alone can't be trusted, because the page writes its own labels. We added a second layer that reads the form the button submits: its destination and the names of the fields it sends.

6. Indic-script visual perception

Aadhaar cards, PAN cards and most Indian government portals carry Devanagari text. MUDRA's on-device OCR reads English and Hindi, so identifiers printed on scanned documents and images are found even when the page's markup contains no trace of them.

Results

MeasureResultIdentifiers detected25 of 25Precision89.3%F1 score94.3%Decoys correctly ignored12 of 15 — the 3 exceptions are no-checksum formats, over-sealed by designValues leaked after redaction0Canary tokens escaped0 (3 planted per observation)Data sent per agent step602 bytes median · 1.26 KB p95Local detection latency (text pipeline)11.1 ms median · 100.6 ms p95Peak browser memory132.6 MBAutomated tests417 (318 extension · 99 backend)Visual-pass latency & image-fixture scoresbeing measured

The decoy result, broken down by format, shows the reasoning:

DecoyHas a checksum?Result12-digit order and invoice numbersYes, VerhoeffPassed through16-digit ticket and batch numbersYes, LuhnPassed throughGSTIN (contains a valid PAN)Structure, whole-tokenPassed throughVehicle registrationsStructurePassed throughPAN-shaped SKU codesNo checksum existsOver-sealed, deliberatelyIFSC-shaped branch codesNo checksum existsOver-sealed, deliberately

How we work: measurement over assertion

Our evaluation harness is public and reproducible in one command. It runs synthetic fixture pages through the real pipeline and publishes recall, precision, leak count, decoy behaviour, latency, memory and payload size, together with the machine, browser and model state they were measured on.

The harness earned its place early. Our first two versions scored perfectly on nothing: one because a payload that omits values looks identical to one that redacted them, and the other because the headless test browser had no layout engine and every page was discarded before scoring. Getting the measurement right took three attempts, and every wrong version flattered us. That is why every number above comes with its denominator.

The bugs that taught us the most

BugHow it was foundWhy tests missed itA second copy of the Aadhaar detector had no Verhoeff check, so every 12-digit order number was sealed as an IDThe evaluation's decoy setEach detector passed its own tests; the duplicate was the problemA prompt injection could get a "Transfer" click approved under an ordinary form-filling grantBuilding the attack for the demoNo test had played the attackerChrome's real error for a lost offscreen-document race matched none of the expected phrasing, so the first observation after a cold start would have been droppedWiring the real pipeline end to endMocks returned the phrasing we expected, not the phrasing Chrome usesWith no OCR engine, the canary stream was skipped, disabling the tracer on the path most likely to leakWriting the canary testsThe degraded path was assumed safe