SuperClaw
Voice-first AI agent that places real orders for you
SuperClaw is a voice-first assistant that runs errands instead of describing them. You hold the mic and say what you need (groceries from Instamart, a food order, a table through Dineout) and an AI agent drives a real browser session on your own account while you watch it work, step by step, from the app. When it needs a decision it stops and asks; when it is time to pay it hands you to your own UPI app rather than holding money itself.
The Problem
Assistants that only talk still leave the work to you. Doing the work means driving real consumer websites that were never built for agents (sessions expire, layouts shift, checkout demands a human), so the hard part is not the language model but keeping a long, failure-prone browser task legible and interruptible for the person watching it.
The Solution
An LLM browser agent drives Chrome over CDP against the user's own signed-in session, streaming its progress to the phone as live screenshots with step-by-step narration. A router decides per request whether the job is a direct answer, a browser task, or a connected-account lookup through Gmail, Calendar or Drive. Long-term preferences persist in PostgreSQL with pgvector behind a confidence threshold, so the agent narrows its questions over time instead of re-asking.
Architecture
- Flutter app (Riverpod, clean architecture) on iOS and Android with Firebase Auth, Remote Config and push notifications
- Python/FastAPI agent backend; a Go gateway with a master agent and capability resolver is in progress
- Browser agent: browser-use driving Chrome over CDP on Azure OpenAI, including realtime voice
- Request router fanning out to a direct LLM answer, the browser agent, or Gmail/Calendar/Drive via Composio OAuth
- PostgreSQL + pgvector for long-term user memory with confidence-scored preferences
- React admin console for access approval and versioned, admin-managed prompts
- Azure VMs behind nginx with TLS, separate stage / preprod / prod, versioned releases with one-step rollback
- GitHub Actions CI shipping every main push to TestFlight
Challenges & How I Solved Them
A question the agent never got to ask
When the agent paused for input (a size, a seat, an OTP), the request died at three minutes on a framework timeout, so the user's answer arrived to nobody. Fixing the timeout was half of it; the rest was designing the wait: a push notification, a reminder at five minutes, and an explicit 'stopped waiting' message at ten instead of a silent failure.
An agent that went blind at checkout
Full-page captures were too heavy for payment pages, so the agent was reasoning without seeing exactly where it mattered most. Switching to lighter screenshots restored its view of the step that decides whether a task succeeds.
Memory that learned the wrong things
Preferences were being written from plain confirmations, so a bare 'yes' could harden into a stated interest: the agent once recommended badminton on that basis. Storage now requires a real preference signal above a confidence threshold, not an acknowledgement.
'Lost contact' that hadn't lost anything
On Android a transient network error was being handled as the task having finished, ending live runs that were still going. Separating connection failure from task completion kept long browser tasks alive across flaky mobile networks.
My Contributions
- SuperBrowse live task view: streamed agent screenshots with step narration, stop/end controls and retry on failure
- Human-in-the-loop question flow across push notifications, reminder and timeout states
- Long-term user memory on PostgreSQL + pgvector with confidence thresholds
- Waitlist and access control: a Remote Config gate plus admin approval, enforced server-side on every REST route and the voice socket
- Release engineering: release-per-commit deploys with preflight checks, one-command rollback and 100+ self-checking regression scripts
- Leading the in-progress backend migration from Python to Go: master agent, capability resolver, versioned prompts and a flight-booking agent
Impact
- Live on Google Play (com.suprclaw.suprclaw), ordering on the user's own account with payment handed off to UPI
- Every commit on main ships to TestFlight; stage, preprod and prod run versioned releases with one-step rollback
- 100+ regression checks, each verified to fail when the bug it guards against returns
Lessons Learned
An agent that acts on your behalf is judged on what it does while you wait. Most of the work that mattered was not model quality but the seams: showing the task honestly, asking at the right moment, and failing in a way the person can understand and retry.