Rembrr is a voice and chat assistant that runs tasks on connected accounts: read and draft Gmail, schedule on Calendar, look up Drive, search the web with citations, and manage tasks.
My part: the realtime voice and agent backend — the live-conversation pipeline, tool calling, long-term memory, and encryption of secrets. The mobile app and automation layer are owned by the rest of the team. I have been on the product since September 2025.
In production: App Store · Google Play · rembrr.net.
The problem
An STT → LLM → TTS cascade feels slow and breaks on interruptions. Gemini Live cannot mix google_search with function_declarations. Gmail and Calendar tokens cannot sit in plaintext. Memory cannot bloat the voice request.
Architecture
Flutter app
→ LiveKit (audio / WebSocket proxy)
→ Gemini Multimodal Live
→ tool dispatcher (FastAPI)
→ Gmail / Calendar / Drive / Tasks
→ isolated search sub-agent (REST, search-only model)
→ RAG: Gemini embeddings + pgvector + hybrid ranking
(fact extraction off the request path)
→ secrets: envelope KEK/DEK, AES-256-GCM, KEK in Cloud KMSReal-time voice
Voice runs on Gemini Multimodal Live instead of a stitched STT → LLM → TTS pipeline: fewer hops, less accumulated latency, native barge-in. Audio travels over LiveKit.
Long-term memory (RAG)
Gemini embeddings over pgvector with hybrid ranking (dense + lexical). Fact extraction runs in the background so it adds no latency to the turn. Retrieval is injected as relevance-bounded context, not full history.
Decisions
- Isolated search sub-agent — a separate REST path runs the model with only search and returns
{ answer, citations }. Voice and chat get sources without breaking function calling. - Envelope encryption with KMS — connected-account tokens encrypted in-app (KEK/DEK, AES-256-GCM); KEK lives in Cloud KMS.
- Prompts as data — from f-strings to a Supabase-backed loader, editable without a redeploy.
- Explicit connection lifecycle — for realtime voice, owning the session beats an opaque managed socket.
What I discarded
- Full conversation history in the prompt vs. relevance-bounded context.
- Extracting facts on the voice request vs. a background job.
- One model with search + tools vs. a separate search sub-agent (the API forbids the mix).
Operations
Ongoing work is turn latency, LiveKit session stability, and the tool layer. Prompts edit without a redeploy. Secrets do not travel in plaintext. If Gemini or LiveKit fail, the session has an explicit lifecycle — not a socket that lives until it dies.
Similar systems (voice + tools + memory + secrets) can be commissioned. Rembrr is an existing product; I built this layer.