Andrés Navarro designed and built Rembrr’s voice and agent backend. Rembrr is a personal voice and chat assistant: you talk to it and it runs real tasks across your connected accounts — reading and drafting email in Gmail, scheduling on Google Calendar, looking things up in Google Drive, searching the web with cited sources, and managing tasks. Similar production voice and chat systems can be commissioned.
I have worked on the product since September 2025. My part is the real-time voice and agent backend: the pipeline that carries a live conversation, the tool-calling layer, long-term memory, and encryption of sensitive data. The mobile app and the automation layer are owned by the rest of the team.
Real-time voice
Voice runs on Gemini Multimodal Live rather than a stitched STT → LLM → TTS pipeline: fewer hops, less accumulated latency, and native interruption handling (barge-in), which is what makes a spoken conversation feel natural. Audio travels over LiveKit through a WebSocket proxy.
Long-term memory (RAG)
The assistant remembers across conversations. Memory is built with Gemini
embeddings over pgvector and hybrid ranking (dense + lexical); fact
extraction runs in the background, on a schedule, off the request path so it
adds no latency to the conversation. Retrieval is injected as
relevance-bounded context, not as full history.
Notable decisions
- Isolated search sub-agent — Gemini 2.5/Live can’t mix
google_searchwithfunction_declarations. A separate REST path runs the model with only search enabled and returns{ answer, citations }, so voice and chat get cited results regardless of the main model tier. - Envelope encryption with KMS — connected-account tokens and other sensitive data are encrypted at the application level with a KEK/DEK pattern (AES-256-GCM), with the KEK in GCP Cloud KMS.
- Prompt management as data, not code — migrating from f-string prompts to a Supabase-backed loader, editable without a redeploy.
- Explicit connection lifecycle — for a real-time voice product, controlling the connection matters more than the convenience of a managed realtime service.
Status
In production, published on the App Store and Google Play. Ongoing work is evolving the tool layer, controlling latency, and keeping real-time sessions stable.