How I run a fleet of AI agents from my phone

Most of my agent work now starts on my phone. I send a message or a voice note to a Telegram bot. On the other end is a full Claude Code session on my desktop, with my files, my tools and my rules. This post covers how that works, and what broke on the way.

Why a phone, and why Telegram

I had a Discord bridge first. It disconnected often, it was restrictive by design, and it had no way to run long jobs. A long job made the bot go silent until it finished, so I could not ask it anything else.

In July I installed an open-source Telegram bridge and started patching it. The first four patches were plumbing. The service could not find Node, so every Node tool it spawned failed. Logs only went to the system journal, so the log command was broken. A restart took 30 seconds; now it takes 5. And I added a one-word project switcher, so I can jump into any repo from the chat.

One group per project

The biggest change was group-per-project. Each project gets its own private Telegram group, bound to its folder with one command. The car-sourcing platform has a group. So does personal admin. Each group keeps its own session, so context from one project never leaks into another.

That change was 704 lines and 18 new tests. The suite was 70 of 70 when it shipped.

The background lane

The rule that makes this usable: decide before the first tool call whether a job will take more than about two minutes. If it will, hand it off.

A handoff is one command. It returns instantly and the job runs in a separate session. I stay free to keep talking to the main chat. When the job finishes, the report comes back to the chat session, not straight to me. The session reads it, finishes anything left undone, and sends me a short update in its own words.

Background jobs can now run for 8 hours. They used to be capped at 30 minutes, and the cap was killing video renders after the output already existed. They also start fresh each time. The old persistent workers piled up 836,000 tokens of context in a single day.

Routing: who does the work

Every session follows the same split. A strong model drives. It reads the request, plans, and writes the final answer. Cheap, uncapped lanes do the typing and the browsing from a spec the driver writes. A read-only scout does wide searches. A reviewer with fresh context checks anything that matters.

The split is enforced by hooks, not by good intentions. A dispatch to an expensive worker has to carry a written reason for escalating, or the hook blocks it. I added that after an audit showed the policy being ignored when it only lived in a prompt.

Background jobs had their own version of this problem. The config had no model set for them, so every unattended job fell back to the most expensive model. It was eating my top-tier allowance every day. The fix was one config key that pins background lanes to a cheaper model and outranks any per-chat override. I confirmed it from the run log after a restart.

What broke

Voice notes were the first surprise. For weeks they were silently not transcribed. A key lookup returned nothing, the transcriber returned nothing, and the raw audio went to the model, which then went looking for a transcription tool on every message. It looked like a missing feature. It was a one-line bug. Voice now goes through a transcription command with a silence guard, so a note from a dead microphone cannot turn into an invented prompt.

The second one could have cost money. On August 3, I typed "approve all" in reply to one message. The session held three separate approval queues at the time, and it resolved all of them. That time it was harmless. Now, when I use Telegram's Reply, the bridge attaches the quoted message and the approval applies to that message only. A bare "approve all" with more than one queue pending gets a list back and a question.

The third lesson came from merging upstream changes into my patched copy. The merge was clean: no conflict markers, all tests green. A reviewer pass still found two real bugs in the clean parts. The upstream code assumed there was exactly one chat lane, and mine has one per project. Every group message would have spawned as a background job. Neither bug would have shown up until the second restart. Now I audit the clean hunks on every merge, because a quick "restarted fine" check would have passed both.

How the work splits

A quick question, like "what did the parts yards reply?", gets answered inline. A full ad-account audit goes to the background lane. The chat tells me what it started, and I keep working in the same thread. The report comes back as a short summary with file paths I can tap.

Recurring jobs sit in a small scheduler. "Every morning at 8, summarize yesterday's commits" runs as a task. "Remind me in 90 minutes" is just a ping. Both are one command, and the daemon picks them up within a minute.

Upkeep

The bridge is one small service on my desktop. It needs one habit. When it goes quiet, check whether the system's out-of-memory killer took it. When that happens, the request it was working on is lost, so I restart the service and send the request again.