Self-Healing Agent Fleet
20+ AI agents running unattended across two machines — surviving reboots, crashes and their own operator
An AI agent in a chat window does what you ask while you watch. That is not the same thing as AI doing work, and the gap between them is entirely unglamorous: what happens when nobody is watching.
Every real failure I hit was operational, not intelligent:
The machine reboots. Every agent is gone, and nothing notices.
An agent goes idle with work still queued. It finished its task and is now waiting to be spoken to.
An agent crashes but its session survives. Every health check reports green while the thing is dead — the expensive one.
An agent dies holding a claimed job. The job is marked in-progress forever and no other agent will touch it.
A task needs credentials. Now the blast radius of one bad instruction is every client you have.
None of this is solved by a better prompt. It is ordinary systems reliability work, and it is the difference between a demo and infrastructure — in my experience the actual thing blocking companies from putting agents into production, rather than model capability.
Dispatch through a queue, not through attention. Work goes into a SQLite queue tagged with which agent owns it. Agents claim, work, respond. A scheduled job pokes each session once a minute so an idle agent picks up queued work without anyone prompting it. Producing work and consuming it are fully decoupled — I can queue twenty tasks at midnight and go to bed.
A supervisor with an opinion about what “alive” means. A control tool handles status, ensure-all, restart, a pane dump for when an agent goes quiet, and a doctor command for config drift and cron coverage. It runs every five minutes and once at boot. Anything down comes back without me.
The important design decision: liveness is not “the session exists.” That check is exactly what made crashed agents invisible. The supervisor inspects what is actually running inside the session, because the failure mode I most needed to catch was the one that looks healthy.
Stuck-job recovery, tuned per machine. A reaper releases jobs claimed by an agent that died holding them. Its timeout is 20 minutes on the GPU box rather than 15, because a transcription batch or a render legitimately runs longer — and re-dispatching a live job wastes the GPU it was waiting for.
Separation by blast radius. Client work runs on the always-on server with the client credentials. Experimental work runs on the GPU workstation with a separate queue and separate database, and the supervisor config there refuses client agents by design. Both fleets run with permission prompts disabled — which is precisely why the boundary is enforced at the machine level rather than by intention.
I built the supervisor on the remote server and ported it to the workstation. It failed three times, and every failure presented the same way: session exists, agent does nothing.
1. The trust prompt was inverted between machines. One pre-selects “Yes, I trust this folder”; the other lists “No, exit” first. The supervisor answered by keystroke position, so on the second machine it confidently selected No. The agent quit, the session survived as a plain shell, and every health check stayed green. Fixed by selecting the option by its label — also version-independent, so the next upstream reordering will not break it.
2. Launching in a directory is not the same as being in it. The shell profile on the workstation landed each new session back in the home directory. The agents opened the wrong workspace, found no specification file, and ran with no instructions or boundaries whatsoever — the most dangerous of the three, and the most silent. Fixed with an explicit directory change before launch.
3. A health check that always warns is a health check nobody reads. I consolidated one keepalive script per agent into a single parameterised script. The doctor command identified agents by finding their name in a scheduled-job line — and a generic script names none of them, so every agent warned permanently. A check that always fires trains you to ignore it, which is worse than no check at all.
The general lesson, and why I write these down: an agent that is silently misconfigured is worse than one that is plainly dead. A dead agent gets restarted. A misconfigured one keeps accepting work.
Running continuously for months, doing real work daily, recovering from reboots and crashes without intervention. The hard parts were reliability, isolation and observability — not model capability. Anyone can start an agent; keeping twenty of them working unattended is an operations problem.
Any team that has proven an AI workflow in a chat window and now needs it to run unattended, on a schedule, at volume — without a person babysitting it, and knowing when it breaks rather than hearing it from a client. Also directly applicable wherever agent work must be isolated from production credentials, which is most regulated environments.
Private Call Analysis Pipeline