Argo OS Build Swarm
Distributed binary package compilation for Argo OS
Live Swarm Status LIVE
Gateway
API endpoint & job dispatch
Orchestrators
Queue management & failover
Build Drones
Parallel compilation workers
Currently Building
The Problem: Gentoo's Biggest Hurdle
Gentoo Linux is powerful. Source-based compilation means every package is optimized for your exact hardware. USE flags let you strip out bloat and include only what you need. It's the ultimate customization—if you have unlimited time.
The catch? Compile times are brutal. A fresh Gentoo desktop install can take 12-24 hours of continuous compilation. Updating Firefox? 45 minutes. LibreOffice? An hour. The Rust compiler? Don't even ask.
For a single system, this is manageable. Set up a compile overnight, check it in the morning. But what if you're running five Gentoo systems? Or ten? What if you want to deploy a custom Linux distro across an entire homelab? Traditional Gentoo doesn't scale.
The Solution: Distributed Binary Compilation
The Build Swarm flips the Gentoo model on its head. Instead of each machine compiling its own packages, a central swarm of worker nodes compiles everything in parallel, then distributes binary packages to client systems.
The math is simple: 5 drones with 66 CPU cores compile packages simultaneously. What took one machine 12 hours now takes the swarm 1-2 hours—and client machines install those binaries in seconds.
But this isn't just "Gentoo with binaries." The swarm maintains full USE flag compatibility, handles dependency resolution, manages the binary package repository, and provides real-time monitoring of all builds. It's an entire build infrastructure.
What It Does For Me
90% Faster System Updates
My desktop gets updates in minutes, not hours. The swarm pre-compiles everything
overnight, and when I run emerge --sync, binaries are ready and waiting.
Multi-System Consistency
All my Argo OS machines run identical packages—same versions, same USE flags, same optimizations. No more "works on my desktop but breaks on the server."
Automatic Dependency Handling
When a core library updates, the swarm rebuilds everything that depends on it. I never hit broken library links or version mismatches.
Real-Time Build Visibility
Live dashboards show what's building, which drone is handling it, and estimated completion. No more wondering if that big update is still running.
Overnight Builds, Morning Binaries
Schedule world updates to run at 2 AM. Wake up to fresh binaries, pull them to any client machine in under a minute.
Safe Rollbacks
Binary packages are versioned. If an update breaks something, I can instantly roll back to the previous build without recompiling.
What It Can Do For Others
The Build Swarm isn't just for my homelab. It's designed as an open-source blueprint for anyone who wants Gentoo's power without the compile-time tax.
For Homelabbers
Running multiple Gentoo/Funtoo systems? Set up your own swarm with as few as 2 drones. Even modest hardware—a couple old desktops or VMs—dramatically cuts update times across your fleet.
For Small Teams
Development teams using Gentoo can share a build swarm. Developers get instant package updates while a dedicated build server handles compilation in the background.
For Education
Teaching Linux internals? Students can experiment with Gentoo without the compile wait. The swarm provides binaries for the teaching environment while advanced students can study the swarm architecture itself.
Why This Makes Gentoo Actually Usable
Gentoo's reputation problem is real. Ask any Linux user: "Would you run Gentoo as your daily driver?" Most will laugh. The compile times are legendary. The maintenance overhead is notorious.
But those problems aren't inherent to Gentoo. They're artifacts of the single-machine compilation model. The Build Swarm removes that bottleneck entirely.
With the swarm running, Gentoo becomes as easy to update as Arch or Debian—emerge --sync && emerge -uDN @world
pulls binaries in seconds. You keep all of Gentoo's advantages (USE flags, optimization,
rolling releases) without the downside.
The Result
Gentoo as a daily driver becomes practical. Not for masochists. Not just for servers where you update once a month. Practical for desktops, laptops, workstations—any machine where you want performance and customization without the wait.
Deploying Tailored Linux to Many Machines
The real power of the Build Swarm emerges when you're deploying to multiple machines with different needs. This isn't just about having binaries—it's about having the right binaries for each system.
🖥️ Desktop Profile
Full GUI stack, media codecs, gaming libraries
USE="X wayland pulseaudio vulkan" 🖧 Server Profile
Headless, minimal, security-focused
USE="-X -gtk -qt5 hardened" 🤖 Container Host Profile
Docker/Podman optimized, overlay filesystems
USE="containers cgroups overlay" 🔬 Development Profile
Compilers, debuggers, language runtimes
USE="debug rust python nodejs" How It Works
Each profile defines a set of USE flags and package configurations. The swarm maintains separate binary repositories for each profile. When you deploy a new machine:
- Boot the Argo OS installer
- Select your target profile (desktop, server, container, dev)
- Point to your binhost
- Pull packages—the profile-matched binaries install in minutes
- You have a fully configured, optimized Gentoo system
Scaling becomes trivial. Need to deploy 10 identical container hosts? The swarm already has the binaries. Need to add a desktop to the fleet? Same process, different profile. The compile work is done once, deployed infinitely.
The Bigger Picture: Argo OS
The Build Swarm is a core component of Argo OS—my custom Gentoo-based distribution designed for power users who want control without compromise.
Argo OS combines Gentoo's flexibility with modern tooling: binary packages via the swarm, OpenRC for init (no systemd), optimized defaults for workstations and servers, and documentation that doesn't assume you have infinite time to read wikis.
The goal? Make source-based Linux practical for everyone—not just the patient few who can afford to wait for compiles.
Learn More About Argo OS →Key Features
Parallel Compilation
Distribute builds across 5+ worker nodes with 60+ CPU cores for massive parallelization.
Automatic Failover
Primary/standby orchestrator architecture ensures builds continue even if a node fails.
Binary Packages
Compiled packages are stored on a binhost for instant deployment to client systems.
Sweeper Drones
Specialized workers handle long-running builds (LLVM, Rust, Firefox) without blocking the queue.
Cross-Network
Works across multiple sites via Tailscale mesh VPN for geographically distributed builds.
Real-time Monitoring
Live dashboards show build progress, drone status, and queue depth.
Technology Stack
CLI Reference
The build-swarm CLI v4 provides a single entry point for all swarm operations.
Every command is designed to be safe by default—destructive operations require explicit confirmation.
$ build-swarm status # Show swarm status + next recommended action
$ build-swarm fresh # Clean start: reset → sync → build
$ build-swarm monitor # Interactive TUI dashboard
$ build-swarm verify # Safety check with Risk Score
$ build-swarm finalize # Move staging → production (atomic)
$ build-swarm fix-blocked # Diagnose blocked packages
$ build-swarm sync # Sync portage tree across all drones
$ build-swarm reset # Reset all drones to idle
_status Shows drone states, queue depth, active builds, and a recommended next action based on current swarm state. The first command you run. Read-only fresh Performs a full clean start: resets all drones to idle, syncs the portage tree across all nodes, then kicks off a new build cycle. The "just make it go" command. Resets state monitor Opens an interactive TUI dashboard with real-time build progress, drone utilization, package queue, and estimated completion. Press q to exit. Read-only verify Runs a comprehensive safety check on completed builds. Computes a Risk Score (0-100) based on failed packages, missing dependencies, and version conflicts. Score under 15 = safe to finalize. Read-only finalize Atomically moves packages from staging to the production binhost. Uses directory swap so clients never see a partial update. Only runs if verify passes. Modifies prod fix-blocked Diagnoses packages stuck in "blocked" state. Identifies circular dependencies, missing USE flags, and slot conflicts. Suggests specific resolution commands. Read-only sync Syncs the portage tree (emerge --sync) on all drones in parallel. Ensures every node has the same package definitions before a build cycle. Updates tree reset Resets all drones to idle state. Clears in-progress tasks and returns them to the queue. Does not delete any compiled packages. Resets state Daily Workflow
A typical swarm build cycle from start to finish. Most days this takes 6-8 hours of unattended build time, plus about 2 minutes of actual human interaction.
Start Clean Build
build-swarm fresh Resets all drones, syncs the portage tree, and kicks off a full @world build.
One command to start the entire pipeline. Run this in the morning or schedule it overnight.
Watch Progress (Optional)
build-swarm monitor Opens the interactive TUI dashboard. Watch drones pick up packages, see build times, track failures in real time. Satisfying to watch, but entirely optional—the swarm runs itself.
Check Completion
build-swarm status Shows the current state: how many packages built, how many failed, queue depth, and a recommended next action. If the queue is empty and drones are idle, you're done.
Safety Check
build-swarm verify Runs the verification suite and computes a Risk Score (0-100). Checks for missing dependencies, version mismatches, and failed critical packages. Under 15 = safe to ship.
Release to Production
build-swarm finalize Atomically swaps the staging binhost with production. Client machines immediately see the new packages. Rollback is instant if anything goes wrong—the old production dir is kept.
Update Desktop
sudo apkg update Pull the freshly built binaries to your local machine. What took the swarm hours to compile installs in seconds. Your desktop stays current without ever pegging its own CPU.
Self-Healing Architecture
Drones fail. Networks drop. Machines reboot. The swarm handles all of it automatically—no human intervention required for routine failures.
Circuit Breakers
5 consecutive build failures on a single drone triggers a circuit breaker. The drone is grounded for 5 minutes—removed from the active pool to prevent cascading failures. After cooldown, it's automatically re-tested and restored.
Auto-Reclaim
When a drone gets grounded, any packages it was building are automatically returned to the queue. Another drone picks them up within seconds. No work is lost, no manual reassignment needed.
Maintenance Loop
A 60-second maintenance sweep runs continuously, checking for offline drones, stale tasks, and orphaned builds. Anything stuck for more than 2 minutes gets reclaimed and requeued automatically.
Auto-Reboot
Grounded drones don't just sit there. The orchestrator SSHs into the drone and restarts the build service. If the drone comes back healthy, it's ungrounded and returned to the active pool.
Staging/Production Split
All builds go to a staging directory first. Client machines never see
work-in-progress. Only after build-swarm finalize are packages promoted to
the production binhost via an atomic directory swap.
Idempotent Operations
Every swarm operation is safe to re-run. build-swarm fresh can be called
multiple times without side effects. Failed builds are retried, successful ones are
skipped. The swarm converges to completion.
Performance Metrics
Real numbers from production builds. These aren't benchmarks—they're averages from months of daily operation across the full fleet.
Hardware Requirements
You don't need a rack of servers. A couple of spare machines and SSH access is enough to get started.
Getting Started
- 2 drones — any Linux machine with
emerge - 1 gateway — can share a drone host
- SSH access between all nodes
- Shared storage — NFS, rsync, or HTTP binhost
- ~20 cores total for meaningful speedup
Production Ready
- 3+ drones — dedicated build machines
- Dedicated orchestrator — separate from drones
- 50+ total cores — for 10x+ speedup
- Tailscale — for multi-site connectivity
- Shared binhost — accessible to all clients
Current Production Fleet
| Node | Role | Cores | Site | Status |
|---|---|---|---|---|
| Izar | Orchestrator + Drone | 16 | Local | Primary |
| Tau-Beta | Drone | 8 | Local | Active |
| Tarn | Orchestrator + Drone | 14 | Remote | Active |
| Meridian | Drone | 24 | Remote | Active |
| Capella | Sweeper Drone | 8 | Local | Sweeper |
| Total | 70 | 2 sites, connected via Tailscale | ||
Related Pages
Related Blog Posts
The full story of building, breaking, and hardening the swarm—told in real time across these posts.