OpenClaw Playground Monitoring System

Status: ✅ Operational Deployed: 2026-03-05 Cost: $1.50-$6.00/month (configurable)

Overview

Automated monitoring system for playground labs using OpenClaw. Performs hourly health checks and scheduled functional tests on all 5 labs (linux-sandbox, terminal, networking, iac, argo-os) with automatic investigation when issues occur.

Architecture

┌─────────────────────────────────────────────────────┐
│         OpenClaw Gateway (<internal-host>)            │
├─────────────────────────────────────────────────────┤
│                                                       │
│  ┌──────────────────────────────────────────────┐  │
│  │  Cron Jobs (Scheduled Monitoring)            │  │
│  │  • Health Check (Hourly - FREE)              │  │
│  │  • Full Test (Every 6h - $6/month)           │  │
│  │  • Investigation (On-demand - FREE)          │  │
│  └──────────────────────────────────────────────┘  │
│                      │                               │
│  ┌──────────────────▼───────────────────────────┐  │
│  │  Skills (Executable Logic)                   │  │
│  │  • playground-health-check (Cerebras)        │  │
│  │  • playground-full-test (Gemini)             │  │
│  │  • playground-investigate (Cerebras)         │  │
│  └──────────────────────────────────────────────┘  │
│                      │                               │
│  ┌──────────────────▼───────────────────────────┐  │
│  │  Telegram Alerts                             │  │
│  │  @argobox_oc_bot → Your Telegram Chat        │  │
│  └──────────────────────────────────────────────┘  │
│                                                       │
└─────────────────────────────────────────────────────┘

What Gets Monitored

Health Check (Hourly, Free)

• Runs: Every hour automatically
• Checks: Is lab engine responding?
• Model: Cerebras (free tier)
• Time: ~30 seconds
• Alert: Only if engine is DOWN

Full Functional Test (Every 6 Hours, ~$0.20/run)

• Runs: Every 6 hours on schedule
• Tests: All 5 labs + all documented assignments
• Labs Tested:
  • linux-sandbox: 7 assignments
  • terminal: 7 assignments
  • networking: 6 assignments
  • iac: 4 assignments
  • argo-os: 4 assignments
• Model: Gemini
• Time: 5-10 minutes
• Alert: If any test fails

Investigation (On-Demand, Free)

• Trigger: Automatically on test failure
• Actions:
  • Reads DocHub docs (/docs/playgrounds/)
  • Reads actual code (/src/pages/api/labs/)
  • Analyzes root cause
  • Reports with file locations
• Model: Cerebras (free tier)
• Time: 3 minutes
• Output: Detailed findings + recommended fixes

Cost & Configuration

Current Setup (Balanced)

• Health check: $0/month (Cerebras, free)
• Full test: $6/month (4 tests/day × $0.05/test)
• Investigation: $0/month (Cerebras, free)
• Total: $6/month

Cost Scenarios

Scenario	Health	Full Test	Freq	Cost/month
Maximum	Hourly	4x daily	0:00,6:00,12:00,18:00	$6.00
Balanced	Hourly	1x daily	**0 6 * * ***	$1.50
Budget	6-hourly	Weekly	0 6 * * 1	$0.21
Free	6-hourly	Disabled	Manual only	$0.00

How to Adjust

Edit cron schedule in config:

ssh user@<server-ip>
nano /opt/openclaw/data/config/cron/playground-monitoring.json
# Change "schedule" field in cron-playground-full-test
# Examples:
# "0 6 * * *"           → Daily at 6 AM ($1.50/month)
# "0 6,18 * * *"        → Twice daily ($3/month)
# "0 0,6,12,18 * * *"   → Every 6 hours ($6/month)
# "0 6 * * 1"           → Weekly Monday ($0.21/month)
# Set enabled: false    → Disable ($0/month)

Then restart:

cd /opt/openclaw && docker compose restart openclaw-openclaw-gateway-1

Using the System

Receive Alerts

Monitor your Telegram for alerts from @argobox_oc_bot:

Health Check Alert (Hourly):

✅ Playground Health - HEALTHY
Engine: Up ✓
Response Time: 234ms

Full Test Alert (6-hourly):

✅ All Tests Passed!
linux-sandbox: ✓ PASS (7/7 assignments)
terminal: ✓ PASS (7/7 assignments)
networking: ✓ PASS (6/6 assignments)
iac: ✓ PASS (4/4 assignments)
argo-os: ✓ PASS (4/4 assignments)

All 5 labs ready for users! 🎉

Failure Alert + Investigation:

⚠️ Lab Test Failed - linux-sandbox
Error: Provisioning timeout (30s)

🔍 Investigation Complete
Root Cause: Container resource limit
Expected: 2-3 seconds
Actual: 30+ seconds timeout

Files to check:
- src/pages/api/labs/create.ts:145
- src/components/labs/LabLauncher.astro:67

Ready for fix - Launch Claude Code

Check Status

View latest test results:

ssh user@<server-ip>
docker logs openclaw-openclaw-gateway-1 | grep -i "playground\|test" | tail -50

View cron configuration:

cat /opt/openclaw/data/config/cron/playground-monitoring.json | jq .

Troubleshooting

No alerts in Telegram?

1. Verify bot is connected: docker logs ... | grep telegram | head -5
2. Check cron jobs loaded: docker logs ... | grep cron
3. Wait for next scheduled test (hourly health check)

Tests failing?

1. Check health check: "Is engine up?"
2. Read investigation report from Telegram
3. Check file locations mentioned in report
4. Use Claude Code to implement fix

Adjusting frequency?

1. Edit schedule in playground-monitoring.json
2. Restart OpenClaw
3. Cost updates automatically based on schedule

How It Was Built

Skills Deployed

• Location: /opt/openclaw/data/workspace/skills/playground-*/
• Language: TypeScript
• Each skill handles one monitoring function

Cron Configuration

• Location: /opt/openclaw/data/config/cron/playground-monitoring.json
• Format: cron schedule + agent config
• Runs automatically on specified schedule

Integration

• OpenClaw executes on cron schedule
• Uses configured LLM models (Cerebras, Gemini)
• Sends results to Telegram
• Auto-investigates on failures

Incident History

2026-03-05: Deployment challenges resolved

• Fixed gateway auth issues
• Resolved Telegram bot token injection
• Documented lessons learned
• System now stable and operational

See: /home/user/Development/openclaw-config/DEPLOYMENT-INCIDENTS-AND-FIXES.md

Files & Documentation

File	Purpose
DEPLOYMENT-INCIDENTS-AND-FIXES.md	Technical incident report + fixes
PLAYGROUND-COST-CONFIG.md	Cost optimization guide
DEPLOY-COMPLETE-SYSTEM.sh	Fresh deployment script
Journal entry	High-level incident log

Next Steps

1. Wait for alerts - First test will run within 1-6 hours
2. Adjust frequency - If needed, change cron schedule based on results
3. Fix issues - Use Claude Code when investigations reveal problems
4. Monitor - Watch Telegram for ongoing health

Security Notes

• System is behind Cloudflare Access
• Telegram token stored in .env on secure server
• Investigation reads from DocHub (public docs) and codebase
• No sensitive data in alerts

---

Status: ✅ Ready to monitor Last Updated: 2026-03-05 Contact: Check incident log or system logs if issues arise