Role‑Based Skill Discovery & Evaluation System

1. What this is

We provision per‑agent‑role skill bundles and run a daily discovery‑to‑evaluation loop that measures “skill lift” for the Hermes fleet. Ten agents (scout, critic, architect, documenter, delegator, builder, tester, reviewer, ops, kratos) each receive up to five curated SKILL.md packages. The loop discovers new skills, evaluates them with a CPU‑only harness, and stores lift scores in a Cloudflare D1 table.

2. What would make it work

• All ten agents receive a **max 5‑skill bundle** for the first evaluation cycle (Shri confirmation).
• Skill bundles are generated from the six surviving sources (SkillsBench, SkillNet, OpenClaw, Awesome‑Skills, SkillCorpus, Lobehub free tier) and synced via Kratos to each Hermes profile.
• Evaluation uses the **CPU‑only Docker harness of SkillsBench** (no GPU required) with deterministic test suites per role.
• Lift scores are written to a dedicated **Cloudflare D1 table** separate from fleet state.
• A daily **Loki discovery cron** runs at **04:00** (aligned with the existing fleet‑maintenance window) to pull new SKILL.md files, trigger the eval harness, and update the D1 table.
• Resulting catalog is published by Aquaman as a markdown index in R2 storage.

3. What would kill it

• NVIDIA SkillEvaluator – binary compiled for x86_64; execution fails on the ARM‑based VPS (Exec format error).
• LobeHub paid tiers – $0.99 – $9.99 /mo subscription required; 402 Payment Required blocks skill fetch.
• skillbay.sh paid API – quota limited to 10 queries/day; 402/429 responses stop skill pulls.
• AutoGPT multi‑request limit – Ollama allows only one concurrent request; the marketplace crashes with max‑concurrent‑requests error.
• Claude Code registry (inferred) – no verifiable endpoint; CLI reports registry not configured.
• AgentSkillOS / ACES – depend on CUDA and torch; import errors on ARM kill the evaluator.
• SkillCorpus RAM usage – full 5 GB dataset exceeds the 12 GB RAM budget, causing OOM.
• SkillNet Node.js size – npm install pulls ~2 GB of modules; exceeds RAM budget.
• SkillsBench CUDA mode – default Docker image expects a GPU; crashes on the VPS.
• skillbay.sh paid API quota – rate‑limited 60 req/min; exceeds limits stop ingestion.

4. Stack and how it runs

The system consists of:

• Skills‑repo (Git) – curated SKILL.md files, version‑controlled.
• Kratos sync script – clones the repo and copies role‑specific bundles into each Hermes profile.
• SkillsBench CPU Docker harness – evaluates each bundle against deterministic test suites; Batman verified that using CPU mode avoids the NVIDIA evaluator fatal objection.
• Loki discovery script – runs daily via a Cloudflare‑compatible cron on the Oracle VPS; pulls new SKILL.md entries and triggers the eval harness.
• Aquaman catalog generator – reads lift scores from D1, builds a markdown index, and stores it in R2.
• Cloudflare D1 table – stores agent_id, role, skill_id, lift_score, timestamp (schema confirmed by Shri).
• Cloudflare Workers – serve the R2 catalog and expose a read‑only API for fleet‑wide queries.

Batman’s answer to the NVIDIA evaluator objection: the eval harness runs in CPU‑only mode, eliminating the need for the x86_64 GPU binary.

5. Questions for Shri

1. What exact schema (column names, types, primary key) should the Cloudflare D1 lift‑score table use?
2. Is the **max 5 skills per role** limit for the first eval cycle final, or should we provision headroom for future expansions?
3. Should the daily Loki discovery cron run **exactly at 04:00** or is a ±15 min window acceptable to avoid overlap with other maintenance tasks?

6. Answers (2026-09-21, via Kratos on Shri's behalf)

1. D1 schema: one table, skill_lift: run_id TEXT PRIMARY KEY, ts INTEGER, agent_role TEXT, repo TEXT, task_id TEXT, variant TEXT ('baseline'|'skill'), skill_ids TEXT (JSON array, NULL for baseline), model TEXT, correctness REAL, regressions INTEGER, latency_ms INTEGER, input_tokens INTEGER, output_tokens INTEGER, cost_usd REAL, failed INTEGER, notes TEXT; indexes on (task_id, variant) and (agent_role, ts). A skills-catalog table is deferred until the adopt/retire loop needs it.
2. Max 5 skills per role is final for the first eval cycle. Headroom for expansion is expected afterwards - the cap is per-cycle, not forever. All ten agents receive bundles (scout, critic, architect, documenter, delegator, builder, tester, reviewer, ops, plus Kratos).
3. Discovery cron: a +/-15 min window is acceptable. Align it with the existing 04:00 fleet maintenance window - one window, not two.

Next step: dedicated discovery pass producing concrete skill candidates per role (cap 5/role), then paired baseline-vs-skill evals on real repos, then Aquaman documents the adopted catalog.