Platform
How REI Skills routes tasks and reports benchmarks
Understand REI’s hybrid task router, interpret the held-out Recall@3 and Precision@3 results, and build an evaluation set for your own tasks.

Give the router concrete context
The recommend command takes a task description, and the optional files argument supplies stack hints. Use a real action and target artifact, such as checking incoming sensor timestamps against a delayed batch. File names should reflect the tools in the project and exclude sensitive paths if the command will be shared.
The hybrid router combines a lexical index with local lightweight vector features. Its normalization accounts for English and Turkish task phrasing. Ranked suggestions still need a review of scope, tools and the actual success criteria before use.
Read the repository benchmark correctly
The repository documents 260 benchmark tasks across train, development and held-out evaluation splits. In the 120-task evaluation split, the hybrid router reports 97.32% Recall@3 and 58.9% Precision@3. Recall tracks how many labeled relevant skills were retrieved near the top. Precision tracks how many returned skills match the labels. Both inform how much review a shortlist still needs.
These figures describe the pinned repository benchmark. Task selection, labeling, language mix, repository changes and hardware can alter results. The report includes split-level numbers and source data for a separate reading.
npx rei-skills@6.6.1 recommend "Review event time in IoT telemetry" --files replay.csv --jsonnpx rei-skills@6.6.1 evaluate --split evalMeasure fit on your workload
Create a small test set across your recurring tasks, with accepted skills and deliberate out-of-scope cases. Save task wording, file hints, accepted labels and the rationale for borderline examples. Measure shortlist usefulness and false positives separately. Report failure cases beside the aggregate.
Hold a portion of the data back while tuning. Repeat against raw task wording and a second phrasing to detect keyword dependence. When reporting latency, capture command, operating system and sample size so someone else can explain variance.
Route, then inspect the skill
Open the returned SKILL.md and follow its references only after confirming scope. Validate that the tool and operating environment exist. A rank is a retrieval result. Use the next stage to produce a plan and proof obligations for the actual task.
Skills in the catalog
Repository sources
References are pinned to the same upstream commit as the public catalog.