REI Skills

Engineering field notes

Reproducible research workflows with AI agents

Create an evidence ledger for an agent-assisted engineering study. Separate observations, simulations and claims before using the optional PaperOrchestra integration.

Three checkpoints in the guide: Research question, Repeatable run, Claim and artifact
Rootcastle field notes · Engineering field notes

Start with a question that can fail

Write the expected comparison, deciding metric, baseline and conditions in which the claim would be rejected. For a telemetry forecast, use a time-based holdout and make the target horizon explicit. For a mechanical experiment, record fixture revision, calibration and uncertainty before recording results.

Freeze the primary outcomes before analyzing the holdout. If exploratory results lead to a new hypothesis, label it exploratory and propose a separate confirmatory test. Preserve both runs and their configuration so the reasoning can be audited.

Build the reproducibility record

Save dataset and source revisions, tool versions, seed policy, tolerance, stopping criterion and the exact measurement procedure. Include a script or command that regenerates summary tables from primary observations. If hardware or a proprietary tool is required, specify the dependency and which steps can run independently.

Give every eventual claim an ID, a figure or table, and the artifact used to produce it. Record negative results and known exclusions alongside the positive evidence. Check whether the same source covers both the quantitative claim and the boundary conditions under which it was measured.

Use PaperOrchestra after the evidence exists

The upstream project documents an optional PaperOrchestra bridge. Its default path is a preflight. Work that uses external model providers requires a reviewed data scope and explicit authorization. Treat outlines, literature checks and chapter suggestions as drafts for an author to review.

Re-read primary sources for each citation and inspect figures against the data. Save information that identifies what was generated and which parts were verified by an author. Check citations, units, uncertainty and reproducibility before preparing external publication.

Close with independently checkable artifacts

A useful handoff includes the inputs or a controlled access description, scripts and their versions, an evidence ledger, a limitations section and exact verification instructions. Leave any unverified provider-backed process clearly identified in the release notes.

Example file

The template links each claim to a baseline, dataset revision and output artifact. Record the measured result and its uncertainty after running the study.

Download the research evidence ledger

Skills in the catalog

Repository sources

References are pinned to the same upstream commit as the public catalog.