Exploratory
A question and a setup written down, then run, with no criterion fixed beforehand. Reported as it came out, with code and seeds. A reason to design a real experiment, never a result of record.
Scouting
Criteria fixed before the run, on a separate reimplementation at toy scale. A reason to design a real experiment, not a result of record.
Recorded
Criteria written in the public changelog before the run; model calls cached under a hash of each request and committed, so the run replays byte for byte. The freeze itself is not publicly timestamped: these runs predate the public repository.
Committed
Everything in Recorded, plus the freeze commit and the SHA-256 of the sealed test corpus published here, in the public repository, before the run. Nobody can move the goalposts afterwards.
The rules behind all of it: pass criteria are fixed before each run; data that has been seen becomes
development data; the system under test never screens its own test data; and every model call is cached
under a hash of the full request, so recorded runs replay from the public repository with no API calls.
The full log is CHANGELOG_EXPERIMENTS.md.