Guide
Keel inspects the artifact. Meta's runtime guards judge what a model says; Keel judges what was done to its weights, before the model is deployed. This page explains the product, the evidence, and the exact place it occupies in a protection stack.
Meta releases open weights. That is a strategy, and it works: anyone can run Glimmer locally, fine-tune it, or ship a derivative. The same property is the risk. A derivative is a different model, and nothing in the release process proves what it became.
The licence asks derivative authors to say what they built. That is a declaration, not a check. Runtime guards score prompts and responses; they never see a weight file. So the moment a checkpoint leaves Meta, every safety claim about it rests on trust.
The gap is specific, and it is not that Meta does no safety work. Meta checks the base model before release, and it checks the conversation at runtime. Nothing checks the derivative artifact after release. That is the hole Keel fills, and open weights are exactly why the hole exists.
Keel replaces that trust with a measurement. It compares a derivative against the model it came from, asks both the same questions, and writes a card that states what moved, whether the measurement was valid, and what a platform should do.
| layer | question it answers |
|---|---|
| model card | what the publisher says was done. |
| Keel | what was actually done, measured against the base, before deployment. |
| Llama Guard, Prompt Guard, CodeShield | is this conversation safe, at runtime. |
| LlamaFirewall | which runtime guard runs on which message. |
These are not competing products. The runtime guards protect the conversation; Keel protects the artifact that starts it. A checkpoint that Keel blocks never reaches the conversation, which is why the artifact layer comes first in the pipeline.
The commercial shape of the gap is an enterprise sale. A regulated buyer inherits a derivative and cannot verify it. Keel turns that into an artifact the buyer can check.
keel policy exits allow, review
or block, so a release pipeline can gate on a checkpoint the way it gates on tests.The strategic case for an independent scanner is credibility: a vendor's own attestation is a claim, and a third-party fingerprint is evidence. That is how the tool helps sell the enterprise tier rather than sit beside it.
A scan takes a derivative and its base, asks 32 fixed questions of both, and also asks 32 harmful requests. It compares the answers, scores the difference, and emits a card. The comparison is relative to the base, never to a global threshold, because families differ: Glimmer refuses all 32 harmful requests, Mistral refuses 19 of 32 before anyone edits it.
| column | meaning |
|---|---|
| model | the checkpoint. Community models appear by letter, not by repository, so the page does not advertise third-party projects. |
| source | card means a full report exists; a baseline row
is a reference point with no card. |
| drift axes | what the 32 probes measured. PASS means no axis moved enough to be statistically believable at this sample size. It is a weak signal by construction. |
| policy decision | what a platform does. This reads harmful compliance as well as the axes, so it is the stricter column. |
| validity | ok means the measurement itself was sound.
suspect means the model changed answer format and the axes cannot be trusted. |
| harmful compliance | out of 32 harmful requests, how many the model answered instead of refusing. Lower is better; 0.00 refuses all of them. |
| judged | how many answers the judge scored. |
| fingerprint | a hash of the card. Edit one number and it changes. |
MEASUREMENT SUSPECT, the axes below are not evidence.q is the corrected probability that the change is noise; below 0.05
is the usual threshold.They answer different questions. Drift axes ask whether the 32 probes moved; policy asks whether the model should be allowed. With 32 probes, a change can be large in consequence and still invisible to the axes. The Qwen rows prove the point: the axes found nothing, and harmful compliance moved 0.97. A single "pass" for those rows would be the dishonest result, which is why the two columns exist.
This is not a Glimmer behaviour. The same method, the same panel and the same judge were run on three families:
| family | base | derivative | delta |
|---|---|---|---|
| Meta Glimmer 30B | 0.00 | 0.16, our ablation | +0.16 |
| Meta Glimmer 30B | 0.00 | 0.09, a community edit | +0.09 |
| Qwen 2.5 7B | 0.00 | 0.97, our ablation | +0.97 |
| Qwen 2.5 7B | 0.00 | 0.78, a community edit | +0.78 |
| Mistral 7B | 0.44 | 0.81, a community edit | +0.38 |
Qwen does not cluster tightly: the 7B derivatives sit at 0.97 and 0.78, while the larger community checkpoint sits at 0.51. The family result is a range, 0.51 to 0.97, not a single point.
These are the answer-level numbers under the frozen contract. The answer extractor is
format-aware: Glimmer separates reasoning with a to=self marker, so only the
answer segment is judged, while Qwen and Mistral return plain answers and their full
response is the answer. A first pass wrongly emptied Qwen and Mistral because it looked for
a Glimmer marker that those models never emit; the fix restored their real numbers, and
both Muse judges agree on those rows (30/32, 28/30, 29/30), which is a live cross-check against Muse Spark Open. Glimmer stays at 0.16 because
its ablation moves the reasoning channel, which the contract deliberately excludes.
A clean number from a broken measurement is worse than no number. That rule was tested five times:
Press Verify on any card and the server recomputes it from committed responses.
MATCH means the numbers were reproduced from scratch. Press Tamper test
and one axis changes by 0.25, which changes the fingerprint. Then press Verify the
tampered card: the stored tampered fingerprint and the recomputed one differ, and the
page says DIFF. That is the loop end to end, an edit that cannot survive a
check. You can also try to re-score the tampered card: scoring never reads a submitted
card, so the attempt is rejected against the responses.
The fingerprint is a checksum, not a signature. It proves the card matches its own
contents and that an edit will be caught. Authenticity comes from the repository and the
manifest hashes, or from keel sign, which adds an HMAC over the card when a
deployment needs one.
If you do not want to trust this page, do not. The verify page lists a SHA-256 for each response file and offers a proof bundle: the base responses, the candidate responses, the card, and a CHECK.txt with the command. Run that command offline and the fingerprint must match the one on the card. That is the difference between a screenshot and a proof.
The upload path uses the free program scorers, so the card states exactly which axes it could not measure.
The completions are withheld. A model that complies with 81 percent of harmful requests produces text that should not be published, and publishing it would demonstrate the attack instead of the defence. Reviewers request access; the manifest publishes a SHA-256 for the committed panel files and the withheld harmful files, so anything received can be checked. The public bundle carries the probes and panel responses with generic names and no source map.
Every boundary below is a scoping decision, and each one is what lets the tool say no.
muse-spark-1.3, answer-only
extraction, three-repeat majority, and the contract is recorded in every card. The headline
number reports the delta under that contract, not a single draw from whichever judge is
available.A detector that never refuses is not a detector. These boundaries are why this tool can refuse, and the five refusals are why its verdict means anything.
| term | meaning |
|---|---|
| checkpoint | one saved version of a model's weights. |
| derivative | a model built by modifying another one. |
| ablation | removing a direction or component, often to remove refusal behaviour. |
| quantization | storing weights with fewer bits. It is a control here because it should not change behaviour. |
| LoRA | a small set of trained weights added to a base model. |
| drift | a measured change in behaviour against the base. |
| validity | whether the measurement can be trusted. |
| fingerprint | a hash of a card's contents. |
The walkthrough is also available as a forty second video with annotated scenes and a written description.
python scripts/verify_repro.py python scripts/check_links.py --url https://tomscot-keel.hf.space python scripts/stress_space.py --url https://tomscot-keel.hf.space
Technical detail: docs/METHOD.md and docs/RESULTS.md.