# CaveatDiff research note

Research date: 2026-08-25.

## Problem

Model cards are part of the evidence people use to decide whether an AI model
fits a task. They commonly document intended use, out-of-scope use, known
limitations, bias, training data, license, and evaluation results. Those cards
can change as model repositories evolve, but a normal text diff does not tell a
reviewer that a limitation disappeared, a license changed, or an evaluation
number moved.

## Landscape search

The scoped search found:

- model-card templates, editors, generators, and validation toolkits;
- generic Markdown and semantic diff tools;
- model-behavior, data-drift, and benchmark monitoring products;
- tools for comparing prompts and model outputs.

It did not find a focused deterministic tool that compares two model-card
versions and elevates changes to disclosure sections and Hugging Face metadata.
This supports a useful narrow project, not proof that no comparable software
exists.

Two earlier candidates were rejected during this research cycle. MCP tool drift
already has dedicated scanners and proxies. Bilingual invariant checking is
already present in local translation-QA products. CaveatDiff avoids repeating
those products.

## Product decision

CaveatDiff will be a local-only static web app plus a dependency-free Node.js
CLI. A reviewer supplies two Markdown model cards. The tool compares structured
disclosure categories, selected front-matter fields, removed risk statements,
and changed evaluation numbers. It reports facts about documentation changes
and does not infer whether a model became safer or worse.

## Sources

- Hugging Face model-card documentation:
  https://huggingface.co/docs/hub/main/model-cards
- Hugging Face annotated model-card template:
  https://huggingface.co/docs/hub/en/model-card-annotated
- Mitchell et al., *Model Cards for Model Reporting*:
  https://arxiv.org/abs/1810.03993
- Google Model Card Toolkit:
  https://github.com/tensorflow/model-card-toolkit
- NIST AI Risk Management Framework:
  https://www.nist.gov/itl/ai-risk-management-framework