setgrep: grep for file sets

setgrep is a registry-driven recogniser for multipart file patterns. It reads the gruppe registry and, for a directory of loose files, answers "which of these files form one object, and what is each one's role?"

New to multipart file patterns? gruppe's plain-language explainer walks one shapefile from the files on disk to the grouping record.

Where grep finds lines and siegfried classifies single files, setgrep groups files into sets. The registry is the data; setgrep's generic engine, poc/recognize.py, executes it, with no format names in its control flow, and writes the grouping record for each recognised set in grex, the record format gruppe specifies.

Status: Python proof of concept; a Rust engine is planned. No release yet.

Install

You need git and Python 3.12; the demo also needs jq and column. On Debian or Ubuntu, column is in the bsdextrautils package (on some other distributions it is in util-linux):

sudo apt-get install git jq bsdextrautils

setgrep reads the registry from a gruppe clone, so clone both repositories side by side:

git clone https://opencommit.eu/gruppe/gruppe
git clone https://opencommit.eu/gruppe/setgrep
cd setgrep
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt      # PyYAML and jsonschema, pinned

poc/recognize.py and the demo look for gruppe at ../gruppe; set GRUPPE_DIR to use a clone elsewhere. poc/recognize_mpf001.py runs without gruppe, except for its --self-check against gruppe's committed record. mise install sets up the pinned Python too.

Usage

Run these from the setgrep directory. ROADS is a shapefile set committed in gruppe.

ROADS=../gruppe/examples/MPF-001-shapefile/sample/roads

# Generic, registry-driven recognition: a JSON report of every candidate set
python3 poc/recognize.py "$ROADS"
python3 poc/recognize.py "$ROADS" --quiet    # the summary line only

The registry defaults to mpf-registry.yaml in the gruppe clone; --registry FILE uses another. Exit codes follow grep: 0 recognised, 1 nothing, 2 error.

# The grouping record: a JSON list of grex records, one per recognised set, each with
# one content-addressed manifestation_id for the set, and each member's role, PUID and
# content address
python3 poc/recognize.py "$ROADS" --grex > roads.grex.json

# The same, with siegfried identifying each member from its bytes (the sample's files
# are stand-ins, so here it finds nothing; see "Real samples" below)
python3 poc/recognize.py "$ROADS" --grex --sf

# Check it against gruppe's grex schema and rules, and against the record gruppe
# commits for the same sample (10 checks)
python3 poc/check_record.py roads.grex.json --against ../gruppe/grex/conformance/MPF-001-shapefile.grex.json

# Export a recognised, complete shapefile set as a METS structMap. The producer still
# reads the earlier record shape, from a shapefile-only recogniser
python3 poc/recognize_mpf001.py "$ROADS" > roads.record.json
python3 -m producer.mets roads.record.json -o roads.mets.xml

# Tests: the unit suite, and the conformance corpus (the recognition decisions,
# kept as golden records a future engine must reproduce)
python3 -m unittest discover -s tests
python3 tests/conformance/conformance.py --check

What the grouping record is today. recognize.py --grex writes grex version 0.2, the record format gruppe specifies in grex/SPEC.md, and the tests validate its records against gruppe's grex/grex.schema.json. Each PUID in it is the one the pattern expects for the member, and the record says so (puid_resolved_by: pattern-expected); no tool read the bytes, so the record names no identification tool and no signature file. With --sf, siegfried reads each member's bytes: a PUID it matches by signature, and that the pattern expects, is recorded as identified by signature, and the record names siegfried and its signature files; any other answer stays a note beside the pattern's PUID. The record carries no entity_id yet: gruppe's registry does not say which members make a pattern's lasting identity, and the engine writes one only for a pattern that declares its identity_basis. The producer (METS and signing) still reads an earlier shape, from poc/recognize_mpf001.py, where the set id is called object_id. For PUIDs identified from real bytes, see "Real samples" below.

The conformance harness finds gruppe's grex schema beside setgrep; set GREX_SCHEMA to point at it elsewhere.

Signing (optional). The producer can also wrap a grouping record as a signed in-toto Statement: a DSSE envelope signed with cosign, plus an RFC 3161 timestamp from a local timestamp authority. It needs cosign and openssl. scripts/setup-signing.sh creates a local key and timestamp authority under .keys/ (git ignores it); then python3 -m producer.emit "$ROADS" --verify writes the Statement, envelope and timestamp to demo-out/ and checks them. See docs/packagingfacts-producer-readout.md.

Real samples. The committed examples are synthetic. scripts/fetch-samples.sh downloads real, freely licensed samples into samples/ (git ignores it), and scripts/validate_real.py mpf-001 checks the recogniser against the real bytes, with siegfried's byte-signature layer if siegfried is installed. Sources and licences are in docs/sample-sources.md. The fetch needs curl and unzip (Debian/Ubuntu: sudo apt-get install curl unzip). For the byte-signature checks, install siegfried (sf) from its project page, then run sf -update once to fetch its signature file; validate_real.py finds sf on the PATH, or takes --sf PATH. Without siegfried, the name-only checks still run and the byte-signature checks report SKIP.

Demo

demo/run-demo.sh is a presenter-paced terminal demo in seven sections, in the order of gruppe's plain-language explainer: the problem, what PRONOM answers and what it cannot, the registry, generic recognition, the record; then real bytes (a public-domain shapefile downloaded and identified with siegfried, skipped when it cannot run) and what comes next. It runs live on inputs it generates. See demo/README.md.

Layout

Path	What it is
poc/	The Python engine: recognize.py (generic, registry-driven; writes the grex record), check_record.py (checks a record against gruppe's grex schema), recognize_mpf001.py (the earlier record shape the producer reads, MPF-001 only), and the experiment that scored the engine (RESULTS.md).
producer/	A grouping record to a METS structMap (mets.py), and to a signed in-toto Statement (emit.py, sign.py).
tests/	Unit tests. tests/conformance/ is the language-agnostic corpus a future engine must pass.
demo/	The presenter demo.
docs/	Design records: when to rewrite and the rewrite plan, the XML-parsing decision, the structure-only-core finding, prior art, records of the producer and the METS emitter, sample sources.
scripts/	Real-sample fetch and validation; local signing setup.

Relationship to gruppe

• gruppe is the registry setgrep reads, and the home of grex, the output contract for a recognised set.
• setgrep works on its own. Its producer can also sign a grouping record in the attestation format of Provenant, a separate project; acceptance by Provenant's verifier (provenant-verify) is not yet shown (see the producer record linked above).

Status and next steps

• Now: a public Python proof of concept, and a demo that ends on its generic output.
• Next: the producer reads grex records; export a recognised set as an Archivematica structMap (demo v2); a standalone command-line tool.
• Later: the Rust engine. Its gate and plan are in docs/decision-poc-success-criteria.md and docs/engine-rewrite-plan.md.

This is a prototype. The Python dependencies are pinned, and CI runs a dependency audit (pip-audit) as advisory only: it reports and never fails the build. The Rust engine replaces the Python core later.

CI. CI of record runs on the maintainers' sovereign runner; the public copy runs the same workflows where the host allows. The workflows are in .forgejo/workflows/: ci.yml (compiles the proof of concept, runs the unit tests with the pinned requirements, advisory pip-audit) and compliance.yml (REUSE lint and the DCO sign-off check). The conformance corpus is not in CI yet; run it as shown above.

AI assistance

This repository was written with AI assistance (Anthropic's Claude), and that is disclosed: AI-assisted commits carry a Co-Authored-By: trailer. The policy mirrors the Linux kernel's. AI-assisted contributions are permitted and must be disclosed. An AI cannot provide the Signed-off-by: a human signs off every commit on main and thereby takes responsibility for the contribution's origin and licensing under the DCO. Nothing reaches main on an AI's authority alone.

Licence and contributing

REUSE-compliant: code is Apache-2.0, prose is CC-BY-4.0 (see LICENSES/). Fork it, try it out, and open issues; pull requests are not accepted for now. See CONTRIBUTING.md.