You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@sourcelume.apache.org by an...@apache.org on 2026/09/08 19:58:43 UTC
(sourcelume-spec) branch minimal-mimir updated (372b552 -> 134026a)
This is an automated email from the ASF dual-hosted git repository.
ANierbeck pushed a change to branch minimal-mimir
in repository https://gitbox.apache.org/repos/asf/sourcelume-spec.git
discard 372b552 Correct Mímir training-corpus premise; author 12 direct-input records
new 134026a Correct Mímir training-corpus premise; author 12 direct-input records
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (372b552)
\
N -- N -- N refs/heads/minimal-mimir (134026a)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
notes/stress-test-dfm-mimir.md | 16 +++++++---------
spec/0.0.1/index.md | 6 +++---
2 files changed, 10 insertions(+), 12 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@sourcelume.apache.org
For additional commands, e-mail: commits-help@sourcelume.apache.org
(sourcelume-spec) 01/01: Correct Mímir training-corpus premise; author 12 direct-input records
Posted by an...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ANierbeck pushed a commit to branch minimal-mimir
in repository https://gitbox.apache.org/repos/asf/sourcelume-spec.git
commit 134026a59bb8819d242f824ed5f9e4d28274ec90
Author: anierbeck <ac...@codecentric.de>
AuthorDate: Tue Sep 8 21:54:17 2026 +0200
Correct Mímir training-corpus premise; author 12 direct-input records
An earlier pass built the stress-test example, the manifest, and ~28 dynaword provenance records on the premise that DFM Mímir trains on the danish-pd/PleIAs dataset and on raw danish-dynaword sources. Verification against the Mímir v1 technical report (arXiv 2608.13517, Appendix A, Table 10 — all 161 training datasets) showed this premise is wrong: no danish-pd or PleIAs dataset appears in the corpus, and Mímir does not train on raw dynaword — only on four synthetic derivatives (schn [...]
This commit corrects the record: removes the 27 wrong-premise example files (danish-pd best-effort + 8 committed representative dynaword records + 19 untracked raw-dynaword records) and authors 12 new ProvenanceRecords for datasets Mímir actually consumes directly — the sapientinc HRM-Text core, laerebogen, dfm-dyna-instruct, opus-da-en-permissive, wiki-instruct-da, dfm8-openhermes-da, both agreement-supplied sources (DBC, Lex.dk), and all four danish-dynaword-* derivatives. All 13 ex [...]
Also restores two references that were broken on this branch: notes/stress-test-dfm-mimir.md (rewritten around the correct danish-dynaword-denoising example; surfaces a new gap-1, the agreement-supplied license-IRI convention, in addition to the original three) and examples/minimal-record.jsonld. The manifest notes/mimir-dynaword-sources.md is rewritten to describe Mímir's direct inputs rather than dynaword's 48 sub-sources. spec/0.0.1/index.md license-field prose now documents both i [...]
Apache-ai: Yes
Generated-by: pi (anthropic/claude-opus-4-1)
Reviewed-by: anierbeck
---
.../dfm-mimir-danish-dynaword-denoising.jsonld | 47 ++++
...mir-danish-dynaword-paragraph-reordering.jsonld | 47 ++++
...imir-danish-dynaword-prefix-continuation.jsonld | 47 ++++
.../dfm-mimir-danish-dynaword-span-filling.jsonld | 47 ++++
examples/dfm-mimir-danish-pd-best-effort.jsonld | 37 ----
examples/dfm-mimir-dbc.jsonld | 37 ++++
examples/dfm-mimir-dfm-dyna-instruct.jsonld | 42 ++++
examples/dfm-mimir-dfm8-openhermes-da.jsonld | 42 ++++
examples/dfm-mimir-dynaword-adl.jsonld | 42 ----
examples/dfm-mimir-dynaword-ai-aktindsigt.jsonld | 42 ----
examples/dfm-mimir-dynaword-dakultur.jsonld | 37 ----
...m-mimir-dynaword-folketingets-dokumenter.jsonld | 52 -----
examples/dfm-mimir-dynaword-kalliope.jsonld | 47 ----
examples/dfm-mimir-dynaword-ncc-parliament.jsonld | 47 ----
.../dfm-mimir-dynaword-retsinformationdk.jsonld | 42 ----
examples/dfm-mimir-dynaword-wikipedia.jsonld | 47 ----
examples/dfm-mimir-hrm-text-data-io-cleaned.jsonld | 42 ++++
examples/dfm-mimir-laerebogen.jsonld | 52 +++++
examples/dfm-mimir-lexdk-articles.jsonld | 37 ++++
examples/dfm-mimir-opus-da-en-permissive.jsonld | 42 ++++
examples/dfm-mimir-wiki-instruct-da.jsonld | 42 ++++
examples/minimal-record.jsonld | 25 +++
notes/mimir-dynaword-sources.md | 243 +++++++--------------
notes/stress-test-dfm-mimir.md | 128 +++++++++++
spec/0.0.1/index.md | 6 +-
25 files changed, 761 insertions(+), 558 deletions(-)
diff --git a/examples/dfm-mimir-danish-dynaword-denoising.jsonld b/examples/dfm-mimir-danish-dynaword-denoising.jsonld
new file mode 100644
index 0000000..56fc9d8
--- /dev/null
+++ b/examples/dfm-mimir-danish-dynaword-denoising.jsonld
@@ -0,0 +1,47 @@
+{
+ "@context": "../context/0.0.1/sourcelume.jsonld",
+ "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/danish-dynaword-denoising",
+ "type": "ProvenanceRecord",
+ "identifier": "https://huggingface.co/datasets/schneiderkamplab/danish-dynaword-denoising",
+ "name": "danish-dynaword-denoising",
+ "version": "20260515",
+ "license": "https://www.apache.org/licenses/LICENSE-2.0",
+ "creator": [
+ {
+ "type": "schema:Organization",
+ "name": "Danish Foundation Models (danish-foundation-models/danish-dynaword) — original text corpus",
+ "role": "originator"
+ },
+ {
+ "type": "schema:Person",
+ "name": "Peter Schneider-Kamp (schneiderkamplab)",
+ "role": "curator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "schneiderkamplab",
+ "role": "distributor"
+ }
+ ],
+ "created": "2026-09-08T00:00:00Z",
+ "added": "2026-05-15T00:00:00Z",
+ "contentCreated": "2026-05-15T00:00:00Z",
+ "origin": "Chat-template-ready Danish denoising / text-restoration instruction pairs derived from the Danish Dynaword corpus. Rows were generated by corrupting text from Dynaword source files (adl, ai-aktindsigt, botxt, cellar) and judged accepted/rejected with google/gemma-4-31B-it. Contains only judge-accepted rows: 65,518 rows from 69,907 audited (4,389 rejected). Form: synthetic + audited. Contributed 323M sampled tokens per epoch to the DFM Mimir v1 corpus.",
+ "custodyChain": [
+ {
+ "agent": "https://huggingface.co/datasets/danish-foundation-models/danish-dynaword",
+ "action": "provided-source-corpus",
+ "startTime": "2025-08-18T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/google/gemma-4-31B-it",
+ "action": "judged-generated-rows",
+ "startTime": "2026-05-15T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/schneiderkamplab",
+ "action": "published-accepted-rows-as-danish-dynaword-denoising",
+ "startTime": "2026-05-15T00:00:00Z"
+ }
+ ]
+}
diff --git a/examples/dfm-mimir-danish-dynaword-paragraph-reordering.jsonld b/examples/dfm-mimir-danish-dynaword-paragraph-reordering.jsonld
new file mode 100644
index 0000000..c958aa4
--- /dev/null
+++ b/examples/dfm-mimir-danish-dynaword-paragraph-reordering.jsonld
@@ -0,0 +1,47 @@
+{
+ "@context": "../context/0.0.1/sourcelume.jsonld",
+ "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/danish-dynaword-paragraph-reordering",
+ "type": "ProvenanceRecord",
+ "identifier": "https://huggingface.co/datasets/schneiderkamplab/danish-dynaword-paragraph-reordering",
+ "name": "danish-dynaword-paragraph-reordering",
+ "version": "20260515",
+ "license": "https://www.apache.org/licenses/LICENSE-2.0",
+ "creator": [
+ {
+ "type": "schema:Organization",
+ "name": "Danish Foundation Models (danish-foundation-models/danish-dynaword) — original text corpus",
+ "role": "originator"
+ },
+ {
+ "type": "schema:Person",
+ "name": "Peter Schneider-Kamp (schneiderkamplab)",
+ "role": "curator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "schneiderkamplab",
+ "role": "distributor"
+ }
+ ],
+ "created": "2026-09-08T00:00:00Z",
+ "added": "2026-05-15T00:00:00Z",
+ "contentCreated": "2026-05-15T00:00:00Z",
+ "origin": "Chat-template-ready Danish paragraph-reordering instruction pairs derived from the Danish Dynaword corpus. Paragraphs from Dynaword source files (adl, botxt, cellar) were shuffled and the model asked to restore the original order; rows were judged accepted/rejected with google/gemma-4-31B-it. Contains only judge-accepted rows: 55,340 rows. Form: synthetic + audited. Contributed 160M sampled tokens per epoch to the DFM Mimir v1 corpus.",
+ "custodyChain": [
+ {
+ "agent": "https://huggingface.co/datasets/danish-foundation-models/danish-dynaword",
+ "action": "provided-source-corpus",
+ "startTime": "2025-08-18T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/google/gemma-4-31B-it",
+ "action": "judged-generated-rows",
+ "startTime": "2026-05-15T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/schneiderkamplab",
+ "action": "published-accepted-rows-as-danish-dynaword-paragraph-reordering",
+ "startTime": "2026-05-15T00:00:00Z"
+ }
+ ]
+}
diff --git a/examples/dfm-mimir-danish-dynaword-prefix-continuation.jsonld b/examples/dfm-mimir-danish-dynaword-prefix-continuation.jsonld
new file mode 100644
index 0000000..179e244
--- /dev/null
+++ b/examples/dfm-mimir-danish-dynaword-prefix-continuation.jsonld
@@ -0,0 +1,47 @@
+{
+ "@context": "../context/0.0.1/sourcelume.jsonld",
+ "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/danish-dynaword-prefix-continuation",
+ "type": "ProvenanceRecord",
+ "identifier": "https://huggingface.co/datasets/schneiderkamplab/danish-dynaword-prefix-continuation",
+ "name": "danish-dynaword-prefix-continuation",
+ "version": "20260515",
+ "license": "https://www.apache.org/licenses/LICENSE-2.0",
+ "creator": [
+ {
+ "type": "schema:Organization",
+ "name": "Danish Foundation Models (danish-foundation-models/danish-dynaword) — original text corpus",
+ "role": "originator"
+ },
+ {
+ "type": "schema:Person",
+ "name": "Peter Schneider-Kamp (schneiderkamplab)",
+ "role": "curator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "schneiderkamplab",
+ "role": "distributor"
+ }
+ ],
+ "created": "2026-09-08T00:00:00Z",
+ "added": "2026-05-15T00:00:00Z",
+ "contentCreated": "2026-05-15T00:00:00Z",
+ "origin": "Chat-template-ready Danish prefix-continuation instruction pairs derived from the Danish Dynaword corpus. Text from Dynaword source files (adl, ai-aktindsigt) was split so the model continues a given prefix; rows were judged accepted/rejected with google/gemma-4-31B-it. Contains only judge-accepted rows: 105,317 rows from 110,396 audited. Form: synthetic + audited. Contributed 252M sampled tokens per epoch to the DFM Mimir v1 corpus.",
+ "custodyChain": [
+ {
+ "agent": "https://huggingface.co/datasets/danish-foundation-models/danish-dynaword",
+ "action": "provided-source-corpus",
+ "startTime": "2025-08-18T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/google/gemma-4-31B-it",
+ "action": "judged-generated-rows",
+ "startTime": "2026-05-15T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/schneiderkamplab",
+ "action": "published-accepted-rows-as-danish-dynaword-prefix-continuation",
+ "startTime": "2026-05-15T00:00:00Z"
+ }
+ ]
+}
diff --git a/examples/dfm-mimir-danish-dynaword-span-filling.jsonld b/examples/dfm-mimir-danish-dynaword-span-filling.jsonld
new file mode 100644
index 0000000..30d9e51
--- /dev/null
+++ b/examples/dfm-mimir-danish-dynaword-span-filling.jsonld
@@ -0,0 +1,47 @@
+{
+ "@context": "../context/0.0.1/sourcelume.jsonld",
+ "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/danish-dynaword-span-filling",
+ "type": "ProvenanceRecord",
+ "identifier": "https://huggingface.co/datasets/schneiderkamplab/danish-dynaword-span-filling",
+ "name": "danish-dynaword-span-filling",
+ "version": "20260515",
+ "license": "https://www.apache.org/licenses/LICENSE-2.0",
+ "creator": [
+ {
+ "type": "schema:Organization",
+ "name": "Danish Foundation Models (danish-foundation-models/danish-dynaword) — original text corpus",
+ "role": "originator"
+ },
+ {
+ "type": "schema:Person",
+ "name": "Peter Schneider-Kamp (schneiderkamplab)",
+ "role": "curator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "schneiderkamplab",
+ "role": "distributor"
+ }
+ ],
+ "created": "2026-09-08T00:00:00Z",
+ "added": "2026-05-15T00:00:00Z",
+ "contentCreated": "2026-05-15T00:00:00Z",
+ "origin": "Chat-template-ready Danish span-filling instruction pairs derived from the Danish Dynaword corpus. Spans were masked out of text from Dynaword source files (adl, ai-aktindsigt) and the model asked to fill them; rows were judged accepted/rejected with google/gemma-4-31B-it. Contains only judge-accepted rows: 54,968 rows. Form: synthetic + audited. Contributed 251M sampled tokens per epoch to the DFM Mimir v1 corpus.",
+ "custodyChain": [
+ {
+ "agent": "https://huggingface.co/datasets/danish-foundation-models/danish-dynaword",
+ "action": "provided-source-corpus",
+ "startTime": "2025-08-18T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/google/gemma-4-31B-it",
+ "action": "judged-generated-rows",
+ "startTime": "2026-05-15T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/schneiderkamplab",
+ "action": "published-accepted-rows-as-danish-dynaword-span-filling",
+ "startTime": "2026-05-15T00:00:00Z"
+ }
+ ]
+}
diff --git a/examples/dfm-mimir-danish-pd-best-effort.jsonld b/examples/dfm-mimir-danish-pd-best-effort.jsonld
deleted file mode 100644
index 58c75dd..0000000
--- a/examples/dfm-mimir-danish-pd-best-effort.jsonld
+++ /dev/null
@@ -1,37 +0,0 @@
-{
- "@context": "../context/0.0.1/sourcelume.jsonld",
- "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/danish-pd",
- "type": "ProvenanceRecord",
- "identifier": "https://huggingface.co/datasets/PleIAs/Danish-PD",
- "name": "PleIAs - Danish Public Domain",
- "version": "danish-pd@huggingface-2024-03",
- "license": "https://creativecommons.org/publicdomain/mark/1.0/",
- "creator": [
- {
- "type": "schema:Organization",
- "name": "LANGU:IA",
- "role": "curator"
- },
- {
- "type": "schema:Organization",
- "name": "PleIAs",
- "role": "distributor"
- }
- ],
- "created": "2026-09-05T00:00:00Z",
- "added": "2024-03-01T00:00:00Z",
- "contentCreated": "1700-01-01T00:00:00Z",
- "origin": "3113 Danish monographs/periodicals whose copyright has expired, recovered from Internet Archive and various European national libraries and cultural heritage institutions. The original authors/rightsholders are not individually identifiable at this scale and are omitted from creator/custodyChain; see this field for that gap.",
- "custodyChain": [
- {
- "agent": "https://huggingface.co/datasets/PleIAs",
- "action": "aggregated",
- "startTime": "2024-03-01T00:00:00Z"
- },
- {
- "agent": "https://huggingface.co/danish-foundation-models",
- "action": "repackaged-into-training-corpus",
- "startTime": "2026-01-01T00:00:00Z"
- }
- ]
-}
diff --git a/examples/dfm-mimir-dbc.jsonld b/examples/dfm-mimir-dbc.jsonld
new file mode 100644
index 0000000..1df2401
--- /dev/null
+++ b/examples/dfm-mimir-dbc.jsonld
@@ -0,0 +1,37 @@
+{
+ "@context": "../context/0.0.1/sourcelume.jsonld",
+ "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/dbc",
+ "type": "ProvenanceRecord",
+ "identifier": "https://sourcelume.apache.org/records/dfm-mimir-v1/dbc",
+ "name": "DBC (agreement-supplied)",
+ "version": "2026",
+ "license": "https://sourcelume.apache.org/ns#agreement-supplied",
+ "creator": [
+ {
+ "type": "schema:Organization",
+ "name": "DBC (Danish Library Center / Dansk Bibliotekscenter)",
+ "role": "originator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "Danish Foundation Models",
+ "role": "curator"
+ }
+ ],
+ "created": "2026-09-08T00:00:00Z",
+ "added": "2026-01-01T00:00:00Z",
+ "contentCreated": "1800-01-01T00:00:00Z",
+ "origin": "Agreement-supplied Danish text corpus provided to the Danish Foundation Models project by DBC (Dansk Bibliotekscenter) under a Danish Foundation Model agreement whose licensing does not permit public sharing. No Hugging Face identifier exists; the data is not publicly redistributed. Per the Mimir technical report, DFM uses such agreement-supplied datasets to derive synthetic instruction-tuning data rather than training directly on the data. Contributed 356M sampled tokens pe [...]
+ "custodyChain": [
+ {
+ "agent": "https://www.dbc.dk/",
+ "action": "supplied-to-dfm-under-agreement",
+ "startTime": "2026-01-01T00:00:00Z"
+ },
+ {
+ "agent": "https://www.foundationmodels.dk",
+ "action": "held-and-derived-synthetic-instruction-data",
+ "startTime": "2026-01-01T00:00:00Z"
+ }
+ ]
+}
diff --git a/examples/dfm-mimir-dfm-dyna-instruct.jsonld b/examples/dfm-mimir-dfm-dyna-instruct.jsonld
new file mode 100644
index 0000000..e88f1ab
--- /dev/null
+++ b/examples/dfm-mimir-dfm-dyna-instruct.jsonld
@@ -0,0 +1,42 @@
+{
+ "@context": "../context/0.0.1/sourcelume.jsonld",
+ "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/dfm-dyna-instruct",
+ "type": "ProvenanceRecord",
+ "identifier": "https://huggingface.co/datasets/danish-foundation-models/dfm-dyna-instruct",
+ "name": "DFM dyna-instruct",
+ "version": "0.1.3",
+ "license": "https://opendatacommons.org/licenses/by/1-0/",
+ "creator": [
+ {
+ "type": "schema:Organization",
+ "name": "Danish Foundation Models",
+ "role": "originator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "Danish Foundation Models",
+ "role": "curator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "Danish Foundation Models",
+ "role": "distributor"
+ }
+ ],
+ "created": "2026-09-08T00:00:00Z",
+ "added": "2026-06-03T00:00:00Z",
+ "contentCreated": "1993-01-01T00:00:00Z",
+ "origin": "A collection (version 0.1.3) of openly-licensed instruction-tuning datasets for Danish (and English, French, German, Italian), built using the Dynaword approach. 4.40M samples, 2.85B Llama-3 tokens across 8 task types: general, translation, reasoning, refusal, code, tool_use, instruction_following, qa. Sub-sources include apertus-sft-mixture (ODC-By, 2.30B), wiki-instruct-da (Unknown license, 190M), wildchat-qwen (CC-BY 4.0, 172M), translation-100k (CC-BY-SA 4.0, 101M), agen [...]
+ "custodyChain": [
+ {
+ "agent": "https://www.foundationmodels.dk",
+ "action": "collected-and-curated-sub-sources",
+ "startTime": "2026-06-03T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/danish-foundation-models",
+ "action": "published-dfm-dyna-instruct-v0-1-3",
+ "startTime": "2026-06-03T00:00:00Z"
+ }
+ ]
+}
diff --git a/examples/dfm-mimir-dfm8-openhermes-da.jsonld b/examples/dfm-mimir-dfm8-openhermes-da.jsonld
new file mode 100644
index 0000000..9d7f1d2
--- /dev/null
+++ b/examples/dfm-mimir-dfm8-openhermes-da.jsonld
@@ -0,0 +1,42 @@
+{
+ "@context": "../context/0.0.1/sourcelume.jsonld",
+ "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/dfm8-openhermes-da",
+ "type": "ProvenanceRecord",
+ "identifier": "https://huggingface.co/datasets/schneiderkamplab/dfm8-openhermes-da",
+ "name": "dfm8-openhermes-da",
+ "version": "20260515",
+ "license": "https://www.apache.org/licenses/LICENSE-2.0",
+ "creator": [
+ {
+ "type": "schema:Organization",
+ "name": "teknium/OpenHermes — original English instruction dataset",
+ "role": "originator"
+ },
+ {
+ "type": "schema:Person",
+ "name": "Peter Schneider-Kamp (schneiderkamplab)",
+ "role": "curator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "schneiderkamplab",
+ "role": "distributor"
+ }
+ ],
+ "created": "2026-09-08T00:00:00Z",
+ "added": "2026-05-15T00:00:00Z",
+ "contentCreated": "2024-01-01T00:00:00Z",
+ "origin": "DFM8 repaired OpenHermes dataset: a Danish-translated/repaired version of the OpenHermes instruction-tuning corpus (originally GPT-4-generated synthetic data from open sources, assembled by teknium). The dataset was translated to Danish and audited/repaired by schneiderkamplab. 967,334 rows. Form: translated + audited. Contributed 922M sampled tokens per epoch to the DFM Mimir v1 corpus (1.31%). The underlying OpenHermes data carries no single explicit license; the schneider [...]
+ "custodyChain": [
+ {
+ "agent": "https://huggingface.co/datasets/teknium/openhermes",
+ "action": "published-original-openhermes-english",
+ "startTime": "2024-01-01T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/schneiderkamplab",
+ "action": "translated-to-danish-and-audited-as-dfm8-openhermes-da",
+ "startTime": "2026-05-15T00:00:00Z"
+ }
+ ]
+}
diff --git a/examples/dfm-mimir-dynaword-adl.jsonld b/examples/dfm-mimir-dynaword-adl.jsonld
deleted file mode 100644
index fede5f0..0000000
--- a/examples/dfm-mimir-dynaword-adl.jsonld
+++ /dev/null
@@ -1,42 +0,0 @@
-{
- "@context": "../context/0.0.1/sourcelume.jsonld",
- "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/dynaword-adl",
- "type": "ProvenanceRecord",
- "identifier": "https://tekster.kb.dk/text?editorial=no&f%5Bsubcollection_ssi%5D%5B%5D=adl&match=one&search_field=Alt",
- "name": "Archive for Danish Literature (ADL)",
- "version": "adl@dynaword-1.2.23",
- "license": "https://creativecommons.org/publicdomain/zero/1.0/legalcode.en",
- "creator": [
- {
- "type": "schema:Organization",
- "name": "Det Kongelige Bibliotek (Royal Danish Library)",
- "role": "originator"
- },
- {
- "type": "schema:Organization",
- "name": "Danish Foundation Models",
- "role": "curator"
- }
- ],
- "created": "2026-09-08T00:00:00Z",
- "added": "2020-09-14T00:00:00Z",
- "contentCreated": "1700-01-01T00:00:00Z",
- "origin": "Selected works from 78 Danish authors from the Middle Ages to the mid-20th century, reproduced from standard printed editions and digitised by the Royal Danish Library (Det Kongelige Bibliotek) as the Archive for Danish Literature (ADL). 498 samples, 58.49M Llama-3 tokens. The dataset was originally distributed as part of the Danish Gigaword corpus and carried forward into dynaword.",
- "custodyChain": [
- {
- "agent": "https://tekster.kb.dk/",
- "action": "digitised-and-published",
- "startTime": "1700-01-01T00:00:00Z"
- },
- {
- "agent": "https://huggingface.co/danish-foundation-models",
- "action": "incorporated-into-danish-gigaword",
- "startTime": "2020-09-14T00:00:00Z"
- },
- {
- "agent": "https://huggingface.co/datasets/danish-foundation-models/danish-dynaword",
- "action": "incorporated-into-dynaword",
- "startTime": "2024-01-01T00:00:00Z"
- }
- ]
-}
diff --git a/examples/dfm-mimir-dynaword-ai-aktindsigt.jsonld b/examples/dfm-mimir-dynaword-ai-aktindsigt.jsonld
deleted file mode 100644
index cc9f527..0000000
--- a/examples/dfm-mimir-dynaword-ai-aktindsigt.jsonld
+++ /dev/null
@@ -1,42 +0,0 @@
-{
- "@context": "../context/0.0.1/sourcelume.jsonld",
- "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/dynaword-ai-aktindsigt",
- "type": "ProvenanceRecord",
- "identifier": "https://huggingface.co/datasets/AI-aktindsigt/Skrabet_kommunale_hjemmesider",
- "name": "AI Aktindsigt (scraped municipal websites)",
- "version": "ai-aktindsigt@skrabet-2025-03",
- "license": "https://www.apache.org/licenses/LICENSE-2.0",
- "creator": [
- {
- "type": "schema:Organization",
- "name": "AI-aktindsigt project",
- "role": "originator"
- },
- {
- "type": "schema:Organization",
- "name": "Danish Foundation Models",
- "role": "curator"
- }
- ],
- "created": "2026-09-08T00:00:00Z",
- "added": "2025-03-24T00:00:00Z",
- "contentCreated": "2010-01-01T00:00:00Z",
- "origin": "Multiple web scrapes of Danish municipal websites, collected across different municipal domains as part of the AI-aktindsigt project (ai-aktindsigt.dk). The dynaword subset is derived from the source dataset AI-aktindsigt/Skrabet_kommunale_hjemmesider. 200.91K samples, 139.23M Llama-3 tokens; document created-dates range from 2010 to March 2024.",
- "custodyChain": [
- {
- "agent": "https://ai-aktindsigt.dk",
- "action": "scraped-municipal-websites",
- "startTime": "2010-01-01T00:00:00Z"
- },
- {
- "agent": "https://huggingface.co/datasets/AI-aktindsigt/Skrabet_kommunale_hjemmesider",
- "action": "published-scrape-dataset",
- "startTime": "2025-03-24T00:00:00Z"
- },
- {
- "agent": "https://huggingface.co/datasets/danish-foundation-models/danish-dynaword",
- "action": "incorporated-into-dynaword",
- "startTime": "2025-03-24T00:00:00Z"
- }
- ]
-}
diff --git a/examples/dfm-mimir-dynaword-dakultur.jsonld b/examples/dfm-mimir-dynaword-dakultur.jsonld
deleted file mode 100644
index 6b69d99..0000000
--- a/examples/dfm-mimir-dynaword-dakultur.jsonld
+++ /dev/null
@@ -1,37 +0,0 @@
-{
- "@context": "../context/0.0.1/sourcelume.jsonld",
- "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/dynaword-dakultur",
- "type": "ProvenanceRecord",
- "identifier": "https://huggingface.co/datasets/NLPnorth/dakultur",
- "name": "DaKultur (Danish cultural-knowledge queries)",
- "version": "dakultur@nlpnorth-2025",
- "license": "https://opensource.org/license/mit",
- "creator": [
- {
- "type": "schema:Organization",
- "name": "NLPnorth (DaKultur study authors)",
- "role": "originator"
- },
- {
- "type": "schema:Organization",
- "name": "Danish Foundation Models",
- "role": "distributor"
- }
- ],
- "created": "2026-09-08T00:00:00Z",
- "added": "2026-07-20T00:00:00Z",
- "contentCreated": "2024-06-01T00:00:00Z",
- "origin": "Danish user queries from the DaKultur study, in which native Danish speakers wrote prompts to test the cultural awareness of Danish language models. Covers food, traditions, lifestyle, trivia, social norms, politics, language, arts, geography, transport, education, and work. Only the queries are included; model responses and human judgments from the original evaluation are not part of this release. 344 samples, 5.49K Llama-3 tokens. Published as arXiv:2504.02403 (Müller-Eber [...]
- "custodyChain": [
- {
- "agent": "https://huggingface.co/datasets/NLPnorth/dakultur",
- "action": "collected-and-published",
- "startTime": "2024-06-01T00:00:00Z"
- },
- {
- "agent": "https://huggingface.co/datasets/danish-foundation-models/danish-dynaword",
- "action": "incorporated-into-dynaword",
- "startTime": "2026-07-20T00:00:00Z"
- }
- ]
-}
diff --git a/examples/dfm-mimir-dynaword-folketingets-dokumenter.jsonld b/examples/dfm-mimir-dynaword-folketingets-dokumenter.jsonld
deleted file mode 100644
index 1e05c5c..0000000
--- a/examples/dfm-mimir-dynaword-folketingets-dokumenter.jsonld
+++ /dev/null
@@ -1,52 +0,0 @@
-{
- "@context": "../context/0.0.1/sourcelume.jsonld",
- "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/dynaword-folketingets-dokumenter",
- "type": "ProvenanceRecord",
- "identifier": "https://doi.org/10.5279/dk-ra-14004",
- "name": "Folketingets dokumenter (Danish parliamentary records)",
- "version": "folketingets-dokumenter@rigsarkivet-14004",
- "license": "https://creativecommons.org/licenses/by/4.0/",
- "creator": [
- {
- "type": "schema:Organization",
- "name": "Folketinget (Danish Parliament)",
- "role": "originator"
- },
- {
- "type": "schema:Organization",
- "name": "Rigsarkivet (Danish National Archives)",
- "role": "curator"
- },
- {
- "type": "schema:Organization",
- "name": "Digitaliseringsstyrelsen (Danish Agency for Digitisation)",
- "role": "curator"
- },
- {
- "type": "schema:Organization",
- "name": "Danish Foundation Models",
- "role": "distributor"
- }
- ],
- "created": "2026-09-08T00:00:00Z",
- "added": "2026-08-21T00:00:00Z",
- "contentCreated": "1848-01-01T00:00:00Z",
- "origin": "Records of the Danish Parliament (Folketinget) and former Landstinget from the 1848/49 parliamentary year through May 2026: bills, proposed resolutions, proceedings, and questions from MPs to ministers. Combines born-digital modern documents with OCR-processed historical material. Rigsarkivet published 125,406 documents as delivery 14004 (DOI 10.5279/dk-ra-14004). Personal names were pseudonymised before release; naturalisation/citizenship documents were excluded because for [...]
- "custodyChain": [
- {
- "agent": "https://www.folketingstidende.dk/da/e-folketingstidende",
- "action": "produced-and-archived",
- "startTime": "1848-01-01T00:00:00Z"
- },
- {
- "agent": "https://digidata.rigsarkivet.dk/aflevering/14004",
- "action": "published-as-delivery-14004",
- "startTime": "2026-08-21T00:00:00Z"
- },
- {
- "agent": "https://huggingface.co/datasets/danish-foundation-models/danish-dynaword",
- "action": "incorporated-into-dynaword",
- "startTime": "2026-08-21T00:00:00Z"
- }
- ]
-}
diff --git a/examples/dfm-mimir-dynaword-kalliope.jsonld b/examples/dfm-mimir-dynaword-kalliope.jsonld
deleted file mode 100644
index 36a431a..0000000
--- a/examples/dfm-mimir-dynaword-kalliope.jsonld
+++ /dev/null
@@ -1,47 +0,0 @@
-{
- "@context": "../context/0.0.1/sourcelume.jsonld",
- "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/dynaword-kalliope",
- "type": "ProvenanceRecord",
- "identifier": "https://kalliope.org/da/about/kalliope",
- "name": "Kalliope (digital library of Danish poetry)",
- "version": "kalliope@thabz-71b9deb",
- "license": "https://creativecommons.org/publicdomain/mark/1.0/",
- "creator": [
- {
- "type": "schema:Organization",
- "name": "Kalliope (digital library)",
- "role": "originator"
- },
- {
- "type": "schema:Person",
- "name": "thabz (Kalliope repository maintainer)",
- "role": "curator"
- },
- {
- "type": "schema:Organization",
- "name": "Danish Foundation Models",
- "role": "distributor"
- }
- ],
- "created": "2026-09-08T00:00:00Z",
- "added": "2026-08-22T00:00:00Z",
- "contentCreated": "1772-01-01T00:00:00Z",
- "origin": "Older Danish poetry from the Kalliope digital library, which states that its texts are public domain. DFM drew the texts from the source repository github.com/thabz/Kalliope at revision 71b9deb. Kalliope is described as a growing collection rather than a complete record of Danish poetry. 22.27K samples, 14.01M Llama-3 tokens; document created-dates range as far back as 1772.",
- "custodyChain": [
- {
- "agent": "https://kalliope.org/da/about/kalliope",
- "action": "published-public-domain-poetry",
- "startTime": "1772-01-01T00:00:00Z"
- },
- {
- "agent": "https://github.com/thabz/Kalliope/commit/71b9debf43b53a8e9c55d7bff9110f084348b9f9",
- "action": "source-repository-snapshot",
- "startTime": "2026-08-22T00:00:00Z"
- },
- {
- "agent": "https://huggingface.co/datasets/danish-foundation-models/danish-dynaword",
- "action": "incorporated-into-dynaword",
- "startTime": "2026-08-22T00:00:00Z"
- }
- ]
-}
diff --git a/examples/dfm-mimir-dynaword-ncc-parliament.jsonld b/examples/dfm-mimir-dynaword-ncc-parliament.jsonld
deleted file mode 100644
index bfeddf1..0000000
--- a/examples/dfm-mimir-dynaword-ncc-parliament.jsonld
+++ /dev/null
@@ -1,47 +0,0 @@
-{
- "@context": "../context/0.0.1/sourcelume.jsonld",
- "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/dynaword-ncc-parliament",
- "type": "ProvenanceRecord",
- "identifier": "https://huggingface.co/datasets/NbAiLab/NCC",
- "name": "Norwegian Colossal Corpus — parliament (Danish subset)",
- "version": "ncc_parliament@ncc-2021",
- "license": "https://data.norge.no/nlod/en/2.0",
- "creator": [
- {
- "type": "schema:Organization",
- "name": "Nasjonalbiblioteket (National Library of Norway)",
- "role": "originator"
- },
- {
- "type": "schema:Organization",
- "name": "NbAiLab (National Library of Norway AI Lab)",
- "role": "distributor"
- },
- {
- "type": "schema:Organization",
- "name": "Danish Foundation Models",
- "role": "curator"
- }
- ],
- "created": "2026-09-08T00:00:00Z",
- "added": "2025-05-08T00:00:00Z",
- "contentCreated": "2021-01-01T00:00:00Z",
- "origin": "Parliamentary records from the Norwegian parliament, OCR-derived, extracted from the Norwegian Colossal Corpus (NCC) published by the National Library of Norway. DFM filtered the NCC parliament subset to Danish: language classified as Danish with threshold 0.75, plus minimum-length, alpha-ratio, stop-word, and duplicate filters. The filtering reduced 9,528 parliament documents to 1,077 (1.08K samples, 338.87M Llama-3 tokens). Recurring OCR errors and historic expressions in [...]
- "custodyChain": [
- {
- "agent": "https://www.nb.no/en/",
- "action": "collected-and-digitised-parliament-records",
- "startTime": "2021-01-01T00:00:00Z"
- },
- {
- "agent": "https://huggingface.co/datasets/NbAiLab/NCC",
- "action": "published-as-norwegian-colossal-corpus",
- "startTime": "2021-01-01T00:00:00Z"
- },
- {
- "agent": "https://huggingface.co/datasets/danish-foundation-models/danish-dynaword",
- "action": "filtered-to-danish-and-incorporated-into-dynaword",
- "startTime": "2025-05-08T00:00:00Z"
- }
- ]
-}
diff --git a/examples/dfm-mimir-dynaword-retsinformationdk.jsonld b/examples/dfm-mimir-dynaword-retsinformationdk.jsonld
deleted file mode 100644
index 2ae8a26..0000000
--- a/examples/dfm-mimir-dynaword-retsinformationdk.jsonld
+++ /dev/null
@@ -1,42 +0,0 @@
-{
- "@context": "../context/0.0.1/sourcelume.jsonld",
- "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/dynaword-retsinformationdk",
- "type": "ProvenanceRecord",
- "identifier": "https://www.retsinformation.dk",
- "name": "retsinformation.dk (Danish legal information)",
- "version": "retsinformationdk@retsinformation-2025-06",
- "license": "https://www.retsinformation.dk/forms/r0710.aspx?id=164796",
- "creator": [
- {
- "type": "schema:Organization",
- "name": "Folketinget (Danish Parliament) and Danish government authorities",
- "role": "originator"
- },
- {
- "type": "schema:Organization",
- "name": "Digitaliseringsstyrelsen (Danish Agency for Digitisation)",
- "role": "distributor"
- },
- {
- "type": "schema:Organization",
- "name": "Danish Foundation Models",
- "role": "curator"
- }
- ],
- "created": "2026-09-08T00:00:00Z",
- "added": "2025-06-26T00:00:00Z",
- "contentCreated": "2025-06-25T00:00:00Z",
- "origin": "Danish legislation, administrative regulations, executive orders, preparatory works (forarbejder), historical versions, and case-law references from retsinformation.dk, the official legal information system of Denmark. Under Danish Copyright Law section 9 (lov ophavsret, retsinformation.dk/forms/r0710.aspx?id=164796), laws, administrative regulations, court decisions and similar public acts are not subject to copyright. 100.52K samples, 818.25M Llama-3 tokens.",
- "custodyChain": [
- {
- "agent": "https://www.retsinformation.dk",
- "action": "published-official-legal-information",
- "startTime": "2000-01-01T00:00:00Z"
- },
- {
- "agent": "https://huggingface.co/datasets/danish-foundation-models/danish-dynaword",
- "action": "incorporated-into-dynaword",
- "startTime": "2025-06-26T00:00:00Z"
- }
- ]
-}
diff --git a/examples/dfm-mimir-dynaword-wikipedia.jsonld b/examples/dfm-mimir-dynaword-wikipedia.jsonld
deleted file mode 100644
index 65605dc..0000000
--- a/examples/dfm-mimir-dynaword-wikipedia.jsonld
+++ /dev/null
@@ -1,47 +0,0 @@
-{
- "@context": "../context/0.0.1/sourcelume.jsonld",
- "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/dynaword-wikipedia",
- "type": "ProvenanceRecord",
- "identifier": "https://dumps.wikimedia.org/dawiki/latest/",
- "name": "Danish Wikipedia",
- "version": "wikipedia@wikimedia-dawiki-latest",
- "license": "https://creativecommons.org/licenses/by-sa/4.0/deed.en",
- "creator": [
- {
- "type": "schema:Organization",
- "name": "Wikimedia Foundation contributors (da.wikipedia.org)",
- "role": "originator"
- },
- {
- "type": "schema:Organization",
- "name": "Wikimedia Foundation",
- "role": "distributor"
- },
- {
- "type": "schema:Organization",
- "name": "Danish Foundation Models",
- "role": "curator"
- }
- ],
- "created": "2026-09-08T00:00:00Z",
- "added": "2025-08-18T00:00:00Z",
- "contentCreated": "2024-06-16T00:00:00Z",
- "origin": "The Danish subsection of Wikipedia, pulled by DFM from the latest Wikimedia database dump (dawiki) and parsed with the wtf_wikipedia javascript parser (chosen over mwparserfromhell, mediawiki_dump, and wikiextractor after empirical comparison). Text is contributed by Wikimedia editors under CC-BY-SA 4.0. 309.64K samples, 173.33M Llama-3 tokens.",
- "custodyChain": [
- {
- "agent": "https://da.wikipedia.org/",
- "action": "authored-by-editors",
- "startTime": "2001-01-01T00:00:00Z"
- },
- {
- "agent": "https://dumps.wikimedia.org/dawiki/latest/",
- "action": "published-database-dump",
- "startTime": "2025-08-18T00:00:00Z"
- },
- {
- "agent": "https://huggingface.co/datasets/danish-foundation-models/danish-dynaword",
- "action": "parsed-and-incorporated-into-dynaword",
- "startTime": "2025-08-18T00:00:00Z"
- }
- ]
-}
diff --git a/examples/dfm-mimir-hrm-text-data-io-cleaned.jsonld b/examples/dfm-mimir-hrm-text-data-io-cleaned.jsonld
new file mode 100644
index 0000000..13bfa27
--- /dev/null
+++ b/examples/dfm-mimir-hrm-text-data-io-cleaned.jsonld
@@ -0,0 +1,42 @@
+{
+ "@context": "../context/0.0.1/sourcelume.jsonld",
+ "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/hrm-text-data-io-cleaned",
+ "type": "ProvenanceRecord",
+ "identifier": "https://huggingface.co/datasets/sapientinc/HRM-Text-data-io-cleaned-20260515",
+ "name": "HRM-Text-data-io-cleaned-20260515",
+ "version": "20260515",
+ "license": "https://opendatacommons.org/licenses/by/1-0/",
+ "creator": [
+ {
+ "type": "schema:Person",
+ "name": "Guan Wang (sapientinc)",
+ "role": "originator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "sapientinc",
+ "role": "curator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "sapientinc",
+ "role": "distributor"
+ }
+ ],
+ "created": "2026-09-08T00:00:00Z",
+ "added": "2026-05-15T00:00:00Z",
+ "contentCreated": "2024-01-01T00:00:00Z",
+ "origin": "Pre-built HRM-Text pretraining dataset produced from raw data using the data_io cleaning scripts (github.com/sapientinc/data_io). English, size category 100B<n<1T rows. Consists mainly of Flan, Platypus, and tasksource sub-collections (per the Mimir technical report's description of the Sapient mixed collection). The largest single source in the Mimir corpus: 11.92B sampled tokens per epoch (16.91% of the corpus). Form: curated + reformatted. Described in the HRM-Text paper [...]
+ "custodyChain": [
+ {
+ "agent": "https://github.com/sapientinc/data_io",
+ "action": "cleaned-raw-data-into-hrm-text-pretraining-set",
+ "startTime": "2026-05-15T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/sapientinc",
+ "action": "published-hrm-text-data-io-cleaned-20260515",
+ "startTime": "2026-05-15T00:00:00Z"
+ }
+ ]
+}
diff --git a/examples/dfm-mimir-laerebogen.jsonld b/examples/dfm-mimir-laerebogen.jsonld
new file mode 100644
index 0000000..ed0aa2c
--- /dev/null
+++ b/examples/dfm-mimir-laerebogen.jsonld
@@ -0,0 +1,52 @@
+{
+ "@context": "../context/0.0.1/sourcelume.jsonld",
+ "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/laerebogen",
+ "type": "ProvenanceRecord",
+ "identifier": "https://huggingface.co/datasets/danish-foundation-models/laerebogen",
+ "name": "Lærebogen",
+ "version": "2026",
+ "license": "https://www.apache.org/licenses/LICENSE-2.0",
+ "creator": [
+ {
+ "type": "schema:Person",
+ "name": "Dan Saattrup Smart (saattrupdan)",
+ "role": "originator"
+ },
+ {
+ "type": "schema:Person",
+ "name": "Sofie Helene Bruun (sofiehb)",
+ "role": "originator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "Alexandra Institute",
+ "role": "curator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "Danish Foundation Models",
+ "role": "distributor"
+ }
+ ],
+ "created": "2026-09-08T00:00:00Z",
+ "added": "2026-01-01T00:00:00Z",
+ "contentCreated": "2026-01-01T00:00:00Z",
+ "origin": "A Danish instruction-following dataset of 5 million multi-turn conversation examples, generated synthetically with Qwen3-235B-A22B-Instruct-2507-FP8. Pipeline: 176 manual Danish seed prompts (adapted from English Self-Instruct seeds and EU Horizon TrustLLM crowdsourced prompts) -> Alpaca-recipe expansion to 1M with structured generation and MinHash deduplication -> grammar correction/translation to Danish -> quality-improvement rewrite -> 4 generations of Evol-Instruct -> ad [...]
+ "custodyChain": [
+ {
+ "agent": "https://alexandra.dk",
+ "action": "authored-seed-prompts-and-generated-dataset",
+ "startTime": "2026-01-01T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507-FP8",
+ "action": "generated-and-evolved-instructions",
+ "startTime": "2026-01-01T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/danish-foundation-models",
+ "action": "published-on-hugging-face",
+ "startTime": "2026-01-01T00:00:00Z"
+ }
+ ]
+}
diff --git a/examples/dfm-mimir-lexdk-articles.jsonld b/examples/dfm-mimir-lexdk-articles.jsonld
new file mode 100644
index 0000000..70d69cb
--- /dev/null
+++ b/examples/dfm-mimir-lexdk-articles.jsonld
@@ -0,0 +1,37 @@
+{
+ "@context": "../context/0.0.1/sourcelume.jsonld",
+ "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/lexdk-articles",
+ "type": "ProvenanceRecord",
+ "identifier": "https://sourcelume.apache.org/records/dfm-mimir-v1/lexdk-articles",
+ "name": "Lex.dk articles (agreement-supplied)",
+ "version": "2026",
+ "license": "https://sourcelume.apache.org/ns#agreement-supplied",
+ "creator": [
+ {
+ "type": "schema:Organization",
+ "name": "Lex.dk (Danish encyclopedia)",
+ "role": "originator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "Danish Foundation Models",
+ "role": "curator"
+ }
+ ],
+ "created": "2026-09-08T00:00:00Z",
+ "added": "2026-01-01T00:00:00Z",
+ "contentCreated": "2000-01-01T00:00:00Z",
+ "origin": "Agreement-supplied Danish encyclopedia article corpus from Lex.dk, provided to the Danish Foundation Models project under a Danish Foundation Model agreement whose licensing does not permit public sharing. No Hugging Face identifier exists; the data is not publicly redistributed. Per the Mimir technical report, DFM uses such agreement-supplied datasets to derive synthetic instruction-tuning data rather than training directly on the data. Contributed 313M sampled tokens per e [...]
+ "custodyChain": [
+ {
+ "agent": "https://lex.dk/",
+ "action": "supplied-to-dfm-under-agreement",
+ "startTime": "2026-01-01T00:00:00Z"
+ },
+ {
+ "agent": "https://www.foundationmodels.dk",
+ "action": "held-and-derived-synthetic-instruction-data",
+ "startTime": "2026-01-01T00:00:00Z"
+ }
+ ]
+}
diff --git a/examples/dfm-mimir-opus-da-en-permissive.jsonld b/examples/dfm-mimir-opus-da-en-permissive.jsonld
new file mode 100644
index 0000000..22e22b8
--- /dev/null
+++ b/examples/dfm-mimir-opus-da-en-permissive.jsonld
@@ -0,0 +1,42 @@
+{
+ "@context": "../context/0.0.1/sourcelume.jsonld",
+ "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/opus-da-en-permissive",
+ "type": "ProvenanceRecord",
+ "identifier": "https://huggingface.co/datasets/schneiderkamplab/opus-da-en-permissive",
+ "name": "OPUS da-en permissive subset",
+ "version": "20260515",
+ "license": "https://opus.nlpl.eu/",
+ "creator": [
+ {
+ "type": "schema:Organization",
+ "name": "OPUS (Tiedemann, J. — University of Helsinki / opus.nlpl.eu) — original corpora aggregator",
+ "role": "originator"
+ },
+ {
+ "type": "schema:Person",
+ "name": "Peter Schneider-Kamp (schneiderkamplab)",
+ "role": "curator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "schneiderkamplab",
+ "role": "distributor"
+ }
+ ],
+ "created": "2026-09-08T00:00:00Z",
+ "added": "2026-05-15T00:00:00Z",
+ "contentCreated": "1990-01-01T00:00:00Z",
+ "origin": "A processed Danish-English sentence-pair subset harvested from OPUS (opus.nlpl.eu), selected for permissive Creative Commons or public-domain-style licensing and converted to JSONL.GZ for DFM/HRM-Text training. Schema: {id, da, en, source}. Sources include DGT (EU Directorate-General for Translation memory, e.g. EU treaties), among other OPUS da-en corpora. Intended to preserve the exact rows used by the local DFM data pipeline so DFM8 can be rebuilt without local-only files [...]
+ "custodyChain": [
+ {
+ "agent": "https://opus.nlpl.eu/",
+ "action": "aggregated-and-published-opus-corpora",
+ "startTime": "2012-01-01T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/schneiderkamplab",
+ "action": "filtered-to-permissive-subset-and-converted-to-jsonlg",
+ "startTime": "2026-05-15T00:00:00Z"
+ }
+ ]
+}
diff --git a/examples/dfm-mimir-wiki-instruct-da.jsonld b/examples/dfm-mimir-wiki-instruct-da.jsonld
new file mode 100644
index 0000000..bc6d1ea
--- /dev/null
+++ b/examples/dfm-mimir-wiki-instruct-da.jsonld
@@ -0,0 +1,42 @@
+{
+ "@context": "../context/0.0.1/sourcelume.jsonld",
+ "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/wiki-instruct-da",
+ "type": "ProvenanceRecord",
+ "identifier": "https://huggingface.co/datasets/synquid/wiki-instruct-da",
+ "name": "wiki-instruct-da",
+ "version": "20260515",
+ "license": "https://en.wikipedia.org/wiki/Wikipedia:Text_of_the_Creative_Commons_Attribution-ShareAlike_4.0_Unported_License",
+ "creator": [
+ {
+ "type": "schema:Organization",
+ "name": "Wikimedia Foundation contributors (da.wikipedia.org) — original text",
+ "role": "originator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "synquid",
+ "role": "curator"
+ },
+ {
+ "type": "schema:Organization",
+ "name": "synquid",
+ "role": "distributor"
+ }
+ ],
+ "created": "2026-09-08T00:00:00Z",
+ "added": "2026-05-15T00:00:00Z",
+ "contentCreated": "2001-01-01T00:00:00Z",
+ "origin": "Danish instruction-following pairs derived from Danish Wikipedia, where prompts ask for a short encyclopedic article and the target is the corresponding Wikipedia text. The underlying Danish Wikipedia content is licensed CC-BY-SA 4.0 (Wikimedia). A same-named sub-source (190.21M tokens, license listed as Unknown) also appears inside the dfm-dyna-instruct collection; the standalone synquid/wiki-instruct-da dataset is the larger, separately-versioned release. Form: reformatted [...]
+ "custodyChain": [
+ {
+ "agent": "https://da.wikipedia.org/",
+ "action": "authored-by-editors-under-cc-by-sa-4-0",
+ "startTime": "2001-01-01T00:00:00Z"
+ },
+ {
+ "agent": "https://huggingface.co/synquid",
+ "action": "derived-instruction-pairs-and-published",
+ "startTime": "2026-05-15T00:00:00Z"
+ }
+ ]
+}
diff --git a/examples/minimal-record.jsonld b/examples/minimal-record.jsonld
new file mode 100644
index 0000000..8641645
--- /dev/null
+++ b/examples/minimal-record.jsonld
@@ -0,0 +1,25 @@
+{
+ "@context": "../context/0.0.1/sourcelume.jsonld",
+ "id": "https://sourcelume.apache.org/records/minimal-example",
+ "type": "ProvenanceRecord",
+ "identifier": "https://example.org/datasets/minimal-example",
+ "name": "Minimal example record",
+ "license": "https://www.apache.org/licenses/LICENSE-2.0",
+ "creator": [
+ {
+ "type": "schema:Organization",
+ "name": "Example Organization"
+ }
+ ],
+ "created": "2026-09-08T00:00:00Z",
+ "added": "2026-09-08T00:00:00Z",
+ "contentCreated": "2026-09-08T00:00:00Z",
+ "origin": "A minimal conformant ProvenanceRecord demonstrating the smallest valid 0.0.1 shape: one creator, one license, one origin statement, and a single custody-chain entry.",
+ "custodyChain": [
+ {
+ "agent": "https://example.org/",
+ "action": "created",
+ "startTime": "2026-09-08T00:00:00Z"
+ }
+ ]
+}
diff --git a/notes/mimir-dynaword-sources.md b/notes/mimir-dynaword-sources.md
index 72c4435..ab41590 100644
--- a/notes/mimir-dynaword-sources.md
+++ b/notes/mimir-dynaword-sources.md
@@ -1,162 +1,81 @@
-# Mímir (DFM) training corpus — dynaword source manifest
-
-This is a working manifest of the constituent datasets that make up the
-Danish Dynaword corpus, against which the DFM Mímir model is trained. It is
-used to track authoring of `ProvenanceRecord` examples in this repository.
-
-Source: <https://huggingface.co/datasets/danish-foundation-models/danish-dynaword>
-(version 1.2.23 at time of writing). Dynaword is continually developed; sizes
-and even membership change over time. Pin a revision for reproducibility.
-
-The corpus totals 9.81B tokens (Llama 3) across 48 sources. The license below
-is applied to the constituent text; the collection (metadata, quality
-control) is CC-0. Each row's datasheet lives at
-`data/<source>/<source>.md` under the dynaword repo — that datasheet is the
-authoritative source for originator/curator/custody detail when authoring a
-`ProvenanceRecord` for that source.
-
-## License families and representative picks
-
-Eight license classes appear across the 48 sources. We author one
-representative `ProvenanceRecord` per class first, to prove the 0.0.1 schema
-handles every license dynaword uses. The other 40 sources are authored in a
-follow-on pass using the same conventions.
-
-| License class | IRI / note | Picked source | Tokens | Domain | Datasheet |
-|-----------------------------------------|-------------------------------------------------|--------------------------|----------|--------|----------|
-| CC-0 (public domain dedication) | https://creativecommons.org/publicdomain/zero/1.0/legalcode.en | adl | 58.49M | Books | data/adl/adl.md |
-| CC-BY 4.0 | https://creativecommons.org/licenses/by/4.0/deed.en | folketingets-dokumenter | 2.81B | Other | data/folketingets-dokumenter/folketingets-dokumenter.md |
-| CC-BY-SA 4.0 | https://creativecommons.org/licenses/by-sa/4.0/deed.en | wikipedia | 173.33M | Encyclopedic | data/wikipedia/wikipedia.md |
-| Public domain (by expiration, not CC-0) | CC Public Domain Mark — see spec prose convention | kalliope | 14.01M | Books | data/kalliope/kalliope.md |
-| Danish Copyright Law (state edict) | No standard IRI — documented convention | retsinformationdk | 818.25M | Legal | data/retsinformationdk/retsinformationdk.md |
-| NLOD 2.0 (Norwegian Licence for Open Government Data) | https://data.norge.no/nlod/en/2.0 | ncc_parliament | 338.87M | Other | data/ncc_parliament/ncc_parliament.md |
-| Apache 2.0 | https://www.apache.org/licenses/LICENSE-2.0 | ai-aktindsigt | 139.23M | Web | data/ai-aktindsigt/ai-aktindsigt.md |
-| MIT | https://opensource.org/license/mit | dakultur | 5.49K | Conversation | data/dakultur/dakultur.md |
-
-danish-pd (already authored, CC Public Domain Mark) overlaps the "Public
-domain" row; kalliope is a second PD source but by a different mechanism
-(explicit PD release of poetry), so it still serves as the representative
-for the PD-by-means-other-than-CC-0 case.
-
-## Full source list (48)
-
-Token counts are Llama-3 tokens. "Datasheet" column is the relative path
-within the dynaword repo.
-
-### Other (domain) — 3.22B
-
-| Source | Tokens | License | Datasheet |
-|---|---|---|---|
-| ncc_parliament | 338.87M | NLOD 2.0 | data/ncc_parliament/ncc_parliament.md |
-| dannet | 1.48M | DanNet 1.0 (custom) | data/dannet/dannet.md |
-| depbank | 185.45K | CC-BY-SA 4.0 | data/depbank/depbank.md |
-| synne | 52.02K | CC-0 | data/synne/synne.md |
-| historical-danish-handwriting | 5.20M | CC-BY 4.0 | data/historical-danish-handwriting/historical-danish-handwriting.md |
-| kb_historical_letters | 14.75M | CC-0 | data/kb_historical_letters/kb_historical_letters.md |
-| tidsskrift-dk | 50.03M | CC-BY 4.0 | data/tidsskrift-dk/tidsskrift-dk.md |
-| folketingets-dokumenter | 2.81B | CC-BY 4.0 | data/folketingets-dokumenter/folketingets-dokumenter.md |
-
-### Legal (domain) — 3.18B
-
-| Source | Tokens | License | Datasheet |
-|---|---|---|---|
-| cellar | 1.15B | CC-BY-SA 4.0 | data/cellar/cellar.md |
-| eur-lex-sum-da | 31.37M | CC-BY-SA 4.0 | data/eur-lex-sum-da/eur-lex-sum-da.md |
-| fm-udgivelser | 50.34M | CC-BY-SA 4.0 | data/fm-udgivelser/fm-udgivelser.md |
-| retsinformationdk | 818.25M | Danish Copyright Law | data/retsinformationdk/retsinformationdk.md |
-| skat | 122.11M | CC-0 | data/skat/skat.md |
-| retspraksis | 56.26M | CC-0 | data/retspraksis/retspraksis.md |
-| domsdatabasen | 86.35M | Danish Copyright Law | data/domsdatabasen/domsdatabasen.md |
-| kb_administrative_publication | 844.58M | CC-0 | data/kb_administrative_publication/kb_administrative_publication.md |
-| municipality_meetings | 22.64M | CC-0 | data/municipality_meetings/municipality_meetings.md |
-
-### News (domain) — 1.09B
-
-| Source | Tokens | License | Datasheet |
-|---|---|---|---|
-| enevaeldens_nyheder | 1.03B | CC-BY-SA 4.0 | data/enevaeldens_nyheder/enevaeldens_nyheder.md |
-| ncc_newspaper | 1.05M | CC-0 | data/ncc_newspaper/ncc_newspaper.md |
-| tv2r | 21.67M | CC-BY-SA 4.0 | data/tv2r/tv2r.md |
-| nordjyllandnews | 37.90M | CC-0 | data/nordjyllandnews/nordjyllandnews.md |
-
-### Books (domain) — 747.93M
-
-| Source | Tokens | License | Datasheet |
-|---|---|---|---|
-| grundtvig | 10.53M | CC-0 | data/grundtvig/grundtvig.md |
-| ncc_books | 531.97M | CC-0 | data/ncc_books/ncc_books.md |
-| memo | 113.74M | CC-BY-SA 4.0 | data/memo/memo.md |
-| adl | 58.49M | CC-0 | data/adl/adl.md |
-| wikibooks | 7.63M | CC-BY-SA 4.0 | data/wikibooks/wikibooks.md |
-| jvj | 3.55M | CC-BY-SA 4.0 | data/jvj/jvj.md |
-| gutenberg | 6.76M | Gutenberg (custom) | data/gutenberg/gutenberg.md |
-| relig | 1.24M | CC-0 | data/relig/relig.md |
-| kalliope | 14.01M | Public domain | data/kalliope/kalliope.md |
-
-### Conversation (domain) — 497.11M
-
-| Source | Tokens | License | Datasheet |
-|---|---|---|---|
-| danske-taler | 8.72M | CC-0 | data/danske-taler/danske-taler.md |
-| opensubtitles | 271.60M | CC-0 | data/opensubtitles/opensubtitles.md |
-| ep | 100.84M | CC-0 | data/ep/ep.md |
-| ft | 114.09M | CC-0 | data/ft/ft.md |
-| spont | 1.56M | CC-0 | data/spont/spont.md |
-| naat | 286.68K | CC-0 | data/naat/naat.md |
-| dakultur | 5.49K | MIT | data/dakultur/dakultur.md |
-| mosel_youtubecommons | 7.09K | CC-BY 4.0 | data/mosel_youtubecommons/mosel_youtubecommons.md |
-
-### Social Media (domain) — 389.32M
-
-| Source | Tokens | License | Datasheet |
-|---|---|---|---|
-| hest | 389.32M | CC-0 | data/hest/hest.md |
-
-### Web (domain) — 295.93M
-
-| Source | Tokens | License | Datasheet |
-|---|---|---|---|
-| ai-aktindsigt | 139.23M | Apache 2.0 | data/ai-aktindsigt/ai-aktindsigt.md |
-| ncc_maalfrid | 29.26M | NLOD 2.0 | data/ncc_maalfrid/ncc_maalfrid.md |
-| miljoeportalen | 127.38M | CC-0 | data/miljoeportalen/miljoeportalen.md |
-| hvadvilduhelst | 57.22K | CC-BY 4.0 | data/hvadvilduhelst/hvadvilduhelst.md |
-
-### Encyclopedic (domain) — 185.75M
-
-| Source | Tokens | License | Datasheet |
-|---|---|---|---|
-| wikisource | 6.28M | CC-BY-SA 4.0 | data/wikisource/wikisource.md |
-| wikipedia | 173.33M | CC-BY-SA 4.0 | data/wikipedia/wikipedia.md |
-| wiki-comments | 6.14M | CC-BY-SA 4.0 | data/wiki-comments/wiki-comments.md |
-
-### Speeches (domain) — 161.33M
-
-| Source | Tokens | License | Datasheet |
-|---|---|---|---|
-| mosel_voxpopuli | 161.33M | CC-BY 4.0 | data/mosel_voxpopuli/mosel_voxpopuli.md |
-
-### Medical (domain) — 27.07M
-
-| Source | Tokens | License | Datasheet |
-|---|---|---|---|
-| health_hovedstaden | 27.07M | CC-0 | data/health_hovedstaden/health_hovedstaden.md |
-
-### Readaloud (domain) — 7.30M
-
-| Source | Tokens | License | Datasheet |
-|---|---|---|---|
-| nota | 7.30M | CC-0 | data/nota/nota.md |
-
-### Dialect (domain) — 847.97K
-
-| Source | Tokens | License | Datasheet |
-|---|---|---|---|
-| botxt | 847.97K | CC-0 | data/botxt/botxt.md |
-
-## Note on danish-pd
-
-`examples/dfm-mimir-danish-pd-best-effort.jsonld` is already authored. It
-is not itself a dynaword source — it is the upstream corpus that PleIAs
-aggregated and that DFM drew from for the public-domain portion of
-dynaword. It is included here as the reference record shape and as the
-Public-Domain-Mark representative.
+# DFM Mímir v1 — direct training-input manifest
+
+This is a working manifest of the datasets that the DFM Mímir v1 model is
+*directly* trained on, against which we author `ProvenanceRecord` examples in
+this repository.
+
+**Corrected understanding (2026-09-08).** An earlier version of this manifest
+described the 48 constituent sources of the
+[`danish-foundation-models/danish-dynaword`](https://huggingface.co/datasets/danish-foundation-models/danish-dynaword)
+corpus as if they were Mímir's training data. That was wrong. Mímir does **not**
+train on raw dynaword. Per the Mímir v1 technical report (arXiv 2608.13517,
+Appendix A, Table 10), Mímir is trained on **161 datasets**; dynaword enters
+the corpus only indirectly, through four synthetic derivatives published under
+`schneiderkamplab/danish-dynaword-*` (denoising, span-filling,
+prefix-continuation, paragraph-reordering), which are derived from dynaword
+text and judge-filtered with `google/gemma-4-31B-it`.
+
+Dynaword itself (9.81B Llama-3 tokens across 48 sources, v1.2.23) remains a
+useful upstream-of-Mímir corpus, but its 48 sub-sources are **not** direct
+Mímir training inputs and are out of scope for this branch's example records.
+
+## The 12 authored direct-input records
+
+Each row below has a corresponding `examples/dfm-mimir-*.jsonld` record in this
+repository. Token counts and shares are sampled tokens per epoch, from Table 10
+of the technical report.
+
+| # | Dataset (HF id) | Form | Tokens/epoch | Share | Record |
+|---|---|---|---|---|---|
+| 1 | `sapientinc/HRM-Text-data-io-cleaned-20260515` | Curated + reformatted | 11.92B | 16.91% | `dfm-mimir-hrm-text-data-io-cleaned.jsonld` |
+| 2 | `danish-foundation-models/laerebogen` | Reformatted | 8.32B | 11.81% | `dfm-mimir-laerebogen.jsonld` |
+| 5 | `danish-foundation-models/dfm-dyna-instruct` | Reformatted | 3.54B | 5.03% | `dfm-mimir-dfm-dyna-instruct.jsonld` |
+| 7 | `schneiderkamplab/opus-da-en-permissive` | Reformatted | 2.90B | 4.12% | `dfm-mimir-opus-da-en-permissive.jsonld` |
+| 16 | `synquid/wiki-instruct-da` | Reformatted | 988M | 1.40% | `dfm-mimir-wiki-instruct-da.jsonld` |
+| 17 | `schneiderkamplab/dfm8-openhermes-da` | Translated + audited | 922M | 1.31% | `dfm-mimir-dfm8-openhermes-da.jsonld` |
+| 34 | DBC (agreement-supplied) | Agreement-supplied | 356M | 0.505% | `dfm-mimir-dbc.jsonld` |
+| 35 | `schneiderkamplab/danish-dynaword-denoising` | Synthetic + audited | 323M | 0.459% | `dfm-mimir-danish-dynaword-denoising.jsonld` |
+| 36 | Lex.dk articles (agreement-supplied) | Agreement-supplied | 313M | 0.445% | `dfm-mimir-lexdk-articles.jsonld` |
+| 38 | `schneiderkamplab/danish-dynaword-prefix-continuation` | Synthetic + audited | 252M | 0.357% | `dfm-mimir-danish-dynaword-prefix-continuation.jsonld` |
+| 39 | `schneiderkamplab/danish-dynaword-span-filling` | Synthetic + audited | 251M | 0.356% | `dfm-mimir-danish-dynaword-span-filling.jsonld` |
+| 49 | `schneiderkamplab/danish-dynaword-paragraph-reordering` | Synthetic + audited | 160M | 0.228% | `dfm-mimir-danish-dynaword-paragraph-reordering.jsonld` |
+
+These 12 cover: the English pretraining core (sapientinc), the three largest
+Danish instruction sources (laerebogen, dfm-dyna-instruct, opus), the
+Danish-Wikipedia-derived instruction set, the translated OpenHermes derivative,
+both agreement-supplied sources, and all four dynaword-derived synthetic task
+datasets. Together they represent ~41% of the Mímir corpus by sampled tokens
+and span every "form" category the report uses for Danish-relevant data.
+
+## The remaining 149 datasets
+
+The other 149 entries in Table 10 are predominantly English
+instruction/reasoning/tool-use corpora (OpenMathInstruct, Nemotron, Dolci,
+Tulu, etc.) and a long tail of small `schneiderkamplab/sapient-synth-*` and
+`oliverkinch/*` Danish datasets. They are not authored as records in this pass.
+The full 161-entry table is reproduced in Appendix A of the Mímir v1 technical
+report (arXiv 2608.13517) and is the authoritative source for any future
+record authoring.
+
+## Dynaword's sub-sources (out of scope, kept for reference)
+
+The 48 constituent sources of `danish-foundation-models/danish-dynaword`
+(adl, folketingets-dokumenter, wikipedia, kalliope, retsinformationdk, ncc_parliament,
+ai-aktindsigt, dakultur, and 40 others) are upstream of dynaword and therefore
+*upstream-of-upstream* of Mímir. They are not direct Mímir training inputs.
+Their per-source datasheets live at `data/<source>/<source>.md` under the
+dynaword repo. If a future crosswalk or registry query needs to trace Mímir's
+Danish text back to its ultimate origin, those sub-sources would each warrant
+their own `ProvenanceRecord` linked from the four `danish-dynaword-*`
+derivative records' `custodyChain` entries — but that is a larger, separate
+effort, not part of this 0.0.1 teaching artifact.
+
+## License vocabulary used across the 12 records
+
+| License | IRI | Records |
+|---|---|---|
+| Apache 2.0 | https://www.apache.org/licenses/LICENSE-2.0 | laerebogen, 4x danish-dynaword-* derivatives, dfm8-openhermes-da |
+| ODC-By 1.0 | https://opendatacommons.org/licenses/by/1-0/ | hrm-text-data-io-cleaned, dfm-dyna-instruct (dominant sub-source) |
+| CC-BY-SA 4.0 (Wikipedia) | https://creativecommons.org/licenses/by-sa/4.0/deed.en | wiki-instruct-da (underlying text) |
+| OPUS aggregated (mixed permissive) | https://opus.nlpl.eu/ | opus-da-en-permissive |
+| Agreement-supplied (interim convention) | https://sourcelume.apache.org/ns#agreement-supplied | DBC, Lex.dk articles |
diff --git a/notes/stress-test-dfm-mimir.md b/notes/stress-test-dfm-mimir.md
new file mode 100644
index 0000000..94b87cb
--- /dev/null
+++ b/notes/stress-test-dfm-mimir.md
@@ -0,0 +1,128 @@
+# Stress test: DFM Mímir v1 training data vs. Sourcelume 0.0.1 schema
+
+Status: **Findings from an internal stress test, not yet sent to the list.**
+Gaps 1–3 below were folded directly into `0.0.1` (which had not yet been released, so nothing
+needed a version bump) rather than deferred — see the resolution notes on each. Gap 4 remains an
+open scope question that could be raised on `dev@sourcelume.apache.org` if/when we choose to.
+
+## What we stress-tested
+
+[DFM Mímir v1](https://huggingface.co/danish-foundation-models/DFM-Mimir) is a 1B-parameter
+Danish/English Hierarchical Reasoning Model (HRM-Text) trained from scratch on **161 datasets**
+that the project states are "permissible and openly licensed, made available by agreement, or
+allowed by the EU text-and-data-mining exception for research institutions" (Mímir v1 technical
+report, arXiv 2608.13517). Mímir is an unusually good fit for Sourcelume's purpose: its entire
+value proposition is *verifiable, ethically-sourced training data*, yet today all of that
+provenance and licensing information lives as human-readable prose — in the arXiv report's
+Appendix A (Table 10, the full 161-dataset manifest), the Hugging Face model card, and per-dataset
+datasheets. None of it is machine-checkable.
+
+We took a real source dataset from Mímir's training corpus —
+[`schneiderkamplab/danish-dynaword-denoising`](https://huggingface.co/datasets/schneiderkamplab/danish-dynaword-denoising)
+(323M sampled tokens/epoch, 0.46% of the corpus) — and attempted to express it as a Sourcelume
+`ProvenanceRecord` under the current 0.0.1 schema. This dataset is one of four synthetic
+derivatives that the Mímir project derives from the
+[`danish-foundation-models/danish-dynaword`](https://huggingface.co/datasets/danish-foundation-models/danish-dynaword)
+raw text corpus (the others being `-span-filling`, `-prefix-continuation`, and
+`-paragraph-reordering`); Mímir trains on the derivatives, not on raw dynaword directly.
+
+## What worked
+
+A structurally valid 0.0.1 record can be produced
+([`examples/dfm-mimir-danish-dynaword-denoising.jsonld`](../examples/dfm-mimir-danish-dynaword-denoising.jsonld)).
+It passes both JSON Schema and SHACL validation, and, after the fixes below, records the
+identifiable custody hops (dynaword source-corpus provision, Gemma-4 judging of generated rows,
+schneiderkamplab publication of accepted rows) and role-tagged creators (DFM/dynaword as
+`originator`, Peter Schneider-Kamp as `curator`, schneiderkamplab as `distributor`).
+
+## What the schema could not represent
+
+Four concrete gaps surfaced against this one real dataset. Each would have been hypothetical if
+we had only inspected the 0.0.1 field list; mapping a real datasheet made them immediate.
+
+1. **"Agreement-supplied" has no clean license IRI.** Two of the 161 Mímir datasets — "DBC
+ (agreement-supplied)" and "Lex.dk articles" — are provided to the Danish Foundation Models
+ project under agreements whose licensing does not permit public sharing. SPDX has no term
+ for *agreement-bound, non-redistributable data*. Our best-effort records use an interim
+ Sourcelume convention IRI (`https://sourcelume.apache.org/ns#agreement-supplied`) as a
+ placeholder, which is pragmatic but not a precise license term.
+ **Resolved for 0.0.1:** documented this as the interim convention in `spec/0.0.1/index.md`'s
+ `license` field description, pending a more precise term if/when the list takes it up — not a
+ schema change, since inventing a `sourcelume:`-native license term without list consensus would
+ be exactly the kind of thing to raise there first.
+
+2. **Single `custodyEvent` collapses a multi-hop chain.** This dataset actually has at least three
+ custodians: the original text sources aggregated by dynaword (e.g. ADL, ai-aktindsigt, botxt,
+ cellar — each a separate corpus), the DFM dynaword corpus (which aggregated them), and
+ schneiderkamplab (which generated the synthetic denoising task and judged rows with
+ Gemma-4-31B-it before publishing accepted rows). The single-`custodyEvent` shape could record
+ only the most recent hop and silently dropped the earlier ones.
+ **Resolved for 0.0.1:** `custodyEvent` (object) replaced with `custodyChain` (array, min 1).
+ The updated example now records the dynaword-provision and schneiderkamplab-publication hops;
+ the underlying original authors/rightsholders (per dynaword sub-source) still can't be recorded
+ as a hop within *this* derivative's record since they have no single identifiable agent or
+ timestamp at this scale — a data-availability limit, not a schema one. Those sub-sources, if
+ needed, would each warrant their own `ProvenanceRecord` linked from this one's `custodyChain`.
+
+3. **`creator` conflates originator and curator.** `creator` was a single object. For
+ `danish-dynaword-denoising` there are at least three distinct roles: DFM/dynaword (the original
+ corpus originator), Peter Schneider-Kamp/schneiderkamplab (the curator who derived and
+ audited the synthetic task), and schneiderkamplab (the distributor on Hugging Face). The
+ single-object shape could not distinguish these roles.
+ **Resolved for 0.0.1:** `creator` is now an array (min 1) of parties, each with an optional
+ `role` (`originator`/`curator`/`distributor`). The updated example tags DFM/dynaword as
+ `originator`, Peter Schneider-Kamp as `curator`, and schneiderkamplab as `distributor`. Full
+ alignment with a richer provenance vocabulary (PROV) was considered and deferred — the
+ three-value enum covers this dataset's needs without committing to PROV's larger model.
+
+4. **No place for downstream usage or audit metadata.** Mímir's technical report documents a
+ formal memorisation audit across four data risk categories (agreement-backed synthetic,
+ opt-out-uncertain, high-confidence-no-opt-out, low-risk synthetic/reasoning). This is exactly
+ the kind of claim Sourcelume should make checkable, but it belongs to the *model producer's*
+ view of the dataset, not the *dataset producer's* record. The 0.0.1 schema has no field for
+ it. This raises a scope question: should `ProvenanceRecord` stay dataset-centric
+ (and let model usage/audits live in a separate, later type), or should it grow a `usage: [...]`
+ block recording how specific models consumed and audited the dataset?
+
+## Bug found in our own validator
+
+While building the stress-test records, we discovered that the JSON Schema validator
+(NetworkNT `json-schema-validator`, used in an earlier Java-based validator) was **not enforcing
+`format: uri`** on the `license` field by default — JSON Schema 2020-12 treats `format` as
+annotation-only unless assertion mode is explicitly enabled. A record with
+`"license": "Agreement-supplied"` (a plain string, not a URI) silently passed validation. Fix
+applied at the time in `ValidateExamplesTest.java`:
+`SchemaValidatorsConfig.builder().formatAssertionsEnabled(true)`. With the fix, the invalid
+record correctly failed with `/license: does not match the uri pattern`.
+
+Validation tooling has since moved to Python (`tools/validate.py`, see `SETUP.md`), which has
+the analogous gap: the `jsonschema` library also treats `format` as non-enforcing unless a
+format checker is supplied. `tools/validate.py` passes
+`format_checker=Draft202012Validator.FORMAT_CHECKER` and `pyproject.toml` depends on
+`jsonschema[format]` (which pulls in `rfc3987`) specifically to keep this check active — verified
+by re-running the same broken-license case against the Python validator.
+
+A separate, deeper issue remains: even with format assertions on, JSON-LD's `@type: @id`
+coercion resolves *any* non-absolute string as a relative IRI against the document base, so a
+plain string like `"Agreement-supplied"` still becomes *some* absolute IRI at the RDF level.
+SHACL's `sh:nodeKind sh:IRI` therefore cannot catch "the author typed words instead of a URI."
+Real license validation likely needs an enum/pattern against a known vocabulary (e.g. the SPDX
+license list), not just a URI-shape check.
+
+## Open questions (not yet raised on the list)
+
+- **Usage/audit scope decision** — decide whether `ProvenanceRecord` stays dataset-centric or
+ grows a `usage` block; if the former (current default in `spec/0.0.1/index.md`'s non-goals),
+ schedule a separate model-audit record type for a later version.
+- **License vocabulary beyond the interim convention** — the agreement-supplied and
+ public-domain conventions above are stopgaps; consider a proper enum/pattern against a known
+ license list (e.g. SPDX) so `format: uri` isn't the only check, per the deeper JSON-LD coercion
+ issue noted below.
+
+## Inputs used
+
+- `examples/dfm-mimir-danish-dynaword-denoising.jsonld` — the conformant best-effort record.
+- Mímir v1 technical report, arXiv 2608.13517 (Appendix A, Table 10 — the 161-dataset manifest).
+- `schneiderkamplab/danish-dynaword-denoising` README on Hugging Face (license, row counts,
+ source files, judge model).
+- DFM-Mimir Hugging Face model card.
diff --git a/spec/0.0.1/index.md b/spec/0.0.1/index.md
index 05755b5..a553c9c 100644
--- a/spec/0.0.1/index.md
+++ b/spec/0.0.1/index.md
@@ -7,7 +7,7 @@ the minimal viable shape: enough to publish a checkable claim about one dataset'
or more role-tagged creators, its license, its origin, and an ordered custody chain. It
intentionally leaves out cryptographic attestation and downstream usage/audit metadata — those
are planned for later, once Sourcelume Attest exists and once the dataset-vs-model-usage scope
-question is settled on the dev list (see `notes/stress-test-dfm-mimir.md`).
+question is settled (see `notes/stress-test-dfm-mimir.md`).
## Purpose
@@ -25,7 +25,7 @@ claim's accuracy — it gives the claim a consistent shape so it *can* be verifi
| `identifier` | yes | string (IRI or DOI) | Identifier for the dataset being described. |
| `name` | yes | string | Human-readable dataset name. |
| `version` | no | string | Dataset version, if versioned. |
-| `license` | yes | IRI | License under which the dataset is claimed to be distributed (e.g. an SPDX license URL). For a public-domain-by-copyright-expiration claim, for which SPDX has no direct term, use the CC Public Domain Mark IRI (`https://creativecommons.org/publicdomain/mark/1.0/`) as the current best-effort convention pending a more precise term. |
+| `license` | yes | IRI | License under which the dataset is claimed to be distributed (e.g. an SPDX license URL). Two interim conventions are documented for cases SPDX has no direct term: (a) for a public-domain-by-copyright-expiration claim, use the CC Public Domain Mark IRI (`https://creativecommons.org/publicdomain/mark/1.0/`); (b) for agreement-supplied data whose licensing does not permit public sharing, use `https://sourcelume.apache.org/ns#agreement-supplied`. Both are best-effor [...]
| `creator` | yes | array (min 1) | One or more parties involved in producing this dataset — each a `schema:Organization` or `schema:Person` with a `name`, and an optional `role` (`originator`, `curator`, or `distributor`) to distinguish who made the underlying data from who aggregated or redistributed it. Omit `role` for a single, undifferentiated creator. |
| `created` | yes | xsd:dateTime | When this provenance record was authored (record metadata, not content provenance). Distinct from `added` and `contentCreated` below. |
| `added` | yes | xsd:dateTime | When the dataset was added to the collection being described (e.g. when a source was incorporated into dynaword). Mirrors the `added` field in dynaword datasheets. |
@@ -41,7 +41,7 @@ claim's accuracy — it gives the claim a consistent shape so it *can* be verifi
claims, and verification is a downstream concern for registries/auditors, not the record itself.
- **No usage/audit metadata.** `ProvenanceRecord` is deliberately dataset-centric. A model
producer's downstream usage claims about a dataset (e.g. a memorisation audit) belong to a
- separate, later record type, not to this one — this is a scope decision worth defending on the
+ separate, later record type, not to this one — this is a scope decision worth raising on the
dev list rather than a settled fact, see `notes/stress-test-dfm-mimir.md`.
## See also
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@sourcelume.apache.org
For additional commands, e-mail: commits-help@sourcelume.apache.org