(datasketches-tck) branch main updated: docs: explain snapshot maintenance workflow (#6)

[ti...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

tisonkun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datasketches-tck.git


The following commit(s) were added to refs/heads/main by this push:
     new f63a0e6  docs: explain snapshot maintenance workflow (#6)
f63a0e6 is described below

commit f63a0e637fd1c886c99899bdcddf04a78b7e612a
Author: tison <wa...@gmail.com>
AuthorDate: Tue Sep 1 08:51:29 2026 +0800

    docs: explain snapshot maintenance workflow (#6)
    
    Signed-off-by: tison <wa...@gmail.com>
---
 README.md                       |   2 +
 docs/serialization-snapshots.md | 110 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 112 insertions(+)

diff --git a/README.md b/README.md
index 7ea3004..0168d81 100644
--- a/README.md
+++ b/README.md
@@ -12,6 +12,8 @@ The structure (or image) of a serialized sketch is independent of the language f
 
 This repository contains snapshots of serialized sketches, which a particular DataSketches implementation should be able to read. Snapshot generators are also included.
 
+See [Serialization snapshots](docs/serialization-snapshots.md) for the repository design, the snapshot generation commands, and the maintainer workflow for updating the corpus from an upstream DataSketches implementation.
+
 ## Contribute
 
 Please visit the main [DataSketches website](https://datasketches.apache.org) for more information.
diff --git a/docs/serialization-snapshots.md b/docs/serialization-snapshots.md
new file mode 100644
index 0000000..d44c415
--- /dev/null
+++ b/docs/serialization-snapshots.md
@@ -0,0 +1,110 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements. See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License. You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+# Serialization snapshots
+
+The serialization corpus is a compatibility boundary between DataSketches implementations. Each directory under `serialization/<language>/snapshots` contains sketches produced by one implementation and intended to be read and validated by the others.
+
+This repository generates snapshots from exact upstream commits instead of following the latest branch. Pinning makes a checkout reproducible and ensures that changes to the compatibility corpus receive normal code review.
+
+## Set up the toolchain
+
+Install [mise](https://mise.jdx.dev/), then install the pinned toolchain and inspect the available snapshot commands:
+
+```shell
+mise install
+mise run tck -- snapshots --help
+```
+
+Mise supplies Go, CMake and CTest, Java, and Maven. Git is required for every source language, a C++ compiler is required for C++, and Make is required for Go.
+
+Commands accept `cpp`, `go`, `java`, or `all` as the source language.
+
+## Update snapshots from upstream
+
+Updating the corpus is an intentional maintainer operation because selecting an upstream revision and accepting compatibility changes require review. There is no GitHub Actions workflow that discovers newer revisions or opens snapshot update pull requests.
+
+To update one source language:
+
+1. Choose the upstream commit to adopt. Prefer a commit on the implementation's main development branch whose generator represents the compatibility behavior being adopted.
+2. Change that language's `commit` field in `internal/snapshots/generator.go`. If the upstream build or output layout changed, update the corresponding adapter in `internal/snapshots/<language>.go` as well.
+3. Regenerate the complete snapshot directory:
+
+   ```shell
+   mise run tck -- snapshots update go
+   ```
+
+   Update mode generates from the new pin and atomically replaces `serialization/go/snapshots`; it is not an incremental copy, so removed upstream outputs become visible as deletions.
+
+4. Review the pin and corpus together:
+
+   ```shell
+   git diff --stat
+   git diff -- internal/snapshots/generator.go
+   git status --short serialization/go/snapshots
+   ```
+
+   Added and deleted files change the set of compatibility cases. Unexpected changes to deterministic files should be understood from the upstream change before they are accepted.
+
+5. Verify that generation is reproducible at the new pin and run the repository checks:
+
+   ```shell
+   mise run tck -- snapshots check go
+   mise run check
+   ```
+
+   A second generation may report allowed modifications for known probabilistic snapshots. The file set and deterministic contents must reproduce.
+
+Use `all` instead of a language only when intentionally refreshing every source implementation. Updating languages separately usually produces smaller, easier-to-review pull requests.
+
+## Check the pinned corpus
+
+Use check mode to regenerate snapshots from the currently pinned commit and compare them with the committed corpus:
+
+```shell
+mise run tck -- snapshots check go
+```
+
+Check mode does not modify the repository. It fails for added or deleted files and for content changes to deterministic snapshots. It reports, but allows, content changes to existing snapshots classified as probabilistic by `internal/snapshots/stability.go`.
+
+This command answers whether the repository matches its pin; it does not determine whether the pin is the latest upstream commit.
+
+## Use the corpus from an implementation
+
+An implementation consumes the `.sk` files as test fixtures. Its compatibility tests should load snapshots produced by the other source languages, deserialize each supported sketch family, and assert observable results with tolerances appropriate to that algorithm.
+
+This repository centralizes the fixture corpus and source-side generation. Consumer tests remain in the individual DataSketches implementation repositories.
+
+## Review policy for probabilistic snapshots
+
+Some upstream generators contain randomness, so byte-for-byte reproduction is not a valid invariant for every file. The stability policy is source-specific because upstream implementations do not always seed or exercise an algorithm in the same way.
+
+Only modifications to existing probabilistic files are allowed in check mode. Additions and deletions always block the check because they alter the compatibility corpus, and modifications to deterministic files block because they indicate either a compatibility change or a non-reproducible generator.
+
+The probabilistic classification only controls byte-level comparison in this repository. Upstream generators remain responsible for constructing valid sketches, and consumers remain responsible for algorithm-appropriate assertions.
+
+## GitHub Actions
+
+`.github/workflows/check.yml` runs `mise run check` for pull requests and pushes to `main`. It validates the Go implementation of the TCK tooling, but it does not run the upstream snapshot generators or modify committed snapshots.
+
+Snapshot generation is kept out of the required workflow because it clones and builds three external projects, and because an automated update cannot decide whether an upstream compatibility change should be adopted. A pull request that updates a pin should include the generated corpus changes and record which source-language checks were run locally.
+
+## Implementation notes
+
+For each requested language, the `tck` command reads the repository and commit from `internal/snapshots/generator.go`, checks out that revision in a temporary workspace, and invokes the source-specific adapter in `internal/snapshots/<language>.go`. It then compares the generated output with `serialization/<language>/snapshots`; update mode atomically replaces that directory.
+
+The command-line interface and change report live in `cmd/tck`. Reconciliation and file comparison live in `internal/snapshots`, where `stability.go` classifies deterministic and known probabilistic outputs.


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org