(opendal-yinyang) branch main updated: docs: define the YinYang Format (#41)

[xu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

Xuanwo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opendal-yinyang.git


The following commit(s) were added to refs/heads/main by this push:
     new 60582b1  docs: define the YinYang Format (#41)
60582b1 is described below

commit 60582b1c530acfd0a418414ef17809197a060c57
Author: Xuanwo <gi...@xuanwo.io>
AuthorDate: Wed Sep 2 12:37:34 2026 +0800

    docs: define the YinYang Format (#41)
    
    * docs: define the managed core model
    
    * docs: define RFC and specification lifecycle
    
    * docs: define the YinYang Format
---
 AGENTS.md                   |  19 +++++
 rfcs/0041_yinyang_format.md | 202 ++++++++++++++++++++++++++++++++++++++++++++
 specs/README.md             |  23 +++++
 3 files changed, 244 insertions(+)

diff --git a/AGENTS.md b/AGENTS.md
index 973ddc1..eff52cc 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -21,6 +21,25 @@ cargo x licenses
 Keep commits focused and use semantic commit and pull request titles such as
 `feat:`, `fix:`, `docs:`, `build:`, or `ci:`.
 
+## Documentation lifecycle
+
+- `rfcs/` records design decisions and their historical context. An RFC may be
+  edited while its pull request is under review. Once the RFC is merged into
+  `main`, treat that file as immutable: do not edit, rename, move, or delete it.
+  Correct or supersede an accepted RFC with a new RFC that links to the earlier
+  decision.
+- `specs/` is the source of truth for current supported contracts and
+  implementation details. Update the relevant specification in the same change
+  whenever an API, behavior, wire format, invariant, compatibility rule, or
+  implementation boundary changes.
+- Specifications describe the current contract. Keep decision history,
+  rejected alternatives, discussion, and migration rationale in RFCs. A
+  specification may cite RFCs for rationale, but it must remain self-contained.
+- When code, tests, and a specification disagree, establish the intended current
+  contract before editing. Fix the implementation or update the specification
+  in the same change; never rewrite a merged RFC to make it match current
+  behavior.
+
 ## Important notes
 
 - Minimum Rust version is 1.91.0, configured in `Cargo.toml`. The development
diff --git a/rfcs/0041_yinyang_format.md b/rfcs/0041_yinyang_format.md
new file mode 100644
index 0000000..ded6781
--- /dev/null
+++ b/rfcs/0041_yinyang_format.md
@@ -0,0 +1,202 @@
+- Proposal Name: `yinyang_format`
+- Start Date: 2026-09-01
+- RFC PR: [apache/opendal-yinyang#41](https://github.com/apache/opendal-yinyang/pull/41)
+- Tracking Issue: [apache/opendal-yinyang#19](https://github.com/apache/opendal-yinyang/issues/19)
+
+# Summary
+
+Define the Apache OpenDAL™ YinYang Format as a materialized logical tree,
+immutable filesystem versions, idempotent commit facts, and verifiable
+references to immutable bytes. These are the stable semantics of the format;
+snapshots, delta batches, patch layers, record streams, and other persistence
+structures are encoding choices.
+
+The YinYang Format is the internal storage contract of the `Managed` Volume
+Model defined by RFC-0016. This RFC does not change the user-facing Volume Model
+vocabulary, the `Direct` model, runtime behavior, or any persistent encoding.
+
+# Problem
+
+The YinYang Format needs stable names for the state observed by filesystem
+operations. An encoding may represent that state as compacted snapshots, delta
+streams, layered file mappings, and immutable objects. Those structures answer
+how state is stored; they do not define what a file, filesystem version, or
+successful commit means.
+
+Treating encoding structures as format semantics makes the contract depend on
+one compaction and framing strategy. It also obscures three independent
+boundaries: logical identity versus path, a successful publication versus the
+resulting filesystem version, and logical content versus its physical placement.
+
+# Proposed Design
+
+## Format structure
+
+The YinYang Format has one bootstrap record and three semantic layers:
+
+```text
+Bootstrap   FsFormat
+Logical     Tree / Node / File
+Versioning  FsHead / FsVersion / Commit
+Physical    BlobRef
+```
+
+`FsFormat` is created once. `FsHead` is the only mutable publication cell.
+`FsVersion` and every byte region reached through a `BlobRef` are immutable.
+
+## Bootstrap
+
+```text
+FsFormat {
+  fs: FsId,
+  root: NodeId,
+  decodings: [Extension],
+  head_extension: Extension?,
+}
+```
+
+`FsFormat` contains information required before a reader can locate the head or
+interpret file content. Writer packing targets and other operational policy do
+not belong to the format contract.
+
+## Logical state
+
+```text
+Tree = ordered map Path -> Node
+
+Node {
+  id: NodeId,
+  generation: Generation,
+  attrs: NodeAttrs,
+  body: Dir | File,
+}
+
+Dir {
+  entries_generation: Generation,
+}
+
+File {
+  version: FileVersionId,
+  content: ContentId,
+  parts: [FilePart],
+}
+
+FilePart {
+  range: FileRange,
+  source_offset: u64,
+  source: FileSource,
+}
+
+FileSource {
+  stored: BlobRef,
+  decoded: [ContentId],
+}
+
+ContentId {
+  digest: Digest,
+  length: u64,
+}
+```
+
+`Path` identifies a location while `NodeId` identifies a node across rename.
+`Node.generation` protects node content and attributes;
+`Dir.entries_generation` independently protects the directory membership set.
+`FileVersionId` identifies one content publication, including a publication
+whose logical bytes equal an older version. `ContentId` identifies the complete
+logical byte sequence independently of its placement.
+
+`FilePart` is a mapping from a file range into a source. `FileSource` is the
+larger verification and decoding unit, so multiple parts may reuse one source
+through different offsets. Parts are ordered, non-overlapping, and exactly
+cover `[0, File.content.length)`; an empty file has no parts. The logical source
+content is the last decoded identity, or the stored identity when the decoding
+list is empty.
+
+## Versioning and publication
+
+```text
+FsHead {
+  current: FsVersionRef,
+  gc_epoch: GcEpoch,
+  min_retained: VersionNumber,
+}
+
+FsVersionRef {
+  number: VersionNumber,
+  blob: BlobRef,
+}
+
+FsVersion {
+  fs: FsId,
+  number: VersionNumber,
+  tree: Tree,
+  commits: [Commit],
+}
+
+Commit {
+  id: CommitId,
+  version: VersionNumber,
+}
+```
+
+An observation returns `FsHead` and the condition token needed to replace the
+same head state. Publication first makes the new immutable version and all of
+its referenced bytes durable, then conditionally replaces the complete head.
+Advancing the garbage-collection fence uses the same conditional cell.
+
+`VersionNumber` is the publication order. `FsVersion.fs` must equal
+`FsFormat.fs`, `FsVersionRef.number` must equal the referenced version number,
+and `FsHead.min_retained` must not exceed the current number.
+
+A caller assigns `CommitId` before attempting publication. A `Commit` exists
+only after that attempt succeeds and binds the identifier to the published
+version. Retained commits cover a contiguous version range ending at the
+current version, contain one commit per non-genesis version in that range, and
+contain no duplicate `CommitId`. This allows a retry to distinguish a committed
+attempt from a conflict or an outcome older than the retained range.
+
+## Physical boundary
+
+`BlobRef` is the only physical concept exposed by the YinYang Format:
+
+> A persistent, verifiable reference to a contiguous byte region in one
+> immutable blob.
+
+The reference encapsulates location, byte range, and the integrity information
+required by its persistent representation. The format does not depend on object
+keys, framing, checksums, stream kinds, object classes, or garbage-collection
+sharding.
+
+## Persistence projection
+
+An encoding may compact or partition format values without changing their
+meaning:
+
+| Format value | Possible persistence representation |
+| --- | --- |
+| `FsVersion.tree` | Tree snapshot plus ordered delta batches |
+| `FsVersion.commits` | One or more commit batches |
+| `File.parts` | Base and patch layers with inline and continued mappings |
+| `BlobRef` | Whole-object, framed-payload, or stored-range reference |
+
+Snapshot cursors, batch boundaries, compaction weights, patch levels, bounding
+ranges, streams, and continuation records belong to the encoding. Materializing
+them must produce the format values and invariants defined above.
+
+Format naming follows the semantic boundary: `Fs*` names bootstrap and
+publication state; `Tree`, `Node`, `Dir`, and `File` name logical state;
+`Commit*` names idempotent publication facts; and `BlobRef` names the physical
+reference. Persistence-specific terms such as `Snapshot`, `Batch`, `Layer`,
+`Stream`, and `Locator` remain inside encodings.
+
+# Compatibility and Migration
+
+This RFC changes no API behavior or wire bytes. Existing and in-progress
+encodings may retain their current tuple fields, type tags, object keys, and
+integrity checks. Their readers materialize the YinYang Format values at the
+encoding boundary, and their writers lower those values back into snapshots,
+batches, layers, and references.
+
+RFC-0016 continues to use `Volume` for the user-facing choice between `Direct`
+and `Managed`. YinYang Format names the internal storage contract of a Managed
+volume and does not rename that architecture-level concept.
diff --git a/specs/README.md b/specs/README.md
new file mode 100644
index 0000000..0a2c879
--- /dev/null
+++ b/specs/README.md
@@ -0,0 +1,23 @@
+# Specifications
+
+This directory contains the current, maintained contracts for Apache OpenDAL™
+YinYang.
+
+A specification describes the currently supported behavior, APIs, wire formats,
+invariants, compatibility rules, and implementation boundaries. Specifications
+evolve with the implementation and must be updated in the same change as the
+contract they describe.
+
+RFCs in [`../rfcs`](../rfcs) preserve design decisions and their historical
+context. An RFC may change while it is under review, but its file becomes
+immutable once merged into `main`. Correct or supersede an accepted RFC with a
+new RFC rather than editing the historical document.
+
+A specification may cite RFCs for rationale, but it must be self-contained and
+is authoritative for current behavior. Keep discussion, rejected alternatives,
+and decision history in RFCs instead of specifications.
+
+When the implementation, tests, and a specification disagree, first establish
+the intended current contract. Update the implementation, tests, and
+specification together as needed; do not repair the mismatch by rewriting a
+merged RFC.