You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "symious (via GitHub)" <gi...@apache.org> on 2026/07/21 02:25:57 UTC

[PR] HDDS-15728. Add OEP doc [ozone]

symious opened a new pull request, #10822:
URL: https://github.com/apache/ozone/pull/10822

   ## What changes were proposed in this pull request?
   
   This is the OEP doc for S3 versioning feature.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-15728
   
   ## How was this patch tested?
   
   No need test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3694985890


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,389 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited; markers
+count toward the limit), `noncurrentVersionExpiration` (opt-in), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed) — all enforced by a new **VersionCleanupService**
+following the `KeyDeletingService` pattern. A per-bucket `maxVersionsPolicy`
+selects between `TRIM` (default: reclaim oldest-first, asynchronously rather than
+inside the write transaction) and `REJECT` (the write fails instead and the
+operator chooses what to delete; opt-in, since S3 has no error code for the
+condition). All versions count against the bucket space quota;
+version count, marker count, and noncurrent bytes are exposed via
+`ozone sh bucket info` and Recon. `maxVersions` is documented as an Ozone
+extension over S3 semantics (comparable to Lifecycle `NewerNoncurrentVersions`).
+
+## Interaction with Ozone snapshots

Review Comment:
   Filed HDDS-16060 for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on PR #10822:
URL: https://github.com/apache/ozone/pull/10822#issuecomment-5112401872

   @aswinshakil  Thanks for the questions.
   
   > why not `KeyInfo.keyLocationList`
   
   We evaluated it and rejected it early on (it's under Alternatives in the OEP):
   
   - All versions in one RocksDB value means every PUT/DELETE is a full read-modify-write of a record that grows with the version count, instead of one small KV write per version operation.
   - A location group carries no per-version dataSize / mtime / ETag, and cannot express a delete marker or the null version.
   
   > Snapshot
   
   I've added an "Interaction with Ozone snapshots" section to the OEP. PTAL.
   
   Given how many edge cases sit in this intersection, we're inclined to gate it in phase 1: reject `CreateSnapshot` on buckets with versioning Enabled/Suspended. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3764442027


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited; markers
+count toward the limit), `noncurrentVersionExpiration` (opt-in), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed) — all enforced by a new **VersionCleanupService**
+following the `KeyDeletingService` pattern. Versions beyond `maxVersions` are

Review Comment:
   > even lifecycle is disabled, we still need to clean for "maxVersion" and the marker
   
   This makes sense. 
   
   But imaging if both the KeyLifecycleService and VersionCleanupService are enabled, and user has defined the NoncurrentVersionExpiration properly, then some of their function are overlapped, and the same bucket can be scanned twice in this case. How can we save one service from do the duplication work, that's one improvement that we can think of.   How about always enable the KeyLifecycleService if version feature is finalized, and create a built-in(or hidden) lifecycle for bucket when it's versioning is enabled, use the maxVersions as value in configuration?  This way, it can satisfy both the S3 spec compatibility, and also no another bulk of similar logic codes, extra bucket scan, and execution time.  Because to improve the S3 compatibility, KeyLifecycleService naturally need to do these things. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3756888550


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version

Review Comment:
   What will be the result if delete an delete marker? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3680082542


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,376 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}/{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` reserved for the null slot and `1` for the first-version sentinel.
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current; if
+  `maxVersions` is exceeded, trim the oldest noncurrent version in the same batch
+  (its blocks go to deletedTable).
+- **PUT (Suspended)**: locate the null slot (current, or an `isNullVersion`
+  record in versionedKeyTable) and replace it; versions accumulated while Enabled
+  are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker overwrites the
+  null slot.) **DELETE ?versionId=x**: permanently delete that version (blocks to
+  deletedTable); if it was current, trigger version promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited;
+synchronous trim inside the write transaction; markers count toward the limit),
+`noncurrentVersionExpiration` (opt-in, enforced by a new
+**VersionCleanupService** following the `KeyDeletingService` pattern), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed). All versions count against the bucket space quota;
+version count, marker count, and noncurrent bytes are exposed via
+`ozone sh bucket info` and Recon. `maxVersions` is documented as an Ozone
+extension over S3 semantics (comparable to Lifecycle `NewerNoncurrentVersions`).
+
+## Interaction with Ozone snapshots
+
+A snapshot checkpoints the entire OM RocksDB, so versionedKeyTable is captured
+automatically along with the bucket's versioning status: a snapshot holds the
+complete version history as of its creation. Snapshot creation and deletion need
+no change — `SnapshotDeletingService` moves only deletedTable, deletedDirTable
+and snapshotRenamedTable entries between snapshots, and permanently deleted
+versions travel through deletedTable, the path already used when an overwrite
+reclaims the previous blocks. Multiple versions of one key queued for deletion
+share a deletedTable entry, which is a `RepeatedOmKeyInfo` list evaluated one
+record at a time, so no new structure is required there either.
+
+What does change is **block reclamation**. Snapshots share physical blocks with
+the active object store, so a snapshot keeps its data only because
+`KeyDeletingService` refuses to reclaim blocks that a previous snapshot still
+references. That decision is made per deletedTable record by
+`ReclaimableKeyFilter`, which resolves the key's path in the previous snapshot
+and looks it up **in keyTable only**
+(`KeyManagerImpl.getPreviousSnapshotOzoneKeyInfo`), then compares objectID and
+block locations. With versioning a key's live records span two tables, so the
+lookup misses: permanently deleting a noncurrent version resolves to whatever is
+current in the previous snapshot — a different record with different blocks —
+which reads as "not present in the previous snapshot", i.e. reclaimable. Blocks
+a snapshot still points at would be deleted. objectID does not rescue the
+comparison either way: `prepareFileInfo` keeps it stable across a key's versions,
+so it matches while the block lists do not.
+
+The fix is to make that lookup version-aware. A record carrying a `versionId`
+resolves to the exact dbKey `/{volume}/{bucket}/{keyName}/{MAX - versionId}` in
+the previous snapshot's versionedKeyTable, falling back to keyTable when that
+version was current there. Because versionId is frozen at creation and never
+reused, this is an exact point lookup that *replaces* — rather than extends — the
+objectID-plus-block-list heuristic for versioned buckets. The same lookup feeds
+`calculateExclusiveSize`, keeping snapshot exclusive-size reporting correct for
+noncurrent versions.
+
+This makes deletedTable the single choke point for version reclamation: every
+path that removes a version permanently — `DELETE ?versionId=`, the `maxVersions`
+trim, and VersionCleanupService — writes the record to deletedTable and lets
+KeyDeletingService apply the snapshot-aware filter. None of them may reclaim
+blocks directly.
+
+Snapshot diff remains a **current-version diff**; versionedKeyTable is
+deliberately not added to it. keyTable still holds exactly one record per key, so
+diff cost and semantics do not change as versions accumulate. One rule does need
+stating: **a key whose current version is a delete marker counts as absent**.
+Without it a delete surfaces as a MODIFY (if the marker carries the key's
+objectID) or as a DELETE plus a phantom CREATE (if it carries a new one), instead
+of the DELETE users expect; restoring an object by removing its marker reports
+CREATE correspondingly.
+
+Reads inside a snapshot go through `OmSnapshot`, which implements
+`IOmMetadataReader`, whose `lookupKey(OmKeyArgs)` has no version dimension.
+`?versionId=` is plumbed through it so versions retained in a snapshot are
+actually readable; `ListObjectVersions` against a snapshot is out of scope for
+the first version.
+
+## S3 Gateway and native API surface
+
+New endpoints: `PutBucketVersioning`, `GetBucketVersioning`,
+`ListObjectVersions`. Extended: GET/HEAD/DELETE with `?versionId=` (including the
+literal `null`), `x-amz-version-id` / `x-amz-delete-marker` response headers,
+per-entry version semantics in batch `DeleteObjects`, `CopyObject` versioning
+behavior, and version-id headers on `PutObject` / `CompleteMultipartUpload`. The
+native interfaces (`ozone sh`, `OzoneBucket` API) are extended in parallel so
+versions are manageable outside the S3 path.
+
+## Compatibility and upgrade
+
+Wire protocol changes are limited to optional proto fields, so old records and
+old clients are unaffected. The new column family and request surface are gated
+by `OMLayoutFeature.OBJECT_VERSIONING`; before cluster finalization, enabling
+versioning is rejected with `NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION`.
+Snapshots taken before the feature is used are unaffected, and snapshot
+interaction is covered above. Buckets without versioning behave exactly as today
+(verified by benchmarks).
+
+# Alternatives
+
+- **Single-record layout** (versions embedded in the existing
+  `keyLocationVersions` list): every PUT/DELETE becomes a full read-modify-write
+  of one growing RocksDB value — write amplification linear in the version count
+  (hundreds of KB per write at ~100 versions of a large key), degraded
+  compaction, and no per-version object metadata (size / mtime / ETag) without
+  rebuilding an object record inside the location structure. Rejected in favor of
+  the two-table design, where each version operation writes one small KV (O(1))
+  and each version is a complete `OmKeyInfo`.
+- **A two-state model** (dropping Suspended): "disabled after being enabled" must
+  retain historical versions anyway — otherwise a single state change silently
+  destroys data — yet the bucket would present itself as never-versioned and
+  `GetBucketVersioning` could not answer `Suspended`. Since the semantics are
+  unavoidable, keep the standard three states and full S3 compatibility.
+- **Reusing `objectID` as the version identity**: code verification showed the
+  overwrite path (`OMKeyRequest.prepareFileInfo`) reuses the existing record via
+  `toBuilder()`, so objectID is intentionally stable across overwrites — snapshot
+  diff relies on it to distinguish "modified" from "deleted + recreated".
+  **Reusing `updateID`**: hsync re-commits the same version repeatedly, so it
+  drifts, while a version identity must be frozen at creation. Hence the
+  dedicated `versionId` field.
+- **S3-style random version IDs**: equivalent for reuse-prevention, but they
+  destroy the temporal clustering of versionedKeyTable — version promotion would
+  degrade from one seek to a scan-and-sort (with mtime tiebreak problems), and
+  `GET ?versionId=` from a point lookup to a prefix scan. Unpredictability, the
+  only remaining benefit, is provided by the external encoding layer instead.
+- **A per-key sequential counter** for readable first versions: requires a
+  persistent allocator and reuses IDs after full deletion of a key. Replaced by the
+  pinned-first generator, which special-cases only the first version and needs no
+  allocator.
+- **A version-level snapshot diff** (reporting noncurrent versions created or
+  removed between two snapshots): the diff is keyed by objectID, which a key's
+  versions share, so it would need a different identity and a version dimension
+  in the report; versionedKeyTable would also have to join
+  `RocksDBCheckpointDiffer.COLUMN_FAMILIES_TO_TRACK_IN_DAG`, paying
+  compaction-log and SST retention cost for information already reachable through
+  the snapshot's own version listing. Deferred; the diff stays a current-version
+  diff with an explicit delete-marker rule.
+- **Pinning the generator into bucket metadata** (so a bucket keeps the generator
+  it was created with): removes the risk of a mid-life generator change, but adds a
+  proto field, a write path in both bucket create and set-property, and leaves the
+  generator un-changeable even when an operator legitimately wants to switch.
+  Replaced by the commit-time ordering check, which turns the risk into a loud,
+  per-key failure with a clear remedy and costs no persistent state.
+
+# Plan
+
+Implemented as one umbrella Jira with ten tasks (33 sub-tasks, each roughly one
+PR), in dependency order `T1 → T2 → T3 → T4 → T5 → (T6 ∥ T7) → (T8 ∥ T9) → T10`:
+
+| Task | Scope |
+|---|---|
+| T1 Metadata foundation | proto three-state enum + legacy-boolean sync, set-property state machine, `OmKeyInfo` version fields, versionedKeyTable column family, layout feature gate |
+| T2 VersionId generator framework | `VersionIdGenerator` interface with class-name configuration, transaction-index default, commit-time ordering check, pinned-first generator |
+| T3 ENABLED write paths | PUT two-table update, DELETE marker insertion, quota accounting |
+| T4 Read / permanent delete / promotion | `?versionId=` reads (including through `OmSnapshot`), permanent delete with the version-aware reclamation lookup, version promotion |
+| T5 SUSPENDED semantics | null-slot overwrite, null markers, zero-migration legacy keys |
+| T6 S3 Gateway endpoints | bucket versioning endpoints, object `versionId` support, batch delete |
+| T7 ListObjectVersions | OM merged listing, protocol plumbing, gateway `?versions` |
+| T8 Reclamation | `maxVersions` trim, VersionCleanupService, expired marker cleanup |

Review Comment:
   Recommend to implement T8 before the S3 endpoints are exposed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3850607894


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,614 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest.
+
+```mermaid
+flowchart LR
+    subgraph KT["keyTable — one entry per key"]
+        K1["/vol/buck/photo.jpg<br/>versionId 104 · current"]
+    end
+    subgraph VKT["versionedKeyTable — noncurrent, newest → oldest"]
+        V1["/vol/buck/photo.jpg 0x00 MAX-103<br/>delete marker 103"]
+        V2["/vol/buck/photo.jpg 0x00 MAX-102<br/>object 102"]
+        V3["/vol/buck/photo.jpg 0x00 MAX-101<br/>object 101"]
+    end
+    K1 -. "one seek reaches the newest noncurrent version" .-> V1
+    V1 --> V2 --> V3
+```
+
+Because the suffix is `MAX - versionId`, a single `seek` on the key's prefix lands
+on the newest noncurrent version: `ListObjectVersions` and version promotion are a
+seek plus a sequential read, never a scan-and-sort. The table is registered in
+`OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## How a key's versions evolve
+
+The two mechanisms that are hardest to read out of prose — the delete marker and
+the null version — are easiest to follow as one continuous story of a single key
+`k`. `obj(n)` is a data version with versionId `n`, `marker(n)` a delete marker,
+and `null` marks the record occupying the key's null slot.
+
+| # | Operation | Bucket state | keyTable (current) | versionedKeyTable (newest → oldest) |
+|---|---|---|---|---|
+| 0 | `PUT k` | Unversioned | `obj(—)` no versionId | – |
+| 1 | `PutBucketVersioning(Enabled)` | Enabled | `obj(—)` read as the null version | – |
+| 2 | `PUT k` | Enabled | `obj(101)` | `obj(—)` |
+| 3 | `PUT k` | Enabled | `obj(102)` | `obj(101)`, `obj(—)` |
+| 4 | `DELETE k` | Enabled | `marker(103)` | `obj(102)`, `obj(101)`, `obj(—)` |
+| 5 | `DELETE k?versionId=103` | Enabled | `obj(102)` **promoted back** | `obj(101)`, `obj(—)` |
+| 6 | `PutBucketVersioning(Suspended)` | Suspended | `obj(102)` | `obj(101)`, `obj(—)` |
+| 7 | `PUT k` | Suspended | `obj(104, null)` | `obj(102)`, `obj(101)` |
+| 8 | `PUT k` | Suspended | `obj(105, null)` overwritten in place | unchanged |
+| 9 | `DELETE k` | Suspended | `marker(106, null)` overwritten in place | unchanged |
+| 10 | `PutBucketVersioning(Enabled)`, `PUT k` | Enabled | `obj(107)` | `marker(106, null)`, `obj(102)`, `obj(101)` |
+
+Three things to read out of it:
+
+- **A delete marker is an ordinary version that happens to carry no blocks**
+  (step 4). It becomes the current version, so a plain `GET` returns 404 with
+  `x-amz-delete-marker: true` while every earlier version stays intact and
+  addressable by versionId. Deleting the marker by versionId (step 5) is exactly
+  S3's "restore an object": the newest remaining version is promoted back into
+  keyTable.
+- **Suspended does not stop versioning, it stops *accumulating*** (steps 7–9).
+  Each write lands in the key's single null slot: the first suspended write pushes
+  the previous current version down into versionedKeyTable and drops the key's
+  older null record (step 7); every later suspended write overwrites the null slot
+  in place, so the version count stops growing. Versions accumulated while Enabled
+  are never touched.
+- **The null version is a slot, not a reserved id** (step 7 onwards). It carries a
+  normally generated versionId like any other version and is marked by
+  `isNullVersion`. Pinning it to a fixed low id would misorder a null created
+  between two versioned writes, which is the middle version of the key, not its
+  oldest. Step 10 shows a null delete marker sliding into the noncurrent chain in
+  its correct chronological position once the bucket is Enabled again.
+
+### Where a write lands
+
+The table above is one trace; the rule behind it is the same for every write that
+carries no versionId. Suspended is the branch worth reading twice: it does not
+stop versioning, it collapses every write into the key's one null slot, which is
+why the version count stops growing while the versions accumulated under Enabled
+stay untouched.
+
+```mermaid
+flowchart TB
+    W["PUT k / DELETE k<br/>(no versionId)"] --> S{"bucket state"}
+    S -->|Unversioned| U["overwrite keyTable in place<br/>old blocks → deletedTable<br/><i>unchanged from today</i>"]
+    S -->|Enabled| E["move current version → versionedKeyTable<br/>write the new record into keyTable"]
+    S -->|Suspended| N{"is the current version<br/>already the null slot?"}
+    N -->|yes| N1["overwrite the null record in place<br/>its blocks → deletedTable, a marker carries none<br/><b>version count does not grow</b>"]
+    N -->|no| N2["move current version → versionedKeyTable<br/>write the new record as current, isNullVersion<br/>drop the key's older null record, if any"]
+    E --> R["new current version:<br/>an object on PUT, a delete marker on DELETE"]
+    N1 --> R
+    N2 --> R
+```
+
+### The life of a delete marker
+
+A delete marker is not a tombstone the read path has to special-case away — it is
+an ordinary version that carries no blocks, and it occupies the current slot like
+any other. That single fact produces all of its externally visible behaviour:
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    Obj: current = obj(102)
+    Marker: current = marker(103)
+    Gone: key no longer exists
+    [*] --> Obj: PUT k
+    Obj --> Marker: DELETE k (no versionId)
+    Marker --> Obj: DELETE k?versionId=103<br/>promotion restores obj(102)
+    Marker --> Gone: lifecycle ExpiredObjectDeleteMarker<br/>once it is the only version left
+    Obj --> Obj: GET k → 200
+    Marker --> Marker: GET k → 404, x-amz-delete-marker
+```
+
+Deleting the marker by versionId is exactly S3's "restore an object": nothing is
+rewritten, the newest remaining version is moved back into keyTable.
+
+### Version promotion
+
+The invariant is that keyTable always holds a key's current version. When a
+permanent delete removes the current version, promotion restores the invariant in
+the same WriteBatch, under the bucket write lock:
+
+```mermaid
+flowchart TB
+    A["DELETE k?versionId = current"] --> B["remove the current entry from keyTable<br/>blocks → deletedTable"]
+    B --> C{"seek versionedKeyTable<br/>on k's prefix"}
+    C -->|"a noncurrent version exists"| D["move the newest one back into keyTable<br/>record content stays frozen — a pure positional move"]
+    C -->|"none left"| E["the key disappears entirely"]
+```
+
+## VersionId generation
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and read when an OM
+starts, so a change takes effect on every bucket once the OMs have been restarted;
+it is not recorded in bucket metadata. Every generator must satisfy, **for itself**:
+strictly increasing within a key (a later version's id is always greater than every
+earlier version's id of that key); frozen once assigned; and `0` never handed out,
+being the unset value of the optional field. `1` is reserved as well, but by the
+pinned generator below rather than by the interface: it is that generator's own
+sentinel, which it assigns and never proposes.
+
+The identifier is deliberately **not** derived from OM's Ratis transaction index.
+The versionId is persisted, externally referenced, and part of the S3 API surface;
+tying it to the replication log would export an internal counter into a public
+identity and pin the feature to one execution model, when the OM execution path is
+itself an abstraction (`ExecutionContext` already carries an index whose term may
+be absent).
+
+Numbering a version by when it was written is what S3-compatible stores generally
+do: a GCS `generation` is a microsecond timestamp, an Azure Blob version id is an
+ISO-8601 instant, and RocksDB's user-defined timestamps order the versions of a key
+by a `key + timestamp` suffix — the same encoding the versionedKeyTable dbKey
+already uses. RocksDB is also the precedent for the split below: it orders by the
+timestamp but leaves keeping them monotonic to the caller.
+
+The shape matches SCM's `SequenceIdGenerator` — a cheap local allocation plus a
+floor that guarantees monotonicity. SCM's class itself is not reusable here: it
+lives in `server-scm` on `SCMHAManager` / `@Replicate`, so it would couple OM's
+write path to SCM across a module boundary. It also needs machinery this does not,
+because `localId` has to increase globally while a `versionId` only has to increase
+within one key, whose floor is already persisted as that key's current version. SCM
+does show the migration path if a persisted sequence is ever wanted:
+`upgradeToSequenceId` seeds one from a future timestamp in this same encoding.
+
+- **`UniqueIdVersionIdGenerator` (default)** — the same time-based scheme Ozone
+  already uses to generate block local IDs (`hdds.utils.UniqueId`): the id is
+  `currentTimeMillis << 16 | counter`, a 16-bit counter disambiguating ids proposed
+  within the same millisecond. It needs no allocator, no persistent state and no
+  coordination; values are small positive longs, so `MAX_VALUE - versionId`
+  ordering in versionedKeyTable stays plain signed arithmetic; and the id sorts by
+  creation time, which is what the version chain wants anyway.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes its own sentinel, `PinnedFirstVersionIdGenerator.FIRST_VERSION_ID
+  = 1`, below any proposed id, so it sorts oldest and can be referenced without
+  listing the key's versions first. All later versions take a normally proposed id. First versions are detected by "no
+  current version in keyTable", reusing the lookup the write path performs anyway.
+  Known trade-off: if every version of a key is permanently deleted and the key is
+  recreated, the new first version takes the sentinel again; only deployments that
+  accept this should configure this generator.
+
+**How the id appears on the wire.** `x-amz-version-id`, `?versionId=` and
+`ListObjectVersions` carry the id as its decimal digits, the way a GCS
+`generation` is rendered: `117121056706265088` for a proposed id, `1` for a
+pinned first version. Decimal digits cannot collide with `null`, the identifier
+S3 reserves for a key's null version, so both share the parameter without
+escaping. Clients should still treat the value as opaque. The digits of a proposed
+id do decode to the millisecond of the write, but that is a property of this
+generator rather than of the API, and ids are ordered only within one key —
+comparing them across keys means nothing.
+
+**Where the id is proposed.** The generator runs on the leader in `preExecute` and
+the id is stamped into the request that goes into the replication log, so every OM
+applies an identical value; nothing is proposed during apply. The one apply-time
+decision is the pinned sentinel, which is a constant and is therefore deterministic
+on its own: if the pinned generator is configured and the key has no current
+version at apply time, the version takes `FIRST_VERSION_ID` instead of the proposed
+id.
+
+**Why a proposed id is only a floor.** A time-based id is monotonic within one OM
+process, not across a leader change onto a node whose clock lags, and the "strictly
+increasing within a key" guarantee binds one generator rather than a sequence of
+them. So the generator only proposes, and the id a version is applied with is the
+later of that proposal and the id after the key's current version:
+
+```
+versionId = max(proposedVersionId, currentVersionId + 1)
+```
+
+The write path already holds the current version, so this costs no read, and it
+reads no other key and no global state: it is a pure function of the replicated
+request and the state every OM already shares, so all of them settle on the same id
+without coordinating. A current version predating versioning carries no id, and a
+key in that state has no other versions to order against, so it takes the proposed
+value unchanged — no versionedKeyTable lookup in any case.
+
+Ordering therefore never rests on the clock being trustworthy, and neither does it
+rest on one generator having proposed every id of a key. Under a clock regression, or
+after a change of generator, the ids of an affected key climb by one until proposed
+values overtake them again: the versions stay correctly ordered, and only the id's
+reading as a time degrades — the right one of the two to give up. An earlier draft
+instead rejected such a commit with `INVALID_REQUEST`; that is dropped, because it
+fails a client write over an operational clock problem the client can neither see
+nor fix.
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected.
+- **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+
+```mermaid
+flowchart TB
+    G["GET / HEAD k"] --> Q{"versionId given?"}
+    Q -->|no| C{"current version in keyTable"}
+    C -->|object| C1["200, the current version"]
+    C -->|delete marker| C2["404<br/>x-amz-delete-marker: true"]
+    Q -->|yes| L["check the current version,<br/>then point-look-up versionedKeyTable"]
+    L --> M{"the addressed version"}
+    M -->|object| M1["200, that version"]
+    M -->|delete marker| M2["405<br/>x-amz-delete-marker: true<br/>Allow: DELETE"]
+    M -->|no such version| M3["404"]
+```
+
+The 404/405 split is S3's, and it follows from the marker being addressable: an
+unaddressed read steps over the marker and reports the key absent, while a read
+that names the marker gets told the version exists but has no body to return.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) A batch delete is the same operation over
+  more keys and takes the same path — it goes through a request of its own, which
+  inserts one marker per key from the same implementation, so the two cannot
+  diverge. Hard-deleting there instead would destroy the current version and leave
+  the versions under it in the versionedKeyTable with nothing above them: absent to
+  a plain read, holding quota, and unreachable by promotion, which only runs when a
+  version is deleted by id. **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion as shown above.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation: version-aware lifecycle rules
+
+Version reclamation is expressed as **lifecycle rules on the lifecycle engine
+already in master** (HDDS-8342: `OmLifecycleConfiguration`, `KeyLifecycleService`,
+and the three S3 lifecycle endpoints), not as a service of its own. The engine
+already provides everything version reclamation needs — per-bucket rule storage
+and validation, prefix/tag filters, a resumable background scan with saved scan
+state, batched deletes through Ratis, metrics, and `suspend`/`resume` admin
+commands. Versioning contributes the three actions that the feature documentation
+currently lists as unsupported, and each maps 1:1 onto an S3 element:
+
+| S3 lifecycle element | Effect on a versioned bucket |
+|---|---|
+| `NoncurrentVersionExpiration.NoncurrentDays` | permanently deletes a noncurrent version once it has been noncurrent that long |
+| `NoncurrentVersionExpiration.NewerNoncurrentVersions` | keeps at most N noncurrent versions of a key, reclaiming oldest-first |
+| `Expiration.ExpiredObjectDeleteMarker` | removes a delete marker, and the key with it, once it is the only version left |
+| `Expiration.Days` / `.Date` (existing action) | on a versioned bucket, inserts a delete marker instead of deleting the object, as S3 does — through the same insertion the write requests use, so an expiry and a DELETE leave the same records behind |
+
+Concretely: `LifecycleAction` gains a `noncurrentVersionExpiration` field and
+`LifecycleExpiration` an `expiredObjectDeleteMarker` field — both optional
+additions to existing messages — and `LifecycleActionTask` gains a
+versionedKeyTable scan next to its keyTable scan, reusing the same batching, scan
+state and metrics. Expiration counts from the moment the version that superseded
+this one was committed, as `NoncurrentDays` does, not from when the version itself
+was written.
+
+One deliberate departure from stock S3: **the lifecycle service is enabled by
+default.** `ozone.lifecycle.service.enabled` flips from `false` to `true`. A rule
+that never runs is worse than no rule, and versioning is the first feature that
+depends on reclamation for correct operation rather than for tidiness. OM rejects
+`PutBucketVersioning(Enabled)` while the lifecycle service is disabled: that does
+not promise a bucket's versions are bounded — one whose owner writes no rule
+accumulates them, as on S3 — but it rules out a rule existing that nothing will
+ever act on.
+
+Earlier drafts bounded version growth with a bucket-level `maxVersions` property
+as well, first refusing the write once the limit was reached and later trimming
+oldest-first from the same scan. Both are dropped. A per-bucket cap duplicates
+`NewerNoncurrentVersions` in an Ozone-private property, which cuts against
+expressing reclamation as lifecycle rules at all; and applying it to the buckets
+it existed for — those carrying no lifecycle configuration — meant scheduling scan
+work for buckets the engine otherwise never visits, a second kind of task that
+differs from every other one in what it may assume. A bucket with no rule now
+accumulates versions, which is what S3 does.
+
+## Quota and observability
+
+All versions count against the bucket space quota. Version count, marker count,

Review Comment:
   Yes, it's planed in [T10](https://docs.google.com/document/d/1uT3Hzss5gJkiuxzzeZYVUB_UqjSbqROUApY5pH6k8TI/edit?tab=t.0#heading=h.jni13mkcoukl).
   
   For the second, I'd leave the formula to HDDS-16176 and follow up in this feature once it's fixed. Added to open questions.
   
   



##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,614 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest.
+
+```mermaid
+flowchart LR
+    subgraph KT["keyTable — one entry per key"]
+        K1["/vol/buck/photo.jpg<br/>versionId 104 · current"]
+    end
+    subgraph VKT["versionedKeyTable — noncurrent, newest → oldest"]
+        V1["/vol/buck/photo.jpg 0x00 MAX-103<br/>delete marker 103"]
+        V2["/vol/buck/photo.jpg 0x00 MAX-102<br/>object 102"]
+        V3["/vol/buck/photo.jpg 0x00 MAX-101<br/>object 101"]
+    end
+    K1 -. "one seek reaches the newest noncurrent version" .-> V1
+    V1 --> V2 --> V3
+```
+
+Because the suffix is `MAX - versionId`, a single `seek` on the key's prefix lands
+on the newest noncurrent version: `ListObjectVersions` and version promotion are a
+seek plus a sequential read, never a scan-and-sort. The table is registered in
+`OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## How a key's versions evolve
+
+The two mechanisms that are hardest to read out of prose — the delete marker and
+the null version — are easiest to follow as one continuous story of a single key
+`k`. `obj(n)` is a data version with versionId `n`, `marker(n)` a delete marker,
+and `null` marks the record occupying the key's null slot.
+
+| # | Operation | Bucket state | keyTable (current) | versionedKeyTable (newest → oldest) |
+|---|---|---|---|---|
+| 0 | `PUT k` | Unversioned | `obj(—)` no versionId | – |
+| 1 | `PutBucketVersioning(Enabled)` | Enabled | `obj(—)` read as the null version | – |
+| 2 | `PUT k` | Enabled | `obj(101)` | `obj(—)` |
+| 3 | `PUT k` | Enabled | `obj(102)` | `obj(101)`, `obj(—)` |
+| 4 | `DELETE k` | Enabled | `marker(103)` | `obj(102)`, `obj(101)`, `obj(—)` |
+| 5 | `DELETE k?versionId=103` | Enabled | `obj(102)` **promoted back** | `obj(101)`, `obj(—)` |
+| 6 | `PutBucketVersioning(Suspended)` | Suspended | `obj(102)` | `obj(101)`, `obj(—)` |
+| 7 | `PUT k` | Suspended | `obj(104, null)` | `obj(102)`, `obj(101)` |
+| 8 | `PUT k` | Suspended | `obj(105, null)` overwritten in place | unchanged |
+| 9 | `DELETE k` | Suspended | `marker(106, null)` overwritten in place | unchanged |
+| 10 | `PutBucketVersioning(Enabled)`, `PUT k` | Enabled | `obj(107)` | `marker(106, null)`, `obj(102)`, `obj(101)` |
+
+Three things to read out of it:
+
+- **A delete marker is an ordinary version that happens to carry no blocks**
+  (step 4). It becomes the current version, so a plain `GET` returns 404 with
+  `x-amz-delete-marker: true` while every earlier version stays intact and
+  addressable by versionId. Deleting the marker by versionId (step 5) is exactly
+  S3's "restore an object": the newest remaining version is promoted back into
+  keyTable.
+- **Suspended does not stop versioning, it stops *accumulating*** (steps 7–9).
+  Each write lands in the key's single null slot: the first suspended write pushes
+  the previous current version down into versionedKeyTable and drops the key's
+  older null record (step 7); every later suspended write overwrites the null slot
+  in place, so the version count stops growing. Versions accumulated while Enabled
+  are never touched.
+- **The null version is a slot, not a reserved id** (step 7 onwards). It carries a
+  normally generated versionId like any other version and is marked by
+  `isNullVersion`. Pinning it to a fixed low id would misorder a null created
+  between two versioned writes, which is the middle version of the key, not its
+  oldest. Step 10 shows a null delete marker sliding into the noncurrent chain in
+  its correct chronological position once the bucket is Enabled again.
+
+### Where a write lands
+
+The table above is one trace; the rule behind it is the same for every write that
+carries no versionId. Suspended is the branch worth reading twice: it does not
+stop versioning, it collapses every write into the key's one null slot, which is
+why the version count stops growing while the versions accumulated under Enabled
+stay untouched.
+
+```mermaid
+flowchart TB
+    W["PUT k / DELETE k<br/>(no versionId)"] --> S{"bucket state"}
+    S -->|Unversioned| U["overwrite keyTable in place<br/>old blocks → deletedTable<br/><i>unchanged from today</i>"]
+    S -->|Enabled| E["move current version → versionedKeyTable<br/>write the new record into keyTable"]
+    S -->|Suspended| N{"is the current version<br/>already the null slot?"}
+    N -->|yes| N1["overwrite the null record in place<br/>its blocks → deletedTable, a marker carries none<br/><b>version count does not grow</b>"]
+    N -->|no| N2["move current version → versionedKeyTable<br/>write the new record as current, isNullVersion<br/>drop the key's older null record, if any"]
+    E --> R["new current version:<br/>an object on PUT, a delete marker on DELETE"]
+    N1 --> R
+    N2 --> R
+```
+
+### The life of a delete marker
+
+A delete marker is not a tombstone the read path has to special-case away — it is
+an ordinary version that carries no blocks, and it occupies the current slot like
+any other. That single fact produces all of its externally visible behaviour:
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    Obj: current = obj(102)
+    Marker: current = marker(103)
+    Gone: key no longer exists
+    [*] --> Obj: PUT k
+    Obj --> Marker: DELETE k (no versionId)
+    Marker --> Obj: DELETE k?versionId=103<br/>promotion restores obj(102)
+    Marker --> Gone: lifecycle ExpiredObjectDeleteMarker<br/>once it is the only version left
+    Obj --> Obj: GET k → 200
+    Marker --> Marker: GET k → 404, x-amz-delete-marker
+```
+
+Deleting the marker by versionId is exactly S3's "restore an object": nothing is
+rewritten, the newest remaining version is moved back into keyTable.
+
+### Version promotion
+
+The invariant is that keyTable always holds a key's current version. When a
+permanent delete removes the current version, promotion restores the invariant in
+the same WriteBatch, under the bucket write lock:
+
+```mermaid
+flowchart TB
+    A["DELETE k?versionId = current"] --> B["remove the current entry from keyTable<br/>blocks → deletedTable"]
+    B --> C{"seek versionedKeyTable<br/>on k's prefix"}
+    C -->|"a noncurrent version exists"| D["move the newest one back into keyTable<br/>record content stays frozen — a pure positional move"]
+    C -->|"none left"| E["the key disappears entirely"]
+```
+
+## VersionId generation
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and read when an OM
+starts, so a change takes effect on every bucket once the OMs have been restarted;
+it is not recorded in bucket metadata. Every generator must satisfy, **for itself**:
+strictly increasing within a key (a later version's id is always greater than every
+earlier version's id of that key); frozen once assigned; and `0` never handed out,
+being the unset value of the optional field. `1` is reserved as well, but by the
+pinned generator below rather than by the interface: it is that generator's own
+sentinel, which it assigns and never proposes.
+
+The identifier is deliberately **not** derived from OM's Ratis transaction index.
+The versionId is persisted, externally referenced, and part of the S3 API surface;
+tying it to the replication log would export an internal counter into a public
+identity and pin the feature to one execution model, when the OM execution path is
+itself an abstraction (`ExecutionContext` already carries an index whose term may
+be absent).
+
+Numbering a version by when it was written is what S3-compatible stores generally
+do: a GCS `generation` is a microsecond timestamp, an Azure Blob version id is an
+ISO-8601 instant, and RocksDB's user-defined timestamps order the versions of a key
+by a `key + timestamp` suffix — the same encoding the versionedKeyTable dbKey
+already uses. RocksDB is also the precedent for the split below: it orders by the
+timestamp but leaves keeping them monotonic to the caller.
+
+The shape matches SCM's `SequenceIdGenerator` — a cheap local allocation plus a
+floor that guarantees monotonicity. SCM's class itself is not reusable here: it
+lives in `server-scm` on `SCMHAManager` / `@Replicate`, so it would couple OM's
+write path to SCM across a module boundary. It also needs machinery this does not,
+because `localId` has to increase globally while a `versionId` only has to increase
+within one key, whose floor is already persisted as that key's current version. SCM
+does show the migration path if a persisted sequence is ever wanted:
+`upgradeToSequenceId` seeds one from a future timestamp in this same encoding.
+
+- **`UniqueIdVersionIdGenerator` (default)** — the same time-based scheme Ozone
+  already uses to generate block local IDs (`hdds.utils.UniqueId`): the id is
+  `currentTimeMillis << 16 | counter`, a 16-bit counter disambiguating ids proposed
+  within the same millisecond. It needs no allocator, no persistent state and no
+  coordination; values are small positive longs, so `MAX_VALUE - versionId`
+  ordering in versionedKeyTable stays plain signed arithmetic; and the id sorts by
+  creation time, which is what the version chain wants anyway.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes its own sentinel, `PinnedFirstVersionIdGenerator.FIRST_VERSION_ID
+  = 1`, below any proposed id, so it sorts oldest and can be referenced without
+  listing the key's versions first. All later versions take a normally proposed id. First versions are detected by "no
+  current version in keyTable", reusing the lookup the write path performs anyway.
+  Known trade-off: if every version of a key is permanently deleted and the key is
+  recreated, the new first version takes the sentinel again; only deployments that
+  accept this should configure this generator.
+
+**How the id appears on the wire.** `x-amz-version-id`, `?versionId=` and
+`ListObjectVersions` carry the id as its decimal digits, the way a GCS
+`generation` is rendered: `117121056706265088` for a proposed id, `1` for a
+pinned first version. Decimal digits cannot collide with `null`, the identifier
+S3 reserves for a key's null version, so both share the parameter without
+escaping. Clients should still treat the value as opaque. The digits of a proposed
+id do decode to the millisecond of the write, but that is a property of this
+generator rather than of the API, and ids are ordered only within one key —
+comparing them across keys means nothing.
+
+**Where the id is proposed.** The generator runs on the leader in `preExecute` and
+the id is stamped into the request that goes into the replication log, so every OM
+applies an identical value; nothing is proposed during apply. The one apply-time
+decision is the pinned sentinel, which is a constant and is therefore deterministic
+on its own: if the pinned generator is configured and the key has no current
+version at apply time, the version takes `FIRST_VERSION_ID` instead of the proposed
+id.
+
+**Why a proposed id is only a floor.** A time-based id is monotonic within one OM
+process, not across a leader change onto a node whose clock lags, and the "strictly
+increasing within a key" guarantee binds one generator rather than a sequence of
+them. So the generator only proposes, and the id a version is applied with is the
+later of that proposal and the id after the key's current version:
+
+```
+versionId = max(proposedVersionId, currentVersionId + 1)
+```
+
+The write path already holds the current version, so this costs no read, and it
+reads no other key and no global state: it is a pure function of the replicated
+request and the state every OM already shares, so all of them settle on the same id
+without coordinating. A current version predating versioning carries no id, and a
+key in that state has no other versions to order against, so it takes the proposed
+value unchanged — no versionedKeyTable lookup in any case.
+
+Ordering therefore never rests on the clock being trustworthy, and neither does it
+rest on one generator having proposed every id of a key. Under a clock regression, or
+after a change of generator, the ids of an affected key climb by one until proposed
+values overtake them again: the versions stay correctly ordered, and only the id's
+reading as a time degrades — the right one of the two to give up. An earlier draft
+instead rejected such a commit with `INVALID_REQUEST`; that is dropped, because it
+fails a client write over an operational clock problem the client can neither see
+nor fix.
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected.
+- **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+
+```mermaid
+flowchart TB
+    G["GET / HEAD k"] --> Q{"versionId given?"}
+    Q -->|no| C{"current version in keyTable"}
+    C -->|object| C1["200, the current version"]
+    C -->|delete marker| C2["404<br/>x-amz-delete-marker: true"]
+    Q -->|yes| L["check the current version,<br/>then point-look-up versionedKeyTable"]
+    L --> M{"the addressed version"}
+    M -->|object| M1["200, that version"]
+    M -->|delete marker| M2["405<br/>x-amz-delete-marker: true<br/>Allow: DELETE"]
+    M -->|no such version| M3["404"]
+```
+
+The 404/405 split is S3's, and it follows from the marker being addressable: an
+unaddressed read steps over the marker and reports the key absent, while a read
+that names the marker gets told the version exists but has no body to return.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) A batch delete is the same operation over
+  more keys and takes the same path — it goes through a request of its own, which
+  inserts one marker per key from the same implementation, so the two cannot
+  diverge. Hard-deleting there instead would destroy the current version and leave
+  the versions under it in the versionedKeyTable with nothing above them: absent to
+  a plain read, holding quota, and unreachable by promotion, which only runs when a
+  version is deleted by id. **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion as shown above.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation: version-aware lifecycle rules
+
+Version reclamation is expressed as **lifecycle rules on the lifecycle engine
+already in master** (HDDS-8342: `OmLifecycleConfiguration`, `KeyLifecycleService`,
+and the three S3 lifecycle endpoints), not as a service of its own. The engine
+already provides everything version reclamation needs — per-bucket rule storage
+and validation, prefix/tag filters, a resumable background scan with saved scan
+state, batched deletes through Ratis, metrics, and `suspend`/`resume` admin
+commands. Versioning contributes the three actions that the feature documentation
+currently lists as unsupported, and each maps 1:1 onto an S3 element:
+
+| S3 lifecycle element | Effect on a versioned bucket |
+|---|---|
+| `NoncurrentVersionExpiration.NoncurrentDays` | permanently deletes a noncurrent version once it has been noncurrent that long |
+| `NoncurrentVersionExpiration.NewerNoncurrentVersions` | keeps at most N noncurrent versions of a key, reclaiming oldest-first |
+| `Expiration.ExpiredObjectDeleteMarker` | removes a delete marker, and the key with it, once it is the only version left |
+| `Expiration.Days` / `.Date` (existing action) | on a versioned bucket, inserts a delete marker instead of deleting the object, as S3 does — through the same insertion the write requests use, so an expiry and a DELETE leave the same records behind |
+
+Concretely: `LifecycleAction` gains a `noncurrentVersionExpiration` field and
+`LifecycleExpiration` an `expiredObjectDeleteMarker` field — both optional
+additions to existing messages — and `LifecycleActionTask` gains a
+versionedKeyTable scan next to its keyTable scan, reusing the same batching, scan
+state and metrics. Expiration counts from the moment the version that superseded
+this one was committed, as `NoncurrentDays` does, not from when the version itself
+was written.
+
+One deliberate departure from stock S3: **the lifecycle service is enabled by
+default.** `ozone.lifecycle.service.enabled` flips from `false` to `true`. A rule
+that never runs is worse than no rule, and versioning is the first feature that
+depends on reclamation for correct operation rather than for tidiness. OM rejects
+`PutBucketVersioning(Enabled)` while the lifecycle service is disabled: that does
+not promise a bucket's versions are bounded — one whose owner writes no rule
+accumulates them, as on S3 — but it rules out a rule existing that nothing will
+ever act on.
+
+Earlier drafts bounded version growth with a bucket-level `maxVersions` property
+as well, first refusing the write once the limit was reached and later trimming
+oldest-first from the same scan. Both are dropped. A per-bucket cap duplicates
+`NewerNoncurrentVersions` in an Ozone-private property, which cuts against
+expressing reclamation as lifecycle rules at all; and applying it to the buckets
+it existed for — those carrying no lifecycle configuration — meant scheduling scan
+work for buckets the engine otherwise never visits, a second kind of task that
+differs from every other one in what it may assume. A bucket with no rule now
+accumulates versions, which is what S3 does.
+
+## Quota and observability
+
+All versions count against the bucket space quota. Version count, marker count,

Review Comment:
   Yes, it's planed in [T10](https://docs.google.com/document/d/1uT3Hzss5gJkiuxzzeZYVUB_UqjSbqROUApY5pH6k8TI/edit?tab=t.0#heading=h.jni13mkcoukl).
   
   For the second question, I'd leave the formula to HDDS-16176 and follow up in this feature once it's fixed. Added to open questions.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3894723952


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,630 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.

Review Comment:
   Recon impact will come as part to update usage size of keyTable, and also need understand versioning.



##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,630 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+The only inbound edge is `PutBucketVersioning`, as in S3, where `CreateBucket`
+carries no versioning parameter: a bucket is always created Unversioned. OM
+rejects a `CreateBucket` that carries a versioning status, also with
+`INVALID_REQUEST`, so the state a bucket starts in cannot be chosen and every
+status it ever holds has been through the transition check above. The field is
+on `BucketInfo` because that message is the bucket's on-disk record and the
+shape `InfoBucket` and `ListBuckets` return, not because a create needs it.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept, and a status
+derives it (`ENABLED → true`, otherwise `false`). The reverse does not hold: a
+record without the enum reads as `UNVERSIONED` whatever the boolean says, and a
+legacy client setting the boolean on such a bucket leaves it without a status,
+so it keeps using the in-record block version list rather than being opted into
+S3 versioning semantics behind its owner's back. A legacy client's boolean is
+mapped onto the state machine only on a bucket that already carries a status,
+where the two have to stay consistent. Old and new clients/OMs coexist during
+rolling upgrades either way.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest.
+
+```mermaid
+flowchart LR
+    subgraph KT["keyTable — one entry per key"]
+        K1["/vol/buck/photo.jpg<br/>versionId 104 · current"]
+    end
+    subgraph VKT["versionedKeyTable — noncurrent, newest → oldest"]
+        V1["/vol/buck/photo.jpg 0x00 MAX-103<br/>delete marker 103"]
+        V2["/vol/buck/photo.jpg 0x00 MAX-102<br/>object 102"]
+        V3["/vol/buck/photo.jpg 0x00 MAX-101<br/>object 101"]
+    end
+    K1 -. "one seek reaches the newest noncurrent version" .-> V1
+    V1 --> V2 --> V3
+```
+
+Because the suffix is `MAX - versionId`, a single `seek` on the key's prefix lands
+on the newest noncurrent version: `ListObjectVersions` and version promotion are a
+seek plus a sequential read, never a scan-and-sort. The table is registered in
+`OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## How a key's versions evolve
+
+The two mechanisms that are hardest to read out of prose — the delete marker and
+the null version — are easiest to follow as one continuous story of a single key
+`k`. `obj(n)` is a data version with versionId `n`, `marker(n)` a delete marker,
+and `null` marks the record occupying the key's null slot.
+
+| # | Operation | Bucket state | keyTable (current) | versionedKeyTable (newest → oldest) |
+|---|---|---|---|---|
+| 0 | `PUT k` | Unversioned | `obj(—)` no versionId | – |
+| 1 | `PutBucketVersioning(Enabled)` | Enabled | `obj(—)` read as the null version | – |
+| 2 | `PUT k` | Enabled | `obj(101)` | `obj(—)` |
+| 3 | `PUT k` | Enabled | `obj(102)` | `obj(101)`, `obj(—)` |
+| 4 | `DELETE k` | Enabled | `marker(103)` | `obj(102)`, `obj(101)`, `obj(—)` |
+| 5 | `DELETE k?versionId=103` | Enabled | `obj(102)` **promoted back** | `obj(101)`, `obj(—)` |
+| 6 | `PutBucketVersioning(Suspended)` | Suspended | `obj(102)` | `obj(101)`, `obj(—)` |
+| 7 | `PUT k` | Suspended | `obj(104, null)` | `obj(102)`, `obj(101)` |
+| 8 | `PUT k` | Suspended | `obj(105, null)` overwritten in place | unchanged |
+| 9 | `DELETE k` | Suspended | `marker(106, null)` overwritten in place | unchanged |
+| 10 | `PutBucketVersioning(Enabled)`, `PUT k` | Enabled | `obj(107)` | `marker(106, null)`, `obj(102)`, `obj(101)` |
+
+Three things to read out of it:
+
+- **A delete marker is an ordinary version that happens to carry no blocks**
+  (step 4). It becomes the current version, so a plain `GET` returns 404 with
+  `x-amz-delete-marker: true` while every earlier version stays intact and
+  addressable by versionId. Deleting the marker by versionId (step 5) is exactly
+  S3's "restore an object": the newest remaining version is promoted back into
+  keyTable.
+- **Suspended does not stop versioning, it stops *accumulating*** (steps 7–9).
+  Each write lands in the key's single null slot: the first suspended write pushes
+  the previous current version down into versionedKeyTable and drops the key's
+  older null record (step 7); every later suspended write overwrites the null slot
+  in place, so the version count stops growing. Versions accumulated while Enabled
+  are never touched.
+- **The null version is a slot, not a reserved id** (step 7 onwards). It carries a
+  normally generated versionId like any other version and is marked by
+  `isNullVersion`. Pinning it to a fixed low id would misorder a null created
+  between two versioned writes, which is the middle version of the key, not its
+  oldest. Step 10 shows a null delete marker sliding into the noncurrent chain in
+  its correct chronological position once the bucket is Enabled again.
+
+### Where a write lands
+
+The table above is one trace; the rule behind it is the same for every write that
+carries no versionId. Suspended is the branch worth reading twice: it does not
+stop versioning, it collapses every write into the key's one null slot, which is
+why the version count stops growing while the versions accumulated under Enabled
+stay untouched.
+
+```mermaid
+flowchart TB
+    W["PUT k / DELETE k<br/>(no versionId)"] --> S{"bucket state"}
+    S -->|Unversioned| U["overwrite keyTable in place<br/>old blocks → deletedTable<br/><i>unchanged from today</i>"]
+    S -->|Enabled| E["move current version → versionedKeyTable<br/>write the new record into keyTable"]
+    S -->|Suspended| N{"is the current version<br/>already the null slot?"}
+    N -->|yes| N1["overwrite the null record in place<br/>its blocks → deletedTable, a marker carries none<br/><b>version count does not grow</b>"]
+    N -->|no| N2["move current version → versionedKeyTable<br/>write the new record as current, isNullVersion<br/>drop the key's older null record, if any"]
+    E --> R["new current version:<br/>an object on PUT, a delete marker on DELETE"]
+    N1 --> R
+    N2 --> R
+```
+
+### The life of a delete marker
+
+A delete marker is not a tombstone the read path has to special-case away — it is
+an ordinary version that carries no blocks, and it occupies the current slot like
+any other. That single fact produces all of its externally visible behaviour:
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    Obj: current = obj(102)
+    Marker: current = marker(103)
+    Gone: key no longer exists
+    [*] --> Obj: PUT k
+    Obj --> Marker: DELETE k (no versionId)
+    Marker --> Obj: DELETE k?versionId=103<br/>promotion restores obj(102)
+    Marker --> Gone: lifecycle ExpiredObjectDeleteMarker<br/>once it is the only version left
+    Obj --> Obj: GET k → 200
+    Marker --> Marker: GET k → 404, x-amz-delete-marker
+```
+
+Deleting the marker by versionId is exactly S3's "restore an object": nothing is
+rewritten, the newest remaining version is moved back into keyTable.
+
+### Version promotion
+
+The invariant is that keyTable always holds a key's current version. When a
+permanent delete removes the current version, promotion restores the invariant in
+the same WriteBatch, under the bucket write lock:
+
+```mermaid
+flowchart TB
+    A["DELETE k?versionId = current"] --> B["remove the current entry from keyTable<br/>blocks → deletedTable"]
+    B --> C{"seek versionedKeyTable<br/>on k's prefix"}
+    C -->|"a noncurrent version exists"| D["move the newest one back into keyTable<br/>record content stays frozen — a pure positional move"]
+    C -->|"none left"| E["the key disappears entirely"]
+```
+
+## VersionId generation
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,

Review Comment:
   Version will be generated once key is committed in validateUpdateFlow. For simplify,
   1. Keep maxVersionId in keyInfo (with default = 0 for first key)
   2. VersionId = maxVersionId + 1
   when moving version across keyTable, versionKeyTable, can update maxVersionId as max of movement.



##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,630 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+The only inbound edge is `PutBucketVersioning`, as in S3, where `CreateBucket`
+carries no versioning parameter: a bucket is always created Unversioned. OM
+rejects a `CreateBucket` that carries a versioning status, also with
+`INVALID_REQUEST`, so the state a bucket starts in cannot be chosen and every
+status it ever holds has been through the transition check above. The field is
+on `BucketInfo` because that message is the bucket's on-disk record and the
+shape `InfoBucket` and `ListBuckets` return, not because a create needs it.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept, and a status
+derives it (`ENABLED → true`, otherwise `false`). The reverse does not hold: a
+record without the enum reads as `UNVERSIONED` whatever the boolean says, and a
+legacy client setting the boolean on such a bucket leaves it without a status,
+so it keeps using the in-record block version list rather than being opted into
+S3 versioning semantics behind its owner's back. A legacy client's boolean is
+mapped onto the state machine only on a bucket that already carries a status,
+where the two have to stay consistent. Old and new clients/OMs coexist during
+rolling upgrades either way.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current

Review Comment:
   keeping a separate versionedKeyTable is better, but some impact of this may have multi-part table where its keyInfo is splitted.



##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,630 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+The only inbound edge is `PutBucketVersioning`, as in S3, where `CreateBucket`
+carries no versioning parameter: a bucket is always created Unversioned. OM
+rejects a `CreateBucket` that carries a versioning status, also with
+`INVALID_REQUEST`, so the state a bucket starts in cannot be chosen and every
+status it ever holds has been through the transition check above. The field is
+on `BucketInfo` because that message is the bucket's on-disk record and the
+shape `InfoBucket` and `ListBuckets` return, not because a create needs it.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept, and a status
+derives it (`ENABLED → true`, otherwise `false`). The reverse does not hold: a
+record without the enum reads as `UNVERSIONED` whatever the boolean says, and a
+legacy client setting the boolean on such a bucket leaves it without a status,
+so it keeps using the in-record block version list rather than being opted into
+S3 versioning semantics behind its owner's back. A legacy client's boolean is
+mapped onto the state machine only on a bucket that already carries a status,
+where the two have to stay consistent. Old and new clients/OMs coexist during
+rolling upgrades either way.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest.
+
+```mermaid
+flowchart LR
+    subgraph KT["keyTable — one entry per key"]
+        K1["/vol/buck/photo.jpg<br/>versionId 104 · current"]
+    end
+    subgraph VKT["versionedKeyTable — noncurrent, newest → oldest"]
+        V1["/vol/buck/photo.jpg 0x00 MAX-103<br/>delete marker 103"]
+        V2["/vol/buck/photo.jpg 0x00 MAX-102<br/>object 102"]
+        V3["/vol/buck/photo.jpg 0x00 MAX-101<br/>object 101"]
+    end
+    K1 -. "one seek reaches the newest noncurrent version" .-> V1
+    V1 --> V2 --> V3
+```
+
+Because the suffix is `MAX - versionId`, a single `seek` on the key's prefix lands
+on the newest noncurrent version: `ListObjectVersions` and version promotion are a
+seek plus a sequential read, never a scan-and-sort. The table is registered in
+`OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## How a key's versions evolve
+
+The two mechanisms that are hardest to read out of prose — the delete marker and

Review Comment:
   we need support listing with markedDelete=true for listing keys whose current version is marked deleted in this case. we need provide set of impact to all api and new parameter required in this case



##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,630 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+The only inbound edge is `PutBucketVersioning`, as in S3, where `CreateBucket`
+carries no versioning parameter: a bucket is always created Unversioned. OM
+rejects a `CreateBucket` that carries a versioning status, also with
+`INVALID_REQUEST`, so the state a bucket starts in cannot be chosen and every
+status it ever holds has been through the transition check above. The field is
+on `BucketInfo` because that message is the bucket's on-disk record and the
+shape `InfoBucket` and `ListBuckets` return, not because a create needs it.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept, and a status
+derives it (`ENABLED → true`, otherwise `false`). The reverse does not hold: a
+record without the enum reads as `UNVERSIONED` whatever the boolean says, and a
+legacy client setting the boolean on such a bucket leaves it without a status,
+so it keeps using the in-record block version list rather than being opted into
+S3 versioning semantics behind its owner's back. A legacy client's boolean is
+mapped onto the state machine only on a bucket that already carries a status,
+where the two have to stay consistent. Old and new clients/OMs coexist during
+rolling upgrades either way.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest.
+
+```mermaid
+flowchart LR
+    subgraph KT["keyTable — one entry per key"]
+        K1["/vol/buck/photo.jpg<br/>versionId 104 · current"]
+    end
+    subgraph VKT["versionedKeyTable — noncurrent, newest → oldest"]
+        V1["/vol/buck/photo.jpg 0x00 MAX-103<br/>delete marker 103"]
+        V2["/vol/buck/photo.jpg 0x00 MAX-102<br/>object 102"]
+        V3["/vol/buck/photo.jpg 0x00 MAX-101<br/>object 101"]
+    end
+    K1 -. "one seek reaches the newest noncurrent version" .-> V1
+    V1 --> V2 --> V3
+```
+
+Because the suffix is `MAX - versionId`, a single `seek` on the key's prefix lands
+on the newest noncurrent version: `ListObjectVersions` and version promotion are a
+seek plus a sequential read, never a scan-and-sort. The table is registered in
+`OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## How a key's versions evolve
+
+The two mechanisms that are hardest to read out of prose — the delete marker and
+the null version — are easiest to follow as one continuous story of a single key
+`k`. `obj(n)` is a data version with versionId `n`, `marker(n)` a delete marker,
+and `null` marks the record occupying the key's null slot.
+
+| # | Operation | Bucket state | keyTable (current) | versionedKeyTable (newest → oldest) |
+|---|---|---|---|---|
+| 0 | `PUT k` | Unversioned | `obj(—)` no versionId | – |
+| 1 | `PutBucketVersioning(Enabled)` | Enabled | `obj(—)` read as the null version | – |
+| 2 | `PUT k` | Enabled | `obj(101)` | `obj(—)` |
+| 3 | `PUT k` | Enabled | `obj(102)` | `obj(101)`, `obj(—)` |
+| 4 | `DELETE k` | Enabled | `marker(103)` | `obj(102)`, `obj(101)`, `obj(—)` |
+| 5 | `DELETE k?versionId=103` | Enabled | `obj(102)` **promoted back** | `obj(101)`, `obj(—)` |
+| 6 | `PutBucketVersioning(Suspended)` | Suspended | `obj(102)` | `obj(101)`, `obj(—)` |
+| 7 | `PUT k` | Suspended | `obj(104, null)` | `obj(102)`, `obj(101)` |
+| 8 | `PUT k` | Suspended | `obj(105, null)` overwritten in place | unchanged |
+| 9 | `DELETE k` | Suspended | `marker(106, null)` overwritten in place | unchanged |
+| 10 | `PutBucketVersioning(Enabled)`, `PUT k` | Enabled | `obj(107)` | `marker(106, null)`, `obj(102)`, `obj(101)` |
+
+Three things to read out of it:
+
+- **A delete marker is an ordinary version that happens to carry no blocks**
+  (step 4). It becomes the current version, so a plain `GET` returns 404 with
+  `x-amz-delete-marker: true` while every earlier version stays intact and
+  addressable by versionId. Deleting the marker by versionId (step 5) is exactly
+  S3's "restore an object": the newest remaining version is promoted back into
+  keyTable.
+- **Suspended does not stop versioning, it stops *accumulating*** (steps 7–9).
+  Each write lands in the key's single null slot: the first suspended write pushes
+  the previous current version down into versionedKeyTable and drops the key's
+  older null record (step 7); every later suspended write overwrites the null slot
+  in place, so the version count stops growing. Versions accumulated while Enabled
+  are never touched.
+- **The null version is a slot, not a reserved id** (step 7 onwards). It carries a
+  normally generated versionId like any other version and is marked by
+  `isNullVersion`. Pinning it to a fixed low id would misorder a null created
+  between two versioned writes, which is the middle version of the key, not its
+  oldest. Step 10 shows a null delete marker sliding into the noncurrent chain in
+  its correct chronological position once the bucket is Enabled again.
+
+### Where a write lands
+
+The table above is one trace; the rule behind it is the same for every write that
+carries no versionId. Suspended is the branch worth reading twice: it does not
+stop versioning, it collapses every write into the key's one null slot, which is
+why the version count stops growing while the versions accumulated under Enabled
+stay untouched.
+
+```mermaid
+flowchart TB
+    W["PUT k / DELETE k<br/>(no versionId)"] --> S{"bucket state"}
+    S -->|Unversioned| U["overwrite keyTable in place<br/>old blocks → deletedTable<br/><i>unchanged from today</i>"]
+    S -->|Enabled| E["move current version → versionedKeyTable<br/>write the new record into keyTable"]
+    S -->|Suspended| N{"is the current version<br/>already the null slot?"}
+    N -->|yes| N1["overwrite the null record in place<br/>its blocks → deletedTable, a marker carries none<br/><b>version count does not grow</b>"]
+    N -->|no| N2["move current version → versionedKeyTable<br/>write the new record as current, isNullVersion<br/>drop the key's older null record, if any"]
+    E --> R["new current version:<br/>an object on PUT, a delete marker on DELETE"]
+    N1 --> R
+    N2 --> R
+```
+
+### The life of a delete marker
+
+A delete marker is not a tombstone the read path has to special-case away — it is
+an ordinary version that carries no blocks, and it occupies the current slot like
+any other. That single fact produces all of its externally visible behaviour:
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    Obj: current = obj(102)
+    Marker: current = marker(103)
+    Gone: key no longer exists
+    [*] --> Obj: PUT k
+    Obj --> Marker: DELETE k (no versionId)
+    Marker --> Obj: DELETE k?versionId=103<br/>promotion restores obj(102)
+    Marker --> Gone: lifecycle ExpiredObjectDeleteMarker<br/>once it is the only version left
+    Obj --> Obj: GET k → 200
+    Marker --> Marker: GET k → 404, x-amz-delete-marker
+```
+
+Deleting the marker by versionId is exactly S3's "restore an object": nothing is
+rewritten, the newest remaining version is moved back into keyTable.
+
+### Version promotion
+
+The invariant is that keyTable always holds a key's current version. When a
+permanent delete removes the current version, promotion restores the invariant in
+the same WriteBatch, under the bucket write lock:
+
+```mermaid
+flowchart TB
+    A["DELETE k?versionId = current"] --> B["remove the current entry from keyTable<br/>blocks → deletedTable"]
+    B --> C{"seek versionedKeyTable<br/>on k's prefix"}
+    C -->|"a noncurrent version exists"| D["move the newest one back into keyTable<br/>record content stays frozen — a pure positional move"]
+    C -->|"none left"| E["the key disappears entirely"]
+```
+
+## VersionId generation
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and read when an OM
+starts, so a change takes effect on every bucket once the OMs have been restarted;
+it is not recorded in bucket metadata. Every generator must satisfy, **for itself**:
+strictly increasing within a key (a later version's id is always greater than every
+earlier version's id of that key); frozen once assigned; and `0` never handed out,
+being the unset value of the optional field. `1` is reserved as well, but by the
+pinned generator below rather than by the interface: it is that generator's own
+sentinel, which it assigns and never proposes.
+
+The identifier is deliberately **not** derived from OM's Ratis transaction index.
+The versionId is persisted, externally referenced, and part of the S3 API surface;
+tying it to the replication log would export an internal counter into a public
+identity and pin the feature to one execution model, when the OM execution path is
+itself an abstraction (`ExecutionContext` already carries an index whose term may
+be absent).
+
+Numbering a version by when it was written is what S3-compatible stores generally
+do: a GCS `generation` is a microsecond timestamp, an Azure Blob version id is an
+ISO-8601 instant, and RocksDB's user-defined timestamps order the versions of a key
+by a `key + timestamp` suffix — the same encoding the versionedKeyTable dbKey
+already uses. RocksDB is also the precedent for the split below: it orders by the
+timestamp but leaves keeping them monotonic to the caller.
+
+The shape matches SCM's `SequenceIdGenerator` — a cheap local allocation plus a
+floor that guarantees monotonicity. SCM's class itself is not reusable here: it
+lives in `server-scm` on `SCMHAManager` / `@Replicate`, so it would couple OM's
+write path to SCM across a module boundary. It also needs machinery this does not,
+because `localId` has to increase globally while a `versionId` only has to increase
+within one key, whose floor is already persisted as that key's current version. SCM
+does show the migration path if a persisted sequence is ever wanted:
+`upgradeToSequenceId` seeds one from a future timestamp in this same encoding.
+
+- **`UniqueIdVersionIdGenerator` (default)** — the same time-based scheme Ozone
+  already uses to generate block local IDs (`hdds.utils.UniqueId`): the id is
+  `currentTimeMillis << 16 | counter`, a 16-bit counter disambiguating ids proposed
+  within the same millisecond. It needs no allocator, no persistent state and no
+  coordination; values are small positive longs, so `MAX_VALUE - versionId`
+  ordering in versionedKeyTable stays plain signed arithmetic; and the id sorts by
+  creation time, which is what the version chain wants anyway.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes its own sentinel, `PinnedFirstVersionIdGenerator.FIRST_VERSION_ID
+  = 1`, below any proposed id, so it sorts oldest and can be referenced without
+  listing the key's versions first. All later versions take a normally proposed id. First versions are detected by "no
+  current version in keyTable", reusing the lookup the write path performs anyway.
+  Known trade-off: if every version of a key is permanently deleted and the key is
+  recreated, the new first version takes the sentinel again; only deployments that
+  accept this should configure this generator.
+
+**How the id appears on the wire.** `x-amz-version-id`, `?versionId=` and
+`ListObjectVersions` carry the id as its decimal digits, the way a GCS
+`generation` is rendered: `117121056706265088` for a proposed id, `1` for a
+pinned first version. Decimal digits cannot collide with `null`, the identifier
+S3 reserves for a key's null version, so both share the parameter without
+escaping. Clients should still treat the value as opaque. The digits of a proposed
+id do decode to the millisecond of the write, but that is a property of this
+generator rather than of the API, and ids are ordered only within one key —
+comparing them across keys means nothing.
+
+**Where the id is proposed.** The generator runs on the leader in `preExecute` and
+the id is stamped into the request that goes into the replication log, so every OM
+applies an identical value; nothing is proposed during apply. The one apply-time
+decision is the pinned sentinel, which is a constant and is therefore deterministic
+on its own: if the pinned generator is configured and the key has no current
+version at apply time, the version takes `FIRST_VERSION_ID` instead of the proposed
+id.
+
+**Why a proposed id is only a floor.** A time-based id is monotonic within one OM
+process, not across a leader change onto a node whose clock lags, and the "strictly
+increasing within a key" guarantee binds one generator rather than a sequence of
+them. So the generator only proposes, and the id a version is applied with is the
+later of that proposal and the id after the key's current version:
+
+```
+versionId = max(proposedVersionId, currentVersionId + 1)
+```
+
+The write path already holds the current version, so this costs no read, and it
+reads no other key and no global state: it is a pure function of the replicated
+request and the state every OM already shares, so all of them settle on the same id
+without coordinating. A current version predating versioning carries no id, and a
+key in that state has no other versions to order against, so it takes the proposed
+value unchanged — no versionedKeyTable lookup in any case.
+
+Ordering therefore never rests on the clock being trustworthy, and neither does it
+rest on one generator having proposed every id of a key. Under a clock regression, or
+after a change of generator, the ids of an affected key climb by one until proposed
+values overtake them again: the versions stay correctly ordered, and only the id's
+reading as a time degrades — the right one of the two to give up. An earlier draft
+instead rejected such a commit with `INVALID_REQUEST`; that is dropped, because it
+fails a client write over an operational clock problem the client can neither see
+nor fix.
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)

Review Comment:
   we can list down new ozone CLI, ozone api, and S3 api that needs to be handled



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on PR #10822:
URL: https://github.com/apache/ozone/pull/10822#issuecomment-5326593120

   @ChenSammi 
   
   > On decoupling versionId from the Ratis transaction index
   
   The goal here is to stop deriving the id from the replication log — versionId is user-visible and permanent, and the versionedKeyTable layout depends on it being ordered, which is the property most at risk if OM's execution model changes.
   
   First look was SCM's SequenceIdGenerator, since it solves the same problem. It doesn't fit: it lives in server-scm and is built on SCMHAManager / DBTransactionBuffer / @Replicate, so reusing it would couple OM's write path to SCM across a module boundary. Its shape is still instructive though — a cheap local allocation plus a floor that guarantees monotonicity.
   
   So instead, looking at how object stores identify a version: a GCS generation is a microsecond timestamp, an Azure Blob version id is an ISO-8601 instant, and RocksDB's user-defined timestamps order versions by a key + timestamp suffix — the same encoding we already use in the dbKey. All three make the timestamp the sort key.
   
   Proposal: timestamp + max.
   
   The OM handling the request proposes a versionId in preExecute — wall clock in microseconds. preExecute's output is what goes to Ratis, so the id is fixed before replication.
   On apply, VersionIdAllocator settles it: max(proposedVersionId, currentVersionId + 1). A clock can't promise per-key ordering alone (same-microsecond writes, or a lagging clock after leader change), so the proposal is only a floor. The current version is already in hand, so this costs no read and stays a pure function of replicated state.
   
   That gives SCM's local-allocation-plus-floor structure without its batch machinery, since a versionId only has to increase within one key.
   
   Cost: one optional field on the commit / delete / complete-MPU requests. No new table, no extra I/O.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3756861818


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.

Review Comment:
   "**PUT (Unversioned)**: unchanged from today." can have its own new line. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3900041922


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,630 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+The only inbound edge is `PutBucketVersioning`, as in S3, where `CreateBucket`
+carries no versioning parameter: a bucket is always created Unversioned. OM
+rejects a `CreateBucket` that carries a versioning status, also with
+`INVALID_REQUEST`, so the state a bucket starts in cannot be chosen and every
+status it ever holds has been through the transition check above. The field is
+on `BucketInfo` because that message is the bucket's on-disk record and the
+shape `InfoBucket` and `ListBuckets` return, not because a create needs it.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept, and a status
+derives it (`ENABLED → true`, otherwise `false`). The reverse does not hold: a
+record without the enum reads as `UNVERSIONED` whatever the boolean says, and a
+legacy client setting the boolean on such a bucket leaves it without a status,
+so it keeps using the in-record block version list rather than being opted into
+S3 versioning semantics behind its owner's back. A legacy client's boolean is
+mapped onto the state machine only on a bucket that already carries a status,
+where the two have to stay consistent. Old and new clients/OMs coexist during
+rolling upgrades either way.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest.
+
+```mermaid
+flowchart LR
+    subgraph KT["keyTable — one entry per key"]
+        K1["/vol/buck/photo.jpg<br/>versionId 104 · current"]
+    end
+    subgraph VKT["versionedKeyTable — noncurrent, newest → oldest"]
+        V1["/vol/buck/photo.jpg 0x00 MAX-103<br/>delete marker 103"]
+        V2["/vol/buck/photo.jpg 0x00 MAX-102<br/>object 102"]
+        V3["/vol/buck/photo.jpg 0x00 MAX-101<br/>object 101"]
+    end
+    K1 -. "one seek reaches the newest noncurrent version" .-> V1
+    V1 --> V2 --> V3
+```
+
+Because the suffix is `MAX - versionId`, a single `seek` on the key's prefix lands
+on the newest noncurrent version: `ListObjectVersions` and version promotion are a
+seek plus a sequential read, never a scan-and-sort. The table is registered in
+`OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## How a key's versions evolve
+
+The two mechanisms that are hardest to read out of prose — the delete marker and
+the null version — are easiest to follow as one continuous story of a single key
+`k`. `obj(n)` is a data version with versionId `n`, `marker(n)` a delete marker,
+and `null` marks the record occupying the key's null slot.
+
+| # | Operation | Bucket state | keyTable (current) | versionedKeyTable (newest → oldest) |
+|---|---|---|---|---|
+| 0 | `PUT k` | Unversioned | `obj(—)` no versionId | – |
+| 1 | `PutBucketVersioning(Enabled)` | Enabled | `obj(—)` read as the null version | – |
+| 2 | `PUT k` | Enabled | `obj(101)` | `obj(—)` |
+| 3 | `PUT k` | Enabled | `obj(102)` | `obj(101)`, `obj(—)` |
+| 4 | `DELETE k` | Enabled | `marker(103)` | `obj(102)`, `obj(101)`, `obj(—)` |
+| 5 | `DELETE k?versionId=103` | Enabled | `obj(102)` **promoted back** | `obj(101)`, `obj(—)` |
+| 6 | `PutBucketVersioning(Suspended)` | Suspended | `obj(102)` | `obj(101)`, `obj(—)` |
+| 7 | `PUT k` | Suspended | `obj(104, null)` | `obj(102)`, `obj(101)` |
+| 8 | `PUT k` | Suspended | `obj(105, null)` overwritten in place | unchanged |
+| 9 | `DELETE k` | Suspended | `marker(106, null)` overwritten in place | unchanged |
+| 10 | `PutBucketVersioning(Enabled)`, `PUT k` | Enabled | `obj(107)` | `marker(106, null)`, `obj(102)`, `obj(101)` |
+
+Three things to read out of it:
+
+- **A delete marker is an ordinary version that happens to carry no blocks**
+  (step 4). It becomes the current version, so a plain `GET` returns 404 with
+  `x-amz-delete-marker: true` while every earlier version stays intact and
+  addressable by versionId. Deleting the marker by versionId (step 5) is exactly
+  S3's "restore an object": the newest remaining version is promoted back into
+  keyTable.
+- **Suspended does not stop versioning, it stops *accumulating*** (steps 7–9).
+  Each write lands in the key's single null slot: the first suspended write pushes
+  the previous current version down into versionedKeyTable and drops the key's
+  older null record (step 7); every later suspended write overwrites the null slot
+  in place, so the version count stops growing. Versions accumulated while Enabled
+  are never touched.
+- **The null version is a slot, not a reserved id** (step 7 onwards). It carries a
+  normally generated versionId like any other version and is marked by
+  `isNullVersion`. Pinning it to a fixed low id would misorder a null created
+  between two versioned writes, which is the middle version of the key, not its
+  oldest. Step 10 shows a null delete marker sliding into the noncurrent chain in
+  its correct chronological position once the bucket is Enabled again.
+
+### Where a write lands
+
+The table above is one trace; the rule behind it is the same for every write that
+carries no versionId. Suspended is the branch worth reading twice: it does not
+stop versioning, it collapses every write into the key's one null slot, which is
+why the version count stops growing while the versions accumulated under Enabled
+stay untouched.
+
+```mermaid
+flowchart TB
+    W["PUT k / DELETE k<br/>(no versionId)"] --> S{"bucket state"}
+    S -->|Unversioned| U["overwrite keyTable in place<br/>old blocks → deletedTable<br/><i>unchanged from today</i>"]
+    S -->|Enabled| E["move current version → versionedKeyTable<br/>write the new record into keyTable"]
+    S -->|Suspended| N{"is the current version<br/>already the null slot?"}
+    N -->|yes| N1["overwrite the null record in place<br/>its blocks → deletedTable, a marker carries none<br/><b>version count does not grow</b>"]
+    N -->|no| N2["move current version → versionedKeyTable<br/>write the new record as current, isNullVersion<br/>drop the key's older null record, if any"]
+    E --> R["new current version:<br/>an object on PUT, a delete marker on DELETE"]
+    N1 --> R
+    N2 --> R
+```
+
+### The life of a delete marker
+
+A delete marker is not a tombstone the read path has to special-case away — it is
+an ordinary version that carries no blocks, and it occupies the current slot like
+any other. That single fact produces all of its externally visible behaviour:
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    Obj: current = obj(102)
+    Marker: current = marker(103)
+    Gone: key no longer exists
+    [*] --> Obj: PUT k
+    Obj --> Marker: DELETE k (no versionId)
+    Marker --> Obj: DELETE k?versionId=103<br/>promotion restores obj(102)
+    Marker --> Gone: lifecycle ExpiredObjectDeleteMarker<br/>once it is the only version left
+    Obj --> Obj: GET k → 200
+    Marker --> Marker: GET k → 404, x-amz-delete-marker
+```
+
+Deleting the marker by versionId is exactly S3's "restore an object": nothing is
+rewritten, the newest remaining version is moved back into keyTable.
+
+### Version promotion
+
+The invariant is that keyTable always holds a key's current version. When a
+permanent delete removes the current version, promotion restores the invariant in
+the same WriteBatch, under the bucket write lock:
+
+```mermaid
+flowchart TB
+    A["DELETE k?versionId = current"] --> B["remove the current entry from keyTable<br/>blocks → deletedTable"]
+    B --> C{"seek versionedKeyTable<br/>on k's prefix"}
+    C -->|"a noncurrent version exists"| D["move the newest one back into keyTable<br/>record content stays frozen — a pure positional move"]
+    C -->|"none left"| E["the key disappears entirely"]
+```
+
+## VersionId generation
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,

Review Comment:
   Some issues will happen with this approch, one is versionId will be reset when all versions of the key being removed, users will have incorrect value when key being recreated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #10822:
URL: https://github.com/apache/ozone/pull/10822#issuecomment-5410939443

   An idea I had a long time back, is that a versioned bucket effectively gives you infinite snapshots.
   
   If you have the ability to read a bucket "as of" a given time and filter out all the future and past versions, then you can effectively get a consistent read on a bucket and it is almost exactly like a snapshot.
   
   I like in this design how the current key table and the versioned key tabled are separate. Keeping them as the same table with a row per version would result in a listing of the current state being slowed by all the old versions. Having a single row for all versions would result in large values which would be even more problematic. Having the version in the key is important too, as it allows versions we don't care about to be filtered out without deserializing the full value, which is slow.
   
   One thing I am trying to think through - if we wanted to have the ability to read the bucket as of some previous point in time how would that look and are there any changes we may want to make to this design to make that possible, or easier if we later add that?
   
   Lets say you wanted to "snapshot" a versioned bucket using this idea (not the current snapshot implementation). What would that mean?
   
   If we have a monotonically increasing version ID then all you need to so is store the last version_Id somewhere and give it a name:
   
   eg `snapshot versions_bucket_name, daily-backup-2026-08-01`
   
   Then anything that wants to use that snapshot has to use a special version of some list / read API (that does not exist right now) that filters out anything that happened later than version_x. This would just be a scan over the prefix and as we read each row take the latest version of any key which is before the version we care about, and skip any version which is later. Such a listing could be somewhat expensive, but it would be fairly simple to code and potentially quite useful.
   
   Even the snapshot itself isn't really needed it could just be a name attached to a specific version_id. If you have the ability to get the version of some key you wrote and want to see the state of things at that point in time, you just need to use the same list API to list out keys "as of" that version.
   
   If the version_Id is time based, as I saw discussed in some of the comments, there is a possibility to say "show me the state of this bucket as of 5pm yesterday" and it would just work, after adjusting for timezones etc.
   
   One small complication of this approach is that we would need to scan two tables and merge the results, which is slow and tricky code. Reading the table "as of version x" for any key, a version younger than X might be in the key table or the versioned key table.
   
   I feel this is a simpler problem if someone does a point lookup on key1-v123 - where do we look for it? It may be in the current or the versioned table, as we don't know if its the most recent version or not.
   
   What if the design was modified to have the versioned table holding all versions of the key, even the current one? Would it make the point lookup on a version faster and easier (always just read from versioned table for point lookups that include a version). That would mean for a put we would do a double write, but wouldn't need to worry about migrating a record from current to versioned. Same for deletes I think - the tombstone would just go to both tables. One complication is moving a table from non-versioned with existing data to versioned. In that case, the versioned table would not have all the original data. This could be solved with a limitation on how the bucket get created. If you want to have the "list as of" feature, the bucket must have been versioned when it was created.
   
   One thing about using versioned tables in this way - we probably need to ability to prevent deletes of older versions so that the older versions are effectively immutable, except for some retention time. You would also not be able to suspend versioning are the history would be incomplete.
   
   Just to be clear, we don't need to add this sort of stuff to this design now. These are ideas that we might want, as if we can ensure the design now would support them without rework, then it would be nice.
   
   Would it make sense to ensure that the versioned table has all keys? I think it changes this design sightly, but I think the overhead is and complexity is largely the same - we are skipping a migration on new version write, but writing the same key twice to two tables instead.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3757066856


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited; markers
+count toward the limit), `noncurrentVersionExpiration` (opt-in), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed) — all enforced by a new **VersionCleanupService**
+following the `KeyDeletingService` pattern. Versions beyond `maxVersions` are
+reclaimed oldest-first, asynchronously rather than inside the write transaction,
+so the cost of a version write does not grow with the number of versions the key
+already has. An earlier draft also offered a `REJECT` policy that failed the
+write instead; it is dropped. S3 has no error code for the condition and no
+client retries on it, so a default limit enforced that way would break unmodified
+S3 tooling after `maxVersions` overwrites of one key — and reclaiming is what
+S3's own lifecycle rules do. Expiration counts from the moment the version that
+superseded this one was committed, as Lifecycle's `NoncurrentDays` does, not from
+when the version itself was written. All versions count against the bucket space quota;
+version count, marker count, and noncurrent bytes are exposed via
+`ozone sh bucket info` and Recon. `maxVersions` is documented as an Ozone
+extension over S3 semantics (comparable to Lifecycle `NewerNoncurrentVersions`).
+
+## Interaction with Ozone snapshots
+
+**The first version does not allow both features on the same bucket.** The
+version-aware reclamation described in this section is deferred past it, and
+without that lookup the gap below is a data-loss path rather than a rough edge,
+so OM enforces the exclusion instead of documenting an expectation: snapshot
+creation is rejected on an ENABLED or SUSPENDED bucket, and enabling versioning
+or moving SUSPENDED → ENABLED is rejected while any snapshot exists in the

Review Comment:
   Is this drafted by AI?  Maybe we can simply to say that 
   - snapshot creation is only allowed on UNVERSIONED bucket
   - bucket versioning is only allowed to be turned on bucket which doesn't have any snapshot created. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3900103933


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,630 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+The only inbound edge is `PutBucketVersioning`, as in S3, where `CreateBucket`
+carries no versioning parameter: a bucket is always created Unversioned. OM
+rejects a `CreateBucket` that carries a versioning status, also with
+`INVALID_REQUEST`, so the state a bucket starts in cannot be chosen and every
+status it ever holds has been through the transition check above. The field is
+on `BucketInfo` because that message is the bucket's on-disk record and the
+shape `InfoBucket` and `ListBuckets` return, not because a create needs it.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept, and a status
+derives it (`ENABLED → true`, otherwise `false`). The reverse does not hold: a
+record without the enum reads as `UNVERSIONED` whatever the boolean says, and a
+legacy client setting the boolean on such a bucket leaves it without a status,
+so it keeps using the in-record block version list rather than being opted into
+S3 versioning semantics behind its owner's back. A legacy client's boolean is
+mapped onto the state machine only on a bucket that already carries a status,
+where the two have to stay consistent. Old and new clients/OMs coexist during
+rolling upgrades either way.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current

Review Comment:
   Covered: a multipart upload creates its version only at
   CompleteMultipartUpload, so that path demotes the superseded version exactly as a PUT does (T5, and the doc's multipart note). multipartInfoTable and openKeyTable are untouched — versionedKeyTable only holds committed versions, and an in-flight upload is not one yet.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3900088777


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,630 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.

Review Comment:
   Updated, PTAL.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3680043287


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,376 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}/{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` reserved for the null slot and `1` for the first-version sentinel.
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current; if
+  `maxVersions` is exceeded, trim the oldest noncurrent version in the same batch
+  (its blocks go to deletedTable).

Review Comment:
   It may be best to just throw if `maxVersions` is reached instead of implicitly deleted and let the user decide which versions they want to delete.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3681581601


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,376 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}/{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` reserved for the null slot and `1` for the first-version sentinel.
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current; if
+  `maxVersions` is exceeded, trim the oldest noncurrent version in the same batch
+  (its blocks go to deletedTable).

Review Comment:
   Made it configurable, PTAL.



##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,376 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}/{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` reserved for the null slot and `1` for the first-version sentinel.
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current; if
+  `maxVersions` is exceeded, trim the oldest noncurrent version in the same batch
+  (its blocks go to deletedTable).
+- **PUT (Suspended)**: locate the null slot (current, or an `isNullVersion`
+  record in versionedKeyTable) and replace it; versions accumulated while Enabled

Review Comment:
   Updated, PTAL.



##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,376 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}/{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` reserved for the null slot and `1` for the first-version sentinel.
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current; if
+  `maxVersions` is exceeded, trim the oldest noncurrent version in the same batch
+  (its blocks go to deletedTable).
+- **PUT (Suspended)**: locate the null slot (current, or an `isNullVersion`
+  record in versionedKeyTable) and replace it; versions accumulated while Enabled
+  are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker overwrites the
+  null slot.) **DELETE ?versionId=x**: permanently delete that version (blocks to
+  deletedTable); if it was current, trigger version promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited;
+synchronous trim inside the write transaction; markers count toward the limit),
+`noncurrentVersionExpiration` (opt-in, enforced by a new
+**VersionCleanupService** following the `KeyDeletingService` pattern), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed). All versions count against the bucket space quota;
+version count, marker count, and noncurrent bytes are exposed via
+`ozone sh bucket info` and Recon. `maxVersions` is documented as an Ozone
+extension over S3 semantics (comparable to Lifecycle `NewerNoncurrentVersions`).
+
+## Interaction with Ozone snapshots
+
+A snapshot checkpoints the entire OM RocksDB, so versionedKeyTable is captured
+automatically along with the bucket's versioning status: a snapshot holds the
+complete version history as of its creation. Snapshot creation and deletion need
+no change — `SnapshotDeletingService` moves only deletedTable, deletedDirTable
+and snapshotRenamedTable entries between snapshots, and permanently deleted
+versions travel through deletedTable, the path already used when an overwrite
+reclaims the previous blocks. Multiple versions of one key queued for deletion
+share a deletedTable entry, which is a `RepeatedOmKeyInfo` list evaluated one
+record at a time, so no new structure is required there either.
+
+What does change is **block reclamation**. Snapshots share physical blocks with
+the active object store, so a snapshot keeps its data only because
+`KeyDeletingService` refuses to reclaim blocks that a previous snapshot still
+references. That decision is made per deletedTable record by
+`ReclaimableKeyFilter`, which resolves the key's path in the previous snapshot
+and looks it up **in keyTable only**
+(`KeyManagerImpl.getPreviousSnapshotOzoneKeyInfo`), then compares objectID and
+block locations. With versioning a key's live records span two tables, so the
+lookup misses: permanently deleting a noncurrent version resolves to whatever is
+current in the previous snapshot — a different record with different blocks —
+which reads as "not present in the previous snapshot", i.e. reclaimable. Blocks
+a snapshot still points at would be deleted. objectID does not rescue the
+comparison either way: `prepareFileInfo` keeps it stable across a key's versions,
+so it matches while the block lists do not.
+
+The fix is to make that lookup version-aware. A record carrying a `versionId`
+resolves to the exact dbKey `/{volume}/{bucket}/{keyName}/{MAX - versionId}` in
+the previous snapshot's versionedKeyTable, falling back to keyTable when that
+version was current there. Because versionId is frozen at creation and never
+reused, this is an exact point lookup that *replaces* — rather than extends — the
+objectID-plus-block-list heuristic for versioned buckets. The same lookup feeds
+`calculateExclusiveSize`, keeping snapshot exclusive-size reporting correct for
+noncurrent versions.
+
+This makes deletedTable the single choke point for version reclamation: every
+path that removes a version permanently — `DELETE ?versionId=`, the `maxVersions`
+trim, and VersionCleanupService — writes the record to deletedTable and lets
+KeyDeletingService apply the snapshot-aware filter. None of them may reclaim
+blocks directly.
+
+Snapshot diff remains a **current-version diff**; versionedKeyTable is
+deliberately not added to it. keyTable still holds exactly one record per key, so
+diff cost and semantics do not change as versions accumulate. One rule does need
+stating: **a key whose current version is a delete marker counts as absent**.
+Without it a delete surfaces as a MODIFY (if the marker carries the key's
+objectID) or as a DELETE plus a phantom CREATE (if it carries a new one), instead
+of the DELETE users expect; restoring an object by removing its marker reports
+CREATE correspondingly.
+
+Reads inside a snapshot go through `OmSnapshot`, which implements
+`IOmMetadataReader`, whose `lookupKey(OmKeyArgs)` has no version dimension.
+`?versionId=` is plumbed through it so versions retained in a snapshot are
+actually readable; `ListObjectVersions` against a snapshot is out of scope for
+the first version.
+
+## S3 Gateway and native API surface
+
+New endpoints: `PutBucketVersioning`, `GetBucketVersioning`,
+`ListObjectVersions`. Extended: GET/HEAD/DELETE with `?versionId=` (including the
+literal `null`), `x-amz-version-id` / `x-amz-delete-marker` response headers,
+per-entry version semantics in batch `DeleteObjects`, `CopyObject` versioning
+behavior, and version-id headers on `PutObject` / `CompleteMultipartUpload`. The
+native interfaces (`ozone sh`, `OzoneBucket` API) are extended in parallel so
+versions are manageable outside the S3 path.
+
+## Compatibility and upgrade
+
+Wire protocol changes are limited to optional proto fields, so old records and
+old clients are unaffected. The new column family and request surface are gated
+by `OMLayoutFeature.OBJECT_VERSIONING`; before cluster finalization, enabling
+versioning is rejected with `NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION`.
+Snapshots taken before the feature is used are unaffected, and snapshot
+interaction is covered above. Buckets without versioning behave exactly as today
+(verified by benchmarks).
+
+# Alternatives
+
+- **Single-record layout** (versions embedded in the existing
+  `keyLocationVersions` list): every PUT/DELETE becomes a full read-modify-write
+  of one growing RocksDB value — write amplification linear in the version count
+  (hundreds of KB per write at ~100 versions of a large key), degraded
+  compaction, and no per-version object metadata (size / mtime / ETag) without
+  rebuilding an object record inside the location structure. Rejected in favor of
+  the two-table design, where each version operation writes one small KV (O(1))
+  and each version is a complete `OmKeyInfo`.
+- **A two-state model** (dropping Suspended): "disabled after being enabled" must
+  retain historical versions anyway — otherwise a single state change silently
+  destroys data — yet the bucket would present itself as never-versioned and
+  `GetBucketVersioning` could not answer `Suspended`. Since the semantics are
+  unavoidable, keep the standard three states and full S3 compatibility.
+- **Reusing `objectID` as the version identity**: code verification showed the
+  overwrite path (`OMKeyRequest.prepareFileInfo`) reuses the existing record via
+  `toBuilder()`, so objectID is intentionally stable across overwrites — snapshot
+  diff relies on it to distinguish "modified" from "deleted + recreated".
+  **Reusing `updateID`**: hsync re-commits the same version repeatedly, so it
+  drifts, while a version identity must be frozen at creation. Hence the
+  dedicated `versionId` field.
+- **S3-style random version IDs**: equivalent for reuse-prevention, but they
+  destroy the temporal clustering of versionedKeyTable — version promotion would
+  degrade from one seek to a scan-and-sort (with mtime tiebreak problems), and
+  `GET ?versionId=` from a point lookup to a prefix scan. Unpredictability, the
+  only remaining benefit, is provided by the external encoding layer instead.
+- **A per-key sequential counter** for readable first versions: requires a
+  persistent allocator and reuses IDs after full deletion of a key. Replaced by the
+  pinned-first generator, which special-cases only the first version and needs no
+  allocator.
+- **A version-level snapshot diff** (reporting noncurrent versions created or
+  removed between two snapshots): the diff is keyed by objectID, which a key's
+  versions share, so it would need a different identity and a version dimension
+  in the report; versionedKeyTable would also have to join
+  `RocksDBCheckpointDiffer.COLUMN_FAMILIES_TO_TRACK_IN_DAG`, paying
+  compaction-log and SST retention cost for information already reachable through
+  the snapshot's own version listing. Deferred; the diff stays a current-version
+  diff with an explicit delete-marker rule.
+- **Pinning the generator into bucket metadata** (so a bucket keeps the generator
+  it was created with): removes the risk of a mid-life generator change, but adds a
+  proto field, a write path in both bucket create and set-property, and leaves the
+  generator un-changeable even when an operator legitimately wants to switch.
+  Replaced by the commit-time ordering check, which turns the risk into a loud,
+  per-key failure with a clear remedy and costs no persistent state.
+
+# Plan
+
+Implemented as one umbrella Jira with ten tasks (33 sub-tasks, each roughly one
+PR), in dependency order `T1 → T2 → T3 → T4 → T5 → (T6 ∥ T7) → (T8 ∥ T9) → T10`:
+
+| Task | Scope |
+|---|---|
+| T1 Metadata foundation | proto three-state enum + legacy-boolean sync, set-property state machine, `OmKeyInfo` version fields, versionedKeyTable column family, layout feature gate |
+| T2 VersionId generator framework | `VersionIdGenerator` interface with class-name configuration, transaction-index default, commit-time ordering check, pinned-first generator |
+| T3 ENABLED write paths | PUT two-table update, DELETE marker insertion, quota accounting |
+| T4 Read / permanent delete / promotion | `?versionId=` reads (including through `OmSnapshot`), permanent delete with the version-aware reclamation lookup, version promotion |
+| T5 SUSPENDED semantics | null-slot overwrite, null markers, zero-migration legacy keys |
+| T6 S3 Gateway endpoints | bucket versioning endpoints, object `versionId` support, batch delete |
+| T7 ListObjectVersions | OM merged listing, protocol plumbing, gateway `?versions` |
+| T8 Reclamation | `maxVersions` trim, VersionCleanupService, expired marker cleanup |

Review Comment:
   Updated, PTAL.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3764442027


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited; markers
+count toward the limit), `noncurrentVersionExpiration` (opt-in), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed) — all enforced by a new **VersionCleanupService**
+following the `KeyDeletingService` pattern. Versions beyond `maxVersions` are

Review Comment:
   > even lifecycle is disabled, we still need to clean for "maxVersion" and the marker
   
   This makes sense. 
   
   But imaging if both the KeyLifecycleService and VersionCleanupService are enabled, and user has defined the NoncurrentVersionExpiration properly, then some of their function are overlapped, and the same bucket can be scanned twice in this case. How can we save one service from do the duplication work, that's one improvement that we can think of.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3900108670


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,630 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+The only inbound edge is `PutBucketVersioning`, as in S3, where `CreateBucket`
+carries no versioning parameter: a bucket is always created Unversioned. OM
+rejects a `CreateBucket` that carries a versioning status, also with
+`INVALID_REQUEST`, so the state a bucket starts in cannot be chosen and every
+status it ever holds has been through the transition check above. The field is
+on `BucketInfo` because that message is the bucket's on-disk record and the
+shape `InfoBucket` and `ListBuckets` return, not because a create needs it.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept, and a status
+derives it (`ENABLED → true`, otherwise `false`). The reverse does not hold: a
+record without the enum reads as `UNVERSIONED` whatever the boolean says, and a
+legacy client setting the boolean on such a bucket leaves it without a status,
+so it keeps using the in-record block version list rather than being opted into
+S3 versioning semantics behind its owner's back. A legacy client's boolean is
+mapped onto the state machine only on a bucket that already carries a status,
+where the two have to stay consistent. Old and new clients/OMs coexist during
+rolling upgrades either way.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest.
+
+```mermaid
+flowchart LR
+    subgraph KT["keyTable — one entry per key"]
+        K1["/vol/buck/photo.jpg<br/>versionId 104 · current"]
+    end
+    subgraph VKT["versionedKeyTable — noncurrent, newest → oldest"]
+        V1["/vol/buck/photo.jpg 0x00 MAX-103<br/>delete marker 103"]
+        V2["/vol/buck/photo.jpg 0x00 MAX-102<br/>object 102"]
+        V3["/vol/buck/photo.jpg 0x00 MAX-101<br/>object 101"]
+    end
+    K1 -. "one seek reaches the newest noncurrent version" .-> V1
+    V1 --> V2 --> V3
+```
+
+Because the suffix is `MAX - versionId`, a single `seek` on the key's prefix lands
+on the newest noncurrent version: `ListObjectVersions` and version promotion are a
+seek plus a sequential read, never a scan-and-sort. The table is registered in
+`OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## How a key's versions evolve
+
+The two mechanisms that are hardest to read out of prose — the delete marker and

Review Comment:
   This is already the design: ListObjects scans keyTable and skips keys whose current version is a marker, and ListObjectVersions merges both tables and reports markers as DeleteMarker entries — which is how S3 exposes them. A separate markedDelete parameter would be a second, non-standard way to ask the same question. The API surface section lists the affected calls, and the native equivalent is `ozone sh key list --versions`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3681578327


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,376 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}/{Long.MAX_VALUE - versionId}

Review Comment:
   Nice catch. Changed to "\x00".



##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,376 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}/{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` reserved for the null slot and `1` for the first-version sentinel.
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current; if
+  `maxVersions` is exceeded, trim the oldest noncurrent version in the same batch
+  (its blocks go to deletedTable).
+- **PUT (Suspended)**: locate the null slot (current, or an `isNullVersion`
+  record in versionedKeyTable) and replace it; versions accumulated while Enabled
+  are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable.

Review Comment:
   Updated, PTAL.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on PR #10822:
URL: https://github.com/apache/ozone/pull/10822#issuecomment-5423571160

   @sodonnel Thank you for the review.
   
   > Would it make sense to ensure that the versioned table has all keys?
   
   Today versionedKeyTable holds only immutable history: once written, a record is never touched again. Duplicating the current version makes it a mutable record living in two tables, so SetAcl, rename (which changes the dbKey), hsync appends and MPU commits all have to keep both copies in sync, and missing one is a silent divergence.
   
   The migration it would avoid isn't actually costing us anything. On commit the superseded version is already in hand — it's read for the overwrite handling anyway — so moving it is a single put. Both layouts are two puts on an overwrite, and the double write is two puts on a first write as well, where today it's one.
   
   And the point lookup is deterministic rather than a search: check the current version's ID first (that read is needed anyway), and if it doesn't match, do an exact get on versionedKeyTable at (key, versionId). At most two point gets, no scan.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3757009744


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited; markers
+count toward the limit), `noncurrentVersionExpiration` (opt-in), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed) — all enforced by a new **VersionCleanupService**
+following the `KeyDeletingService` pattern. Versions beyond `maxVersions` are

Review Comment:
   We have new service `KeyLifecycleService` now after HDDS-8342 merged into master. Instead of introducing a new service `VersionCleanupService`, we can consider extend the `KeyLifecycleService` service to support the new action **NoncurrentVersionExpiration** to help evaluate and remove old versions, since  `KeyLifecycleService` also need to be extended to support object expiration for versioned buckets, so we can scan the bucket once, and executed all the actions matched(wanted). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3764507111


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited; markers
+count toward the limit), `noncurrentVersionExpiration` (opt-in), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed) — all enforced by a new **VersionCleanupService**
+following the `KeyDeletingService` pattern. Versions beyond `maxVersions` are

Review Comment:
   Or we can decoupling "lifecycle" and "XxxTableScanService", but seems need many changes for this option.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3763348789


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited; markers
+count toward the limit), `noncurrentVersionExpiration` (opt-in), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed) — all enforced by a new **VersionCleanupService**
+following the `KeyDeletingService` pattern. Versions beyond `maxVersions` are
+reclaimed oldest-first, asynchronously rather than inside the write transaction,
+so the cost of a version write does not grow with the number of versions the key
+already has. An earlier draft also offered a `REJECT` policy that failed the
+write instead; it is dropped. S3 has no error code for the condition and no
+client retries on it, so a default limit enforced that way would break unmodified
+S3 tooling after `maxVersions` overwrites of one key — and reclaiming is what
+S3's own lifecycle rules do. Expiration counts from the moment the version that
+superseded this one was committed, as Lifecycle's `NoncurrentDays` does, not from
+when the version itself was written. All versions count against the bucket space quota;
+version count, marker count, and noncurrent bytes are exposed via
+`ozone sh bucket info` and Recon. `maxVersions` is documented as an Ozone
+extension over S3 semantics (comparable to Lifecycle `NewerNoncurrentVersions`).
+
+## Interaction with Ozone snapshots
+
+**The first version does not allow both features on the same bucket.** The
+version-aware reclamation described in this section is deferred past it, and
+without that lookup the gap below is a data-loss path rather than a rough edge,
+so OM enforces the exclusion instead of documenting an expectation: snapshot
+creation is rejected on an ENABLED or SUSPENDED bucket, and enabling versioning
+or moving SUSPENDED → ENABLED is rejected while any snapshot exists in the

Review Comment:
   Updated, PTAL.



##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.

Review Comment:
   Updated, PTAL.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3763320842


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited; markers
+count toward the limit), `noncurrentVersionExpiration` (opt-in), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed) — all enforced by a new **VersionCleanupService**
+following the `KeyDeletingService` pattern. Versions beyond `maxVersions` are

Review Comment:
   I think lifecycle can cover "NoncurrentVersionExpiration", but for "maxVersion" and "deleteMarker", they are kind of seperate with lifecycle feature, even lifecycle is disabled, we still need to clean for "maxVersion" and the marker.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3850116607


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,614 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest.
+
+```mermaid
+flowchart LR
+    subgraph KT["keyTable — one entry per key"]
+        K1["/vol/buck/photo.jpg<br/>versionId 104 · current"]
+    end
+    subgraph VKT["versionedKeyTable — noncurrent, newest → oldest"]
+        V1["/vol/buck/photo.jpg 0x00 MAX-103<br/>delete marker 103"]
+        V2["/vol/buck/photo.jpg 0x00 MAX-102<br/>object 102"]
+        V3["/vol/buck/photo.jpg 0x00 MAX-101<br/>object 101"]
+    end
+    K1 -. "one seek reaches the newest noncurrent version" .-> V1
+    V1 --> V2 --> V3
+```
+
+Because the suffix is `MAX - versionId`, a single `seek` on the key's prefix lands
+on the newest noncurrent version: `ListObjectVersions` and version promotion are a
+seek plus a sequential read, never a scan-and-sort. The table is registered in
+`OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## How a key's versions evolve
+
+The two mechanisms that are hardest to read out of prose — the delete marker and
+the null version — are easiest to follow as one continuous story of a single key
+`k`. `obj(n)` is a data version with versionId `n`, `marker(n)` a delete marker,
+and `null` marks the record occupying the key's null slot.
+
+| # | Operation | Bucket state | keyTable (current) | versionedKeyTable (newest → oldest) |
+|---|---|---|---|---|
+| 0 | `PUT k` | Unversioned | `obj(—)` no versionId | – |
+| 1 | `PutBucketVersioning(Enabled)` | Enabled | `obj(—)` read as the null version | – |
+| 2 | `PUT k` | Enabled | `obj(101)` | `obj(—)` |
+| 3 | `PUT k` | Enabled | `obj(102)` | `obj(101)`, `obj(—)` |
+| 4 | `DELETE k` | Enabled | `marker(103)` | `obj(102)`, `obj(101)`, `obj(—)` |
+| 5 | `DELETE k?versionId=103` | Enabled | `obj(102)` **promoted back** | `obj(101)`, `obj(—)` |
+| 6 | `PutBucketVersioning(Suspended)` | Suspended | `obj(102)` | `obj(101)`, `obj(—)` |
+| 7 | `PUT k` | Suspended | `obj(104, null)` | `obj(102)`, `obj(101)` |
+| 8 | `PUT k` | Suspended | `obj(105, null)` overwritten in place | unchanged |
+| 9 | `DELETE k` | Suspended | `marker(106, null)` overwritten in place | unchanged |
+| 10 | `PutBucketVersioning(Enabled)`, `PUT k` | Enabled | `obj(107)` | `marker(106, null)`, `obj(102)`, `obj(101)` |
+
+Three things to read out of it:
+
+- **A delete marker is an ordinary version that happens to carry no blocks**
+  (step 4). It becomes the current version, so a plain `GET` returns 404 with
+  `x-amz-delete-marker: true` while every earlier version stays intact and
+  addressable by versionId. Deleting the marker by versionId (step 5) is exactly
+  S3's "restore an object": the newest remaining version is promoted back into
+  keyTable.
+- **Suspended does not stop versioning, it stops *accumulating*** (steps 7–9).
+  Each write lands in the key's single null slot: the first suspended write pushes
+  the previous current version down into versionedKeyTable and drops the key's
+  older null record (step 7); every later suspended write overwrites the null slot
+  in place, so the version count stops growing. Versions accumulated while Enabled
+  are never touched.
+- **The null version is a slot, not a reserved id** (step 7 onwards). It carries a
+  normally generated versionId like any other version and is marked by
+  `isNullVersion`. Pinning it to a fixed low id would misorder a null created
+  between two versioned writes, which is the middle version of the key, not its
+  oldest. Step 10 shows a null delete marker sliding into the noncurrent chain in
+  its correct chronological position once the bucket is Enabled again.
+
+### Where a write lands
+
+The table above is one trace; the rule behind it is the same for every write that
+carries no versionId. Suspended is the branch worth reading twice: it does not
+stop versioning, it collapses every write into the key's one null slot, which is
+why the version count stops growing while the versions accumulated under Enabled
+stay untouched.
+
+```mermaid
+flowchart TB
+    W["PUT k / DELETE k<br/>(no versionId)"] --> S{"bucket state"}
+    S -->|Unversioned| U["overwrite keyTable in place<br/>old blocks → deletedTable<br/><i>unchanged from today</i>"]
+    S -->|Enabled| E["move current version → versionedKeyTable<br/>write the new record into keyTable"]
+    S -->|Suspended| N{"is the current version<br/>already the null slot?"}
+    N -->|yes| N1["overwrite the null record in place<br/>its blocks → deletedTable, a marker carries none<br/><b>version count does not grow</b>"]
+    N -->|no| N2["move current version → versionedKeyTable<br/>write the new record as current, isNullVersion<br/>drop the key's older null record, if any"]
+    E --> R["new current version:<br/>an object on PUT, a delete marker on DELETE"]
+    N1 --> R
+    N2 --> R
+```
+
+### The life of a delete marker
+
+A delete marker is not a tombstone the read path has to special-case away — it is
+an ordinary version that carries no blocks, and it occupies the current slot like
+any other. That single fact produces all of its externally visible behaviour:
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    Obj: current = obj(102)
+    Marker: current = marker(103)
+    Gone: key no longer exists
+    [*] --> Obj: PUT k
+    Obj --> Marker: DELETE k (no versionId)
+    Marker --> Obj: DELETE k?versionId=103<br/>promotion restores obj(102)
+    Marker --> Gone: lifecycle ExpiredObjectDeleteMarker<br/>once it is the only version left
+    Obj --> Obj: GET k → 200
+    Marker --> Marker: GET k → 404, x-amz-delete-marker
+```
+
+Deleting the marker by versionId is exactly S3's "restore an object": nothing is
+rewritten, the newest remaining version is moved back into keyTable.
+
+### Version promotion
+
+The invariant is that keyTable always holds a key's current version. When a
+permanent delete removes the current version, promotion restores the invariant in
+the same WriteBatch, under the bucket write lock:
+
+```mermaid
+flowchart TB
+    A["DELETE k?versionId = current"] --> B["remove the current entry from keyTable<br/>blocks → deletedTable"]
+    B --> C{"seek versionedKeyTable<br/>on k's prefix"}
+    C -->|"a noncurrent version exists"| D["move the newest one back into keyTable<br/>record content stays frozen — a pure positional move"]
+    C -->|"none left"| E["the key disappears entirely"]
+```
+
+## VersionId generation
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and read when an OM
+starts, so a change takes effect on every bucket once the OMs have been restarted;
+it is not recorded in bucket metadata. Every generator must satisfy, **for itself**:
+strictly increasing within a key (a later version's id is always greater than every
+earlier version's id of that key); frozen once assigned; and `0` never handed out,
+being the unset value of the optional field. `1` is reserved as well, but by the
+pinned generator below rather than by the interface: it is that generator's own
+sentinel, which it assigns and never proposes.
+
+The identifier is deliberately **not** derived from OM's Ratis transaction index.
+The versionId is persisted, externally referenced, and part of the S3 API surface;
+tying it to the replication log would export an internal counter into a public
+identity and pin the feature to one execution model, when the OM execution path is
+itself an abstraction (`ExecutionContext` already carries an index whose term may
+be absent).
+
+Numbering a version by when it was written is what S3-compatible stores generally
+do: a GCS `generation` is a microsecond timestamp, an Azure Blob version id is an
+ISO-8601 instant, and RocksDB's user-defined timestamps order the versions of a key
+by a `key + timestamp` suffix — the same encoding the versionedKeyTable dbKey
+already uses. RocksDB is also the precedent for the split below: it orders by the
+timestamp but leaves keeping them monotonic to the caller.
+
+The shape matches SCM's `SequenceIdGenerator` — a cheap local allocation plus a
+floor that guarantees monotonicity. SCM's class itself is not reusable here: it
+lives in `server-scm` on `SCMHAManager` / `@Replicate`, so it would couple OM's
+write path to SCM across a module boundary. It also needs machinery this does not,
+because `localId` has to increase globally while a `versionId` only has to increase
+within one key, whose floor is already persisted as that key's current version. SCM
+does show the migration path if a persisted sequence is ever wanted:
+`upgradeToSequenceId` seeds one from a future timestamp in this same encoding.
+
+- **`UniqueIdVersionIdGenerator` (default)** — the same time-based scheme Ozone
+  already uses to generate block local IDs (`hdds.utils.UniqueId`): the id is
+  `currentTimeMillis << 16 | counter`, a 16-bit counter disambiguating ids proposed
+  within the same millisecond. It needs no allocator, no persistent state and no
+  coordination; values are small positive longs, so `MAX_VALUE - versionId`
+  ordering in versionedKeyTable stays plain signed arithmetic; and the id sorts by
+  creation time, which is what the version chain wants anyway.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes its own sentinel, `PinnedFirstVersionIdGenerator.FIRST_VERSION_ID
+  = 1`, below any proposed id, so it sorts oldest and can be referenced without
+  listing the key's versions first. All later versions take a normally proposed id. First versions are detected by "no
+  current version in keyTable", reusing the lookup the write path performs anyway.
+  Known trade-off: if every version of a key is permanently deleted and the key is
+  recreated, the new first version takes the sentinel again; only deployments that
+  accept this should configure this generator.
+
+**How the id appears on the wire.** `x-amz-version-id`, `?versionId=` and
+`ListObjectVersions` carry the id as its decimal digits, the way a GCS
+`generation` is rendered: `117121056706265088` for a proposed id, `1` for a
+pinned first version. Decimal digits cannot collide with `null`, the identifier
+S3 reserves for a key's null version, so both share the parameter without
+escaping. Clients should still treat the value as opaque. The digits of a proposed
+id do decode to the millisecond of the write, but that is a property of this
+generator rather than of the API, and ids are ordered only within one key —
+comparing them across keys means nothing.
+
+**Where the id is proposed.** The generator runs on the leader in `preExecute` and
+the id is stamped into the request that goes into the replication log, so every OM
+applies an identical value; nothing is proposed during apply. The one apply-time
+decision is the pinned sentinel, which is a constant and is therefore deterministic
+on its own: if the pinned generator is configured and the key has no current
+version at apply time, the version takes `FIRST_VERSION_ID` instead of the proposed
+id.
+
+**Why a proposed id is only a floor.** A time-based id is monotonic within one OM
+process, not across a leader change onto a node whose clock lags, and the "strictly
+increasing within a key" guarantee binds one generator rather than a sequence of
+them. So the generator only proposes, and the id a version is applied with is the
+later of that proposal and the id after the key's current version:
+
+```
+versionId = max(proposedVersionId, currentVersionId + 1)
+```
+
+The write path already holds the current version, so this costs no read, and it
+reads no other key and no global state: it is a pure function of the replicated
+request and the state every OM already shares, so all of them settle on the same id
+without coordinating. A current version predating versioning carries no id, and a
+key in that state has no other versions to order against, so it takes the proposed
+value unchanged — no versionedKeyTable lookup in any case.
+
+Ordering therefore never rests on the clock being trustworthy, and neither does it
+rest on one generator having proposed every id of a key. Under a clock regression, or
+after a change of generator, the ids of an affected key climb by one until proposed
+values overtake them again: the versions stay correctly ordered, and only the id's
+reading as a time degrades — the right one of the two to give up. An earlier draft
+instead rejected such a commit with `INVALID_REQUEST`; that is dropped, because it
+fails a client write over an operational clock problem the client can neither see
+nor fix.
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected.
+- **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+
+```mermaid
+flowchart TB
+    G["GET / HEAD k"] --> Q{"versionId given?"}
+    Q -->|no| C{"current version in keyTable"}
+    C -->|object| C1["200, the current version"]
+    C -->|delete marker| C2["404<br/>x-amz-delete-marker: true"]
+    Q -->|yes| L["check the current version,<br/>then point-look-up versionedKeyTable"]
+    L --> M{"the addressed version"}
+    M -->|object| M1["200, that version"]
+    M -->|delete marker| M2["405<br/>x-amz-delete-marker: true<br/>Allow: DELETE"]
+    M -->|no such version| M3["404"]
+```
+
+The 404/405 split is S3's, and it follows from the marker being addressable: an
+unaddressed read steps over the marker and reports the key absent, while a read
+that names the marker gets told the version exists but has no body to return.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) A batch delete is the same operation over
+  more keys and takes the same path — it goes through a request of its own, which
+  inserts one marker per key from the same implementation, so the two cannot
+  diverge. Hard-deleting there instead would destroy the current version and leave
+  the versions under it in the versionedKeyTable with nothing above them: absent to
+  a plain read, holding quota, and unreachable by promotion, which only runs when a
+  version is deleted by id. **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion as shown above.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation: version-aware lifecycle rules
+
+Version reclamation is expressed as **lifecycle rules on the lifecycle engine
+already in master** (HDDS-8342: `OmLifecycleConfiguration`, `KeyLifecycleService`,
+and the three S3 lifecycle endpoints), not as a service of its own. The engine
+already provides everything version reclamation needs — per-bucket rule storage
+and validation, prefix/tag filters, a resumable background scan with saved scan
+state, batched deletes through Ratis, metrics, and `suspend`/`resume` admin
+commands. Versioning contributes the three actions that the feature documentation
+currently lists as unsupported, and each maps 1:1 onto an S3 element:
+
+| S3 lifecycle element | Effect on a versioned bucket |
+|---|---|
+| `NoncurrentVersionExpiration.NoncurrentDays` | permanently deletes a noncurrent version once it has been noncurrent that long |
+| `NoncurrentVersionExpiration.NewerNoncurrentVersions` | keeps at most N noncurrent versions of a key, reclaiming oldest-first |
+| `Expiration.ExpiredObjectDeleteMarker` | removes a delete marker, and the key with it, once it is the only version left |
+| `Expiration.Days` / `.Date` (existing action) | on a versioned bucket, inserts a delete marker instead of deleting the object, as S3 does — through the same insertion the write requests use, so an expiry and a DELETE leave the same records behind |
+
+Concretely: `LifecycleAction` gains a `noncurrentVersionExpiration` field and
+`LifecycleExpiration` an `expiredObjectDeleteMarker` field — both optional
+additions to existing messages — and `LifecycleActionTask` gains a
+versionedKeyTable scan next to its keyTable scan, reusing the same batching, scan
+state and metrics. Expiration counts from the moment the version that superseded
+this one was committed, as `NoncurrentDays` does, not from when the version itself
+was written.
+
+One deliberate departure from stock S3: **the lifecycle service is enabled by
+default.** `ozone.lifecycle.service.enabled` flips from `false` to `true`. A rule
+that never runs is worse than no rule, and versioning is the first feature that
+depends on reclamation for correct operation rather than for tidiness. OM rejects
+`PutBucketVersioning(Enabled)` while the lifecycle service is disabled: that does
+not promise a bucket's versions are bounded — one whose owner writes no rule
+accumulates them, as on S3 — but it rules out a rule existing that nothing will
+ever act on.
+
+Earlier drafts bounded version growth with a bucket-level `maxVersions` property
+as well, first refusing the write once the limit was reached and later trimming
+oldest-first from the same scan. Both are dropped. A per-bucket cap duplicates
+`NewerNoncurrentVersions` in an Ozone-private property, which cuts against
+expressing reclamation as lifecycle rules at all; and applying it to the buckets
+it existed for — those carrying no lifecycle configuration — meant scheduling scan
+work for buckets the engine otherwise never visits, a second kind of task that
+differs from every other one in what it may assume. A bucket with no rule now
+accumulates versions, which is what S3 does.
+
+## Quota and observability
+
+All versions count against the bucket space quota. Version count, marker count,
+and noncurrent bytes are exposed via `ozone sh bucket info` and Recon, and the
+version-aware lifecycle actions report through the existing
+`KeyLifecycleServiceMetrics`.
+
+## Interaction with Ozone snapshots
+
+OM rejects the combination on the same bucket, in both directions:
+
+* a snapshot can only be created on a bucket whose versioning status is
+  UNVERSIONED;
+* versioning can only be enabled on a bucket that has no snapshot.
+
+This is enforced rather than documented as a convention. Snapshots are not
+restricted to any bucket layout in code — `TestOmSnapshotObjectStore` covers OBS
+buckets explicitly — so nothing but enforcement keeps the two apart, and the
+combination has a real gap behind it: snapshot diff is keyed by objectID, which all
+versions of a key share, so a bucket carrying both would present a diff that cannot
+describe what changed. What makes a hard rejection cheap in practice is that the
+two features barely overlap today: snapshots are overwhelmingly used on FSO buckets
+for filesystem-style workloads, while versioning in this first phase is OBS-only.
+The rejection is applied to the source of a linked bucket and evaluated at apply
+time so the two directions cannot race, and a dev-only configuration key lifts it
+for testing. It is lifted for good once version-aware snapshot reclamation and a
+version-aware diff land as follow-up work.
+
+## S3 Gateway and native API surface
+
+New endpoints: `PutBucketVersioning`, `GetBucketVersioning`,
+`ListObjectVersions`. Extended: GET/HEAD/DELETE with `?versionId=` (including the
+literal `null`), `x-amz-version-id` / `x-amz-delete-marker` response headers,
+per-entry version semantics in batch `DeleteObjects` (whose OM-side marker
+insertion lands with the write paths, leaving the gateway to carry the per-entry
+`versionId` and report `DeleteMarker`), `CopyObject` versioning
+behavior, and version-id headers on `PutObject` / `CompleteMultipartUpload`. The
+native interfaces (`ozone sh`, `OzoneBucket` API) are extended in parallel so
+versions are manageable outside the S3 path.
+
+## Compatibility and upgrade
+
+Wire protocol changes are limited to optional proto fields, so old records and
+old clients are unaffected. The new column family and request surface are gated
+by `OMLayoutFeature.OBJECT_VERSIONING`; before cluster finalization, enabling
+versioning is rejected with `NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION`.
+Snapshots taken before the feature is used are unaffected, and snapshot
+interaction is covered above. Buckets without versioning behave exactly as today

Review Comment:
   @chihsuan , versioning is never officially support, while there are some partial codes about versioning.  My suggestion is we defer [HDDS-16127](https://issues.apache.org/jira/browse/HDDS-16127), until current proposed versioning feature is implemented, otherwise there is no meaning. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3900095298


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,630 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+The only inbound edge is `PutBucketVersioning`, as in S3, where `CreateBucket`
+carries no versioning parameter: a bucket is always created Unversioned. OM
+rejects a `CreateBucket` that carries a versioning status, also with
+`INVALID_REQUEST`, so the state a bucket starts in cannot be chosen and every
+status it ever holds has been through the transition check above. The field is
+on `BucketInfo` because that message is the bucket's on-disk record and the
+shape `InfoBucket` and `ListBuckets` return, not because a create needs it.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept, and a status
+derives it (`ENABLED → true`, otherwise `false`). The reverse does not hold: a
+record without the enum reads as `UNVERSIONED` whatever the boolean says, and a
+legacy client setting the boolean on such a bucket leaves it without a status,
+so it keeps using the in-record block version list rather than being opted into
+S3 versioning semantics behind its owner's back. A legacy client's boolean is
+mapped onto the state machine only on a bucket that already carries a status,
+where the two have to stay consistent. Old and new clients/OMs coexist during
+rolling upgrades either way.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest.
+
+```mermaid
+flowchart LR
+    subgraph KT["keyTable — one entry per key"]
+        K1["/vol/buck/photo.jpg<br/>versionId 104 · current"]
+    end
+    subgraph VKT["versionedKeyTable — noncurrent, newest → oldest"]
+        V1["/vol/buck/photo.jpg 0x00 MAX-103<br/>delete marker 103"]
+        V2["/vol/buck/photo.jpg 0x00 MAX-102<br/>object 102"]
+        V3["/vol/buck/photo.jpg 0x00 MAX-101<br/>object 101"]
+    end
+    K1 -. "one seek reaches the newest noncurrent version" .-> V1
+    V1 --> V2 --> V3
+```
+
+Because the suffix is `MAX - versionId`, a single `seek` on the key's prefix lands
+on the newest noncurrent version: `ListObjectVersions` and version promotion are a
+seek plus a sequential read, never a scan-and-sort. The table is registered in
+`OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## How a key's versions evolve
+
+The two mechanisms that are hardest to read out of prose — the delete marker and
+the null version — are easiest to follow as one continuous story of a single key
+`k`. `obj(n)` is a data version with versionId `n`, `marker(n)` a delete marker,
+and `null` marks the record occupying the key's null slot.
+
+| # | Operation | Bucket state | keyTable (current) | versionedKeyTable (newest → oldest) |
+|---|---|---|---|---|
+| 0 | `PUT k` | Unversioned | `obj(—)` no versionId | – |
+| 1 | `PutBucketVersioning(Enabled)` | Enabled | `obj(—)` read as the null version | – |
+| 2 | `PUT k` | Enabled | `obj(101)` | `obj(—)` |
+| 3 | `PUT k` | Enabled | `obj(102)` | `obj(101)`, `obj(—)` |
+| 4 | `DELETE k` | Enabled | `marker(103)` | `obj(102)`, `obj(101)`, `obj(—)` |
+| 5 | `DELETE k?versionId=103` | Enabled | `obj(102)` **promoted back** | `obj(101)`, `obj(—)` |
+| 6 | `PutBucketVersioning(Suspended)` | Suspended | `obj(102)` | `obj(101)`, `obj(—)` |
+| 7 | `PUT k` | Suspended | `obj(104, null)` | `obj(102)`, `obj(101)` |
+| 8 | `PUT k` | Suspended | `obj(105, null)` overwritten in place | unchanged |
+| 9 | `DELETE k` | Suspended | `marker(106, null)` overwritten in place | unchanged |
+| 10 | `PutBucketVersioning(Enabled)`, `PUT k` | Enabled | `obj(107)` | `marker(106, null)`, `obj(102)`, `obj(101)` |
+
+Three things to read out of it:
+
+- **A delete marker is an ordinary version that happens to carry no blocks**
+  (step 4). It becomes the current version, so a plain `GET` returns 404 with
+  `x-amz-delete-marker: true` while every earlier version stays intact and
+  addressable by versionId. Deleting the marker by versionId (step 5) is exactly
+  S3's "restore an object": the newest remaining version is promoted back into
+  keyTable.
+- **Suspended does not stop versioning, it stops *accumulating*** (steps 7–9).
+  Each write lands in the key's single null slot: the first suspended write pushes
+  the previous current version down into versionedKeyTable and drops the key's
+  older null record (step 7); every later suspended write overwrites the null slot
+  in place, so the version count stops growing. Versions accumulated while Enabled
+  are never touched.
+- **The null version is a slot, not a reserved id** (step 7 onwards). It carries a
+  normally generated versionId like any other version and is marked by
+  `isNullVersion`. Pinning it to a fixed low id would misorder a null created
+  between two versioned writes, which is the middle version of the key, not its
+  oldest. Step 10 shows a null delete marker sliding into the noncurrent chain in
+  its correct chronological position once the bucket is Enabled again.
+
+### Where a write lands
+
+The table above is one trace; the rule behind it is the same for every write that
+carries no versionId. Suspended is the branch worth reading twice: it does not
+stop versioning, it collapses every write into the key's one null slot, which is
+why the version count stops growing while the versions accumulated under Enabled
+stay untouched.
+
+```mermaid
+flowchart TB
+    W["PUT k / DELETE k<br/>(no versionId)"] --> S{"bucket state"}
+    S -->|Unversioned| U["overwrite keyTable in place<br/>old blocks → deletedTable<br/><i>unchanged from today</i>"]
+    S -->|Enabled| E["move current version → versionedKeyTable<br/>write the new record into keyTable"]
+    S -->|Suspended| N{"is the current version<br/>already the null slot?"}
+    N -->|yes| N1["overwrite the null record in place<br/>its blocks → deletedTable, a marker carries none<br/><b>version count does not grow</b>"]
+    N -->|no| N2["move current version → versionedKeyTable<br/>write the new record as current, isNullVersion<br/>drop the key's older null record, if any"]
+    E --> R["new current version:<br/>an object on PUT, a delete marker on DELETE"]
+    N1 --> R
+    N2 --> R
+```
+
+### The life of a delete marker
+
+A delete marker is not a tombstone the read path has to special-case away — it is
+an ordinary version that carries no blocks, and it occupies the current slot like
+any other. That single fact produces all of its externally visible behaviour:
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    Obj: current = obj(102)
+    Marker: current = marker(103)
+    Gone: key no longer exists
+    [*] --> Obj: PUT k
+    Obj --> Marker: DELETE k (no versionId)
+    Marker --> Obj: DELETE k?versionId=103<br/>promotion restores obj(102)
+    Marker --> Gone: lifecycle ExpiredObjectDeleteMarker<br/>once it is the only version left
+    Obj --> Obj: GET k → 200
+    Marker --> Marker: GET k → 404, x-amz-delete-marker
+```
+
+Deleting the marker by versionId is exactly S3's "restore an object": nothing is
+rewritten, the newest remaining version is moved back into keyTable.
+
+### Version promotion
+
+The invariant is that keyTable always holds a key's current version. When a
+permanent delete removes the current version, promotion restores the invariant in
+the same WriteBatch, under the bucket write lock:
+
+```mermaid
+flowchart TB
+    A["DELETE k?versionId = current"] --> B["remove the current entry from keyTable<br/>blocks → deletedTable"]
+    B --> C{"seek versionedKeyTable<br/>on k's prefix"}
+    C -->|"a noncurrent version exists"| D["move the newest one back into keyTable<br/>record content stays frozen — a pure positional move"]
+    C -->|"none left"| E["the key disappears entirely"]
+```
+
+## VersionId generation
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and read when an OM
+starts, so a change takes effect on every bucket once the OMs have been restarted;
+it is not recorded in bucket metadata. Every generator must satisfy, **for itself**:
+strictly increasing within a key (a later version's id is always greater than every
+earlier version's id of that key); frozen once assigned; and `0` never handed out,
+being the unset value of the optional field. `1` is reserved as well, but by the
+pinned generator below rather than by the interface: it is that generator's own
+sentinel, which it assigns and never proposes.
+
+The identifier is deliberately **not** derived from OM's Ratis transaction index.
+The versionId is persisted, externally referenced, and part of the S3 API surface;
+tying it to the replication log would export an internal counter into a public
+identity and pin the feature to one execution model, when the OM execution path is
+itself an abstraction (`ExecutionContext` already carries an index whose term may
+be absent).
+
+Numbering a version by when it was written is what S3-compatible stores generally
+do: a GCS `generation` is a microsecond timestamp, an Azure Blob version id is an
+ISO-8601 instant, and RocksDB's user-defined timestamps order the versions of a key
+by a `key + timestamp` suffix — the same encoding the versionedKeyTable dbKey
+already uses. RocksDB is also the precedent for the split below: it orders by the
+timestamp but leaves keeping them monotonic to the caller.
+
+The shape matches SCM's `SequenceIdGenerator` — a cheap local allocation plus a
+floor that guarantees monotonicity. SCM's class itself is not reusable here: it
+lives in `server-scm` on `SCMHAManager` / `@Replicate`, so it would couple OM's
+write path to SCM across a module boundary. It also needs machinery this does not,
+because `localId` has to increase globally while a `versionId` only has to increase
+within one key, whose floor is already persisted as that key's current version. SCM
+does show the migration path if a persisted sequence is ever wanted:
+`upgradeToSequenceId` seeds one from a future timestamp in this same encoding.
+
+- **`UniqueIdVersionIdGenerator` (default)** — the same time-based scheme Ozone
+  already uses to generate block local IDs (`hdds.utils.UniqueId`): the id is
+  `currentTimeMillis << 16 | counter`, a 16-bit counter disambiguating ids proposed
+  within the same millisecond. It needs no allocator, no persistent state and no
+  coordination; values are small positive longs, so `MAX_VALUE - versionId`
+  ordering in versionedKeyTable stays plain signed arithmetic; and the id sorts by
+  creation time, which is what the version chain wants anyway.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes its own sentinel, `PinnedFirstVersionIdGenerator.FIRST_VERSION_ID
+  = 1`, below any proposed id, so it sorts oldest and can be referenced without
+  listing the key's versions first. All later versions take a normally proposed id. First versions are detected by "no
+  current version in keyTable", reusing the lookup the write path performs anyway.
+  Known trade-off: if every version of a key is permanently deleted and the key is
+  recreated, the new first version takes the sentinel again; only deployments that
+  accept this should configure this generator.
+
+**How the id appears on the wire.** `x-amz-version-id`, `?versionId=` and
+`ListObjectVersions` carry the id as its decimal digits, the way a GCS
+`generation` is rendered: `117121056706265088` for a proposed id, `1` for a
+pinned first version. Decimal digits cannot collide with `null`, the identifier
+S3 reserves for a key's null version, so both share the parameter without
+escaping. Clients should still treat the value as opaque. The digits of a proposed
+id do decode to the millisecond of the write, but that is a property of this
+generator rather than of the API, and ids are ordered only within one key —
+comparing them across keys means nothing.
+
+**Where the id is proposed.** The generator runs on the leader in `preExecute` and
+the id is stamped into the request that goes into the replication log, so every OM
+applies an identical value; nothing is proposed during apply. The one apply-time
+decision is the pinned sentinel, which is a constant and is therefore deterministic
+on its own: if the pinned generator is configured and the key has no current
+version at apply time, the version takes `FIRST_VERSION_ID` instead of the proposed
+id.
+
+**Why a proposed id is only a floor.** A time-based id is monotonic within one OM
+process, not across a leader change onto a node whose clock lags, and the "strictly
+increasing within a key" guarantee binds one generator rather than a sequence of
+them. So the generator only proposes, and the id a version is applied with is the
+later of that proposal and the id after the key's current version:
+
+```
+versionId = max(proposedVersionId, currentVersionId + 1)
+```
+
+The write path already holds the current version, so this costs no read, and it
+reads no other key and no global state: it is a pure function of the replicated
+request and the state every OM already shares, so all of them settle on the same id
+without coordinating. A current version predating versioning carries no id, and a
+key in that state has no other versions to order against, so it takes the proposed
+value unchanged — no versionedKeyTable lookup in any case.
+
+Ordering therefore never rests on the clock being trustworthy, and neither does it
+rest on one generator having proposed every id of a key. Under a clock regression, or
+after a change of generator, the ids of an affected key climb by one until proposed
+values overtake them again: the versions stay correctly ordered, and only the id's
+reading as a time degrades — the right one of the two to give up. An earlier draft
+instead rejected such a commit with `INVALID_REQUEST`; that is dropped, because it
+fails a client write over an operational clock problem the client can neither see
+nor fix.
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)

Review Comment:
   Updated, PTAL.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3679957983


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,376 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}/{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` reserved for the null slot and `1` for the first-version sentinel.
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current; if
+  `maxVersions` is exceeded, trim the oldest noncurrent version in the same batch
+  (its blocks go to deletedTable).
+- **PUT (Suspended)**: locate the null slot (current, or an `isNullVersion`
+  record in versionedKeyTable) and replace it; versions accumulated while Enabled
+  are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable.

Review Comment:
   ```suggestion
     first, then point-look-up versionedKeyTable. Return 405 if version is a delete marker.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3700976581


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,389 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited; markers
+count toward the limit), `noncurrentVersionExpiration` (opt-in), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed) — all enforced by a new **VersionCleanupService**
+following the `KeyDeletingService` pattern. A per-bucket `maxVersionsPolicy`
+selects between `TRIM` (default: reclaim oldest-first, asynchronously rather than
+inside the write transaction) and `REJECT` (the write fails instead and the
+operator chooses what to delete; opt-in, since S3 has no error code for the
+condition). All versions count against the bucket space quota;
+version count, marker count, and noncurrent bytes are exposed via
+`ozone sh bucket info` and Recon. `maxVersions` is documented as an Ozone
+extension over S3 semantics (comparable to Lifecycle `NewerNoncurrentVersions`).
+
+## Interaction with Ozone snapshots

Review Comment:
   @smengcl Thanks Siyao — agreed. Updated the OEP, PTAL.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3693999456


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,389 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited; markers
+count toward the limit), `noncurrentVersionExpiration` (opt-in), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed) — all enforced by a new **VersionCleanupService**
+following the `KeyDeletingService` pattern. A per-bucket `maxVersionsPolicy`
+selects between `TRIM` (default: reclaim oldest-first, asynchronously rather than
+inside the write transaction) and `REJECT` (the write fails instead and the
+operator chooses what to delete; opt-in, since S3 has no error code for the
+condition). All versions count against the bucket space quota;
+version count, marker count, and noncurrent bytes are exposed via
+`ozone sh bucket info` and Recon. `maxVersions` is documented as an Ozone
+extension over S3 semantics (comparable to Lifecycle `NewerNoncurrentVersions`).
+
+## Interaction with Ozone snapshots

Review Comment:
   After an offline discussion with the team, we think it would be better to make snapshot and versioning mutually exclusive for now (or explicitly opt-in to enable both for dev testing) since we have some obvious holes to fill. For one, snapshot garbage collection may not be correctly accounting for blocks in key versions which could lead to wrong blocks being reclaimed, which I believe is also mentioned in this section.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "chihsuan (via GitHub)" <gi...@apache.org>.
chihsuan commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3835500060


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,614 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest.
+
+```mermaid
+flowchart LR
+    subgraph KT["keyTable — one entry per key"]
+        K1["/vol/buck/photo.jpg<br/>versionId 104 · current"]
+    end
+    subgraph VKT["versionedKeyTable — noncurrent, newest → oldest"]
+        V1["/vol/buck/photo.jpg 0x00 MAX-103<br/>delete marker 103"]
+        V2["/vol/buck/photo.jpg 0x00 MAX-102<br/>object 102"]
+        V3["/vol/buck/photo.jpg 0x00 MAX-101<br/>object 101"]
+    end
+    K1 -. "one seek reaches the newest noncurrent version" .-> V1
+    V1 --> V2 --> V3
+```
+
+Because the suffix is `MAX - versionId`, a single `seek` on the key's prefix lands
+on the newest noncurrent version: `ListObjectVersions` and version promotion are a
+seek plus a sequential read, never a scan-and-sort. The table is registered in
+`OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## How a key's versions evolve
+
+The two mechanisms that are hardest to read out of prose — the delete marker and
+the null version — are easiest to follow as one continuous story of a single key
+`k`. `obj(n)` is a data version with versionId `n`, `marker(n)` a delete marker,
+and `null` marks the record occupying the key's null slot.
+
+| # | Operation | Bucket state | keyTable (current) | versionedKeyTable (newest → oldest) |
+|---|---|---|---|---|
+| 0 | `PUT k` | Unversioned | `obj(—)` no versionId | – |
+| 1 | `PutBucketVersioning(Enabled)` | Enabled | `obj(—)` read as the null version | – |
+| 2 | `PUT k` | Enabled | `obj(101)` | `obj(—)` |
+| 3 | `PUT k` | Enabled | `obj(102)` | `obj(101)`, `obj(—)` |
+| 4 | `DELETE k` | Enabled | `marker(103)` | `obj(102)`, `obj(101)`, `obj(—)` |
+| 5 | `DELETE k?versionId=103` | Enabled | `obj(102)` **promoted back** | `obj(101)`, `obj(—)` |
+| 6 | `PutBucketVersioning(Suspended)` | Suspended | `obj(102)` | `obj(101)`, `obj(—)` |
+| 7 | `PUT k` | Suspended | `obj(104, null)` | `obj(102)`, `obj(101)` |
+| 8 | `PUT k` | Suspended | `obj(105, null)` overwritten in place | unchanged |
+| 9 | `DELETE k` | Suspended | `marker(106, null)` overwritten in place | unchanged |
+| 10 | `PutBucketVersioning(Enabled)`, `PUT k` | Enabled | `obj(107)` | `marker(106, null)`, `obj(102)`, `obj(101)` |
+
+Three things to read out of it:
+
+- **A delete marker is an ordinary version that happens to carry no blocks**
+  (step 4). It becomes the current version, so a plain `GET` returns 404 with
+  `x-amz-delete-marker: true` while every earlier version stays intact and
+  addressable by versionId. Deleting the marker by versionId (step 5) is exactly
+  S3's "restore an object": the newest remaining version is promoted back into
+  keyTable.
+- **Suspended does not stop versioning, it stops *accumulating*** (steps 7–9).
+  Each write lands in the key's single null slot: the first suspended write pushes
+  the previous current version down into versionedKeyTable and drops the key's
+  older null record (step 7); every later suspended write overwrites the null slot
+  in place, so the version count stops growing. Versions accumulated while Enabled
+  are never touched.
+- **The null version is a slot, not a reserved id** (step 7 onwards). It carries a
+  normally generated versionId like any other version and is marked by
+  `isNullVersion`. Pinning it to a fixed low id would misorder a null created
+  between two versioned writes, which is the middle version of the key, not its
+  oldest. Step 10 shows a null delete marker sliding into the noncurrent chain in
+  its correct chronological position once the bucket is Enabled again.
+
+### Where a write lands
+
+The table above is one trace; the rule behind it is the same for every write that
+carries no versionId. Suspended is the branch worth reading twice: it does not
+stop versioning, it collapses every write into the key's one null slot, which is
+why the version count stops growing while the versions accumulated under Enabled
+stay untouched.
+
+```mermaid
+flowchart TB
+    W["PUT k / DELETE k<br/>(no versionId)"] --> S{"bucket state"}
+    S -->|Unversioned| U["overwrite keyTable in place<br/>old blocks → deletedTable<br/><i>unchanged from today</i>"]
+    S -->|Enabled| E["move current version → versionedKeyTable<br/>write the new record into keyTable"]
+    S -->|Suspended| N{"is the current version<br/>already the null slot?"}
+    N -->|yes| N1["overwrite the null record in place<br/>its blocks → deletedTable, a marker carries none<br/><b>version count does not grow</b>"]
+    N -->|no| N2["move current version → versionedKeyTable<br/>write the new record as current, isNullVersion<br/>drop the key's older null record, if any"]
+    E --> R["new current version:<br/>an object on PUT, a delete marker on DELETE"]
+    N1 --> R
+    N2 --> R
+```
+
+### The life of a delete marker
+
+A delete marker is not a tombstone the read path has to special-case away — it is
+an ordinary version that carries no blocks, and it occupies the current slot like
+any other. That single fact produces all of its externally visible behaviour:
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    Obj: current = obj(102)
+    Marker: current = marker(103)
+    Gone: key no longer exists
+    [*] --> Obj: PUT k
+    Obj --> Marker: DELETE k (no versionId)
+    Marker --> Obj: DELETE k?versionId=103<br/>promotion restores obj(102)
+    Marker --> Gone: lifecycle ExpiredObjectDeleteMarker<br/>once it is the only version left
+    Obj --> Obj: GET k → 200
+    Marker --> Marker: GET k → 404, x-amz-delete-marker
+```
+
+Deleting the marker by versionId is exactly S3's "restore an object": nothing is
+rewritten, the newest remaining version is moved back into keyTable.
+
+### Version promotion
+
+The invariant is that keyTable always holds a key's current version. When a
+permanent delete removes the current version, promotion restores the invariant in
+the same WriteBatch, under the bucket write lock:
+
+```mermaid
+flowchart TB
+    A["DELETE k?versionId = current"] --> B["remove the current entry from keyTable<br/>blocks → deletedTable"]
+    B --> C{"seek versionedKeyTable<br/>on k's prefix"}
+    C -->|"a noncurrent version exists"| D["move the newest one back into keyTable<br/>record content stays frozen — a pure positional move"]
+    C -->|"none left"| E["the key disappears entirely"]
+```
+
+## VersionId generation
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and read when an OM
+starts, so a change takes effect on every bucket once the OMs have been restarted;
+it is not recorded in bucket metadata. Every generator must satisfy, **for itself**:
+strictly increasing within a key (a later version's id is always greater than every
+earlier version's id of that key); frozen once assigned; and `0` never handed out,
+being the unset value of the optional field. `1` is reserved as well, but by the
+pinned generator below rather than by the interface: it is that generator's own
+sentinel, which it assigns and never proposes.
+
+The identifier is deliberately **not** derived from OM's Ratis transaction index.
+The versionId is persisted, externally referenced, and part of the S3 API surface;
+tying it to the replication log would export an internal counter into a public
+identity and pin the feature to one execution model, when the OM execution path is
+itself an abstraction (`ExecutionContext` already carries an index whose term may
+be absent).
+
+Numbering a version by when it was written is what S3-compatible stores generally
+do: a GCS `generation` is a microsecond timestamp, an Azure Blob version id is an
+ISO-8601 instant, and RocksDB's user-defined timestamps order the versions of a key
+by a `key + timestamp` suffix — the same encoding the versionedKeyTable dbKey
+already uses. RocksDB is also the precedent for the split below: it orders by the
+timestamp but leaves keeping them monotonic to the caller.
+
+The shape matches SCM's `SequenceIdGenerator` — a cheap local allocation plus a
+floor that guarantees monotonicity. SCM's class itself is not reusable here: it
+lives in `server-scm` on `SCMHAManager` / `@Replicate`, so it would couple OM's
+write path to SCM across a module boundary. It also needs machinery this does not,
+because `localId` has to increase globally while a `versionId` only has to increase
+within one key, whose floor is already persisted as that key's current version. SCM
+does show the migration path if a persisted sequence is ever wanted:
+`upgradeToSequenceId` seeds one from a future timestamp in this same encoding.
+
+- **`UniqueIdVersionIdGenerator` (default)** — the same time-based scheme Ozone
+  already uses to generate block local IDs (`hdds.utils.UniqueId`): the id is
+  `currentTimeMillis << 16 | counter`, a 16-bit counter disambiguating ids proposed
+  within the same millisecond. It needs no allocator, no persistent state and no
+  coordination; values are small positive longs, so `MAX_VALUE - versionId`
+  ordering in versionedKeyTable stays plain signed arithmetic; and the id sorts by
+  creation time, which is what the version chain wants anyway.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes its own sentinel, `PinnedFirstVersionIdGenerator.FIRST_VERSION_ID
+  = 1`, below any proposed id, so it sorts oldest and can be referenced without
+  listing the key's versions first. All later versions take a normally proposed id. First versions are detected by "no
+  current version in keyTable", reusing the lookup the write path performs anyway.
+  Known trade-off: if every version of a key is permanently deleted and the key is
+  recreated, the new first version takes the sentinel again; only deployments that
+  accept this should configure this generator.
+
+**How the id appears on the wire.** `x-amz-version-id`, `?versionId=` and
+`ListObjectVersions` carry the id as its decimal digits, the way a GCS
+`generation` is rendered: `117121056706265088` for a proposed id, `1` for a
+pinned first version. Decimal digits cannot collide with `null`, the identifier
+S3 reserves for a key's null version, so both share the parameter without
+escaping. Clients should still treat the value as opaque. The digits of a proposed
+id do decode to the millisecond of the write, but that is a property of this
+generator rather than of the API, and ids are ordered only within one key —
+comparing them across keys means nothing.
+
+**Where the id is proposed.** The generator runs on the leader in `preExecute` and
+the id is stamped into the request that goes into the replication log, so every OM
+applies an identical value; nothing is proposed during apply. The one apply-time
+decision is the pinned sentinel, which is a constant and is therefore deterministic
+on its own: if the pinned generator is configured and the key has no current
+version at apply time, the version takes `FIRST_VERSION_ID` instead of the proposed
+id.
+
+**Why a proposed id is only a floor.** A time-based id is monotonic within one OM
+process, not across a leader change onto a node whose clock lags, and the "strictly
+increasing within a key" guarantee binds one generator rather than a sequence of
+them. So the generator only proposes, and the id a version is applied with is the
+later of that proposal and the id after the key's current version:
+
+```
+versionId = max(proposedVersionId, currentVersionId + 1)
+```
+
+The write path already holds the current version, so this costs no read, and it
+reads no other key and no global state: it is a pure function of the replicated
+request and the state every OM already shares, so all of them settle on the same id
+without coordinating. A current version predating versioning carries no id, and a
+key in that state has no other versions to order against, so it takes the proposed
+value unchanged — no versionedKeyTable lookup in any case.
+
+Ordering therefore never rests on the clock being trustworthy, and neither does it
+rest on one generator having proposed every id of a key. Under a clock regression, or
+after a change of generator, the ids of an affected key climb by one until proposed
+values overtake them again: the versions stay correctly ordered, and only the id's
+reading as a time degrades — the right one of the two to give up. An earlier draft
+instead rejected such a commit with `INVALID_REQUEST`; that is dropped, because it
+fails a client write over an operational clock problem the client can neither see
+nor fix.
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected.
+- **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+
+```mermaid
+flowchart TB
+    G["GET / HEAD k"] --> Q{"versionId given?"}
+    Q -->|no| C{"current version in keyTable"}
+    C -->|object| C1["200, the current version"]
+    C -->|delete marker| C2["404<br/>x-amz-delete-marker: true"]
+    Q -->|yes| L["check the current version,<br/>then point-look-up versionedKeyTable"]
+    L --> M{"the addressed version"}
+    M -->|object| M1["200, that version"]
+    M -->|delete marker| M2["405<br/>x-amz-delete-marker: true<br/>Allow: DELETE"]
+    M -->|no such version| M3["404"]
+```
+
+The 404/405 split is S3's, and it follows from the marker being addressable: an
+unaddressed read steps over the marker and reports the key absent, while a read
+that names the marker gets told the version exists but has no body to return.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) A batch delete is the same operation over
+  more keys and takes the same path — it goes through a request of its own, which
+  inserts one marker per key from the same implementation, so the two cannot
+  diverge. Hard-deleting there instead would destroy the current version and leave
+  the versions under it in the versionedKeyTable with nothing above them: absent to
+  a plain read, holding quota, and unreachable by promotion, which only runs when a
+  version is deleted by id. **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion as shown above.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation: version-aware lifecycle rules
+
+Version reclamation is expressed as **lifecycle rules on the lifecycle engine
+already in master** (HDDS-8342: `OmLifecycleConfiguration`, `KeyLifecycleService`,
+and the three S3 lifecycle endpoints), not as a service of its own. The engine
+already provides everything version reclamation needs — per-bucket rule storage
+and validation, prefix/tag filters, a resumable background scan with saved scan
+state, batched deletes through Ratis, metrics, and `suspend`/`resume` admin
+commands. Versioning contributes the three actions that the feature documentation
+currently lists as unsupported, and each maps 1:1 onto an S3 element:
+
+| S3 lifecycle element | Effect on a versioned bucket |
+|---|---|
+| `NoncurrentVersionExpiration.NoncurrentDays` | permanently deletes a noncurrent version once it has been noncurrent that long |
+| `NoncurrentVersionExpiration.NewerNoncurrentVersions` | keeps at most N noncurrent versions of a key, reclaiming oldest-first |
+| `Expiration.ExpiredObjectDeleteMarker` | removes a delete marker, and the key with it, once it is the only version left |
+| `Expiration.Days` / `.Date` (existing action) | on a versioned bucket, inserts a delete marker instead of deleting the object, as S3 does — through the same insertion the write requests use, so an expiry and a DELETE leave the same records behind |
+
+Concretely: `LifecycleAction` gains a `noncurrentVersionExpiration` field and
+`LifecycleExpiration` an `expiredObjectDeleteMarker` field — both optional
+additions to existing messages — and `LifecycleActionTask` gains a
+versionedKeyTable scan next to its keyTable scan, reusing the same batching, scan
+state and metrics. Expiration counts from the moment the version that superseded
+this one was committed, as `NoncurrentDays` does, not from when the version itself
+was written.
+
+One deliberate departure from stock S3: **the lifecycle service is enabled by
+default.** `ozone.lifecycle.service.enabled` flips from `false` to `true`. A rule
+that never runs is worse than no rule, and versioning is the first feature that
+depends on reclamation for correct operation rather than for tidiness. OM rejects
+`PutBucketVersioning(Enabled)` while the lifecycle service is disabled: that does
+not promise a bucket's versions are bounded — one whose owner writes no rule
+accumulates them, as on S3 — but it rules out a rule existing that nothing will
+ever act on.
+
+Earlier drafts bounded version growth with a bucket-level `maxVersions` property
+as well, first refusing the write once the limit was reached and later trimming
+oldest-first from the same scan. Both are dropped. A per-bucket cap duplicates
+`NewerNoncurrentVersions` in an Ozone-private property, which cuts against
+expressing reclamation as lifecycle rules at all; and applying it to the buckets
+it existed for — those carrying no lifecycle configuration — meant scheduling scan
+work for buckets the engine otherwise never visits, a second kind of task that
+differs from every other one in what it may assume. A bucket with no rule now
+accumulates versions, which is what S3 does.
+
+## Quota and observability
+
+All versions count against the bucket space quota. Version count, marker count,

Review Comment:
   Small question here: quota repair recounts a bucket by walking a fixed [set of tables](https://github.com/apache/ozone/blob/8a81f47f66129283cd6124c178fcf16f48539722/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java#L340-L352), and `versionedKeyTable` isn't one of them. If noncurrent versions stay charged but the recount only sees current versions, repair would come out lower and then persist that.
   
   Is that in scope here, or should it be a follow-up? HDDS-15835 was the same shape for incomplete MPU parts.



##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,614 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest.
+
+```mermaid
+flowchart LR
+    subgraph KT["keyTable — one entry per key"]
+        K1["/vol/buck/photo.jpg<br/>versionId 104 · current"]
+    end
+    subgraph VKT["versionedKeyTable — noncurrent, newest → oldest"]
+        V1["/vol/buck/photo.jpg 0x00 MAX-103<br/>delete marker 103"]
+        V2["/vol/buck/photo.jpg 0x00 MAX-102<br/>object 102"]
+        V3["/vol/buck/photo.jpg 0x00 MAX-101<br/>object 101"]
+    end
+    K1 -. "one seek reaches the newest noncurrent version" .-> V1
+    V1 --> V2 --> V3
+```
+
+Because the suffix is `MAX - versionId`, a single `seek` on the key's prefix lands
+on the newest noncurrent version: `ListObjectVersions` and version promotion are a
+seek plus a sequential read, never a scan-and-sort. The table is registered in
+`OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## How a key's versions evolve
+
+The two mechanisms that are hardest to read out of prose — the delete marker and
+the null version — are easiest to follow as one continuous story of a single key
+`k`. `obj(n)` is a data version with versionId `n`, `marker(n)` a delete marker,
+and `null` marks the record occupying the key's null slot.
+
+| # | Operation | Bucket state | keyTable (current) | versionedKeyTable (newest → oldest) |
+|---|---|---|---|---|
+| 0 | `PUT k` | Unversioned | `obj(—)` no versionId | – |
+| 1 | `PutBucketVersioning(Enabled)` | Enabled | `obj(—)` read as the null version | – |
+| 2 | `PUT k` | Enabled | `obj(101)` | `obj(—)` |
+| 3 | `PUT k` | Enabled | `obj(102)` | `obj(101)`, `obj(—)` |
+| 4 | `DELETE k` | Enabled | `marker(103)` | `obj(102)`, `obj(101)`, `obj(—)` |
+| 5 | `DELETE k?versionId=103` | Enabled | `obj(102)` **promoted back** | `obj(101)`, `obj(—)` |
+| 6 | `PutBucketVersioning(Suspended)` | Suspended | `obj(102)` | `obj(101)`, `obj(—)` |
+| 7 | `PUT k` | Suspended | `obj(104, null)` | `obj(102)`, `obj(101)` |
+| 8 | `PUT k` | Suspended | `obj(105, null)` overwritten in place | unchanged |
+| 9 | `DELETE k` | Suspended | `marker(106, null)` overwritten in place | unchanged |
+| 10 | `PutBucketVersioning(Enabled)`, `PUT k` | Enabled | `obj(107)` | `marker(106, null)`, `obj(102)`, `obj(101)` |
+
+Three things to read out of it:
+
+- **A delete marker is an ordinary version that happens to carry no blocks**
+  (step 4). It becomes the current version, so a plain `GET` returns 404 with
+  `x-amz-delete-marker: true` while every earlier version stays intact and
+  addressable by versionId. Deleting the marker by versionId (step 5) is exactly
+  S3's "restore an object": the newest remaining version is promoted back into
+  keyTable.
+- **Suspended does not stop versioning, it stops *accumulating*** (steps 7–9).
+  Each write lands in the key's single null slot: the first suspended write pushes
+  the previous current version down into versionedKeyTable and drops the key's
+  older null record (step 7); every later suspended write overwrites the null slot
+  in place, so the version count stops growing. Versions accumulated while Enabled
+  are never touched.
+- **The null version is a slot, not a reserved id** (step 7 onwards). It carries a
+  normally generated versionId like any other version and is marked by
+  `isNullVersion`. Pinning it to a fixed low id would misorder a null created
+  between two versioned writes, which is the middle version of the key, not its
+  oldest. Step 10 shows a null delete marker sliding into the noncurrent chain in
+  its correct chronological position once the bucket is Enabled again.
+
+### Where a write lands
+
+The table above is one trace; the rule behind it is the same for every write that
+carries no versionId. Suspended is the branch worth reading twice: it does not
+stop versioning, it collapses every write into the key's one null slot, which is
+why the version count stops growing while the versions accumulated under Enabled
+stay untouched.
+
+```mermaid
+flowchart TB
+    W["PUT k / DELETE k<br/>(no versionId)"] --> S{"bucket state"}
+    S -->|Unversioned| U["overwrite keyTable in place<br/>old blocks → deletedTable<br/><i>unchanged from today</i>"]
+    S -->|Enabled| E["move current version → versionedKeyTable<br/>write the new record into keyTable"]
+    S -->|Suspended| N{"is the current version<br/>already the null slot?"}
+    N -->|yes| N1["overwrite the null record in place<br/>its blocks → deletedTable, a marker carries none<br/><b>version count does not grow</b>"]
+    N -->|no| N2["move current version → versionedKeyTable<br/>write the new record as current, isNullVersion<br/>drop the key's older null record, if any"]
+    E --> R["new current version:<br/>an object on PUT, a delete marker on DELETE"]
+    N1 --> R
+    N2 --> R
+```
+
+### The life of a delete marker
+
+A delete marker is not a tombstone the read path has to special-case away — it is
+an ordinary version that carries no blocks, and it occupies the current slot like
+any other. That single fact produces all of its externally visible behaviour:
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    Obj: current = obj(102)
+    Marker: current = marker(103)
+    Gone: key no longer exists
+    [*] --> Obj: PUT k
+    Obj --> Marker: DELETE k (no versionId)
+    Marker --> Obj: DELETE k?versionId=103<br/>promotion restores obj(102)
+    Marker --> Gone: lifecycle ExpiredObjectDeleteMarker<br/>once it is the only version left
+    Obj --> Obj: GET k → 200
+    Marker --> Marker: GET k → 404, x-amz-delete-marker
+```
+
+Deleting the marker by versionId is exactly S3's "restore an object": nothing is
+rewritten, the newest remaining version is moved back into keyTable.
+
+### Version promotion
+
+The invariant is that keyTable always holds a key's current version. When a
+permanent delete removes the current version, promotion restores the invariant in
+the same WriteBatch, under the bucket write lock:
+
+```mermaid
+flowchart TB
+    A["DELETE k?versionId = current"] --> B["remove the current entry from keyTable<br/>blocks → deletedTable"]
+    B --> C{"seek versionedKeyTable<br/>on k's prefix"}
+    C -->|"a noncurrent version exists"| D["move the newest one back into keyTable<br/>record content stays frozen — a pure positional move"]
+    C -->|"none left"| E["the key disappears entirely"]
+```
+
+## VersionId generation
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and read when an OM
+starts, so a change takes effect on every bucket once the OMs have been restarted;
+it is not recorded in bucket metadata. Every generator must satisfy, **for itself**:
+strictly increasing within a key (a later version's id is always greater than every
+earlier version's id of that key); frozen once assigned; and `0` never handed out,
+being the unset value of the optional field. `1` is reserved as well, but by the
+pinned generator below rather than by the interface: it is that generator's own
+sentinel, which it assigns and never proposes.
+
+The identifier is deliberately **not** derived from OM's Ratis transaction index.
+The versionId is persisted, externally referenced, and part of the S3 API surface;
+tying it to the replication log would export an internal counter into a public
+identity and pin the feature to one execution model, when the OM execution path is
+itself an abstraction (`ExecutionContext` already carries an index whose term may
+be absent).
+
+Numbering a version by when it was written is what S3-compatible stores generally
+do: a GCS `generation` is a microsecond timestamp, an Azure Blob version id is an
+ISO-8601 instant, and RocksDB's user-defined timestamps order the versions of a key
+by a `key + timestamp` suffix — the same encoding the versionedKeyTable dbKey
+already uses. RocksDB is also the precedent for the split below: it orders by the
+timestamp but leaves keeping them monotonic to the caller.
+
+The shape matches SCM's `SequenceIdGenerator` — a cheap local allocation plus a
+floor that guarantees monotonicity. SCM's class itself is not reusable here: it
+lives in `server-scm` on `SCMHAManager` / `@Replicate`, so it would couple OM's
+write path to SCM across a module boundary. It also needs machinery this does not,
+because `localId` has to increase globally while a `versionId` only has to increase
+within one key, whose floor is already persisted as that key's current version. SCM
+does show the migration path if a persisted sequence is ever wanted:
+`upgradeToSequenceId` seeds one from a future timestamp in this same encoding.
+
+- **`UniqueIdVersionIdGenerator` (default)** — the same time-based scheme Ozone
+  already uses to generate block local IDs (`hdds.utils.UniqueId`): the id is
+  `currentTimeMillis << 16 | counter`, a 16-bit counter disambiguating ids proposed
+  within the same millisecond. It needs no allocator, no persistent state and no
+  coordination; values are small positive longs, so `MAX_VALUE - versionId`
+  ordering in versionedKeyTable stays plain signed arithmetic; and the id sorts by
+  creation time, which is what the version chain wants anyway.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes its own sentinel, `PinnedFirstVersionIdGenerator.FIRST_VERSION_ID
+  = 1`, below any proposed id, so it sorts oldest and can be referenced without
+  listing the key's versions first. All later versions take a normally proposed id. First versions are detected by "no
+  current version in keyTable", reusing the lookup the write path performs anyway.
+  Known trade-off: if every version of a key is permanently deleted and the key is
+  recreated, the new first version takes the sentinel again; only deployments that
+  accept this should configure this generator.
+
+**How the id appears on the wire.** `x-amz-version-id`, `?versionId=` and
+`ListObjectVersions` carry the id as its decimal digits, the way a GCS
+`generation` is rendered: `117121056706265088` for a proposed id, `1` for a
+pinned first version. Decimal digits cannot collide with `null`, the identifier
+S3 reserves for a key's null version, so both share the parameter without
+escaping. Clients should still treat the value as opaque. The digits of a proposed
+id do decode to the millisecond of the write, but that is a property of this
+generator rather than of the API, and ids are ordered only within one key —
+comparing them across keys means nothing.
+
+**Where the id is proposed.** The generator runs on the leader in `preExecute` and
+the id is stamped into the request that goes into the replication log, so every OM
+applies an identical value; nothing is proposed during apply. The one apply-time
+decision is the pinned sentinel, which is a constant and is therefore deterministic
+on its own: if the pinned generator is configured and the key has no current
+version at apply time, the version takes `FIRST_VERSION_ID` instead of the proposed
+id.
+
+**Why a proposed id is only a floor.** A time-based id is monotonic within one OM
+process, not across a leader change onto a node whose clock lags, and the "strictly
+increasing within a key" guarantee binds one generator rather than a sequence of
+them. So the generator only proposes, and the id a version is applied with is the
+later of that proposal and the id after the key's current version:
+
+```
+versionId = max(proposedVersionId, currentVersionId + 1)
+```
+
+The write path already holds the current version, so this costs no read, and it
+reads no other key and no global state: it is a pure function of the replicated
+request and the state every OM already shares, so all of them settle on the same id
+without coordinating. A current version predating versioning carries no id, and a
+key in that state has no other versions to order against, so it takes the proposed
+value unchanged — no versionedKeyTable lookup in any case.
+
+Ordering therefore never rests on the clock being trustworthy, and neither does it
+rest on one generator having proposed every id of a key. Under a clock regression, or
+after a change of generator, the ids of an affected key climb by one until proposed
+values overtake them again: the versions stay correctly ordered, and only the id's
+reading as a time degrades — the right one of the two to give up. An earlier draft
+instead rejected such a commit with `INVALID_REQUEST`; that is dropped, because it
+fails a client write over an operational clock problem the client can neither see
+nor fix.
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected.
+- **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+
+```mermaid
+flowchart TB
+    G["GET / HEAD k"] --> Q{"versionId given?"}
+    Q -->|no| C{"current version in keyTable"}
+    C -->|object| C1["200, the current version"]
+    C -->|delete marker| C2["404<br/>x-amz-delete-marker: true"]
+    Q -->|yes| L["check the current version,<br/>then point-look-up versionedKeyTable"]
+    L --> M{"the addressed version"}
+    M -->|object| M1["200, that version"]
+    M -->|delete marker| M2["405<br/>x-amz-delete-marker: true<br/>Allow: DELETE"]
+    M -->|no such version| M3["404"]
+```
+
+The 404/405 split is S3's, and it follows from the marker being addressable: an
+unaddressed read steps over the marker and reports the key absent, while a read
+that names the marker gets told the version exists but has no body to return.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) A batch delete is the same operation over
+  more keys and takes the same path — it goes through a request of its own, which
+  inserts one marker per key from the same implementation, so the two cannot
+  diverge. Hard-deleting there instead would destroy the current version and leave
+  the versions under it in the versionedKeyTable with nothing above them: absent to
+  a plain read, holding quota, and unreachable by promotion, which only runs when a
+  version is deleted by id. **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion as shown above.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation: version-aware lifecycle rules
+
+Version reclamation is expressed as **lifecycle rules on the lifecycle engine
+already in master** (HDDS-8342: `OmLifecycleConfiguration`, `KeyLifecycleService`,
+and the three S3 lifecycle endpoints), not as a service of its own. The engine
+already provides everything version reclamation needs — per-bucket rule storage
+and validation, prefix/tag filters, a resumable background scan with saved scan
+state, batched deletes through Ratis, metrics, and `suspend`/`resume` admin
+commands. Versioning contributes the three actions that the feature documentation
+currently lists as unsupported, and each maps 1:1 onto an S3 element:
+
+| S3 lifecycle element | Effect on a versioned bucket |
+|---|---|
+| `NoncurrentVersionExpiration.NoncurrentDays` | permanently deletes a noncurrent version once it has been noncurrent that long |
+| `NoncurrentVersionExpiration.NewerNoncurrentVersions` | keeps at most N noncurrent versions of a key, reclaiming oldest-first |
+| `Expiration.ExpiredObjectDeleteMarker` | removes a delete marker, and the key with it, once it is the only version left |
+| `Expiration.Days` / `.Date` (existing action) | on a versioned bucket, inserts a delete marker instead of deleting the object, as S3 does — through the same insertion the write requests use, so an expiry and a DELETE leave the same records behind |
+
+Concretely: `LifecycleAction` gains a `noncurrentVersionExpiration` field and
+`LifecycleExpiration` an `expiredObjectDeleteMarker` field — both optional
+additions to existing messages — and `LifecycleActionTask` gains a
+versionedKeyTable scan next to its keyTable scan, reusing the same batching, scan
+state and metrics. Expiration counts from the moment the version that superseded
+this one was committed, as `NoncurrentDays` does, not from when the version itself
+was written.
+
+One deliberate departure from stock S3: **the lifecycle service is enabled by
+default.** `ozone.lifecycle.service.enabled` flips from `false` to `true`. A rule
+that never runs is worse than no rule, and versioning is the first feature that
+depends on reclamation for correct operation rather than for tidiness. OM rejects
+`PutBucketVersioning(Enabled)` while the lifecycle service is disabled: that does
+not promise a bucket's versions are bounded — one whose owner writes no rule
+accumulates them, as on S3 — but it rules out a rule existing that nothing will
+ever act on.
+
+Earlier drafts bounded version growth with a bucket-level `maxVersions` property
+as well, first refusing the write once the limit was reached and later trimming
+oldest-first from the same scan. Both are dropped. A per-bucket cap duplicates
+`NewerNoncurrentVersions` in an Ozone-private property, which cuts against
+expressing reclamation as lifecycle rules at all; and applying it to the buckets
+it existed for — those carrying no lifecycle configuration — meant scheduling scan
+work for buckets the engine otherwise never visits, a second kind of task that
+differs from every other one in what it may assume. A bucket with no rule now
+accumulates versions, which is what S3 does.
+
+## Quota and observability
+
+All versions count against the bucket space quota. Version count, marker count,

Review Comment:
   Somewhat related: refunds are computed per block while commit charges are per key, which, for EC, don't match. That predates versioning (HDDS-16176), but the new reclaim path inherits it. Worth settling the formula as part of this design?



##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,614 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and version
+reclamation expressed as lifecycle rules on the lifecycle engine Ozone already
+has — with O(1) metadata cost per version operation and zero regression on
+non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation).
+
+Reclamation is a first-class part of the feature rather than an afterthought. The
+S3 troubleshooting guide documents list degradation and throttling on keys with
+millions of versions and leaves the fix to user-configured Lifecycle rules that
+are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master),
+so versioning does not need a reclamation mechanism of its own: it adds the three
+version-aware actions S3 defines to that engine, and the lifecycle service
+becomes enabled by default, so that a rule a user writes is a rule that runs.
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and
+  `Transition` generally) — Ozone has no storage-class tiering to transition to.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation and the version-aware diff the combination needs are deferred past
+  the first version, so OM rejects the combination outright rather than leaving it
+  to convention. The snapshot section below states the enforcement and what lifts
+  it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*] --> Unversioned: bucket created
+    Unversioned --> Enabled: PutBucketVersioning(Enabled)
+    Enabled --> Suspended: PutBucketVersioning(Suspended)
+    Suspended --> Enabled: PutBucketVersioning(Enabled)
+    Unversioned --> Unversioned: no versions exist
+    Enabled --> Enabled: every write creates a version
+    Suspended --> Suspended: writes reuse the null slot
+```
+
+There is no edge back to Unversioned. Once a bucket has been Enabled it can only
+move between Enabled and Suspended, so no single state change can silently
+destroy the versions the bucket already holds — S3's data-protection promise. OM
+enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED`
+with `INVALID_REQUEST`.
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest.
+
+```mermaid
+flowchart LR
+    subgraph KT["keyTable — one entry per key"]
+        K1["/vol/buck/photo.jpg<br/>versionId 104 · current"]
+    end
+    subgraph VKT["versionedKeyTable — noncurrent, newest → oldest"]
+        V1["/vol/buck/photo.jpg 0x00 MAX-103<br/>delete marker 103"]
+        V2["/vol/buck/photo.jpg 0x00 MAX-102<br/>object 102"]
+        V3["/vol/buck/photo.jpg 0x00 MAX-101<br/>object 101"]
+    end
+    K1 -. "one seek reaches the newest noncurrent version" .-> V1
+    V1 --> V2 --> V3
+```
+
+Because the suffix is `MAX - versionId`, a single `seek` on the key's prefix lands
+on the newest noncurrent version: `ListObjectVersions` and version promotion are a
+seek plus a sequential read, never a scan-and-sort. The table is registered in
+`OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## How a key's versions evolve
+
+The two mechanisms that are hardest to read out of prose — the delete marker and
+the null version — are easiest to follow as one continuous story of a single key
+`k`. `obj(n)` is a data version with versionId `n`, `marker(n)` a delete marker,
+and `null` marks the record occupying the key's null slot.
+
+| # | Operation | Bucket state | keyTable (current) | versionedKeyTable (newest → oldest) |
+|---|---|---|---|---|
+| 0 | `PUT k` | Unversioned | `obj(—)` no versionId | – |
+| 1 | `PutBucketVersioning(Enabled)` | Enabled | `obj(—)` read as the null version | – |
+| 2 | `PUT k` | Enabled | `obj(101)` | `obj(—)` |
+| 3 | `PUT k` | Enabled | `obj(102)` | `obj(101)`, `obj(—)` |
+| 4 | `DELETE k` | Enabled | `marker(103)` | `obj(102)`, `obj(101)`, `obj(—)` |
+| 5 | `DELETE k?versionId=103` | Enabled | `obj(102)` **promoted back** | `obj(101)`, `obj(—)` |
+| 6 | `PutBucketVersioning(Suspended)` | Suspended | `obj(102)` | `obj(101)`, `obj(—)` |
+| 7 | `PUT k` | Suspended | `obj(104, null)` | `obj(102)`, `obj(101)` |
+| 8 | `PUT k` | Suspended | `obj(105, null)` overwritten in place | unchanged |
+| 9 | `DELETE k` | Suspended | `marker(106, null)` overwritten in place | unchanged |
+| 10 | `PutBucketVersioning(Enabled)`, `PUT k` | Enabled | `obj(107)` | `marker(106, null)`, `obj(102)`, `obj(101)` |
+
+Three things to read out of it:
+
+- **A delete marker is an ordinary version that happens to carry no blocks**
+  (step 4). It becomes the current version, so a plain `GET` returns 404 with
+  `x-amz-delete-marker: true` while every earlier version stays intact and
+  addressable by versionId. Deleting the marker by versionId (step 5) is exactly
+  S3's "restore an object": the newest remaining version is promoted back into
+  keyTable.
+- **Suspended does not stop versioning, it stops *accumulating*** (steps 7–9).
+  Each write lands in the key's single null slot: the first suspended write pushes
+  the previous current version down into versionedKeyTable and drops the key's
+  older null record (step 7); every later suspended write overwrites the null slot
+  in place, so the version count stops growing. Versions accumulated while Enabled
+  are never touched.
+- **The null version is a slot, not a reserved id** (step 7 onwards). It carries a
+  normally generated versionId like any other version and is marked by
+  `isNullVersion`. Pinning it to a fixed low id would misorder a null created
+  between two versioned writes, which is the middle version of the key, not its
+  oldest. Step 10 shows a null delete marker sliding into the noncurrent chain in
+  its correct chronological position once the bucket is Enabled again.
+
+### Where a write lands
+
+The table above is one trace; the rule behind it is the same for every write that
+carries no versionId. Suspended is the branch worth reading twice: it does not
+stop versioning, it collapses every write into the key's one null slot, which is
+why the version count stops growing while the versions accumulated under Enabled
+stay untouched.
+
+```mermaid
+flowchart TB
+    W["PUT k / DELETE k<br/>(no versionId)"] --> S{"bucket state"}
+    S -->|Unversioned| U["overwrite keyTable in place<br/>old blocks → deletedTable<br/><i>unchanged from today</i>"]
+    S -->|Enabled| E["move current version → versionedKeyTable<br/>write the new record into keyTable"]
+    S -->|Suspended| N{"is the current version<br/>already the null slot?"}
+    N -->|yes| N1["overwrite the null record in place<br/>its blocks → deletedTable, a marker carries none<br/><b>version count does not grow</b>"]
+    N -->|no| N2["move current version → versionedKeyTable<br/>write the new record as current, isNullVersion<br/>drop the key's older null record, if any"]
+    E --> R["new current version:<br/>an object on PUT, a delete marker on DELETE"]
+    N1 --> R
+    N2 --> R
+```
+
+### The life of a delete marker
+
+A delete marker is not a tombstone the read path has to special-case away — it is
+an ordinary version that carries no blocks, and it occupies the current slot like
+any other. That single fact produces all of its externally visible behaviour:
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    Obj: current = obj(102)
+    Marker: current = marker(103)
+    Gone: key no longer exists
+    [*] --> Obj: PUT k
+    Obj --> Marker: DELETE k (no versionId)
+    Marker --> Obj: DELETE k?versionId=103<br/>promotion restores obj(102)
+    Marker --> Gone: lifecycle ExpiredObjectDeleteMarker<br/>once it is the only version left
+    Obj --> Obj: GET k → 200
+    Marker --> Marker: GET k → 404, x-amz-delete-marker
+```
+
+Deleting the marker by versionId is exactly S3's "restore an object": nothing is
+rewritten, the newest remaining version is moved back into keyTable.
+
+### Version promotion
+
+The invariant is that keyTable always holds a key's current version. When a
+permanent delete removes the current version, promotion restores the invariant in
+the same WriteBatch, under the bucket write lock:
+
+```mermaid
+flowchart TB
+    A["DELETE k?versionId = current"] --> B["remove the current entry from keyTable<br/>blocks → deletedTable"]
+    B --> C{"seek versionedKeyTable<br/>on k's prefix"}
+    C -->|"a noncurrent version exists"| D["move the newest one back into keyTable<br/>record content stays frozen — a pure positional move"]
+    C -->|"none left"| E["the key disappears entirely"]
+```
+
+## VersionId generation
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and read when an OM
+starts, so a change takes effect on every bucket once the OMs have been restarted;
+it is not recorded in bucket metadata. Every generator must satisfy, **for itself**:
+strictly increasing within a key (a later version's id is always greater than every
+earlier version's id of that key); frozen once assigned; and `0` never handed out,
+being the unset value of the optional field. `1` is reserved as well, but by the
+pinned generator below rather than by the interface: it is that generator's own
+sentinel, which it assigns and never proposes.
+
+The identifier is deliberately **not** derived from OM's Ratis transaction index.
+The versionId is persisted, externally referenced, and part of the S3 API surface;
+tying it to the replication log would export an internal counter into a public
+identity and pin the feature to one execution model, when the OM execution path is
+itself an abstraction (`ExecutionContext` already carries an index whose term may
+be absent).
+
+Numbering a version by when it was written is what S3-compatible stores generally
+do: a GCS `generation` is a microsecond timestamp, an Azure Blob version id is an
+ISO-8601 instant, and RocksDB's user-defined timestamps order the versions of a key
+by a `key + timestamp` suffix — the same encoding the versionedKeyTable dbKey
+already uses. RocksDB is also the precedent for the split below: it orders by the
+timestamp but leaves keeping them monotonic to the caller.
+
+The shape matches SCM's `SequenceIdGenerator` — a cheap local allocation plus a
+floor that guarantees monotonicity. SCM's class itself is not reusable here: it
+lives in `server-scm` on `SCMHAManager` / `@Replicate`, so it would couple OM's
+write path to SCM across a module boundary. It also needs machinery this does not,
+because `localId` has to increase globally while a `versionId` only has to increase
+within one key, whose floor is already persisted as that key's current version. SCM
+does show the migration path if a persisted sequence is ever wanted:
+`upgradeToSequenceId` seeds one from a future timestamp in this same encoding.
+
+- **`UniqueIdVersionIdGenerator` (default)** — the same time-based scheme Ozone
+  already uses to generate block local IDs (`hdds.utils.UniqueId`): the id is
+  `currentTimeMillis << 16 | counter`, a 16-bit counter disambiguating ids proposed
+  within the same millisecond. It needs no allocator, no persistent state and no
+  coordination; values are small positive longs, so `MAX_VALUE - versionId`
+  ordering in versionedKeyTable stays plain signed arithmetic; and the id sorts by
+  creation time, which is what the version chain wants anyway.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes its own sentinel, `PinnedFirstVersionIdGenerator.FIRST_VERSION_ID
+  = 1`, below any proposed id, so it sorts oldest and can be referenced without
+  listing the key's versions first. All later versions take a normally proposed id. First versions are detected by "no
+  current version in keyTable", reusing the lookup the write path performs anyway.
+  Known trade-off: if every version of a key is permanently deleted and the key is
+  recreated, the new first version takes the sentinel again; only deployments that
+  accept this should configure this generator.
+
+**How the id appears on the wire.** `x-amz-version-id`, `?versionId=` and
+`ListObjectVersions` carry the id as its decimal digits, the way a GCS
+`generation` is rendered: `117121056706265088` for a proposed id, `1` for a
+pinned first version. Decimal digits cannot collide with `null`, the identifier
+S3 reserves for a key's null version, so both share the parameter without
+escaping. Clients should still treat the value as opaque. The digits of a proposed
+id do decode to the millisecond of the write, but that is a property of this
+generator rather than of the API, and ids are ordered only within one key —
+comparing them across keys means nothing.
+
+**Where the id is proposed.** The generator runs on the leader in `preExecute` and
+the id is stamped into the request that goes into the replication log, so every OM
+applies an identical value; nothing is proposed during apply. The one apply-time
+decision is the pinned sentinel, which is a constant and is therefore deterministic
+on its own: if the pinned generator is configured and the key has no current
+version at apply time, the version takes `FIRST_VERSION_ID` instead of the proposed
+id.
+
+**Why a proposed id is only a floor.** A time-based id is monotonic within one OM
+process, not across a leader change onto a node whose clock lags, and the "strictly
+increasing within a key" guarantee binds one generator rather than a sequence of
+them. So the generator only proposes, and the id a version is applied with is the
+later of that proposal and the id after the key's current version:
+
+```
+versionId = max(proposedVersionId, currentVersionId + 1)
+```
+
+The write path already holds the current version, so this costs no read, and it
+reads no other key and no global state: it is a pure function of the replicated
+request and the state every OM already shares, so all of them settle on the same id
+without coordinating. A current version predating versioning carries no id, and a
+key in that state has no other versions to order against, so it takes the proposed
+value unchanged — no versionedKeyTable lookup in any case.
+
+Ordering therefore never rests on the clock being trustworthy, and neither does it
+rest on one generator having proposed every id of a key. Under a clock regression, or
+after a change of generator, the ids of an affected key climb by one until proposed
+values overtake them again: the versions stay correctly ordered, and only the id's
+reading as a time degrades — the right one of the two to give up. An earlier draft
+instead rejected such a commit with `INVALID_REQUEST`; that is dropped, because it
+fails a client write over an operational clock problem the client can neither see
+nor fix.
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected.
+- **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+
+```mermaid
+flowchart TB
+    G["GET / HEAD k"] --> Q{"versionId given?"}
+    Q -->|no| C{"current version in keyTable"}
+    C -->|object| C1["200, the current version"]
+    C -->|delete marker| C2["404<br/>x-amz-delete-marker: true"]
+    Q -->|yes| L["check the current version,<br/>then point-look-up versionedKeyTable"]
+    L --> M{"the addressed version"}
+    M -->|object| M1["200, that version"]
+    M -->|delete marker| M2["405<br/>x-amz-delete-marker: true<br/>Allow: DELETE"]
+    M -->|no such version| M3["404"]
+```
+
+The 404/405 split is S3's, and it follows from the marker being addressable: an
+unaddressed read steps over the marker and reports the key absent, while a read
+that names the marker gets told the version exists but has no body to return.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) A batch delete is the same operation over
+  more keys and takes the same path — it goes through a request of its own, which
+  inserts one marker per key from the same implementation, so the two cannot
+  diverge. Hard-deleting there instead would destroy the current version and leave
+  the versions under it in the versionedKeyTable with nothing above them: absent to
+  a plain read, holding quota, and unreachable by promotion, which only runs when a
+  version is deleted by id. **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion as shown above.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation: version-aware lifecycle rules
+
+Version reclamation is expressed as **lifecycle rules on the lifecycle engine
+already in master** (HDDS-8342: `OmLifecycleConfiguration`, `KeyLifecycleService`,
+and the three S3 lifecycle endpoints), not as a service of its own. The engine
+already provides everything version reclamation needs — per-bucket rule storage
+and validation, prefix/tag filters, a resumable background scan with saved scan
+state, batched deletes through Ratis, metrics, and `suspend`/`resume` admin
+commands. Versioning contributes the three actions that the feature documentation
+currently lists as unsupported, and each maps 1:1 onto an S3 element:
+
+| S3 lifecycle element | Effect on a versioned bucket |
+|---|---|
+| `NoncurrentVersionExpiration.NoncurrentDays` | permanently deletes a noncurrent version once it has been noncurrent that long |
+| `NoncurrentVersionExpiration.NewerNoncurrentVersions` | keeps at most N noncurrent versions of a key, reclaiming oldest-first |
+| `Expiration.ExpiredObjectDeleteMarker` | removes a delete marker, and the key with it, once it is the only version left |
+| `Expiration.Days` / `.Date` (existing action) | on a versioned bucket, inserts a delete marker instead of deleting the object, as S3 does — through the same insertion the write requests use, so an expiry and a DELETE leave the same records behind |
+
+Concretely: `LifecycleAction` gains a `noncurrentVersionExpiration` field and
+`LifecycleExpiration` an `expiredObjectDeleteMarker` field — both optional
+additions to existing messages — and `LifecycleActionTask` gains a
+versionedKeyTable scan next to its keyTable scan, reusing the same batching, scan
+state and metrics. Expiration counts from the moment the version that superseded
+this one was committed, as `NoncurrentDays` does, not from when the version itself
+was written.
+
+One deliberate departure from stock S3: **the lifecycle service is enabled by
+default.** `ozone.lifecycle.service.enabled` flips from `false` to `true`. A rule
+that never runs is worse than no rule, and versioning is the first feature that
+depends on reclamation for correct operation rather than for tidiness. OM rejects
+`PutBucketVersioning(Enabled)` while the lifecycle service is disabled: that does
+not promise a bucket's versions are bounded — one whose owner writes no rule
+accumulates them, as on S3 — but it rules out a rule existing that nothing will
+ever act on.
+
+Earlier drafts bounded version growth with a bucket-level `maxVersions` property
+as well, first refusing the write once the limit was reached and later trimming
+oldest-first from the same scan. Both are dropped. A per-bucket cap duplicates
+`NewerNoncurrentVersions` in an Ozone-private property, which cuts against
+expressing reclamation as lifecycle rules at all; and applying it to the buckets
+it existed for — those carrying no lifecycle configuration — meant scheduling scan
+work for buckets the engine otherwise never visits, a second kind of task that
+differs from every other one in what it may assume. A bucket with no rule now
+accumulates versions, which is what S3 does.
+
+## Quota and observability
+
+All versions count against the bucket space quota. Version count, marker count,
+and noncurrent bytes are exposed via `ozone sh bucket info` and Recon, and the
+version-aware lifecycle actions report through the existing
+`KeyLifecycleServiceMetrics`.
+
+## Interaction with Ozone snapshots
+
+OM rejects the combination on the same bucket, in both directions:
+
+* a snapshot can only be created on a bucket whose versioning status is
+  UNVERSIONED;
+* versioning can only be enabled on a bucket that has no snapshot.
+
+This is enforced rather than documented as a convention. Snapshots are not
+restricted to any bucket layout in code — `TestOmSnapshotObjectStore` covers OBS
+buckets explicitly — so nothing but enforcement keeps the two apart, and the
+combination has a real gap behind it: snapshot diff is keyed by objectID, which all
+versions of a key share, so a bucket carrying both would present a diff that cannot
+describe what changed. What makes a hard rejection cheap in practice is that the
+two features barely overlap today: snapshots are overwhelmingly used on FSO buckets
+for filesystem-style workloads, while versioning in this first phase is OBS-only.
+The rejection is applied to the source of a linked bucket and evaluated at apply
+time so the two directions cannot race, and a dev-only configuration key lifts it
+for testing. It is lifted for good once version-aware snapshot reclamation and a
+version-aware diff land as follow-up work.
+
+## S3 Gateway and native API surface
+
+New endpoints: `PutBucketVersioning`, `GetBucketVersioning`,
+`ListObjectVersions`. Extended: GET/HEAD/DELETE with `?versionId=` (including the
+literal `null`), `x-amz-version-id` / `x-amz-delete-marker` response headers,
+per-entry version semantics in batch `DeleteObjects` (whose OM-side marker
+insertion lands with the write paths, leaving the gateway to carry the per-entry
+`versionId` and report `DeleteMarker`), `CopyObject` versioning
+behavior, and version-id headers on `PutObject` / `CompleteMultipartUpload`. The
+native interfaces (`ozone sh`, `OzoneBucket` API) are extended in parallel so
+versions are manageable outside the S3 path.
+
+## Compatibility and upgrade
+
+Wire protocol changes are limited to optional proto fields, so old records and
+old clients are unaffected. The new column family and request surface are gated
+by `OMLayoutFeature.OBJECT_VERSIONING`; before cluster finalization, enabling
+versioning is rejected with `NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION`.
+Snapshots taken before the feature is used are unaffected, and snapshot
+interaction is covered above. Buckets without versioning behave exactly as today

Review Comment:
   Could we say something about buckets that already have the old versioning flag set?  They're neither "without versioning" nor the new feature. This came up on HDDS-16127, and the suggestion there was to check with you: reject, migrate, or leave them as they are?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3757066856


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited; markers
+count toward the limit), `noncurrentVersionExpiration` (opt-in), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed) — all enforced by a new **VersionCleanupService**
+following the `KeyDeletingService` pattern. Versions beyond `maxVersions` are
+reclaimed oldest-first, asynchronously rather than inside the write transaction,
+so the cost of a version write does not grow with the number of versions the key
+already has. An earlier draft also offered a `REJECT` policy that failed the
+write instead; it is dropped. S3 has no error code for the condition and no
+client retries on it, so a default limit enforced that way would break unmodified
+S3 tooling after `maxVersions` overwrites of one key — and reclaiming is what
+S3's own lifecycle rules do. Expiration counts from the moment the version that
+superseded this one was committed, as Lifecycle's `NoncurrentDays` does, not from
+when the version itself was written. All versions count against the bucket space quota;
+version count, marker count, and noncurrent bytes are exposed via
+`ozone sh bucket info` and Recon. `maxVersions` is documented as an Ozone
+extension over S3 semantics (comparable to Lifecycle `NewerNoncurrentVersions`).
+
+## Interaction with Ozone snapshots
+
+**The first version does not allow both features on the same bucket.** The
+version-aware reclamation described in this section is deferred past it, and
+without that lookup the gap below is a data-loss path rather than a rough edge,
+so OM enforces the exclusion instead of documenting an expectation: snapshot
+creation is rejected on an ENABLED or SUSPENDED bucket, and enabling versioning
+or moving SUSPENDED → ENABLED is rejected while any snapshot exists in the

Review Comment:
   Is this drafted by AI?  Maybe we can simply to say that 
   - snapshot creation is only allowed on UNVERSIONED bucket
   - bucket versioning is only allowed to be turned on bucket which doesn't have any snapshot created. 
   Basically user choose either versioning or snapshot to meet their requirements. It's not common to see user requires both features.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3763144329


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must

Review Comment:
   It will be guaranted on commiting, versionId less then currentVersion will be rejected.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #10822:
URL: https://github.com/apache/ozone/pull/10822#issuecomment-5406382033

   > @ChenSammi
   > 
   > > On decoupling versionId from the Ratis transaction index
   > 
   > The goal here is to stop deriving the id from the replication log — versionId is user-visible and permanent, and the versionedKeyTable layout depends on it being ordered, which is the property most at risk if OM's execution model changes.
   > 
   > First look was SCM's SequenceIdGenerator, since it solves the same problem. It doesn't fit: it lives in server-scm and is built on SCMHAManager / DBTransactionBuffer / @replicate, so reusing it would couple OM's write path to SCM across a module boundary. Its shape is still instructive though — a cheap local allocation plus a floor that guarantees monotonicity.
   > 
   > So instead, looking at how object stores identify a version: a GCS generation is a microsecond timestamp, an Azure Blob version id is an ISO-8601 instant, and RocksDB's user-defined timestamps order versions by a key + timestamp suffix — the same encoding we already use in the dbKey. All three make the timestamp the sort key.
   > 
   > Proposal: timestamp + max.
   > 
   > The OM handling the request proposes a versionId in preExecute — wall clock in microseconds. preExecute's output is what goes to Ratis, so the id is fixed before replication. On apply, VersionIdAllocator settles it: max(proposedVersionId, currentVersionId + 1). A clock can't promise per-key ordering alone (same-microsecond writes, or a lagging clock after leader change), so the proposal is only a floor. The current version is already in hand, so this costs no read and stays a pure function of replicated state.
   > 
   > That gives SCM's local-allocation-plus-floor structure without its batch machinery, since a versionId only has to increase within one key.
   > 
   > Cost: one optional field on the commit / delete / complete-MPU requests. No new table, no extra I/O.
   
   Thanks @symious  for the further investigation. The timestamp solution looks good to me. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3757066856


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited; markers
+count toward the limit), `noncurrentVersionExpiration` (opt-in), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed) — all enforced by a new **VersionCleanupService**
+following the `KeyDeletingService` pattern. Versions beyond `maxVersions` are
+reclaimed oldest-first, asynchronously rather than inside the write transaction,
+so the cost of a version write does not grow with the number of versions the key
+already has. An earlier draft also offered a `REJECT` policy that failed the
+write instead; it is dropped. S3 has no error code for the condition and no
+client retries on it, so a default limit enforced that way would break unmodified
+S3 tooling after `maxVersions` overwrites of one key — and reclaiming is what
+S3's own lifecycle rules do. Expiration counts from the moment the version that
+superseded this one was committed, as Lifecycle's `NoncurrentDays` does, not from
+when the version itself was written. All versions count against the bucket space quota;
+version count, marker count, and noncurrent bytes are exposed via
+`ozone sh bucket info` and Recon. `maxVersions` is documented as an Ozone
+extension over S3 semantics (comparable to Lifecycle `NewerNoncurrentVersions`).
+
+## Interaction with Ozone snapshots
+
+**The first version does not allow both features on the same bucket.** The
+version-aware reclamation described in this section is deferred past it, and
+without that lookup the gap below is a data-loss path rather than a rough edge,
+so OM enforces the exclusion instead of documenting an expectation: snapshot
+creation is rejected on an ENABLED or SUSPENDED bucket, and enabling versioning
+or moving SUSPENDED → ENABLED is rejected while any snapshot exists in the

Review Comment:
   Maybe we can simply to say that 
   - snapshot creation is only allowed on UNVERSIONED bucket
   - bucket versioning is only allowed to be turned on bucket which doesn't have any snapshot created. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3756888550


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version

Review Comment:
   What will be the result if delete an delete marker? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3756821640


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must

Review Comment:
   The VersionIdGenerator should be linked with a bucket after it's used when versioning is first turn on for this bucket, and cannot be changed after that. Otherwise, it cannot guarantee that versionId is unique across different ID generators and increasing within a key. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3757009744


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,433 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+- **Coexistence with Ozone snapshots on the same bucket** — the version-aware
+  reclamation the combination needs is deferred past the first version, so OM
+  rejects the combination outright rather than leaving it to convention. The
+  snapshot section below states the enforcement and what lifts it.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key
+  names contain `/` verbatim, which would interleave a key's versions with those of
+  keys nested under it), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` and `1` never handed out (`1` is the first-version sentinel).
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute — it
+carries a normally generated id like any other version, since a null created
+between two versioned writes is not the oldest one;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current.
+- **PUT (Suspended)**: the new record takes the null slot and becomes the current
+  version — if the null slot is already the current version, overwrite it in place;
+  otherwise move the current version into versionedKeyTable, write the new record
+  into keyTable as current, and delete the old null record (if any) from
+  versionedKeyTable, blocks to deletedTable in both cases. Versions accumulated
+  while Enabled are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable; if the addressed version is a delete
+  marker, current or not, the gateway returns 405 with `x-amz-delete-marker: true`
+  and `Allow: DELETE`.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker takes the null
+  slot exactly as a suspended PUT does.) **DELETE ?versionId=x**: permanently
+  delete that version (blocks to deletedTable); if it was current, trigger version
+  promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited; markers
+count toward the limit), `noncurrentVersionExpiration` (opt-in), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed) — all enforced by a new **VersionCleanupService**
+following the `KeyDeletingService` pattern. Versions beyond `maxVersions` are

Review Comment:
   We have new service `KeyLifecycleService` now after HDDS-8342 merged into master. Instead of introducing a new service `VersionCleanupService`, we can consider extend the `KeyLifecycleService` service to support the new action **NoncurrentVersionExpiration** to help evaluate and remove old versions, since  `KeyLifecycleService` also need to be extended to support object expiration for versioned buckets, so we can scan the bucket once, executed all the actions matched(wanted). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. Add OEP doc [ozone]

Posted by "aswinshakil (via GitHub)" <gi...@apache.org>.
aswinshakil commented on PR #10822:
URL: https://github.com/apache/ozone/pull/10822#issuecomment-5109925565

   Thank you for working on this, I have few questions. 
   
   1. Ozone already has some level of versioning in `KeyInfo` `repeated KeyLocationList keyLocationList`, This list can have multiple version of keys. Although the current versioning is not an exhaustive implementation. Has this been considered instead of creating and new `versionKeyTable`. 
   2. Snapshot feature supports the old versioning. But the details on the new versioning with `versionKeyTable` and how it will impact snapshot is still unclear. Can you add more details around this area as well?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3679952569


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,376 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}/{Long.MAX_VALUE - versionId}

Review Comment:
   Thanks for the design doc @symious.
   
   Would recommend using specific separator before the versionSuffix (not `/`). 
   With the `/` as a separator say there are 2 objects `/vol/bucket/obj/{versionSuffix}` and `/vol/bucket/obj/001/{versionSuffix}` a prefix scan on `/vol/bucket/obj` will return all objects with that prefix and will include unexpected lexicographic neighbors and client would need a way to further filter through those.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3680069109


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,376 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}/{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` reserved for the null slot and `1` for the first-version sentinel.
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current; if
+  `maxVersions` is exceeded, trim the oldest noncurrent version in the same batch
+  (its blocks go to deletedTable).
+- **PUT (Suspended)**: locate the null slot (current, or an `isNullVersion`
+  record in versionedKeyTable) and replace it; versions accumulated while Enabled

Review Comment:
   Should this be an atomic operation that does
   If null in versionedKeyTable
   - Move currentVersion from keyTable to versionedKeyTable
   - Add the new null key to keyTable 
   - Remove nullKey and any associated delete marker from versionedKeyTable
   If null in keyTable
   - Replace it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15728. [OEP] Support S3-compatible object versioning in Ozone [ozone]

Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on code in PR #10822:
URL: https://github.com/apache/ozone/pull/10822#discussion_r3680082542


##########
hadoop-hdds/docs/content/design/s3-versioning.md:
##########
@@ -0,0 +1,376 @@
+---
+title: S3-compatible Object Versioning
+summary: Bucket-level, S3-compatible object versioning with O(1) version writes and built-in reclamation
+date: 2026-07-21
+jira: HDDS-15728
+status: accepted
+author: Symious
+---
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Summary
+
+Add S3-compatible object versioning to Ozone: the full three-state bucket state
+machine (Unversioned / Enabled / Suspended), per-key version chains with delete
+markers and null versions, the S3 versioning APIs on the S3 Gateway, and built-in
+version reclamation (`maxVersions`, background expiration) — with O(1) metadata
+cost per version operation and zero regression on non-versioned paths.
+
+# Status
+
+Defined in the markdown header.
+
+# Problem statement (Motivation / Abstract)
+
+Amazon S3 provides bucket-level object versioning: a single key can retain multiple
+versions, so users can recover objects that were accidentally overwritten or
+deleted. A large part of the S3 ecosystem (backup software, data lake components,
+DR tooling) depends on the versioning APIs (`PutBucketVersioning`,
+`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an
+S3-compatible API through the S3 Gateway but does not support object versioning
+today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended
+state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record
+rather than object versions, and the gateway has no versioning endpoints.
+
+This proposal implements versioning with S3-compatible semantics, usable by
+standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a
+version operation is decoupled from the number of versions (one extra small KV
+write per operation), and reclamation controls are built into the feature itself
+to avoid the unbounded version accumulation problems commonly seen on S3 (the S3
+troubleshooting guide documents list degradation and throttling on keys with
+millions of versions, and leaves the fix to user-configured Lifecycle rules that
+are often forgotten).
+
+# Non-goals
+
+- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no
+  counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning`
+  returns NotImplemented.
+- **A full S3 Lifecycle rule engine** — only the minimal reclamation capabilities
+  that versioning itself requires are included.
+- **Version-aware cross-cluster replication.**
+- **Versioning for FSO / LEGACY bucket layouts** — the first version supports
+  OBJECT_STORE buckets only; enabling versioning on other layouts returns
+  NotImplemented. Combining FSO's directory/rename semantics with per-key version
+  chains is disproportionately complex, and S3 tooling scenarios essentially use
+  the OBS layout. FSO support can be evaluated as an independent follow-up.
+
+# Technical Description (Architecture and implementation details)
+
+## Bucket state machine
+
+```
+Unversioned (default) ──enable──▶ Enabled ◀──enable── Suspended
+                                     │                     ▲
+                                     └──────suspend────────┘
+        (Once Enabled, a bucket can never return to Unversioned)
+```
+
+A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` /
+`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and
+`BucketArgs`. The legacy `isVersionEnabled` boolean is kept and maintained in
+two-way sync (`ENABLED → true`, otherwise `false`; records without the enum are
+interpreted via the boolean), so old and new clients/OMs coexist during rolling
+upgrades. OM enforces the state machine on `SetBucketProperty`: transitions back
+to `UNVERSIONED` are rejected with `INVALID_REQUEST`, preserving S3's
+data-protection promise that no single state change can silently destroy
+historical versions.
+
+## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent)
+
+A new column family, **versionedKeyTable**, splits responsibilities with keyTable:
+
+- **keyTable** (existing, semantics unchanged) always holds each key's **current
+  version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects
+  read paths are unchanged.
+- **versionedKeyTable** (new) holds all **noncurrent** versions (including
+  noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is
+
+  ```
+  /{volume}/{bucket}/{keyName}/{Long.MAX_VALUE - versionId}
+  ```
+
+  (fixed-width hex suffix), so all versions of a key are physically adjacent and
+  ordered newest to oldest: `ListObjectVersions` and version promotion are a
+  single seek plus a sequential read. The table is registered in
+  `OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so
+  bucket-prefixed iteration and SST filtering resolve its prefix.
+
+`OmKeyInfo` gains three optional proto fields (old records deserialize
+compatibly): `versionId` (int64, assigned once at version creation, then frozen),
+`isDeleteMarker` (a marker is a record with this flag and no data blocks — no
+datanode storage), and `isNullVersion` (the single overwritable "null version"
+slot per key). Keys written before versioning was enabled are interpreted as null
+versions on read — **zero migration**, matching S3's "enabling versioning does
+not change existing objects".
+
+Every keyTable ↔ versionedKeyTable update rides OM's existing atomic
+multi-table WriteBatch commit (the same double-buffer pattern used today for
+keyTable + deletedTable on overwrite): no new transaction mechanism and no
+cross-table consistency problem. The new column family is introduced under the OM
+layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`):
+before finalization, requests carrying a versioning status are rejected.
+
+## VersionId: a pluggable generator
+
+`versionId` generation is abstracted behind a `VersionIdGenerator` interface,
+chosen per cluster by class name (`ozone.om.versioning.version-id-generator`), so
+a deployment can plug in its own. The generator is cluster-wide and may be changed
+on a running cluster; it is not recorded in bucket metadata. Every generator must
+satisfy, **for itself**: strictly increasing within a key (a later version's id is
+always greater than every earlier version's id of that key); frozen once assigned;
+`0` reserved for the null slot and `1` for the first-version sentinel.
+
+That guarantee binds one generator, not a sequence of them, so the write path
+enforces it at commit: a commit whose id does not come after the key's current
+version is rejected (`INVALID_REQUEST`), and the operator deletes the key's
+versions before writing under the new generator. The check costs no read in the
+steady state — the current version holds the key's largest id, so an id above it
+cannot be taken; only a record predating versioning, which carries no id to order
+against, falls back to a versionedKeyTable lookup.
+
+- **`TransactionIndexVersionIdGenerator` (default)** — the OM Ratis transaction
+  index of the committing transaction, used directly. No allocator state of any
+  kind. Externally encoded as an opaque URL-safe string. objectID's epoch bits are
+  deliberately not applied: dropping them keeps versionIds small positive longs, so
+  the versionedKeyTable ordering stays plain signed arithmetic, and uniqueness
+  rests on the monotonicity of the Ratis log index — a log that was reset or rolled
+  back is caught by the commit-time check rather than silently accepted.
+- **`PinnedFirstVersionIdGenerator` (opt-in)** — only the first version is
+  special: it takes the reserved sentinel `FIRST_VERSION_ID = 1`, below any usable
+  transaction index, so it sorts oldest and can be referenced without listing the
+  key's versions first. All later versions are the commit transaction's index; no
+  persistent allocator. First versions are detected by "no current version in
+  keyTable", reusing the lookup the write path performs anyway. Known trade-off: if
+  every version of a key is permanently deleted and the key is recreated, the new
+  first version takes the sentinel again; only deployments that accept this should
+  configure this generator.
+
+How a versionId is rendered on the wire — the opaque encoding, and whether the
+pinned first version is presented as a fixed literal or derived from the keyName —
+is part of the `?versionId=` read path (T4), not of generation.
+
+The null version is not a special ID value but the `isNullVersion` attribute;
+`versionId=null` requests resolve to "locate this key's null slot".
+
+## Request handling
+
+- **PUT (Enabled)**, atomic in one WriteBatch: move the current version (if any)
+  into versionedKeyTable; write the new record into keyTable as current; if
+  `maxVersions` is exceeded, trim the oldest noncurrent version in the same batch
+  (its blocks go to deletedTable).
+- **PUT (Suspended)**: locate the null slot (current, or an `isNullVersion`
+  record in versionedKeyTable) and replace it; versions accumulated while Enabled
+  are unaffected. **PUT (Unversioned)**: unchanged from today.
+- **GET/HEAD**: without versionId, read keyTable; a current delete marker returns
+  404 with `x-amz-delete-marker: true`. With versionId, check the current version
+  first, then point-look-up versionedKeyTable.
+- **DELETE without versionId (Enabled)**: move current into versionedKeyTable and
+  write a delete marker as the new current. (Suspended: the marker overwrites the
+  null slot.) **DELETE ?versionId=x**: permanently delete that version (blocks to
+  deletedTable); if it was current, trigger version promotion.
+- **Version promotion** — the invariant is that keyTable always holds a key's
+  current version. When a permanent delete removes the current version, one
+  `seek` on the key's versionedKeyTable prefix yields the newest noncurrent
+  version (reverse ordering), which is moved back into keyTable unchanged — a
+  pure positional move; the record's content stays frozen. If no noncurrent
+  version remains, the key disappears entirely. Executed in the same WriteBatch
+  as the delete, under the bucket write lock. Deleting a delete marker this way
+  is exactly the S3 "restore an object" flow.
+- **ListObjects**: scans keyTable only, skipping keys whose current version is a
+  marker — list scan volume is not amplified by versions (an advantage over S3's
+  single-namespace implementation). **ListObjectVersions**: merges keyTable and
+  versionedKeyTable in key order (both are key-name prefixed, so the merge is
+  naturally ordered), with `key-marker` / `version-id-marker` pagination.
+
+## Reclamation, quota, observability
+
+Built-in bucket-level controls: `maxVersions` (default 100, 0 = unlimited;
+synchronous trim inside the write transaction; markers count toward the limit),
+`noncurrentVersionExpiration` (opt-in, enforced by a new
+**VersionCleanupService** following the `KeyDeletingService` pattern), and
+expired-delete-marker cleanup (enabled by default: when only a marker remains,
+the whole key is removed). All versions count against the bucket space quota;
+version count, marker count, and noncurrent bytes are exposed via
+`ozone sh bucket info` and Recon. `maxVersions` is documented as an Ozone
+extension over S3 semantics (comparable to Lifecycle `NewerNoncurrentVersions`).
+
+## Interaction with Ozone snapshots
+
+A snapshot checkpoints the entire OM RocksDB, so versionedKeyTable is captured
+automatically along with the bucket's versioning status: a snapshot holds the
+complete version history as of its creation. Snapshot creation and deletion need
+no change — `SnapshotDeletingService` moves only deletedTable, deletedDirTable
+and snapshotRenamedTable entries between snapshots, and permanently deleted
+versions travel through deletedTable, the path already used when an overwrite
+reclaims the previous blocks. Multiple versions of one key queued for deletion
+share a deletedTable entry, which is a `RepeatedOmKeyInfo` list evaluated one
+record at a time, so no new structure is required there either.
+
+What does change is **block reclamation**. Snapshots share physical blocks with
+the active object store, so a snapshot keeps its data only because
+`KeyDeletingService` refuses to reclaim blocks that a previous snapshot still
+references. That decision is made per deletedTable record by
+`ReclaimableKeyFilter`, which resolves the key's path in the previous snapshot
+and looks it up **in keyTable only**
+(`KeyManagerImpl.getPreviousSnapshotOzoneKeyInfo`), then compares objectID and
+block locations. With versioning a key's live records span two tables, so the
+lookup misses: permanently deleting a noncurrent version resolves to whatever is
+current in the previous snapshot — a different record with different blocks —
+which reads as "not present in the previous snapshot", i.e. reclaimable. Blocks
+a snapshot still points at would be deleted. objectID does not rescue the
+comparison either way: `prepareFileInfo` keeps it stable across a key's versions,
+so it matches while the block lists do not.
+
+The fix is to make that lookup version-aware. A record carrying a `versionId`
+resolves to the exact dbKey `/{volume}/{bucket}/{keyName}/{MAX - versionId}` in
+the previous snapshot's versionedKeyTable, falling back to keyTable when that
+version was current there. Because versionId is frozen at creation and never
+reused, this is an exact point lookup that *replaces* — rather than extends — the
+objectID-plus-block-list heuristic for versioned buckets. The same lookup feeds
+`calculateExclusiveSize`, keeping snapshot exclusive-size reporting correct for
+noncurrent versions.
+
+This makes deletedTable the single choke point for version reclamation: every
+path that removes a version permanently — `DELETE ?versionId=`, the `maxVersions`
+trim, and VersionCleanupService — writes the record to deletedTable and lets
+KeyDeletingService apply the snapshot-aware filter. None of them may reclaim
+blocks directly.
+
+Snapshot diff remains a **current-version diff**; versionedKeyTable is
+deliberately not added to it. keyTable still holds exactly one record per key, so
+diff cost and semantics do not change as versions accumulate. One rule does need
+stating: **a key whose current version is a delete marker counts as absent**.
+Without it a delete surfaces as a MODIFY (if the marker carries the key's
+objectID) or as a DELETE plus a phantom CREATE (if it carries a new one), instead
+of the DELETE users expect; restoring an object by removing its marker reports
+CREATE correspondingly.
+
+Reads inside a snapshot go through `OmSnapshot`, which implements
+`IOmMetadataReader`, whose `lookupKey(OmKeyArgs)` has no version dimension.
+`?versionId=` is plumbed through it so versions retained in a snapshot are
+actually readable; `ListObjectVersions` against a snapshot is out of scope for
+the first version.
+
+## S3 Gateway and native API surface
+
+New endpoints: `PutBucketVersioning`, `GetBucketVersioning`,
+`ListObjectVersions`. Extended: GET/HEAD/DELETE with `?versionId=` (including the
+literal `null`), `x-amz-version-id` / `x-amz-delete-marker` response headers,
+per-entry version semantics in batch `DeleteObjects`, `CopyObject` versioning
+behavior, and version-id headers on `PutObject` / `CompleteMultipartUpload`. The
+native interfaces (`ozone sh`, `OzoneBucket` API) are extended in parallel so
+versions are manageable outside the S3 path.
+
+## Compatibility and upgrade
+
+Wire protocol changes are limited to optional proto fields, so old records and
+old clients are unaffected. The new column family and request surface are gated
+by `OMLayoutFeature.OBJECT_VERSIONING`; before cluster finalization, enabling
+versioning is rejected with `NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION`.
+Snapshots taken before the feature is used are unaffected, and snapshot
+interaction is covered above. Buckets without versioning behave exactly as today
+(verified by benchmarks).
+
+# Alternatives
+
+- **Single-record layout** (versions embedded in the existing
+  `keyLocationVersions` list): every PUT/DELETE becomes a full read-modify-write
+  of one growing RocksDB value — write amplification linear in the version count
+  (hundreds of KB per write at ~100 versions of a large key), degraded
+  compaction, and no per-version object metadata (size / mtime / ETag) without
+  rebuilding an object record inside the location structure. Rejected in favor of
+  the two-table design, where each version operation writes one small KV (O(1))
+  and each version is a complete `OmKeyInfo`.
+- **A two-state model** (dropping Suspended): "disabled after being enabled" must
+  retain historical versions anyway — otherwise a single state change silently
+  destroys data — yet the bucket would present itself as never-versioned and
+  `GetBucketVersioning` could not answer `Suspended`. Since the semantics are
+  unavoidable, keep the standard three states and full S3 compatibility.
+- **Reusing `objectID` as the version identity**: code verification showed the
+  overwrite path (`OMKeyRequest.prepareFileInfo`) reuses the existing record via
+  `toBuilder()`, so objectID is intentionally stable across overwrites — snapshot
+  diff relies on it to distinguish "modified" from "deleted + recreated".
+  **Reusing `updateID`**: hsync re-commits the same version repeatedly, so it
+  drifts, while a version identity must be frozen at creation. Hence the
+  dedicated `versionId` field.
+- **S3-style random version IDs**: equivalent for reuse-prevention, but they
+  destroy the temporal clustering of versionedKeyTable — version promotion would
+  degrade from one seek to a scan-and-sort (with mtime tiebreak problems), and
+  `GET ?versionId=` from a point lookup to a prefix scan. Unpredictability, the
+  only remaining benefit, is provided by the external encoding layer instead.
+- **A per-key sequential counter** for readable first versions: requires a
+  persistent allocator and reuses IDs after full deletion of a key. Replaced by the
+  pinned-first generator, which special-cases only the first version and needs no
+  allocator.
+- **A version-level snapshot diff** (reporting noncurrent versions created or
+  removed between two snapshots): the diff is keyed by objectID, which a key's
+  versions share, so it would need a different identity and a version dimension
+  in the report; versionedKeyTable would also have to join
+  `RocksDBCheckpointDiffer.COLUMN_FAMILIES_TO_TRACK_IN_DAG`, paying
+  compaction-log and SST retention cost for information already reachable through
+  the snapshot's own version listing. Deferred; the diff stays a current-version
+  diff with an explicit delete-marker rule.
+- **Pinning the generator into bucket metadata** (so a bucket keeps the generator
+  it was created with): removes the risk of a mid-life generator change, but adds a
+  proto field, a write path in both bucket create and set-property, and leaves the
+  generator un-changeable even when an operator legitimately wants to switch.
+  Replaced by the commit-time ordering check, which turns the risk into a loud,
+  per-key failure with a clear remedy and costs no persistent state.
+
+# Plan
+
+Implemented as one umbrella Jira with ten tasks (33 sub-tasks, each roughly one
+PR), in dependency order `T1 → T2 → T3 → T4 → T5 → (T6 ∥ T7) → (T8 ∥ T9) → T10`:
+
+| Task | Scope |
+|---|---|
+| T1 Metadata foundation | proto three-state enum + legacy-boolean sync, set-property state machine, `OmKeyInfo` version fields, versionedKeyTable column family, layout feature gate |
+| T2 VersionId generator framework | `VersionIdGenerator` interface with class-name configuration, transaction-index default, commit-time ordering check, pinned-first generator |
+| T3 ENABLED write paths | PUT two-table update, DELETE marker insertion, quota accounting |
+| T4 Read / permanent delete / promotion | `?versionId=` reads (including through `OmSnapshot`), permanent delete with the version-aware reclamation lookup, version promotion |
+| T5 SUSPENDED semantics | null-slot overwrite, null markers, zero-migration legacy keys |
+| T6 S3 Gateway endpoints | bucket versioning endpoints, object `versionId` support, batch delete |
+| T7 ListObjectVersions | OM merged listing, protocol plumbing, gateway `?versions` |
+| T8 Reclamation | `maxVersions` trim, VersionCleanupService, expired marker cleanup |

Review Comment:
   Recommend to implement this task before the S3 endpoints are exposed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org