You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "andyhuangdev (via GitHub)" <gi...@apache.org> on 2026/09/21 12:27:50 UTC

[PR] HDDS-16485. Add Object Lock protobuf schemas and retention models [ozone]

andyhuangdev opened a new pull request, #11293:
URL: https://github.com/apache/ozone/pull/11293

   ## What changes were proposed in this pull request?
   
   Add protobuf schemas and domain models for Object Lock retention and legal hold settings.
   
   Buckets gain optional objectLockEnabled and defaultRetention fields. KeyInfo and KeyInfoProtoLight gain retentionDate, retentionConfig, and legalHold fields at indices 23, 24, and 25.
   
   Update OmBucketInfo and OmKeyInfo to preserve these settings through protobuf conversion, metadata codecs, and object copies. Add an immutable Retention model and utilities for retention rule validation and UTC expiration calculations.
   
   Event Hold settings are represented in the schema; state transitions and WORM enforcement remain separate tasks.
   
   ## What is the link to the Apache JIRA
   
   [HDDS-16485](https://issues.apache.org/jira/browse/HDDS-16485)
   
   ## How was this patch tested?
   
   Added tests covering protobuf conversion, metadata copies, open-key and committed-key codecs, Light proto compatibility, and missing optional fields. Retention tests cover date calculations, leap years, expiration boundaries, invalid rules, and overflow.
   
   All 23 tests in TestOmBucketInfo, TestOmKeyInfo, and TestOmKeyInfoCodec passed. All four tests in TestRetention passed. Checkstyle, RAT, and git diff --check passed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16485. Add Object Lock protobuf schemas and retention models [ozone]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #11293:
URL: https://github.com/apache/ozone/pull/11293#discussion_r4062133289


##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmBucketInfo.java:
##########
@@ -426,6 +443,8 @@ public OmBucketInfo withOperationalPropertiesFrom(OmBucketInfo source) {
    * Builder for OmBucketInfo.
    */
   public static class Builder extends WithObjectID.Builder<OmBucketInfo> {
+    private boolean objectLockEnabled;
+    private Retention defaultRetention;

Review Comment:
   same



##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmBucketInfo.java:
##########
@@ -113,8 +113,13 @@ public final class OmBucketInfo extends WithObjectID implements Auditable, CopyO
    */
   private final ImmutableMap<String, String> tags;
 
+  private final boolean objectLockEnabled;
+  private final Retention defaultRetention;
+
   private OmBucketInfo(Builder b) {
     super(b);
+    this.objectLockEnabled = b.objectLockEnabled;
+    this.defaultRetention = b.defaultRetention;

Review Comment:
   nit: place new field at the end(line 143)



##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java:
##########
@@ -112,8 +112,15 @@ public final class OmKeyInfo extends WithParentObjectId
   // been modified.
   private final Long expectedDataGeneration;
 
+  private final Long retentionDate;
+  private final Retention retentionConfig;
+  private final Boolean legalHold;
+
   private OmKeyInfo(Builder b) {
     super(b);
+    this.retentionDate = b.retentionDate;
+    this.retentionConfig = b.retentionConfig;
+    this.legalHold = b.legalHold;

Review Comment:
   same



##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java:
##########
@@ -494,6 +513,9 @@ public String toString() {
    * Builder of OmKeyInfo.
    */
   public static class Builder extends WithParentObjectId.Builder<OmKeyInfo> {
+    private Long retentionDate;
+    private Retention retentionConfig;
+    private Boolean legalHold;

Review Comment:
   same



##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java:
##########
@@ -522,6 +544,9 @@ public Builder() {
     public Builder(OmKeyInfo obj) {
       super(obj);
       this.acls = AclListBuilder.of(obj.acls);
+      this.retentionDate = obj.retentionDate;
+      this.retentionConfig = obj.retentionConfig;
+      this.legalHold = obj.legalHold;

Review Comment:
   elsewhere



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16485. Add Object Lock protobuf schemas and retention models [ozone]

Posted by "shuan1026 (via GitHub)" <gi...@apache.org>.
shuan1026 commented on code in PR #11293:
URL: https://github.com/apache/ozone/pull/11293#discussion_r4247723147


##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -1241,6 +1269,9 @@ message KeyInfo {
   // This allows a key to be created an committed atomically if the original has not
   // been modified.
     optional uint64 expectedDataGeneration = 22;
+    optional uint64 retentionDate = 23;
+    optional RetentionConfig retentionConfig = 24;
+    optional bool legalHold = 25 [default = false];

Review Comment:
   +1 on including Object Lock fields in `computeKeyInfoCompareSignature`.
   
   OEP 16f4074 doesn't mention Snapshot Diff. @chungen0126, worth a one-liner in the design doc that retention / legalHold changes are part of the snapdiff signature (as implemented here), so later phases don't regress it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16485. Add Object Lock protobuf schemas and retention models [ozone]

Posted by "andyhuangdev (via GitHub)" <gi...@apache.org>.
andyhuangdev commented on code in PR #11293:
URL: https://github.com/apache/ozone/pull/11293#discussion_r4247970296


##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -1241,6 +1269,9 @@ message KeyInfo {
   // This allows a key to be created an committed atomically if the original has not
   // been modified.
     optional uint64 expectedDataGeneration = 22;
+    optional uint64 retentionDate = 23;
+    optional RetentionConfig retentionConfig = 24;
+    optional bool legalHold = 25 [default = false];

Review Comment:
   Thanks for pointing this out. Updated the protobuf schemas and retention models to align with OEP 16f4074, including the nested key-level Retention message, separate bucket retention mode, and UTC ISO-8601 retention dates. Snapshot Diff continues to include retention and legal hold changes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16485. Add Object Lock protobuf schemas and retention models [ozone]

Posted by "andyhuangdev (via GitHub)" <gi...@apache.org>.
andyhuangdev commented on code in PR #11293:
URL: https://github.com/apache/ozone/pull/11293#discussion_r4089260088


##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -1241,6 +1269,9 @@ message KeyInfo {
   // This allows a key to be created an committed atomically if the original has not
   // been modified.
     optional uint64 expectedDataGeneration = 22;
+    optional uint64 retentionDate = 23;
+    optional RetentionConfig retentionConfig = 24;
+    optional bool legalHold = 25 [default = false];

Review Comment:
   Thanks for catching this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16485. Add Object Lock protobuf schemas and retention models [ozone]

Posted by "andyhuangdev (via GitHub)" <gi...@apache.org>.
andyhuangdev commented on PR #11293:
URL: https://github.com/apache/ozone/pull/11293#issuecomment-5760486849

   cc @chungen0126 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16485. Add Object Lock protobuf schemas and retention models [ozone]

Posted by "chihsuan (via GitHub)" <gi...@apache.org>.
chihsuan commented on code in PR #11293:
URL: https://github.com/apache/ozone/pull/11293#discussion_r4348312939


##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -833,6 +833,41 @@ message BucketInfo {
     optional uint64 snapshotUsedBytes = 21;
     optional uint64 snapshotUsedNamespace = 22;
     repeated hadoop.hdds.KeyValue tags = 23;
+    optional bool objectLockEnabled = 24 [default = false];
+    optional RetentionConfig defaultRetention = 25;
+}
+
+message RetentionConfig {
+    required RetentionMode retentionMode = 1;
+    optional Rule rule = 2;
+    optional EventHold eventHold = 3;
+}
+
+message Retention {
+    required RetentionMode retentionMode = 1;
+    // UTC ISO-8601 timestamp.
+    optional string retentionDate = 2;

Review Comment:
   This follows an older revision of the design doc. Could we align it with the [latest one](https://github.com/apache/ozone/blob/c6aebcb118f296426f91da7d27eb6702538e3ff8/hadoop-hdds/docs/content/design/s3-object-lock.md#table-changes)? It renames these to `KeyRetention` / `BucketRetention` and stores `retentionDate` as `uint64` epoch millis.



##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/RetentionUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.helpers;
+
+import java.time.DateTimeException;
+import java.time.Instant;
+import java.time.ZoneOffset;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.Rule;
+
+/**
+ * Validation and UTC date calculations for retention rules.
+ */
+public final class RetentionUtil {
+  private RetentionUtil() {
+  }
+
+  public static void validateRule(Rule rule) {
+    if (rule == null || !rule.isInitialized() || rule.getDuration() <= 0) {

Review Comment:
   Should `validateRule` also enforce the upper bound from the design doc? It caps the event hold duration at 36,500 days or 100 years, but this only checks for a positive value.



##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmBucketInfo.java:
##########
@@ -802,7 +839,9 @@ public boolean equals(Object o) {
         Objects.equals(bekInfo, that.bekInfo) &&
         Objects.equals(owner, that.owner) &&
         Objects.equals(defaultReplicationConfig, that.defaultReplicationConfig) &&
-        Objects.equals(tags, that.tags);
+        Objects.equals(tags, that.tags) &&
+        objectLockEnabled == that.objectLockEnabled &&
+        Objects.equals(defaultRetention, that.defaultRetention);

Review Comment:
   Could we add a test where two buckets differ only in `objectLockEnabled` or `defaultRetention`? I found that removing these lines, or `rule` from `Retention.equals`, still keeps all tests green.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16485. Add Object Lock protobuf schemas and retention models [ozone]

Posted by "github-actions[bot] (via GitHub) via issues" <is...@ozone.apache.org>.
github-actions[bot] commented on PR #11293:
URL: https://github.com/apache/ozone/pull/11293#issuecomment-6839688067

   This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16485. Add Object Lock protobuf schemas and retention models [ozone]

Posted by "F64116045 (via GitHub)" <gi...@apache.org>.
F64116045 commented on code in PR #11293:
URL: https://github.com/apache/ozone/pull/11293#discussion_r4083708706


##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -1241,6 +1269,9 @@ message KeyInfo {
   // This allows a key to be created an committed atomically if the original has not
   // been modified.
     optional uint64 expectedDataGeneration = 22;
+    optional uint64 retentionDate = 23;
+    optional RetentionConfig retentionConfig = 24;
+    optional bool legalHold = 25 [default = false];

Review Comment:
   Thanks @andyhuangdev for this patch.
   Do Object Lock metadata changes need to be reported by Snapshot Diff? These fields are currently skipped by [computeKeyInfoCompareSignature()](https://github.com/apache/ozone/blob/a6e7a81699985e1b80d48aecd0c772edffcd148e/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/SnapshotDiffValueParser.java#L83-L133). 
   Not sure if we want to report it
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16485. Add Object Lock protobuf schemas and retention models [ozone]

Posted by "shuan1026 (via GitHub)" <gi...@apache.org>.
shuan1026 commented on code in PR #11293:
URL: https://github.com/apache/ozone/pull/11293#discussion_r4247732635


##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -1241,6 +1269,9 @@ message KeyInfo {
   // This allows a key to be created an committed atomically if the original has not
   // been modified.
     optional uint64 expectedDataGeneration = 22;
+    optional uint64 retentionDate = 23;
+    optional RetentionConfig retentionConfig = 24;
+    optional bool legalHold = 25 [default = false];

Review Comment:
   +1 on including Object Lock fields in `computeKeyInfoCompareSignature`.
   
   OEP 16f4074 doesn't mention Snapshot Diff. @chungen0126, worth a one-liner in the design doc that retention / legalHold changes are part of the snapdiff signature (as implemented here), so later phases don't regress it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16485. Add Object Lock protobuf schemas and retention models [ozone]

Posted by "shuan1026 (via GitHub)" <gi...@apache.org>.
shuan1026 commented on code in PR #11293:
URL: https://github.com/apache/ozone/pull/11293#discussion_r4247723147


##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -1241,6 +1269,9 @@ message KeyInfo {
   // This allows a key to be created an committed atomically if the original has not
   // been modified.
     optional uint64 expectedDataGeneration = 22;
+    optional uint64 retentionDate = 23;
+    optional RetentionConfig retentionConfig = 24;
+    optional bool legalHold = 25 [default = false];

Review Comment:
   +1 on including Object Lock fields in `computeKeyInfoCompareSignature`.
   
   OEP 16f4074 doesn't mention Snapshot Diff. @chungen0126, worth a one-liner in the design doc that retention / legalHold changes are part of the snapdiff signature (as implemented here), so later phases don't regress it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16485. Add Object Lock protobuf schemas and retention models [ozone]

Posted by "shuan1026 (via GitHub)" <gi...@apache.org>.
shuan1026 commented on code in PR #11293:
URL: https://github.com/apache/ozone/pull/11293#discussion_r4247707616


##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -1241,6 +1269,9 @@ message KeyInfo {
   // This allows a key to be created an committed atomically if the original has not
   // been modified.
     optional uint64 expectedDataGeneration = 22;
+    optional uint64 retentionDate = 23;
+    optional RetentionConfig retentionConfig = 24;
+    optional bool legalHold = 25 [default = false];

Review Comment:
   Against OEP 16f4074, key-level lock state is now a single nested message:
   
   ```protobuf
   optional Retention retention = 23;
   optional bool legalHold = 24 [default = false];
   
   message Retention {
     required RetentionMode retentionMode = 1;
     optional string retentionDate = 2;  // OEP; see type note below
     optional EventHold eventHold = 3;
   }
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org