You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "spacemonkd (via GitHub)" <gi...@apache.org> on 2026/04/09 18:47:45 UTC

[PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

spacemonkd opened a new pull request, #10062:
URL: https://github.com/apache/ozone/pull/10062

   ## What changes were proposed in this pull request?
   HDDS-14665. Add upgrade handling to multipart requests
   
   Please describe your PR in detail:
   * This PR adds a new layout version to define the split table behaviour for Multipart Uploads
   * It also adds new checks which fails early in case the finalization is not done and the writes try to use the new split table
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-14665
   
   ## How was this patch tested?
   Patch was tested using unit tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd merged PR #10062:
URL: https://github.com/apache/ozone/pull/10062


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3457488690


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3InitiateMultipartUploadRequest.java:
##########
@@ -72,6 +72,7 @@ public class S3InitiateMultipartUploadRequest extends OMKeyRequest {
 
   private static final Logger LOG =
       LoggerFactory.getLogger(S3InitiateMultipartUploadRequest.class);
+  private static final int LEGACY_MPU_SCHEMA_VERSION = 0;

Review Comment:
   Addressed in the latest commit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3633388419


##########
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmMultipartPartInfo.java:
##########
@@ -63,6 +63,7 @@ public void testGetFromProtoRejectsMissingRequiredFields() {
         () -> OmMultipartPartInfo.getFromProto(base.toBuilder().clearPartName().build()));
     assertThrows(IllegalArgumentException.class,
         () -> OmMultipartPartInfo.getFromProto(base.toBuilder().clearPartNumber().build()));
+    // Behaviour change: eTag is now a mandatory field for all clients.

Review Comment:
   Thanks @errose28 for pointing this out. Yeah I missed out on these comments in the tests, claude has been a bit eager with the notes.
   Addressed in the latest commit!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on PR #10062:
URL: https://github.com/apache/ozone/pull/10062#issuecomment-4658330468

   Thanks for the reviews @errose28, @kerneltime and @amaliujia.
   
   > Looks like nothing is setting schema version right now which makes it difficult to review this end to end. 
   
   I can add the implementation to this PR itself, but then the code changes on this would increase.
   I plan on adding the last PR to this chain after this one which will introduce the implementation and usage of the schemaVersion.
   Would that be okay? If not I can merge those changes as part of this PR @errose28 
   
   Could you take another look at the new changes? CI is green.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on PR #10062:
URL: https://github.com/apache/ozone/pull/10062#issuecomment-5032958866

   >Comment is fine for now, but lets also file a follow-up jira to change it.
   
   I have modified the tests to finalize it in the PR itself instead of a separate JIRA. Hopefully all looks good now.
   @errose28 could you take another look at the changes?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #10062:
URL: https://github.com/apache/ozone/pull/10062#issuecomment-4696583242

   Thanks for the update @spacemonkd. We should only be checking MLV to set the schema when the request is first initiated and before it is submitted to Ratis. All subsequent requests should then use the schema version in the proto metadata as the source of truth. This can be done in a pre-processor because it only needs to mutate the incoming proto's schema version based on the current MLV, then all future processing will obey that schema version.
   
   Finalization goes through Ratis so we don't need to worry about MLV diverging between OMs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3457492761


##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartRequest.java:
##########
@@ -302,6 +306,46 @@ protected OMRequest doPreExecuteCompleteMPU(
 
   }
 
+  /**
+   * Initiate an MPU and optionally rewrite the stored multipart metadata to a
+   * specific schema version.
+   *
+   * <p>The schema version rewrite lets tests emulate post-finalization MPU
+   * entries without needing the rest of the upgrade pipeline.</p>
+   */
+  protected String initiateMultipartUploadWithSchemaVersion(
+      String volumeName, String bucketName, String keyName,
+      byte schemaVersion) throws Exception {

Review Comment:
   Addressed in latest commit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3378211571


##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartKeyInfo.java:
##########
@@ -507,12 +514,21 @@ public MultipartKeyInfo getProto() {
     }
 
     builder.addAllAcls(OzoneAclUtil.toProtobuf(acls));
-    if (schemaVersion == 0) {
+    if (schemaVersion == LEGACY_SCHEMA_VERSION) {
       builder.addAllPartKeyInfoList(partKeyInfoMap);
     }
     return builder.build();
   }
 
+  private static byte validateAndConvertSchemaVersion(int schemaVersion) {
+    if (schemaVersion != LEGACY_SCHEMA_VERSION
+        && schemaVersion != SPLIT_PARTS_SCHEMA_VERSION) {
+      throw new IllegalArgumentException("Unsupported schemaVersion: "
+          + schemaVersion + ". Expected one of [0, 1].");
+    }
+    return (byte) schemaVersion;

Review Comment:
   Yes, I have reverted to `int`.
   Initially I kept as `byte` since we only support the values as 0 or 1 so we do not need to allocate memory for an integer, but I guess it is easier to read with it being int.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #10062:
URL: https://github.com/apache/ozone/pull/10062#issuecomment-4964931744

   I already merged https://github.com/apache/ozone/pull/10588 to unblock this patch. We can go ahead.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on PR #10062:
URL: https://github.com/apache/ozone/pull/10062#issuecomment-4630527162

   @kerneltime @ivandika3 it would be great if you could take a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on PR #10062:
URL: https://github.com/apache/ozone/pull/10062#issuecomment-4773555246

   @errose28  could you take a look as well?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3626558142


##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartPartInfo.java:
##########
@@ -70,9 +69,10 @@ private OmMultipartPartInfo(Builder b) {
     if (b.partNumber <= 0) {
       throw new IllegalArgumentException("partNumber is required and > 0");
     }
-    if (StringUtils.isBlank(b.eTag)) {
-      throw new IllegalArgumentException("eTag is required");
-    }
+    // eTag is optional: not all MPU clients supply an ETag at commit time
+    // (e.g. the Ozone native client), matching the legacy inline flow which
+    // never required it. It is stored when present and used for validation
+    // during CompleteMultipartUpload only when the client provides one.

Review Comment:
   My understanding is that for the old path etags were not enforced on the OM server side, and on the new path they were, and we want to make both paths consistent. The PR currently chose to make both match the old path with no OM server enforcement, but this makes it possible for a native client to create MPUs with no Etag which will appear as corrupted to S3 clients. Actually the OM server side enforcement for the new path was correct and that should be added in the old path as well.
   
   The original tests are wrong and easy to update since the etag just needs to be non-empty to pass the original validation. We can do this in a separate Jira, but it should land before this one to keep the master branch correct. I'm also ok with putting combining it in this PR though. I just don't want the enforcement removal merged into master. Probably something like this would suffice for the tests:
   
   ```diff
   diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotTests.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotTests.java
   index 44131d81c8..27fcd764f1 100644
   --- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotTests.java
   +++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotTests.java
   @@ -129,6 +129,7 @@
    import org.apache.hadoop.ozone.client.OzoneSnapshot;
    import org.apache.hadoop.ozone.client.OzoneSnapshotDiff;
    import org.apache.hadoop.ozone.client.OzoneVolume;
   +import org.apache.hadoop.ozone.client.io.KeyMetadataAware;
    import org.apache.hadoop.ozone.client.io.OzoneDataStreamOutput;
    import org.apache.hadoop.ozone.client.io.OzoneInputStream;
    import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
   @@ -3261,18 +3262,21 @@ public void testSnapshotDiffWithCreateMultipartKeys() throws Exception {
        try (OzoneOutputStream stream = bucket.createMultipartKey(
            regularPartsKey, regularPart.length, 1, regularMpuInfo.getUploadID())) {
          stream.write(regularPart);
   +      setPartETag(stream);
        }
    
        byte[] streamPart = "stream data".getBytes(UTF_8);
        try (OzoneDataStreamOutput streamOut = bucket.createMultipartStreamKey(
            streamPartsKey, streamPart.length, 1, streamMpuInfo.getUploadID())) {
          streamOut.write(streamPart);
   +      setPartETag(streamOut);
        }
    
        byte[] mixedPart = "mixed data".getBytes(UTF_8);
        try (OzoneOutputStream mixedStream = bucket.createMultipartKey(
            mixedPartsKey, mixedPart.length, 1, mixedMpuInfo.getUploadID())) {
          mixedStream.write(mixedPart);
   +      setPartETag(mixedStream);
        }
    
        assertEquals(1,
   @@ -3322,14 +3326,17 @@ public void testSnapshotDiffWithAbortMultipartUpload() throws Exception {
        try (OzoneOutputStream part1Stream = bucket.createMultipartKey(
            partialAbortKey, part1Data.length, 1, partialInfo.getUploadID())) {
          part1Stream.write(part1Data);
   +      setPartETag(part1Stream);
        }
        try (OzoneOutputStream part2Stream = bucket.createMultipartKey(
            partialAbortKey, part2Data.length, 2, partialInfo.getUploadID())) {
          part2Stream.write(part2Data);
   +      setPartETag(part2Stream);
        }
        try (OzoneDataStreamOutput part3Stream = bucket.createMultipartStreamKey(
            partialAbortKey, part3Data.length, 3, partialInfo.getUploadID())) {
          part3Stream.write(part3Data);
   +      setPartETag(part3Stream);
        }
    
        OzoneMultipartUploadPartListParts partsList = bucket.listParts(
   @@ -3348,10 +3355,12 @@ public void testSnapshotDiffWithAbortMultipartUpload() throws Exception {
        try (OzoneOutputStream stream = bucket.createMultipartKey(
            multiAbortKey1, part1Data.length, 1, multiInfo1.getUploadID())) {
          stream.write(part1Data);
   +      setPartETag(stream);
        }
        try (OzoneDataStreamOutput stream = bucket.createMultipartStreamKey(
            multiAbortKey2, part2Data.length, 1, multiInfo2.getUploadID())) {
          stream.write(part2Data);
   +      setPartETag(stream);
        }
    
        bucket.abortMultipartUpload(multiAbortKey1, multiInfo1.getUploadID());
   @@ -3398,10 +3407,12 @@ public void testSnapshotDiffWithCompleteInvisibleMPULifecycle() throws Exception
        try (OzoneOutputStream stream = bucket.createMultipartKey(
            mpuKey1, regularData1.length, 1, mpuInfo1.getUploadID())) {
          stream.write(regularData1);
   +      setPartETag(stream);
        }
        try (OzoneOutputStream stream = bucket.createMultipartKey(
            mpuKey1, regularData2.length, 2, mpuInfo1.getUploadID())) {
          stream.write(regularData2);
   +      setPartETag(stream);
        }
    
        byte[] streamData1 = "Stream multipart data 1".getBytes(UTF_8);
   @@ -3410,10 +3421,12 @@ public void testSnapshotDiffWithCompleteInvisibleMPULifecycle() throws Exception
        try (OzoneDataStreamOutput stream = bucket.createMultipartStreamKey(
            mpuKey2, streamData1.length, 1, mpuInfo2.getUploadID())) {
          stream.write(streamData1);
   +      setPartETag(stream);
        }
        try (OzoneDataStreamOutput stream = bucket.createMultipartStreamKey(
            mpuKey2, streamData2.length, 2, mpuInfo2.getUploadID())) {
          stream.write(streamData2);
   +      setPartETag(stream);
        }
    
    
   @@ -3423,10 +3436,12 @@ public void testSnapshotDiffWithCompleteInvisibleMPULifecycle() throws Exception
        try (OzoneOutputStream stream = bucket.createMultipartKey(
            mpuKey3, mixedRegular.length, 1, mpuInfo3.getUploadID())) {
          stream.write(mixedRegular);
   +      setPartETag(stream);
        }
        try (OzoneDataStreamOutput stream = bucket.createMultipartStreamKey(
            mpuKey3, mixedStream.length, 2, mpuInfo3.getUploadID())) {
          stream.write(mixedStream);
   +      setPartETag(stream);
        }
    
        assertEquals(2,
   @@ -3470,6 +3485,7 @@ private void completeSinglePartMPU(OzoneBucket bucket, String keyName, String da
        byte[] partData = createLargePartData(data, MIN_PART_SIZE);
        OzoneOutputStream partStream = bucket.createMultipartKey(keyName, partData.length, 1, uploadId);
        partStream.write(partData);
   +    setPartETag(partStream);
        partStream.close();
    
        OzoneMultipartUploadPartListParts partsList = bucket.listParts(keyName, uploadId, 0, 100);
   @@ -3491,6 +3507,7 @@ private void completeMultiplePartMPU(
          try (OzoneOutputStream partStream = bucket.createMultipartKey(
              keyName, partData.length, partNum, uploadId)) {
            partStream.write(partData);
   +        setPartETag(partStream);
          }
        }
    
   @@ -3514,12 +3531,14 @@ private void completeMixedPartMPU(
        try (OzoneOutputStream partStream = bucket.createMultipartKey(
            keyName, part1Data.length, 1, uploadId)) {
          partStream.write(part1Data);
   +      setPartETag(partStream);
        }
    
        byte[] part2Data = createLargePartData(streamData, MIN_PART_SIZE);
        try (OzoneDataStreamOutput partStream = bucket.createMultipartStreamKey(
            keyName, part2Data.length, 2, uploadId)) {
          partStream.write(part2Data);
   +      setPartETag(partStream);
        }
    
        OzoneMultipartUploadPartListParts partsList = bucket.listParts(keyName, uploadId, 0, 2);
   @@ -3547,6 +3566,7 @@ private void completeMPUWithReplication(
        try (OzoneOutputStream partStream = bucket.createMultipartKey(
            keyName, partData.length, 1, uploadId)) {
          partStream.write(partData);
   +      setPartETag(partStream);
        }
    
        OzoneMultipartUploadPartListParts partsList = bucket.listParts(keyName, uploadId, 0, 1);
   @@ -3566,6 +3586,7 @@ private void completeMPUWithMetadata(OzoneBucket bucket, String keyName,
        byte[] partData = createLargePartData("MPU with metadata and tags", MIN_PART_SIZE);
        OzoneOutputStream partStream = bucket.createMultipartKey(keyName, partData.length, 1, uploadId);
        partStream.write(partData);
   +    setPartETag(partStream);
        partStream.close();
    
        OzoneMultipartUploadPartListParts partsList = bucket.listParts(keyName, uploadId, 0, 100);
   @@ -3596,6 +3617,10 @@ private byte[] createLargePartData(String baseContent, int targetSize) {
        return result.getBytes(UTF_8);
      }
    
   +  private static void setPartETag(KeyMetadataAware stream) {
   +    stream.getMetadata().put(OzoneConsts.ETAG, UUID.randomUUID().toString());
   +  }
   +
      @Test
      public void testSnapshotDiffMPUCreateNewKey() throws Exception {
        String testVolumeName = "vol-create-new-" + counter.incrementAndGet();
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3375266258


##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartKeyInfo.java:
##########
@@ -87,6 +87,8 @@ public final class OmMultipartKeyInfo extends WithObjectID implements CopyObject
   // This stores the schema version of the multipart key.
   // 0 - Legacy Schema -> Uses the same table to store the multipart part info
   // 1 - New Schema -> Uses a separate table to store the multipart part info
+  private static final int LEGACY_SCHEMA_VERSION = 0;
+  private static final int SPLIT_PARTS_SCHEMA_VERSION = 1;
   private final byte schemaVersion;

Review Comment:
   Might as well make this an int if that is what we are using for the constants. I don't think we need to be concerned with this level of memory savings.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadAbortRequest.java:
##########
@@ -165,6 +165,17 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, Execut
             OMException.ResultCodes.NO_SUCH_MULTIPART_UPLOAD_ERROR);
       }
 
+      // This gate runs in the replicated apply path, so the check must remain
+      // deterministic across replicas for a given log index. MLV advances only
+      // via the Ratis-logged finalize-upgrade request.
+      if (!ozoneManager.getVersionManager()
+          .isAllowed(OMLayoutFeature.MPU_PARTS_TABLE_SPLIT)
+          && multipartKeyInfo.getSchemaVersion() != 0) {
+        throw new OMException("MPU parts-table split behavior is not allowed " +
+          "before cluster finalization.",
+          OMException.ResultCodes.NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION);
+      }

Review Comment:
   This looks like it should be a request pre-processor like [this](https://github.com/apache/ozone/blob/631050666dc77ff052261af388299d88fea338e6/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java#L393). We will be revamping how these pre and post processors are constructed as part of ZDU but we will handle that migration on our feature branch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3456555236


##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadAbortRequest.java:
##########
@@ -95,6 +99,95 @@ public void testValidateAndUpdateCache() throws Exception {
 
   }
 
+  @Test
+  public void testValidateAndUpdateCacheUsesSchemaVersionOneBeforeFinalization()
+      throws Exception {
+    String volumeName = UUID.randomUUID().toString();
+    String bucketName = UUID.randomUUID().toString();
+    String keyName = getKeyName();
+
+    OMRequestTestUtils.addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager, getBucketLayout());
+
+    createParentPath(volumeName, bucketName);
+
+    String multipartUploadID =
+        initiateMultipartUploadWithSchemaVersion(volumeName, bucketName,
+            keyName, (byte) 1);
+
+    String multipartKey = omMetadataManager.getMultipartKey(volumeName,
+        bucketName, keyName, multipartUploadID);
+    OmMultipartKeyInfo multipartKeyInfo = omMetadataManager
+        .getMultipartInfoTable().get(multipartKey);
+    assertNotNull(multipartKeyInfo);
+    assertEquals(1, multipartKeyInfo.getSchemaVersion());
+
+    OMRequest abortMPURequest =
+        doPreExecuteAbortMPU(volumeName, bucketName, keyName,
+            multipartUploadID);
+
+    S3MultipartUploadAbortRequest s3MultipartUploadAbortRequest =
+        getS3MultipartUploadAbortReq(abortMPURequest);
+
+    OMClientResponse omClientResponse =
+        s3MultipartUploadAbortRequest.validateAndUpdateCache(ozoneManager, 2L);
+
+    assertNotEquals(OzoneManagerProtocolProtos.Status
+        .NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION,
+        omClientResponse.getOMResponse().getStatus());
+    assertEquals(OzoneManagerProtocolProtos.Status.OK,
+        omClientResponse.getOMResponse().getStatus());

Review Comment:
   The status can only have one value, so the prior `assertNotEquals` call is redundant given the `assertEquals` following it. The status also jumps out visually more than the `Not` in the method name so removing the other assertion makes this easier when skimming.
   
   There's a similar pattern in other tests added as well.
   
   ```suggestion
    assertEquals(OzoneManagerProtocolProtos.Status.OK,
           omClientResponse.getOMResponse().getStatus());
   ```



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3InitiateMultipartUploadRequest.java:
##########
@@ -72,6 +72,7 @@ public class S3InitiateMultipartUploadRequest extends OMKeyRequest {
 
   private static final Logger LOG =
       LoggerFactory.getLogger(S3InitiateMultipartUploadRequest.class);
+  private static final int LEGACY_MPU_SCHEMA_VERSION = 0;

Review Comment:
   Can we define the schema versions in one place as public constants? Right now definitions are duplicated between this class and `OmMultipartKeyInfo`.



##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadCommitPartRequestWithFSO.java:
##########
@@ -71,6 +76,31 @@ protected String getKeyName() {
     return dirName + UUID.randomUUID().toString();
   }
 
+  @Test
+  public void testValidateAndUpdateCacheDirectoryNotFound() throws Exception {

Review Comment:
   Is this test related to the MPU version somehow?



##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartRequest.java:
##########
@@ -302,6 +306,46 @@ protected OMRequest doPreExecuteCompleteMPU(
 
   }
 
+  /**
+   * Initiate an MPU and optionally rewrite the stored multipart metadata to a
+   * specific schema version.
+   *
+   * <p>The schema version rewrite lets tests emulate post-finalization MPU
+   * entries without needing the rest of the upgrade pipeline.</p>
+   */
+  protected String initiateMultipartUploadWithSchemaVersion(
+      String volumeName, String bucketName, String keyName,
+      byte schemaVersion) throws Exception {

Review Comment:
   nit. `schemaVersion` is an int now.



##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadCommitPartRequest.java:
##########
@@ -151,6 +248,10 @@ public void testValidateAndUpdateCacheMultipartNotFound() throws Exception {
 
     // Add key to open key table.
     addKeyToOpenKeyTable(volumeName, bucketName, keyName, clientID);
+    String openKey = getOpenKey(volumeName, bucketName, keyName, clientID);

Review Comment:
   Why were these last parts after the new test added? They don't seem directly related to the feature.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3InitiateMultipartUploadRequest.java:
##########
@@ -284,6 +291,35 @@ protected void logResult(OzoneManager ozoneManager,
     }
   }
 
+  protected int resolveMultipartSchemaVersion(
+      MultipartInfoInitiateRequest multipartInfoInitiateRequest) {
+    if (!multipartInfoInitiateRequest.hasSchemaVersion()) {
+      return LEGACY_MPU_SCHEMA_VERSION;
+    }
+    return (int) multipartInfoInitiateRequest.getSchemaVersion();
+  }
+
+  @RequestFeatureValidator(
+      conditions = {
+          ValidationCondition.CLUSTER_NEEDS_FINALIZATION,
+          ValidationCondition.CLUSTER_HAS_MPU_PARTS_TABLE_SPLIT
+      },
+      processingPhase = RequestProcessingPhase.PRE_PROCESS,
+      requestType = Type.InitiateMultiPartUpload
+  )
+  public static OMRequest setSchemaVersionOnInitiateMultipartUpload(
+      OMRequest req, ValidationContext ctx) {
+    MultipartInfoInitiateRequest initiateRequest =
+        req.getInitiateMultiPartUploadRequest();
+
+    // Keep newly initiated MPUs on legacy schema until split parts-table
+    // write/read paths are fully implemented.
+    return req.toBuilder()
+        .setInitiateMultiPartUploadRequest(initiateRequest.toBuilder()
+            .setSchemaVersion(LEGACY_MPU_SCHEMA_VERSION))

Review Comment:
   I may have missed this in an earlier comment, but this change has no effect if we commit it with the new layout version but a no-op action attached to that. Say a release goes out with this PR as is. The new MPU version will get finalized, but nothing changes. In the next release we would then need to add a second MPU layout version to check because the first one was basically wasted. The HBase feature made this mistake while trying to develop on master, that's why there's a "deprecated" HBase layout feature and the actual one comes later.
   
   So if we merge this in its current state the master branch is unreleasable. We either need to hold off on this change and leave the new schema hardcoded to off until it is ready for release, or decide that work is complete and flip the switch to enable it for finalized clusters in this PR.



##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadAbortRequest.java:
##########
@@ -95,6 +99,95 @@ public void testValidateAndUpdateCache() throws Exception {
 
   }
 
+  @Test
+  public void testValidateAndUpdateCacheUsesSchemaVersionOneBeforeFinalization()

Review Comment:
   It's somewhat unintuitive how the parent class sets the version manager to pre-finalized by default and tests need to opt in to finalize it. If it's too much change to reverse that in this PR, let's at least add a comment about it on these tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3378205894


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadAbortRequest.java:
##########
@@ -165,6 +165,17 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, Execut
             OMException.ResultCodes.NO_SUCH_MULTIPART_UPLOAD_ERROR);
       }
 
+      // This gate runs in the replicated apply path, so the check must remain
+      // deterministic across replicas for a given log index. MLV advances only
+      // via the Ratis-logged finalize-upgrade request.
+      if (!ozoneManager.getVersionManager()
+          .isAllowed(OMLayoutFeature.MPU_PARTS_TABLE_SPLIT)
+          && multipartKeyInfo.getSchemaVersion() != 0) {
+        throw new OMException("MPU parts-table split behavior is not allowed " +
+          "before cluster finalization.",
+          OMException.ResultCodes.NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION);
+      }

Review Comment:
   This check depends on `multipartKeyInfo.getSchemaVersion()` which is fetched from OM metadata for the upload row.
   The validator context right now doesn't have this information exposed in PRE_PROCESS, so the pre-processor isn't implemented as you had provided in the example.
   
   I did change it to use the layout version stamped by the leader in preExecute, so that way it will be more akin to the behaviour of a pre-processor i.e. more deterministic



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3633202419


##########
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmMultipartPartInfo.java:
##########
@@ -63,6 +63,7 @@ public void testGetFromProtoRejectsMissingRequiredFields() {
         () -> OmMultipartPartInfo.getFromProto(base.toBuilder().clearPartName().build()));
     assertThrows(IllegalArgumentException.class,
         () -> OmMultipartPartInfo.getFromProto(base.toBuilder().clearPartNumber().build()));
+    // Behaviour change: eTag is now a mandatory field for all clients.

Review Comment:
   nit. Here an elsewhere: AI has a tendency to reference past changes where it doesn't make sense in the context of the code base. This is purely a test of present behavior which makes the comment misleading outside of the original agent session that made the change.
   ```suggestion
       // eTag is a mandatory field for all clients.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3378215959


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java:
##########
@@ -161,6 +161,33 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, Execut
             openKey + "entry is not found in the openKey table",
             KEY_NOT_FOUND);
       }
+
+      if (multipartKeyInfo == null) {
+        // This can occur when user started uploading part by the time commit
+        // of that part happens, in between the user might have requested
+        // abort multipart upload. If we just throw exception, then the data
+        // will not be garbage collected, so move this part to delete table
+        // and throw error.
+        throw new OMException("No such Multipart upload is with specified " +
+            "uploadId " + uploadID,
+            OMException.ResultCodes.NO_SUCH_MULTIPART_UPLOAD_ERROR);
+      }
+
+      // This gate runs in the replicated apply path, so the check must remain
+      // deterministic across replicas for a given log index. MLV advances only
+      // via the Ratis-logged finalize-upgrade request.
+      if (!ozoneManager.getVersionManager()
+          .isAllowed(OMLayoutFeature.MPU_PARTS_TABLE_SPLIT)
+          && multipartKeyInfo.getSchemaVersion() != 0) {
+        throw new OMException("MPU parts-table split behavior is not allowed " +
+            "before cluster finalization for commit part request.",
+            OMException.ResultCodes.NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION);
+      }
+      if (multipartKeyInfo.getSchemaVersion() == 1) {

Review Comment:
   Yes, I had missed the extra check. However I have modified this now.
   Please refer to [this](https://github.com/apache/ozone/pull/10062#discussion_r3378205894) for more context.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on PR #10062:
URL: https://github.com/apache/ozone/pull/10062#issuecomment-4663011187

   @errose28 I have added the schemaVersion wiring as well.
   Could you take another look?
   
   Also:
   ```
   if (splitPartsFeatureAllowed
             && multipartKeyInfo.getSchemaVersion() == 1) {
           throw new OMException("MPU parts-table split commit path is not " +
               "supported in this write flow.",
               OMException.ResultCodes.NOT_SUPPORTED_OPERATION);
         }
   ```
   
   added this block so that the current builds for Ozone doesn't give error as we do not have the implementation for the actual split writing of the table. I will remove this in the follow up PR to this which will introduce the wiring.
   
   @amaliujia could you please take a look as well?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3621390930


##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3InitiateMultipartUploadRequest.java:
##########
@@ -113,6 +117,158 @@ public void testValidateAndUpdateCache() throws Exception {
 
   }
 
+  @Test
+  public void testValidateAndUpdateCacheSetsSchemaVersionZeroBeforeFinalization()
+      throws Exception {
+    String volumeName = UUID.randomUUID().toString();
+    String bucketName = UUID.randomUUID().toString();
+    String keyName = UUID.randomUUID().toString();
+
+    OMRequestTestUtils.addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager, getBucketLayout());
+
+    OMRequest modifiedRequest = doPreExecuteInitiateMPU(volumeName,

Review Comment:
   Addressed in the latest commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #10062:
URL: https://github.com/apache/ozone/pull/10062#issuecomment-4598872770

   ## Review
   
   Thanks for landing the upgrade gate ahead of the parts-table-split write path — gating first is the right sequencing. The layout-feature wiring looks correct (no upgrade action / MLV constant needed for a behavior-only feature, and `TestOMVersionManager`'s contiguity check still passes), and the gate's pre-finalization determinism holds: MLV is advanced only by the Ratis-logged `OMFinalizeUpgradeRequest`, so `isAllowed` resolves identically across replicas at a given apply index.
   
   I think there's one blocker, and it's not in the gate itself — it's in the `CommitPart` null-check reorder that came along with it.
   
   ### Blocker — `CommitPart`: moving the `multipartKeyInfo == null` check above `getOmKeyInfo()` crashes the OM on the commit-after-abort path
   
   To read `multipartKeyInfo.getSchemaVersion()` safely, the `multipartKeyInfo == null -> NO_SUCH_MULTIPART_UPLOAD_ERROR` check was moved up to right after `getMultipartInfoTable().get()`, ahead of `omKeyInfo = getOmKeyInfo(...)`. But the `NO_SUCH` path is special: `S3MultipartUploadCommitPartResponse.checkAndUpdateDB()` overrides the flush to GC the orphaned open part and unconditionally dereferences `openPartKeyInfoToBeDeleted` (the `omKeyInfo` handed in by the request) — `S3MultipartUploadCommitPartResponse.java:88-97` (`.getUpdateID()`, `.getObjectID()`).
   
   - On `master`, `omKeyInfo` is loaded and null-checked (-> `KEY_NOT_FOUND`) **before** the `NO_SUCH` throw, so a `NO_SUCH` response always carried a non-null part to delete.
   - With this PR, `NO_SUCH` is thrown while `omKeyInfo` is still `null`; the `catch` passes that null into the response. Error responses are enqueued unconditionally (`RequestHandler.handleWriteRequest`), and `OzoneManagerDoubleBuffer` calls `checkAndUpdateDB` for them and `terminate()`s the OM on any `Throwable`. So the flush NPEs -> `ExitUtils.terminate` -> OM exits, and followers re-applying the same entry exit identically.
   
   This is exactly the scenario the GC branch exists for — a part commit arriving after the upload was aborted. `testValidateAndUpdateCacheMultipartNotFound` already sets up the precondition (random `uploadId` + `addKeyToOpenKeyTable`); it stays green only because it asserts the response status and never flushes the double buffer, so `checkAndUpdateDB` is never run in the unit test. It also affects legacy (schemaVersion 0) uploads, so it's independent of the new feature.
   
   Suggested fix: leave the `multipartKeyInfo == null` check at its original position (after `omKeyInfo` is resolved) and put the new gate immediately after it — the gate only needs `multipartKeyInfo` non-null, which the original ordering already guaranteed there. A regression test that actually flushes the double buffer (or a MiniOzoneCluster commit-after-abort case) would lock it down.
   
   ### Concerns
   
   - **The allowed (post-finalization) branch is never tested.** The base test harness mocks `OMLayoutVersionManager`, so `isAllowed(MPU_PARTS_TABLE_SPLIT)` is always `false` (and `MPU_PARTS_TABLE_SPLIT` appears in no test). The predicate `!isAllowed && schemaVersion != 0` is only ever driven from the left operand — a future inversion (dropped `!`, wrong feature) would still pass CI. Please add an `isAllowed -> true` case per request (Abort can assert full success; for Commit/Complete at least `assertNotEquals(NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION, ...)`).
   - **Unrelated change bundled in.** The `existingKeyInfo` swap to `getOmKeyInfoFromKeyTable(...)` in `S3MultipartUploadCompleteRequest` is behaviorally inert today (base impl equals the old `getKeyTable(...).get(dbOzoneKey)`; the FSO override only adds `setKeyName`, which `validateAtomicRewrite`/`validateIfMatchETag` don't read) and is unrelated to finalization — and line 333 still uses the raw lookup for the same key. Suggest dropping it here or splitting into its own JIRA.
   - **In-apply `isAllowed` read departs from the established pattern.** Every other layout gate — including the EC `CLUSTER_NEEDS_FINALIZATION` validators in these same two files — runs as a leader-only PRE_PROCESS `@RequestFeatureValidator`. This is the first to read `isAllowed` inside the replicated `validateAndUpdateCache` path, which the method's Javadoc cautions against. It's safe today (MLV advances synchronously via the logged finalize), but that relies on the finalization executor staying synchronous (it's pluggable). Worth considering gating on the request-stamped `LayoutVersion` from `preExecute`, or at least a comment documenting the determinism assumption.
   - **Dropped FSO coverage.** Rewriting `testValidateAndUpdateCacheKeyNotFound` to initiate a real MPU removes the FSO `DIRECTORY_NOT_FOUND` branch the old assertion covered; that path is now unasserted in the commit-part suite. Could be restored as a small separate test.
   
   ### Probably for the write-side PR (flagging so they aren't silent regressions later)
   
   Not reachable until a sibling PR writes `schemaVersion != 0`, so not blockers here:
   
   - `KeyManagerImpl.listParts` reads the inline `getPartKeyInfoMap()` (empty for split entries) -> would return 200 with zero parts.
   - The background `S3ExpiredMultipartUploadsAbortRequest` isn't gated and derives quota from the empty inline map -> on a downgraded cluster it would delete the row releasing 0 quota while user-facing Abort is gated off.
   - `NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION` has no case in `S3ErrorTable.translateResultCode` -> falls to `default` -> HTTP 500 to S3 clients.
   
   ### Nits
   
   - Complete gate message reads "...for commit part request." — should be the complete path.
   - Double blank line after the gate block in `S3MultipartUploadAbortRequest`.
   - `schemaVersion` is `uint32` on the wire but narrowed to `byte` on load; gate tests `!= 0` while the model special-cases `== 1`. Not reachable today, but a multiple-of-256 value would truncate to 0 (fail-open) — worth validating to {0,1} at the boundary.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on PR #10062:
URL: https://github.com/apache/ozone/pull/10062#issuecomment-5053975462

   Thanks @errose28, @ivandika3 , @amaliujia for the reviews.
   Merging this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3617679947


##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3InitiateMultipartUploadRequest.java:
##########
@@ -113,6 +117,158 @@ public void testValidateAndUpdateCache() throws Exception {
 
   }
 
+  @Test
+  public void testValidateAndUpdateCacheSetsSchemaVersionZeroBeforeFinalization()
+      throws Exception {
+    String volumeName = UUID.randomUUID().toString();
+    String bucketName = UUID.randomUUID().toString();
+    String keyName = UUID.randomUUID().toString();
+
+    OMRequestTestUtils.addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager, getBucketLayout());
+
+    OMRequest modifiedRequest = doPreExecuteInitiateMPU(volumeName,

Review Comment:
   We should also check the schema version here to make sure it is being set in `preExecute` and not apply. Along those lines the title of this test is somewhat misleading as the value is set in `preExecute` and `validateAndUpdateCache` only forwards it. Also true for most of the following request unit tests.



##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3InitiateMultipartUploadRequest.java:
##########
@@ -113,6 +117,158 @@ public void testValidateAndUpdateCache() throws Exception {
 
   }
 
+  @Test
+  public void testValidateAndUpdateCacheSetsSchemaVersionZeroBeforeFinalization()

Review Comment:
   Can we split these four tests into two pairs parameterized by finalized (true/false) and the corresponding expected schema version? This looks doable for most of the new request unit tests added.



##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartPartInfo.java:
##########
@@ -70,9 +69,10 @@ private OmMultipartPartInfo(Builder b) {
     if (b.partNumber <= 0) {
       throw new IllegalArgumentException("partNumber is required and > 0");
     }
-    if (StringUtils.isBlank(b.eTag)) {
-      throw new IllegalArgumentException("eTag is required");
-    }
+    // eTag is optional: not all MPU clients supply an ETag at commit time
+    // (e.g. the Ozone native client), matching the legacy inline flow which
+    // never required it. It is stored when present and used for validation
+    // during CompleteMultipartUpload only when the client provides one.

Review Comment:
   Why is this making etags optional? I see we are keeping them required at the gateway to be consistent with AWS but I don't understand why this change is necessary in general or in the context of this PR.
   
   > It is stored when present and used for validation
       // during CompleteMultipartUpload only when the client provides one.
   
   It should also be returned when the parts for an in-progress upload are listed by the client.



##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadAbortRequest.java:
##########
@@ -95,6 +99,95 @@ public void testValidateAndUpdateCache() throws Exception {
 
   }
 
+  @Test
+  public void testValidateAndUpdateCacheUsesSchemaVersionOneBeforeFinalization()

Review Comment:
   >     // Tests must explicitly bump metadata layout version to simulate finalized OM.
   
   Comment is fine for now, but lets also file a follow-up jira to change it.



##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadCommitPartRequest.java:
##########
@@ -139,6 +140,97 @@ public void testValidateAndUpdateCacheSuccess() throws Exception {
         .get(partKey));
   }
 
+  @Test
+  public void testValidateAndUpdateCacheUsesSchemaVersionOneBeforeFinalization()
+      throws Exception {
+    String volumeName = UUID.randomUUID().toString();
+    String bucketName = UUID.randomUUID().toString();
+    String keyName = getKeyName();
+
+    // The base test fixture is pre-finalized by default.
+    OMRequestTestUtils.addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager, getBucketLayout());
+
+    createParentPath(volumeName, bucketName);
+
+    String multipartUploadID =
+        initiateMultipartUploadWithSchemaVersion(volumeName, bucketName,
+            keyName, 1);

Review Comment:
   nit. Let's use the the constant for schema versions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3457492234


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3InitiateMultipartUploadRequest.java:
##########
@@ -284,6 +291,35 @@ protected void logResult(OzoneManager ozoneManager,
     }
   }
 
+  protected int resolveMultipartSchemaVersion(
+      MultipartInfoInitiateRequest multipartInfoInitiateRequest) {
+    if (!multipartInfoInitiateRequest.hasSchemaVersion()) {
+      return LEGACY_MPU_SCHEMA_VERSION;
+    }
+    return (int) multipartInfoInitiateRequest.getSchemaVersion();
+  }
+
+  @RequestFeatureValidator(
+      conditions = {
+          ValidationCondition.CLUSTER_NEEDS_FINALIZATION,
+          ValidationCondition.CLUSTER_HAS_MPU_PARTS_TABLE_SPLIT
+      },
+      processingPhase = RequestProcessingPhase.PRE_PROCESS,
+      requestType = Type.InitiateMultiPartUpload
+  )
+  public static OMRequest setSchemaVersionOnInitiateMultipartUpload(
+      OMRequest req, ValidationContext ctx) {
+    MultipartInfoInitiateRequest initiateRequest =
+        req.getInitiateMultiPartUploadRequest();
+
+    // Keep newly initiated MPUs on legacy schema until split parts-table
+    // write/read paths are fully implemented.
+    return req.toBuilder()
+        .setInitiateMultiPartUploadRequest(initiateRequest.toBuilder()
+            .setSchemaVersion(LEGACY_MPU_SCHEMA_VERSION))

Review Comment:
   Hmm, I raised https://github.com/apache/ozone/pull/10588 to implement the split table writes.
   Maybe we can merge this PR after that to avoid this situation?
   
   Could you take a look at that PR then, when you get the time?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3457496363


##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadAbortRequest.java:
##########
@@ -95,6 +99,95 @@ public void testValidateAndUpdateCache() throws Exception {
 
   }
 
+  @Test
+  public void testValidateAndUpdateCacheUsesSchemaVersionOneBeforeFinalization()
+      throws Exception {
+    String volumeName = UUID.randomUUID().toString();
+    String bucketName = UUID.randomUUID().toString();
+    String keyName = getKeyName();
+
+    OMRequestTestUtils.addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager, getBucketLayout());
+
+    createParentPath(volumeName, bucketName);
+
+    String multipartUploadID =
+        initiateMultipartUploadWithSchemaVersion(volumeName, bucketName,
+            keyName, (byte) 1);
+
+    String multipartKey = omMetadataManager.getMultipartKey(volumeName,
+        bucketName, keyName, multipartUploadID);
+    OmMultipartKeyInfo multipartKeyInfo = omMetadataManager
+        .getMultipartInfoTable().get(multipartKey);
+    assertNotNull(multipartKeyInfo);
+    assertEquals(1, multipartKeyInfo.getSchemaVersion());
+
+    OMRequest abortMPURequest =
+        doPreExecuteAbortMPU(volumeName, bucketName, keyName,
+            multipartUploadID);
+
+    S3MultipartUploadAbortRequest s3MultipartUploadAbortRequest =
+        getS3MultipartUploadAbortReq(abortMPURequest);
+
+    OMClientResponse omClientResponse =
+        s3MultipartUploadAbortRequest.validateAndUpdateCache(ozoneManager, 2L);
+
+    assertNotEquals(OzoneManagerProtocolProtos.Status
+        .NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION,
+        omClientResponse.getOMResponse().getStatus());
+    assertEquals(OzoneManagerProtocolProtos.Status.OK,
+        omClientResponse.getOMResponse().getStatus());

Review Comment:
   Addressed in latest commit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3377815149


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java:
##########
@@ -161,6 +161,33 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, Execut
             openKey + "entry is not found in the openKey table",
             KEY_NOT_FOUND);
       }
+
+      if (multipartKeyInfo == null) {
+        // This can occur when user started uploading part by the time commit
+        // of that part happens, in between the user might have requested
+        // abort multipart upload. If we just throw exception, then the data
+        // will not be garbage collected, so move this part to delete table
+        // and throw error.
+        throw new OMException("No such Multipart upload is with specified " +
+            "uploadId " + uploadID,
+            OMException.ResultCodes.NO_SUCH_MULTIPART_UPLOAD_ERROR);
+      }
+
+      // This gate runs in the replicated apply path, so the check must remain
+      // deterministic across replicas for a given log index. MLV advances only
+      // via the Ratis-logged finalize-upgrade request.
+      if (!ozoneManager.getVersionManager()
+          .isAllowed(OMLayoutFeature.MPU_PARTS_TABLE_SPLIT)
+          && multipartKeyInfo.getSchemaVersion() != 0) {
+        throw new OMException("MPU parts-table split behavior is not allowed " +
+            "before cluster finalization for commit part request.",
+            OMException.ResultCodes.NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION);
+      }
+      if (multipartKeyInfo.getSchemaVersion() == 1) {

Review Comment:
   Trying to learn the context:
   
   Do we need this to be 
   ```
   if (ozoneManager.getVersionManager()
   .isAllowed(OMLayoutFeature.MPU_PARTS_TABLE_SPLIT) && 
   multipartKeyInfo.getSchemaVersion() == 1)
   ```
   ?



##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartKeyInfo.java:
##########
@@ -507,12 +514,21 @@ public MultipartKeyInfo getProto() {
     }
 
     builder.addAllAcls(OzoneAclUtil.toProtobuf(acls));
-    if (schemaVersion == 0) {
+    if (schemaVersion == LEGACY_SCHEMA_VERSION) {
       builder.addAllPartKeyInfoList(partKeyInfoMap);
     }
     return builder.build();
   }
 
+  private static byte validateAndConvertSchemaVersion(int schemaVersion) {
+    if (schemaVersion != LEGACY_SCHEMA_VERSION
+        && schemaVersion != SPLIT_PARTS_SCHEMA_VERSION) {
+      throw new IllegalArgumentException("Unsupported schemaVersion: "
+          + schemaVersion + ". Expected one of [0, 1].");
+    }
+    return (byte) schemaVersion;

Review Comment:
   Any reason why we still keep `SchemaVersion` as a byte in JVM? In its proto definition schemaVersion is `optional uint32 schemaVersion = 10;`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on PR #10062:
URL: https://github.com/apache/ozone/pull/10062#issuecomment-4701429117

   @errose28 are the latest changes now in line with what you meant?
   I changed it to have a new check for MPU_SPLIT_TABLE allowed.
   Also note that in the new pre-processor check `setSchemaVersionOnInitiateMultipartUpload` runs on both before and after finalize.
   
   This constant `0` will later on be switched based on the condition whether split table is allowed or not - once we have the implementation ready in the next PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on PR #10062:
URL: https://github.com/apache/ozone/pull/10062#issuecomment-4985216618

   @errose28  could you please take another look at this change?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3386871772


##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartKeyInfo.java:
##########
@@ -507,12 +514,21 @@ public MultipartKeyInfo getProto() {
     }
 
     builder.addAllAcls(OzoneAclUtil.toProtobuf(acls));
-    if (schemaVersion == 0) {
+    if (schemaVersion == LEGACY_SCHEMA_VERSION) {
       builder.addAllPartKeyInfoList(partKeyInfoMap);
     }
     return builder.build();
   }
 
+  private static byte validateAndConvertSchemaVersion(int schemaVersion) {
+    if (schemaVersion != LEGACY_SCHEMA_VERSION
+        && schemaVersion != SPLIT_PARTS_SCHEMA_VERSION) {
+      throw new IllegalArgumentException("Unsupported schemaVersion: "
+          + schemaVersion + ". Expected one of [0, 1].");
+    }
+    return (byte) schemaVersion;

Review Comment:
   :-) for only 0 or 1 we can go for one bit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3378206851


##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartKeyInfo.java:
##########
@@ -87,6 +87,8 @@ public final class OmMultipartKeyInfo extends WithObjectID implements CopyObject
   // This stores the schema version of the multipart key.
   // 0 - Legacy Schema -> Uses the same table to store the multipart part info
   // 1 - New Schema -> Uses a separate table to store the multipart part info
+  private static final int LEGACY_SCHEMA_VERSION = 0;
+  private static final int SPLIT_PARTS_SCHEMA_VERSION = 1;
   private final byte schemaVersion;

Review Comment:
   Switched to int



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #10062:
URL: https://github.com/apache/ozone/pull/10062#issuecomment-4661638445

   My understanding is that schema version to use will be assigned by the leader when the request is received based on whether or not the feature is finalized. Since finalization goes through ratis we would not need the request blocking parts of this change. The request processing would just follow whatever schema version it is given.
   
   I do think it would be helpful to have this whole flow in one PR, even if it increases the size.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3457493820


##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadAbortRequest.java:
##########
@@ -95,6 +99,95 @@ public void testValidateAndUpdateCache() throws Exception {
 
   }
 
+  @Test
+  public void testValidateAndUpdateCacheUsesSchemaVersionOneBeforeFinalization()

Review Comment:
   Addressed in latest commit



##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadCommitPartRequest.java:
##########
@@ -151,6 +248,10 @@ public void testValidateAndUpdateCacheMultipartNotFound() throws Exception {
 
     // Add key to open key table.
     addKeyToOpenKeyTable(volumeName, bucketName, keyName, clientID);
+    String openKey = getOpenKey(volumeName, bucketName, keyName, clientID);

Review Comment:
   Addressed in latest commit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3621289797


##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartPartInfo.java:
##########
@@ -70,9 +69,10 @@ private OmMultipartPartInfo(Builder b) {
     if (b.partNumber <= 0) {
       throw new IllegalArgumentException("partNumber is required and > 0");
     }
-    if (StringUtils.isBlank(b.eTag)) {
-      throw new IllegalArgumentException("eTag is required");
-    }
+    // eTag is optional: not all MPU clients supply an ETag at commit time
+    // (e.g. the Ozone native client), matching the legacy inline flow which
+    // never required it. It is stored when present and used for validation
+    // during CompleteMultipartUpload only when the client provides one.

Review Comment:
   >Why is this making etags optional? I see we are keeping them required at the gateway to be consistent with AWS but I don't understand why this change is necessary in general or in the context of this PR.
   
   So basically there are some tests under snapshot feature where the Ozone client also tries multipart upload but without the ETags present.
   Hence the original behaviour was that etags is optional, S3gateway should enforce the mandatory etag, but OM doesn't need to enforce.
   
   So we can either:
   - Change the snapshot tests to remove those cases where null etag is passed (probably raise a follow up JIRA and make ETag mandatory?)
   - Or go ahead with what we did (i.e. make Etags optional) to avoid regression.
   
   > It should also be returned when the parts for an in-progress upload are listed by the client.
   
   This is already being done. `listParts` returns `partInfo.getETag();` for the split schema (ref [here](https://github.com/apache/ozone/blob/917e4b7706241317d8dbf7199f10be0cd43fdd7f/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java#L1204)) and also the legacy path returns it with the null fallback.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on code in PR #10062:
URL: https://github.com/apache/ozone/pull/10062#discussion_r3621389441


##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadCommitPartRequest.java:
##########
@@ -139,6 +140,97 @@ public void testValidateAndUpdateCacheSuccess() throws Exception {
         .get(partKey));
   }
 
+  @Test
+  public void testValidateAndUpdateCacheUsesSchemaVersionOneBeforeFinalization()
+      throws Exception {
+    String volumeName = UUID.randomUUID().toString();
+    String bucketName = UUID.randomUUID().toString();
+    String keyName = getKeyName();
+
+    // The base test fixture is pre-finalized by default.
+    OMRequestTestUtils.addVolumeAndBucketToDB(volumeName, bucketName,
+        omMetadataManager, getBucketLayout());
+
+    createParentPath(volumeName, bucketName);
+
+    String multipartUploadID =
+        initiateMultipartUploadWithSchemaVersion(volumeName, bucketName,
+            keyName, 1);

Review Comment:
   Addressed in the latest commit.



##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3InitiateMultipartUploadRequest.java:
##########
@@ -113,6 +117,158 @@ public void testValidateAndUpdateCache() throws Exception {
 
   }
 
+  @Test
+  public void testValidateAndUpdateCacheSetsSchemaVersionZeroBeforeFinalization()

Review Comment:
   Addressed in the latest commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14665. Add upgrade handling to multipart requests [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #10062:
URL: https://github.com/apache/ozone/pull/10062#issuecomment-4916662749

   @jojochuang This PR is currently blocked. See [here](https://github.com/apache/ozone/pull/10062#discussion_r3457492234).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org