You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "juncevich (via GitHub)" <gi...@apache.org> on 2025/01/08 21:19:52 UTC
Re: [PR] HDDS-11699. Remove unnecessary information about parts when downloading multipart files. [ozone]
juncevich commented on code in PR #7558:
URL: https://github.com/apache/ozone/pull/7558#discussion_r1907877334
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneClientMultipartUploadWithFSO.java:
##########
@@ -854,6 +856,103 @@ public void testListMultipartUpload() throws Exception {
assertEquals(0, expectedList.size());
}
+ @Test
+ void testGetAllPartsWhenZeroPartNumber() throws Exception {
+ String parentDir = "a/b/c/d/e/f/";
+ keyName = parentDir + "file-ABC";
+ OzoneVolume s3volume = store.getVolume("s3v");
+ s3volume.createBucket(bucketName);
+ OzoneBucket s3Bucket = s3volume.getBucket(bucketName);
+
+ Map<Integer, String> partsMap = new TreeMap<>();
+ String uploadID = initiateMultipartUpload(s3Bucket, keyName, RATIS,
+ ONE);
+ Pair<String, String> partNameAndETag1 = uploadPart(s3Bucket, keyName,
+ uploadID, 1, generateData(OzoneConsts.OM_MULTIPART_MIN_SIZE, (byte) 97));
+ partsMap.put(1, partNameAndETag1.getKey());
+
+ Pair<String, String> partNameAndETag2 = uploadPart(s3Bucket, keyName,
+ uploadID, 2, generateData(OzoneConsts.OM_MULTIPART_MIN_SIZE, (byte) 97));
+ partsMap.put(2, partNameAndETag2.getKey());
+
+ Pair<String, String> partNameAndETag3 = uploadPart(s3Bucket, keyName,
+ uploadID, 3, generateData(OzoneConsts.OM_MULTIPART_MIN_SIZE, (byte) 97));
+ partsMap.put(3, partNameAndETag3.getKey());
+
+ s3Bucket.completeMultipartUpload(keyName, uploadID, partsMap);
+
+ OzoneKeyDetails s3KeyDetailsWithAllParts = ozClient.getProxy()
+ .getS3KeyDetails(s3Bucket.getName(), keyName, 0);
+ List<OzoneKeyLocation> ozoneKeyLocations = s3KeyDetailsWithAllParts.getOzoneKeyLocations();
+ assertEquals(6, ozoneKeyLocations.size());
+ }
+
+ @Test
+ void testGetParticularPart() throws Exception {
+ String parentDir = "a/b/c/d/e/f/";
+ keyName = parentDir + "file-ABC";
+ OzoneVolume s3volume = store.getVolume("s3v");
+ s3volume.createBucket(bucketName);
+ OzoneBucket s3Bucket = s3volume.getBucket(bucketName);
+
+ Map<Integer, String> partsMap = new TreeMap<>();
+ String uploadID = initiateMultipartUpload(s3Bucket, keyName, RATIS,
+ ONE);
+ Pair<String, String> partNameAndETag1 = uploadPart(s3Bucket, keyName,
+ uploadID, 1, generateData(OzoneConsts.OM_MULTIPART_MIN_SIZE, (byte) 97));
+ partsMap.put(1, partNameAndETag1.getKey());
+
+ Pair<String, String> partNameAndETag2 = uploadPart(s3Bucket, keyName,
+ uploadID, 2, generateData(OzoneConsts.OM_MULTIPART_MIN_SIZE, (byte) 97));
+ partsMap.put(2, partNameAndETag2.getKey());
+
+ Pair<String, String> partNameAndETag3 = uploadPart(s3Bucket, keyName,
+ uploadID, 3, generateData(OzoneConsts.OM_MULTIPART_MIN_SIZE, (byte) 97));
+ partsMap.put(3, partNameAndETag3.getKey());
+
+ s3Bucket.completeMultipartUpload(keyName, uploadID, partsMap);
+
+ OzoneKeyDetails s3KeyDetailsOneParts = ozClient.getProxy().getS3KeyDetails(bucketName, keyName, 1);
+ assertEquals(2, s3KeyDetailsOneParts.getOzoneKeyLocations().size());
+
+ OzoneKeyDetails s3KeyDetailsTwoParts = ozClient.getProxy().getS3KeyDetails(bucketName, keyName, 2);
+ assertEquals(2, s3KeyDetailsTwoParts.getOzoneKeyLocations().size());
+
+ OzoneKeyDetails s3KeyDetailsThreeParts = ozClient.getProxy().getS3KeyDetails(bucketName, keyName, 3);
+ assertEquals(2, s3KeyDetailsThreeParts.getOzoneKeyLocations().size());
Review Comment:
Sorry for the long answer.
In block allocating time this method
org.apache.hadoop.ozone.om.request.key.OMAllocateBlockRequest#preExecute ->
org.apache.hadoop.ozone.om.request.key.OMKeyRequest#allocateBlock
has SCM block size = 4194304 bytes.
That's why the file part is divided into two parts.
Is it necessary to change this default value in the test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org