You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "tanvipenumudy (via GitHub)" <gi...@apache.org> on 2023/11/08 22:02:23 UTC
[PR] HDDS-9654. Support ranged GET request for a specified part [ozone]
tanvipenumudy opened a new pull request, #5565:
URL: https://github.com/apache/ozone/pull/5565
## What changes were proposed in this pull request?
Currently, we do not support a 'ranged' GET request for a specified part, which is useful for downloading just a part of a multipart key. Even when a part number has been specified, we still download the entire multipart key.
With this fix, we should be able to download just a part of a multipart key. The additional client logic introduced takes advantage of the existing `BlockID` to `partNumber` mapping as part of the [KeyLocation](https://github.com/apache/ozone/blob/master/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto#L1019-L1035) proto message for filtering out the [OmKeyLocationInfo](https://github.com/apache/ozone/blob/master/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyLocationInfo.java) list.
Reference: [Amazon S3 GetObject Documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#AmazonS3-GetObject-request-querystring-PartNumber)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9654
## How was this patch tested?
Added integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9654. Support ranged GET request for a specified part [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5565:
URL: https://github.com/apache/ozone/pull/5565#issuecomment-1805055823
@SaketaChalamchala @xBis7 @vtutrinov can you please take a look as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9654. Support ranged GET request for a specified part [ozone]
Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on PR #5565:
URL: https://github.com/apache/ozone/pull/5565#issuecomment-1805605963
LGTM
@tanvipenumudy thank you for adding the feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9654. Support ranged GET request for a specified part [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5565:
URL: https://github.com/apache/ozone/pull/5565#issuecomment-1817035592
Thanks @tanvipenumudy for the patch, @kerneltime, @vtutrinov for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9654. Support ranged GET request for a specified part [ozone]
Posted by "tanvipenumudy (via GitHub)" <gi...@apache.org>.
tanvipenumudy commented on code in PR #5565:
URL: https://github.com/apache/ozone/pull/5565#discussion_r1396734315
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1702,7 +1687,19 @@ public OzoneKeyDetails getS3KeyDetails(String bucketName, String keyName,
partNumber)
.collect(Collectors.toList());
keyInfo.updateLocationInfoList(filteredKeyLocationInfo, false);
- return getOzoneKeyDetails(keyInfo, true);
+ keyInfo.setDataSize(getTotalBytesRead(keyInfo));
+ return getOzoneKeyDetails(keyInfo);
+ }
+
+ private long getTotalBytesRead(OmKeyInfo keyInfo) throws IOException {
+ OzoneInputStream keyContent = getInputStreamWithRetryFunction(keyInfo);
+ byte[] buffer = new byte[4096];
+ int bytesRead;
+ long totalBytesRead = 0;
+ while ((bytesRead = keyContent.read(buffer)) != -1) {
+ totalBytesRead += bytesRead;
+ }
+ return totalBytesRead;
Review Comment:
That's right, I was not able to see any existing way in which the OmKeyInfo#dataSize could dynamically get updated based on the updated `OmKeyLocationInfo` list.
I am open to suggestions if there is a better way to handle this!
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1702,7 +1687,19 @@ public OzoneKeyDetails getS3KeyDetails(String bucketName, String keyName,
partNumber)
.collect(Collectors.toList());
keyInfo.updateLocationInfoList(filteredKeyLocationInfo, false);
- return getOzoneKeyDetails(keyInfo, true);
+ keyInfo.setDataSize(getTotalBytesRead(keyInfo));
+ return getOzoneKeyDetails(keyInfo);
+ }
+
+ private long getTotalBytesRead(OmKeyInfo keyInfo) throws IOException {
+ OzoneInputStream keyContent = getInputStreamWithRetryFunction(keyInfo);
+ byte[] buffer = new byte[4096];
+ int bytesRead;
+ long totalBytesRead = 0;
+ while ((bytesRead = keyContent.read(buffer)) != -1) {
+ totalBytesRead += bytesRead;
+ }
+ return totalBytesRead;
Review Comment:
That's right, I was not able to see any existing way in which the `OmKeyInfo#dataSize` could dynamically get updated based on the updated `OmKeyLocationInfo` list.
I am open to suggestions if there is a better way to handle this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9654. Support ranged GET request for a specified part [ozone]
Posted by "tanvipenumudy (via GitHub)" <gi...@apache.org>.
tanvipenumudy commented on code in PR #5565:
URL: https://github.com/apache/ozone/pull/5565#discussion_r1396746375
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1702,7 +1687,19 @@ public OzoneKeyDetails getS3KeyDetails(String bucketName, String keyName,
partNumber)
.collect(Collectors.toList());
keyInfo.updateLocationInfoList(filteredKeyLocationInfo, false);
- return getOzoneKeyDetails(keyInfo, true);
+ keyInfo.setDataSize(getTotalBytesRead(keyInfo));
+ return getOzoneKeyDetails(keyInfo);
+ }
+
+ private long getTotalBytesRead(OmKeyInfo keyInfo) throws IOException {
+ OzoneInputStream keyContent = getInputStreamWithRetryFunction(keyInfo);
+ byte[] buffer = new byte[4096];
+ int bytesRead;
+ long totalBytesRead = 0;
+ while ((bytesRead = keyContent.read(buffer)) != -1) {
+ totalBytesRead += bytesRead;
+ }
+ return totalBytesRead;
Review Comment:
As discussed offline with @kerneltime, I think we can have the block lengths summed up for a given `partNumber` instead - I will update the changes accordingly, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9654. Support ranged GET request for a specified part [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #5565:
URL: https://github.com/apache/ozone/pull/5565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9654. Support ranged GET request for a specified part [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #5565:
URL: https://github.com/apache/ozone/pull/5565#discussion_r1396005275
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1702,7 +1687,19 @@ public OzoneKeyDetails getS3KeyDetails(String bucketName, String keyName,
partNumber)
.collect(Collectors.toList());
keyInfo.updateLocationInfoList(filteredKeyLocationInfo, false);
- return getOzoneKeyDetails(keyInfo, true);
+ keyInfo.setDataSize(getTotalBytesRead(keyInfo));
+ return getOzoneKeyDetails(keyInfo);
+ }
+
+ private long getTotalBytesRead(OmKeyInfo keyInfo) throws IOException {
+ OzoneInputStream keyContent = getInputStreamWithRetryFunction(keyInfo);
+ byte[] buffer = new byte[4096];
+ int bytesRead;
+ long totalBytesRead = 0;
+ while ((bytesRead = keyContent.read(buffer)) != -1) {
+ totalBytesRead += bytesRead;
+ }
+ return totalBytesRead;
Review Comment:
Does this mean client reads data for the specified part just to calculate size for `ContentLength`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9654. Support ranged GET request for a specified part [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5565:
URL: https://github.com/apache/ozone/pull/5565#issuecomment-1807603762
Thank you @tanvipenumudy for this change and @vtutrinov for the review. @tanvipenumudy can you please extend the robot test to read a part and validate the size read?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org