You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/08/26 12:18:20 UTC
[GitHub] [hadoop-ozone] maobaolong opened a new pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
maobaolong opened a new pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354
## What changes were proposed in this pull request?
Skip the inputstream while offset larger than zero in s3g.
## What is the link to the Apache JIRA
HDDS-4151
## How was this patch tested?
Download a big file like 10GB, and get the cost time, compare it than without this PR, this PR can get good performance back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#issuecomment-681273117
@bharatviswa504 Thanks for your review, I reply your comments inline. In fact the performance improvement comes from https://github.com/apache/hadoop-ozone/pull/843 , I just do some refactor and do some improvements by #1212 , and unfortunately, I forget to call seek or skip before the `IOUtils.copyLarge`, that's my bad, and I corrected it by this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#discussion_r477698606
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -277,7 +278,8 @@ public Response get(
try (S3WrapperInputStream s3WrapperInputStream =
new S3WrapperInputStream(
key.getInputStream())) {
- IOUtils.copyLarge(s3WrapperInputStream, dest, startOffset,
+ s3WrapperInputStream.seek(startOffset);
+ IOUtils.copyLarge(s3WrapperInputStream, dest, 0,
Review comment:
I understand that we do seek, because we don't do actual read just change the position at client. but using IOUtils.copyLarge we read and skip.
Not sure if my understanding is right? If not, can you explain if I am missing anything here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#issuecomment-683193714
@bharatviswa504 Thanks for review and merge it for me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong closed pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
maobaolong closed pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#issuecomment-682270416
@bharatviswa504 Thanks for your review and suggestion, I push a new commit to address your comment, PTAL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
maobaolong commented on a change in pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#discussion_r477884943
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -277,7 +278,8 @@ public Response get(
try (S3WrapperInputStream s3WrapperInputStream =
new S3WrapperInputStream(
key.getInputStream())) {
- IOUtils.copyLarge(s3WrapperInputStream, dest, startOffset,
+ s3WrapperInputStream.seek(startOffset);
+ IOUtils.copyLarge(s3WrapperInputStream, dest, 0,
Review comment:
The copyLarge related code is here
```java
public static long copyLarge(final InputStream input, final OutputStream output,
final long inputOffset, final long length, final byte[] buffer) throws IOException {
if (inputOffset > 0) {
skipFully(input, inputOffset);
}
if (length == 0) {
return 0;
}
final int bufferLength = buffer.length;
int bytesToRead = bufferLength;
if (length > 0 && length < bufferLength) {
bytesToRead = (int) length;
}
int read;
long totalRead = 0;
while (bytesToRead > 0 && EOF != (read = input.read(buffer, 0, bytesToRead))) {
output.write(buffer, 0, read);
totalRead += read;
if (length > 0) { // only adjust length if not reading to the end
// Note the cast must work because buffer.length is an integer
bytesToRead = (int) Math.min(length - totalRead, bufferLength);
}
}
return totalRead;
}
public static void skipFully(final InputStream input, final long toSkip) throws IOException {
if (toSkip < 0) {
throw new IllegalArgumentException("Bytes to skip must not be negative: " + toSkip);
}
final long skipped = skip(input, toSkip);
if (skipped != toSkip) {
throw new EOFException("Bytes to skip: " + toSkip + " actual: " + skipped);
}
}
public static long skip(final InputStream input, final long toSkip) throws IOException {
if (toSkip < 0) {
throw new IllegalArgumentException("Skip count must be non-negative, actual: " + toSkip);
}
/*
* N.B. no need to synchronize this because: - we don't care if the buffer is created multiple times (the data
* is ignored) - we always use the same size buffer, so if it it is recreated it will still be OK (if the buffer
* size were variable, we would need to synch. to ensure some other thread did not create a smaller one)
*/
if (SKIP_BYTE_BUFFER == null) {
SKIP_BYTE_BUFFER = new byte[SKIP_BUFFER_SIZE];
}
long remain = toSkip;
while (remain > 0) {
// See https://issues.apache.org/jira/browse/IO-203 for why we use read() rather than delegating to skip()
final long n = input.read(SKIP_BYTE_BUFFER, 0, (int) Math.min(remain, SKIP_BUFFER_SIZE));
if (n < 0) { // EOF
break;
}
remain -= n;
}
return toSkip - remain;
}
```
From the above code of IOUtils.java, we can know that the static method skip really read the inputstream, it is unnecessary and impact the performance, instead, using seek can set the position to the right point, Let's imaging that there are a 10GB file, when we want to read the if for offset 1GB and length 1GB, it have to read the first 1GB contents from datanode and drop it, now, we can seek to the position 1GB, and read the inputstream directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
maobaolong commented on a change in pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#discussion_r477891301
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -557,8 +559,14 @@ private Response createMultipartKey(String bucket, String key, long length,
if (range != null) {
RangeHeader rangeHeader =
RangeHeaderParserUtil.parseRangeHeader(range, 0);
- IOUtils.copyLarge(sourceObject, ozoneOutputStream,
- rangeHeader.getStartOffset(),
+ final long skipped =
+ sourceObject.skip(rangeHeader.getStartOffset());
Review comment:
@bharatviswa504 use skip here due to we are not sure the `sourceObject` is a `seekable` inputstream, so I have to use `skip` method, but luckily, I have override the `skip` method in `KeyInputStream.java`, it look like
```java
public long skip(long n) throws IOException {
if (n <= 0) {
return 0;
}
long toSkip = Math.min(n, length - getPos());
seek(getPos() + toSkip);
return toSkip;
}
```
So, for KeyInputStream which is an implementation of InputStream, which implements the skip method to a seek mode, it can bring the better performance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#discussion_r478744360
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -557,8 +559,14 @@ private Response createMultipartKey(String bucket, String key, long length,
if (range != null) {
RangeHeader rangeHeader =
RangeHeaderParserUtil.parseRangeHeader(range, 0);
- IOUtils.copyLarge(sourceObject, ozoneOutputStream,
- rangeHeader.getStartOffset(),
+ final long skipped =
+ sourceObject.skip(rangeHeader.getStartOffset());
Review comment:
But we use OzoneInputStream here, which has not implemented the skip(). So we use from BaseClass InputStream, which does not take advantage of KeyInputStream seek implementation.
public long skip(long n) throws IOException {
long remaining = n;
int nr;
if (n <= 0) {
return 0;
}
int size = (int)Math.min(MAX_SKIP_BUFFER_SIZE, remaining);
byte[] skipBuffer = new byte[size];
while (remaining > 0) {
nr = read(skipBuffer, 0, (int)Math.min(size, remaining));
if (nr < 0) {
break;
}
remaining -= nr;
}
return n - remaining;
}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#issuecomment-682303622
@maobaolong , s3WrapperInputStream.copyLarge implements the skip and copy. You can just leverage this function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#issuecomment-682238379
@maobaolong
Thanks for the detailed explanation.
I have one comment, rest LGTM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#issuecomment-680847535
@elek Please take a look at this PR, it can solve the performance issue which involved by https://github.com/apache/hadoop-ozone/pull/1212
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi edited a comment on pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
ChenSammi edited a comment on pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#issuecomment-682303622
@maobaolong , s3WrapperInputStream.copyLarge has implemented the skip and copy. You can just leverage this function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#issuecomment-682370102
I have discussed with @ChenSammi Offline, `s3WrapperInputStream.copyLarge` is a duplicated code which copy from `IOUtils.copyLarge`, It has been improved by #1212.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
maobaolong commented on a change in pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#discussion_r478778013
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -557,8 +559,14 @@ private Response createMultipartKey(String bucket, String key, long length,
if (range != null) {
RangeHeader rangeHeader =
RangeHeaderParserUtil.parseRangeHeader(range, 0);
- IOUtils.copyLarge(sourceObject, ozoneOutputStream,
- rangeHeader.getStartOffset(),
+ final long skipped =
+ sourceObject.skip(rangeHeader.getStartOffset());
Review comment:
@bharatviswa504 Thanks for your suggestion, indeed, it is necessary to override the skip method of OzoneInputStream, I've done it by the last commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#discussion_r478744360
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -557,8 +559,14 @@ private Response createMultipartKey(String bucket, String key, long length,
if (range != null) {
RangeHeader rangeHeader =
RangeHeaderParserUtil.parseRangeHeader(range, 0);
- IOUtils.copyLarge(sourceObject, ozoneOutputStream,
- rangeHeader.getStartOffset(),
+ final long skipped =
+ sourceObject.skip(rangeHeader.getStartOffset());
Review comment:
But we use OzoneInputStream here, which has not implemented the skip(). So we use from BaseClass InputStream.
public long skip(long n) throws IOException {
long remaining = n;
int nr;
if (n <= 0) {
return 0;
}
int size = (int)Math.min(MAX_SKIP_BUFFER_SIZE, remaining);
byte[] skipBuffer = new byte[size];
while (remaining > 0) {
nr = read(skipBuffer, 0, (int)Math.min(size, remaining));
if (nr < 0) {
break;
}
remaining -= nr;
}
return n - remaining;
}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#discussion_r478744846
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -277,7 +278,8 @@ public Response get(
try (S3WrapperInputStream s3WrapperInputStream =
new S3WrapperInputStream(
key.getInputStream())) {
- IOUtils.copyLarge(s3WrapperInputStream, dest, startOffset,
+ s3WrapperInputStream.seek(startOffset);
+ IOUtils.copyLarge(s3WrapperInputStream, dest, 0,
Review comment:
Got it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#discussion_r477699663
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -557,8 +559,14 @@ private Response createMultipartKey(String bucket, String key, long length,
if (range != null) {
RangeHeader rangeHeader =
RangeHeaderParserUtil.parseRangeHeader(range, 0);
- IOUtils.copyLarge(sourceObject, ozoneOutputStream,
- rangeHeader.getStartOffset(),
+ final long skipped =
+ sourceObject.skip(rangeHeader.getStartOffset());
Review comment:
skip again uses read and moves the position, why not use like above seek and call copylarge?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#issuecomment-683107270
Thank You @maobaolong for the contribution and @ChenSammi for the review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
bharatviswa504 merged pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1354:
URL: https://github.com/apache/hadoop-ozone/pull/1354#discussion_r478744768
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -557,8 +559,14 @@ private Response createMultipartKey(String bucket, String key, long length,
if (range != null) {
RangeHeader rangeHeader =
RangeHeaderParserUtil.parseRangeHeader(range, 0);
- IOUtils.copyLarge(sourceObject, ozoneOutputStream,
- rangeHeader.getStartOffset(),
+ final long skipped =
+ sourceObject.skip(rangeHeader.getStartOffset());
Review comment:
Why can't we add seek to OzoneInputStream like how we have done for S3WrapperInputStream.
As anyway underlying inputStream used in OzoneInputStream is KeyInputStream which has implemented seek.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org