You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "amogh-jahagirdar (via GitHub)" <gi...@apache.org> on 2024/06/03 19:03:11 UTC
[PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
amogh-jahagirdar opened a new pull request, #10433:
URL: https://github.com/apache/iceberg/pull/10433
This is an alternative approach to https://github.com/apache/iceberg/pull/4912/files and https://github.com/apache/iceberg/pull/8221/files#diff-0b632866a3b10fac55c442b08178ec0ac72b3b600878243e15d788a8bd031054
for retrying failures encountered when retrying on the reading of input streams.
This approach defines a `RetryableInputStream` class which will wrap underlying input streams returned by object store APIs.
Upon failures a new stream will be created. Custom exceptions can be passed in, but the default retries are on SocketTimeoutException and SSLException. This change integrates this input stream implementation with S3InputStream, but RetryableINputStream should be able to be used for the other input streams implementations that are provided by Iceberg.
This change relies on the [Failsafe dependency](https://failsafe.dev/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1763890714
##########
aws/src/test/java/org/apache/iceberg/aws/s3/TestFlakyS3InputStream.java:
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.s3;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.spy;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketTimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import javax.net.ssl.SSLException;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import software.amazon.awssdk.awscore.exception.AwsServiceException;
+import software.amazon.awssdk.core.ResponseInputStream;
+import software.amazon.awssdk.core.exception.SdkClientException;
+import software.amazon.awssdk.core.sync.RequestBody;
+import software.amazon.awssdk.core.sync.ResponseTransformer;
+import software.amazon.awssdk.services.s3.S3Client;
+import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
+import software.amazon.awssdk.services.s3.model.CreateBucketResponse;
+import software.amazon.awssdk.services.s3.model.GetObjectRequest;
+import software.amazon.awssdk.services.s3.model.GetObjectResponse;
+import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
+import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
+import software.amazon.awssdk.services.s3.model.PutObjectRequest;
+import software.amazon.awssdk.services.s3.model.PutObjectResponse;
+
+public class TestFlakyS3InputStream extends TestS3InputStream {
Review Comment:
I renamed from fuzzy to flaky since I think flaky is probably a better adjective to describe that it will periodically fail and then succeed. When I think "fuzzy" I think of fuzzy logic in math which is a bit different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1725991357
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java:
##########
@@ -139,7 +140,11 @@ private InputStream readRange(String range) {
S3RequestUtil.configureEncryption(s3FileIOProperties, requestBuilder);
- return s3.getObject(requestBuilder.build(), ResponseTransformer.toInputStream());
+ stream =
+ RetryableInputStream.builderFor(
+ () -> s3.getObject(requestBuilder.build(), ResponseTransformer.toInputStream()))
Review Comment:
I updated this, the RetryableInputStream offers two builder APIs, one for specifying just a new stream initialization and one for a stream initialization plus a position supplier. The stream initialization function takes in a position (the position can be null to handle the range based requests since for range reads with explicit begin/end we don't care about the current position in the stream). cc @danielcweeks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1726971204
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java:
##########
@@ -92,13 +93,13 @@ public void seek(long newPos) {
public int read() throws IOException {
Preconditions.checkState(!closed, "Cannot read: already closed");
positionStream();
-
+ int bytesRead = stream.read();
Review Comment:
I'm guessing this is just aligning things with the other `read` method and isn't a strictly necessary change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2322226419
> I believe https://github.com/apache/iceberg/pull/8221 aimed at addressing both input and output streams. Is there an existing PR for output stream retries?
@edgarRd based on our later evaluation, the AWS SDK will handle all output stream retries, so I think input stream retry is sufficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624920900
##########
aws/src/test/java/org/apache/iceberg/aws/s3/TestFuzzyS3InputStream.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.s3;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.spy;
+
+import dev.failsafe.FailsafeException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketTimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import javax.net.ssl.SSLException;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import software.amazon.awssdk.awscore.exception.AwsServiceException;
+import software.amazon.awssdk.core.ResponseInputStream;
+import software.amazon.awssdk.core.exception.SdkClientException;
+import software.amazon.awssdk.core.sync.RequestBody;
+import software.amazon.awssdk.core.sync.ResponseTransformer;
+import software.amazon.awssdk.services.s3.S3Client;
+import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
+import software.amazon.awssdk.services.s3.model.CreateBucketResponse;
+import software.amazon.awssdk.services.s3.model.GetObjectRequest;
+import software.amazon.awssdk.services.s3.model.GetObjectResponse;
+import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
+import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
+import software.amazon.awssdk.services.s3.model.PutObjectRequest;
+import software.amazon.awssdk.services.s3.model.PutObjectResponse;
+
+public class TestFuzzyS3InputStream extends TestS3InputStream {
Review Comment:
I copied a lot of the test logic from https://github.com/apache/iceberg/pull/4912/files, will mark @jackye1995 as coauthor
##########
aws/src/test/java/org/apache/iceberg/aws/s3/TestFuzzyS3InputStream.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.s3;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.spy;
+
+import dev.failsafe.FailsafeException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketTimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import javax.net.ssl.SSLException;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import software.amazon.awssdk.awscore.exception.AwsServiceException;
+import software.amazon.awssdk.core.ResponseInputStream;
+import software.amazon.awssdk.core.exception.SdkClientException;
+import software.amazon.awssdk.core.sync.RequestBody;
+import software.amazon.awssdk.core.sync.ResponseTransformer;
+import software.amazon.awssdk.services.s3.S3Client;
+import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
+import software.amazon.awssdk.services.s3.model.CreateBucketResponse;
+import software.amazon.awssdk.services.s3.model.GetObjectRequest;
+import software.amazon.awssdk.services.s3.model.GetObjectResponse;
+import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
+import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
+import software.amazon.awssdk.services.s3.model.PutObjectRequest;
+import software.amazon.awssdk.services.s3.model.PutObjectResponse;
+
+public class TestFuzzyS3InputStream extends TestS3InputStream {
+
+ private static final int DATA_SIZE = 3 * 1024 * 1024;
+
+ @ParameterizedTest
+ @MethodSource("retryableExceptions")
+ public void testReadWithFuzzyStreamRetrySucceed(IOException exception) throws Exception {
+ testRead(
+ fuzzyStreamClient(new AtomicInteger(3), exception), new S3FileIOProperties(), DATA_SIZE);
+ }
Review Comment:
This test takes way too long. That's primarily because it internally tests retrying on just read() which means every byte read will fail and be retried 2 times with a 500 ms in between. So essentially that's a second per byte.
I think what we can do is modularize further and only do buffered read tests + with a much smaller data size test the per byte read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1764033379
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java:
##########
@@ -57,6 +64,14 @@ class S3InputStream extends SeekableInputStream implements RangeReadable {
private final Counter readOperations;
private int skipSize = 1024 * 1024;
+ private RetryPolicy<Object> retryPolicy =
+ RetryPolicy.builder()
+ .handle(
+ ImmutableList.of(
+ SSLException.class, SocketTimeoutException.class, SocketException.class))
+ .onFailure(failure -> openStream(true))
+ .withMaxRetries(3)
Review Comment:
We can make these configurable down the line if needed but this covers all the cases that have been reported so far
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Retry S3InputStream reads [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1771831100
##########
aws/src/test/java/org/apache/iceberg/aws/s3/TestS3InputStream.java:
##########
@@ -52,18 +52,15 @@ public void before() {
createBucket("bucket");
}
- @Test
Review Comment:
Mistake here in my recent push this morning, the test method needs to be called passing in the client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2334992007
> > > The S3 team (@ookumuso) just published what they have developed internally and is now used in Amazon EMR, Athena, GlueETL distributions of Iceberg: #11052, is there a way we can see how we could collaborate on this?
> >
> >
> > Sorry for the late reply, Absolutely @jackye1995 ! So it seems like the overlap between this and #11052 is primarily the retrying of the reading of the input stream. #11052 also includes some configurable properties and different configurations for the S3Client.
> > For the retrying of the input stream, I think the key thing I was trying to achieve was adding the RetryableInputStream primitive so that other FileIOs like GCS/Azure can also use that. How do we feel about that @jackye1995 @ookumuso ? And then the other PR we can talk about the configuration/default updates?
>
> No worries and thanks for following up! I am okay with using the RetryableInputStream as long as we can we can close this out soonish since there seems to be a lot of people hitting the same issue. Have 2 options here:
>
> * I can exclude the InputStream changes entirely from my PR and just keep the s3-retry config changes. Might be good to use the same retry configs that we provide in your RetryableInputStream as well
> * I can keep both depending on your timelines but can be replaced by your change, tests are mostly same but might give you some conflicts on InputStream.
>
> What do you think @jackye1995 @amogh-jahagirdar ?
Sounds good, yeah let me get this to closure soon since it has been open for a while. I think I'd propose option 1 and we rebase #11052 on top of that. My only take is we really should try and minimize as much configuration as possible in Iceberg since the S3 properties is already quite complex. But I think we can address that in the other PR since not retrying the input stream is a concrete issue that quite a few folks are hitting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624919847
##########
aws/src/test/java/org/apache/iceberg/aws/s3/TestFuzzyS3InputStream.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.s3;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.spy;
+
+import dev.failsafe.FailsafeException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketTimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import javax.net.ssl.SSLException;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import software.amazon.awssdk.awscore.exception.AwsServiceException;
+import software.amazon.awssdk.core.ResponseInputStream;
+import software.amazon.awssdk.core.exception.SdkClientException;
+import software.amazon.awssdk.core.sync.RequestBody;
+import software.amazon.awssdk.core.sync.ResponseTransformer;
+import software.amazon.awssdk.services.s3.S3Client;
+import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
+import software.amazon.awssdk.services.s3.model.CreateBucketResponse;
+import software.amazon.awssdk.services.s3.model.GetObjectRequest;
+import software.amazon.awssdk.services.s3.model.GetObjectResponse;
+import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
+import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
+import software.amazon.awssdk.services.s3.model.PutObjectRequest;
+import software.amazon.awssdk.services.s3.model.PutObjectResponse;
+
+public class TestFuzzyS3InputStream extends TestS3InputStream {
+
+ private static final int DATA_SIZE = 3 * 1024 * 1024;
+
+ @ParameterizedTest
+ @MethodSource("retryableExceptions")
+ public void testReadWithFuzzyStreamRetrySucceed(IOException exception) throws Exception {
+ testRead(
+ fuzzyStreamClient(new AtomicInteger(3), exception), new S3FileIOProperties(), DATA_SIZE);
+ }
Review Comment:
This test takes way too long. That's primarily because it internally tests retrying on just read() (the non-buffered reads) which means every byte read will fail and be retried 2 times with a 500 ms in between. So essentially that's a second per byte.
I think what we can do is modularize further and only do buffered read tests + with a much smaller data size test the per byte read to exercise that code path in the tests. The buffered read tests are pretty fast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1625011756
##########
api/src/main/java/org/apache/iceberg/io/RetryableInputStream.java:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import dev.failsafe.Failsafe;
+import dev.failsafe.RetryPolicy;
+import dev.failsafe.RetryPolicyBuilder;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketTimeoutException;
+import java.time.Duration;
+import java.util.List;
+import java.util.function.Supplier;
+import javax.net.ssl.SSLException;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+/**
+ * RetryableInputStream wraps over an underlying InputStream and retries failures encountered when
+ * reading through the stream. On retries, the underlying streams will be reinitialized.
+ */
+public class RetryableInputStream extends InputStream {
+
+ private final InputStream underlyingStream;
+ private final RetryPolicy<Object> retryPolicy;
+
+ private RetryableInputStream(InputStream underlyingStream, RetryPolicy<Object> retryPolicy) {
+ this.underlyingStream = underlyingStream;
+ this.retryPolicy = retryPolicy;
+ }
+
+ @Override
+ public int read() throws IOException {
+ return Failsafe.with(retryPolicy).get(() -> underlyingStream.read());
+ }
+
+ @Override
+ public int read(byte[] b, int off, int len) throws IOException {
+ return Failsafe.with(retryPolicy).get(() -> underlyingStream.read(b, off, len));
+ }
+
+ @Override
+ public void close() throws IOException {
+ underlyingStream.close();
+ }
+
+ public static RetryableInputStream.Builder builderFor(Supplier<InputStream> newStreamSupplier) {
+ return new Builder(newStreamSupplier);
+ }
+
+ public static class Builder {
+
+ private InputStream underlyingStream;
+ private final Supplier<InputStream> newStreamProvider;
+ private List<Class<? extends Exception>> retryableExceptions =
+ ImmutableList.of(SSLException.class, SocketTimeoutException.class);
Review Comment:
[QQ]
- Should we retry this 5xx errors like 503 ?
- will it make sense to make this configurable by any chance ?
- Also any thoughts about outputstream ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2332877910
> > I believe #8221 aimed at addressing both input and output streams. Is there an existing PR for output stream retries?
>
> @edgarRd based on our later evaluation, the AWS SDK will handle all output stream retries, so I think input stream retry is sufficient.
This is interesting. Didn't know that the AWS SDK already handles the outputstream retries, if that's the case then I think that's even more of a case not to introduce an additional retry logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1764030252
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java:
##########
@@ -195,14 +230,20 @@ private void openStream() throws IOException {
}
}
- private void closeStream() throws IOException {
+ private void closeStream(boolean closeQuietly) throws IOException {
if (stream != null) {
// if we aren't at the end of the stream, and the stream is abortable, then
// call abort() so we don't read the remaining data with the Apache HTTP client
abortStream();
try {
stream.close();
} catch (IOException e) {
+ if (closeQuietly) {
+ stream = null;
+ LOG.warn("An error occurred while closing the stream", e);
+ return;
+ }
Review Comment:
This is needed so that IOException failures on close during the retry don't count against the retry on the read. It can just be a best effort close.
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java:
##########
@@ -56,7 +63,15 @@ class S3InputStream extends SeekableInputStream implements RangeReadable {
private final Counter readBytes;
private final Counter readOperations;
- private int skipSize = 1024 * 1024;
+ private int skipSize = 1024 * 1024;;
+ private RetryPolicy<Object> retryPolicy =
+ RetryPolicy.builder()
+ .handle(
+ ImmutableList.of(
+ SSLException.class, SocketTimeoutException.class, SocketException.class))
+ .onFailure(failure -> openStream(true))
+ .withMaxRetries(3)
+ .build();
Review Comment:
I ended up removing RetryableInputStream for now; GCS is channel based and we should probably verify the Azure SDK doesn't already retry internally. We can always add the abstraction later if it ends up being needed but there's arguably not a compelling case for adding it up front when inlining Failsafe is simple. cc @danielcweeks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1763885839
##########
aws/src/test/java/org/apache/iceberg/aws/s3/TestFlakyS3InputStream.java:
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.s3;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.spy;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketTimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import javax.net.ssl.SSLException;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import software.amazon.awssdk.awscore.exception.AwsServiceException;
+import software.amazon.awssdk.core.ResponseInputStream;
+import software.amazon.awssdk.core.exception.SdkClientException;
+import software.amazon.awssdk.core.sync.RequestBody;
+import software.amazon.awssdk.core.sync.ResponseTransformer;
+import software.amazon.awssdk.services.s3.S3Client;
+import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
+import software.amazon.awssdk.services.s3.model.CreateBucketResponse;
+import software.amazon.awssdk.services.s3.model.GetObjectRequest;
+import software.amazon.awssdk.services.s3.model.GetObjectResponse;
+import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
+import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
+import software.amazon.awssdk.services.s3.model.PutObjectRequest;
+import software.amazon.awssdk.services.s3.model.PutObjectResponse;
+
+public class TestFlakyS3InputStream extends TestS3InputStream {
+ private static final int DATA_SIZE = 100;
+ private static final int SEEK_SIZE = 4;
+ private static final int SEEK_NEW_POSITION = 25;
+
+ @ParameterizedTest
+ @MethodSource("retryableExceptions")
+ public void testReadWithFuzzyStreamRetrySucceed(IOException exception) throws Exception {
+ testRead(fuzzyStreamClient(new AtomicInteger(3), exception), DATA_SIZE);
+ }
+
+ @ParameterizedTest
+ @MethodSource("retryableExceptions")
+ public void testReadWithFuzzyStreamExhaustedRetries(IOException exception) {
+ assertThatThrownBy(
+ () -> testRead(fuzzyStreamClient(new AtomicInteger(5), exception), DATA_SIZE))
+ .isInstanceOf(exception.getClass())
+ .hasMessage(exception.getMessage());
+ }
+
+ @ParameterizedTest
+ @MethodSource("nonRetryableExceptions")
+ public void testReadWithFuzzyStreamNonRetryableException(IOException exception) {
+ assertThatThrownBy(
+ () -> testRead(fuzzyStreamClient(new AtomicInteger(3), exception), DATA_SIZE))
+ .isInstanceOf(exception.getClass())
+ .hasMessage(exception.getMessage());
+ }
+
+ @Override
+ protected void testRead(S3Client s3, int dataSize) throws Exception {
+ testRead(s3, DATA_SIZE, 4, SEEK_SIZE, SEEK_NEW_POSITION);
+ }
+
+ private static Stream<Arguments> retryableExceptions() {
+ return Stream.of(
+ Arguments.of(
+ new SocketTimeoutException("socket timeout exception"),
+ new SSLException("some ssl exception")));
+ }
+
+ private static Stream<Arguments> nonRetryableExceptions() {
+ return Stream.of(Arguments.of(new IOException("some generic non-retryable IO exception")));
+ }
+
+ private S3ClientWrapper fuzzyStreamClient(AtomicInteger counter, IOException failure) {
+ S3ClientWrapper fuzzyClient = spy(new S3ClientWrapper(s3Client()));
+ doAnswer(invocation -> new FlakyInputStream(invocation.callRealMethod(), counter, failure))
+ .when(fuzzyClient)
+ .getObject(any(GetObjectRequest.class), any(ResponseTransformer.class));
+ return fuzzyClient;
+ }
+
+ /** Wrapper for S3 client, used to mock the final class DefaultS3Client */
+ public static class S3ClientWrapper implements S3Client {
+
+ private final S3Client delegate;
+
+ public S3ClientWrapper(S3Client delegate) {
+ this.delegate = delegate;
+ }
+
+ @Override
+ public String serviceName() {
+ return delegate.serviceName();
+ }
+
+ @Override
+ public void close() {
+ delegate.close();
+ }
+
+ @Override
+ public <ReturnT> ReturnT getObject(
+ GetObjectRequest getObjectRequest,
+ ResponseTransformer<GetObjectResponse, ReturnT> responseTransformer)
+ throws AwsServiceException, SdkClientException {
+ return delegate.getObject(getObjectRequest, responseTransformer);
+ }
+
+ @Override
+ public HeadObjectResponse headObject(HeadObjectRequest headObjectRequest)
+ throws AwsServiceException, SdkClientException {
+ return delegate.headObject(headObjectRequest);
+ }
+
+ @Override
+ public PutObjectResponse putObject(PutObjectRequest putObjectRequest, RequestBody requestBody)
+ throws AwsServiceException, SdkClientException {
+ return delegate.putObject(putObjectRequest, requestBody);
+ }
+
+ @Override
+ public CreateBucketResponse createBucket(CreateBucketRequest createBucketRequest)
+ throws AwsServiceException, SdkClientException {
+ return delegate.createBucket(createBucketRequest);
+ }
+ }
+
+ static class FlakyInputStream extends InputStream {
Review Comment:
I slimmed down this input stream implementation so that we only implement read/close that's required for the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1625018098
##########
api/src/main/java/org/apache/iceberg/io/RetryableInputStream.java:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import dev.failsafe.Failsafe;
+import dev.failsafe.RetryPolicy;
+import dev.failsafe.RetryPolicyBuilder;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketTimeoutException;
+import java.time.Duration;
+import java.util.List;
+import java.util.function.Supplier;
+import javax.net.ssl.SSLException;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+/**
+ * RetryableInputStream wraps over an underlying InputStream and retries failures encountered when
+ * reading through the stream. On retries, the underlying streams will be reinitialized.
+ */
+public class RetryableInputStream extends InputStream {
+
+ private final InputStream underlyingStream;
+ private final RetryPolicy<Object> retryPolicy;
+
+ private RetryableInputStream(InputStream underlyingStream, RetryPolicy<Object> retryPolicy) {
+ this.underlyingStream = underlyingStream;
+ this.retryPolicy = retryPolicy;
+ }
+
+ @Override
+ public int read() throws IOException {
+ return Failsafe.with(retryPolicy).get(() -> underlyingStream.read());
+ }
+
+ @Override
+ public int read(byte[] b, int off, int len) throws IOException {
+ return Failsafe.with(retryPolicy).get(() -> underlyingStream.read(b, off, len));
+ }
+
+ @Override
+ public void close() throws IOException {
+ underlyingStream.close();
+ }
+
+ public static RetryableInputStream.Builder builderFor(Supplier<InputStream> newStreamSupplier) {
+ return new Builder(newStreamSupplier);
+ }
+
+ public static class Builder {
+
+ private InputStream underlyingStream;
+ private final Supplier<InputStream> newStreamProvider;
+ private List<Class<? extends Exception>> retryableExceptions =
+ ImmutableList.of(SSLException.class, SocketTimeoutException.class);
Review Comment:
@singhpk234 I believe the SDK already handles 503 errors and other errors that happen during the initiation of the request. I believe output stream is different (I don't think we've heard of issues on that side), but I would handle that in a separate PR if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "SandeepSinghGahir (via GitHub)" <gi...@apache.org>.
SandeepSinghGahir commented on PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2353901444
> > > > The S3 team (@ookumuso) just published what they have developed internally and is now used in Amazon EMR, Athena, GlueETL distributions of Iceberg: #11052, is there a way we can see how we could collaborate on this?
> > >
> > >
> > > Sorry for the late reply, Absolutely @jackye1995 ! So it seems like the overlap between this and #11052 is primarily the retrying of the reading of the input stream. #11052 also includes some configurable properties and different configurations for the S3Client.
> > > For the retrying of the input stream, I think the key thing I was trying to achieve was adding the RetryableInputStream primitive so that other FileIOs like GCS/Azure can also use that. How do we feel about that @jackye1995 @ookumuso ? And then the other PR we can talk about the configuration/default updates?
> >
> >
> > No worries and thanks for following up! I am okay with using the RetryableInputStream as long as we can we can close this out soonish since there seems to be a lot of people hitting the same issue. Have 2 options here:
> >
> > * I can exclude the InputStream changes entirely from my PR and just keep the s3-retry config changes. Might be good to use the same retry configs that we provide in your RetryableInputStream as well
> > * I can keep both depending on your timelines but can be replaced by your change, tests are mostly same but might give you some conflicts on InputStream.
> >
> > What do you think @jackye1995 @amogh-jahagirdar ?
>
> Sounds good, yeah let me get this to closure soon since it has been open for a while. I think I'd propose option 1 and we rebase #11052 on top of that. My only take is we really should try and minimize as much configuration as possible in Iceberg since the S3 properties is already quite complex. But I think we can address that in the other PR since not retrying the input stream is a concrete issue that quite a few folks are hitting.
Sounds good. Yes, a lot of folks are hitting this issue, also mentioned here: https://github.com/apache/iceberg/issues/10340
Do we have any expected time to merge this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1625032460
##########
api/src/main/java/org/apache/iceberg/io/RetryableInputStream.java:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import dev.failsafe.Failsafe;
+import dev.failsafe.RetryPolicy;
+import dev.failsafe.RetryPolicyBuilder;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketTimeoutException;
+import java.time.Duration;
+import java.util.List;
+import java.util.function.Supplier;
+import javax.net.ssl.SSLException;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+/**
+ * RetryableInputStream wraps over an underlying InputStream and retries failures encountered when
+ * reading through the stream. On retries, the underlying streams will be reinitialized.
+ */
+public class RetryableInputStream extends InputStream {
+
+ private final InputStream underlyingStream;
+ private final RetryPolicy<Object> retryPolicy;
+
+ private RetryableInputStream(InputStream underlyingStream, RetryPolicy<Object> retryPolicy) {
+ this.underlyingStream = underlyingStream;
+ this.retryPolicy = retryPolicy;
+ }
+
+ @Override
+ public int read() throws IOException {
+ return Failsafe.with(retryPolicy).get(() -> underlyingStream.read());
+ }
+
+ @Override
+ public int read(byte[] b, int off, int len) throws IOException {
+ return Failsafe.with(retryPolicy).get(() -> underlyingStream.read(b, off, len));
+ }
+
+ @Override
+ public void close() throws IOException {
+ underlyingStream.close();
+ }
+
+ public static RetryableInputStream.Builder builderFor(Supplier<InputStream> newStreamSupplier) {
+ return new Builder(newStreamSupplier);
+ }
+
+ public static class Builder {
+
+ private InputStream underlyingStream;
+ private final Supplier<InputStream> newStreamProvider;
+ private List<Class<? extends Exception>> retryableExceptions =
+ ImmutableList.of(SSLException.class, SocketTimeoutException.class);
Review Comment:
> I believe the SDK already handles 503 errors and other errors that happen during the initiation of the request
yup, they do, but we have defaults of SDK at the moment right ? wondering if we should either expose conf's to overwrite it or control it via here.
Also was wondering if we can go as exhaustive a retry policy and have it at one place, as this : https://github.com/apache/hadoop/blob/b108e9e2d814cc009c4b663799e056a65bf24766/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java#L193
In past we have been reached for tickets like this : https://github.com/apache/iceberg/issues/9124#issuecomment-1821822583
> I believe output stream is different (I don't think we've heard of issues on that side), but I would handle that in a separate PR if necessary
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624925629
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java:
##########
@@ -139,7 +140,11 @@ private InputStream readRange(String range) {
S3RequestUtil.configureEncryption(s3FileIOProperties, requestBuilder);
- return s3.getObject(requestBuilder.build(), ResponseTransformer.toInputStream());
+ stream =
+ RetryableInputStream.builderFor(
+ () -> s3.getObject(requestBuilder.build(), ResponseTransformer.toInputStream()))
Review Comment:
any failures that occur on the retry during the getObject request (not the stream read) should just be handled by the SDK. Don't think we need to add anything custom for that since the S3 client is already pluggable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "SandeepSinghGahir (via GitHub)" <gi...@apache.org>.
SandeepSinghGahir commented on PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2282666300
When will this merged? I'm getting this issue while reading iceberg tables in glue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624953864
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java:
##########
@@ -139,7 +140,11 @@ private InputStream readRange(String range) {
S3RequestUtil.configureEncryption(s3FileIOProperties, requestBuilder);
- return s3.getObject(requestBuilder.build(), ResponseTransformer.toInputStream());
+ stream =
+ RetryableInputStream.builderFor(
+ () -> s3.getObject(requestBuilder.build(), ResponseTransformer.toInputStream()))
Review Comment:
I feel like there's an issue here. When the stream gets recreated, the stream will reset back to the original position and we continue from there as if we're at the right place in the stream.
The `pos` won't reflect the new position of the stream, if I'm reading this correctly. I would think the following retry would need to start from `next` to reflect where the next read should start. There's a small problem with the single byte read method because we increment the positions prior to read, so that would likely need to be adjusted to be after the read like in the range read method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1625017201
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java:
##########
@@ -139,7 +140,11 @@ private InputStream readRange(String range) {
S3RequestUtil.configureEncryption(s3FileIOProperties, requestBuilder);
- return s3.getObject(requestBuilder.build(), ResponseTransformer.toInputStream());
+ stream =
+ RetryableInputStream.builderFor(
+ () -> s3.getObject(requestBuilder.build(), ResponseTransformer.toInputStream()))
Review Comment:
I'll double check this, I think you're right although I'm not sure why the first test case wouldn't surface that (since the content would be different). We may need to seek properly on the input stream initialization during retry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1763770630
##########
core/src/main/java/org/apache/iceberg/io/RetryableInputStream.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import dev.failsafe.Failsafe;
+import dev.failsafe.FailsafeException;
+import dev.failsafe.RetryPolicy;
+import dev.failsafe.RetryPolicyBuilder;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketException;
+import java.net.SocketTimeoutException;
+import java.time.Duration;
+import java.util.List;
+import java.util.function.Supplier;
+import javax.net.ssl.SSLException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+/**
+ * RetryableInputStream wraps over an underlying InputStream and retries failures encountered when
+ * reading through the stream. On retries, the underlying streams will be reinitialized.
+ */
Review Comment:
I think that we need to document that the `newStreamSupplier` must track the position within the stream and restart it at the appropriate location. This class will not track/reposition the stream upon recreation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2332876745
> The S3 team (@ookumuso) just published what they have developed internally and is now used in Amazon EMR, Athena, GlueETL distributions of Iceberg: #11052, is there a way we can see how we could collaborate on this?
Sorry for the late reply, Absolutely @jackye1995 ! So it seems like the overlap between this and #11052 is primarily the retrying of the reading of the input stream. #11052 also includes some configurable properties and different configurations for the S3Client.
For the retrying of the input stream, I think the key thing I was trying to achieve was adding the RetryableInputStream primitive so that other FileIOs like GCS/Azure can also use that. How do we feel about that @jackye1995 @ookumuso ? And then the other PR we can talk about the configuration/default updates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1716246358
##########
api/src/main/java/org/apache/iceberg/io/RetryableInputStream.java:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import dev.failsafe.Failsafe;
+import dev.failsafe.RetryPolicy;
+import dev.failsafe.RetryPolicyBuilder;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketTimeoutException;
+import java.time.Duration;
+import java.util.List;
+import java.util.function.Supplier;
+import javax.net.ssl.SSLException;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+/**
+ * RetryableInputStream wraps over an underlying InputStream and retries failures encountered when
+ * reading through the stream. On retries, the underlying streams will be reinitialized.
+ */
+public class RetryableInputStream extends InputStream {
+
+ private final InputStream underlyingStream;
+ private final RetryPolicy<Object> retryPolicy;
+
+ private RetryableInputStream(InputStream underlyingStream, RetryPolicy<Object> retryPolicy) {
+ this.underlyingStream = underlyingStream;
+ this.retryPolicy = retryPolicy;
+ }
+
+ @Override
+ public int read() throws IOException {
+ return Failsafe.with(retryPolicy).get(() -> underlyingStream.read());
+ }
+
+ @Override
+ public int read(byte[] b, int off, int len) throws IOException {
+ return Failsafe.with(retryPolicy).get(() -> underlyingStream.read(b, off, len));
+ }
+
+ @Override
+ public void close() throws IOException {
+ underlyingStream.close();
+ }
+
+ public static RetryableInputStream.Builder builderFor(Supplier<InputStream> newStreamSupplier) {
+ return new Builder(newStreamSupplier);
+ }
+
+ public static class Builder {
+
+ private InputStream underlyingStream;
+ private final Supplier<InputStream> newStreamProvider;
+ private List<Class<? extends Exception>> retryableExceptions =
+ ImmutableList.of(SSLException.class, SocketTimeoutException.class);
Review Comment:
Really sorry, I raised this draft and never followed through:
@singhpk234 You're correct that Iceberg itself does not have a configuration to overwrite the default SDK retry policy. I'd say that's separate to this and wouldn't club the two together imo. But my general thought is that AwsProperties has gotten a bit complex with all the knobs that are exposed, and I think in Iceberg we'd want to avoid the complexity that Hadoop s3a gets into. It's a classic "knobs enable users control" vs "simple/reducing maintenance burden" problem and personally I think Iceberg should attempt to be in the latter category. At least for the 503 retry case, users can already provide a s3client that's configured with their desired retry policy.
On retrying outputstream, I'd say we can address that separately and when it becomes an issue. All the cases, I've so far have really just been on the read side.
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java:
##########
@@ -139,7 +140,11 @@ private InputStream readRange(String range) {
S3RequestUtil.configureEncryption(s3FileIOProperties, requestBuilder);
- return s3.getObject(requestBuilder.build(), ResponseTransformer.toInputStream());
+ stream =
+ RetryableInputStream.builderFor(
+ () -> s3.getObject(requestBuilder.build(), ResponseTransformer.toInputStream()))
Review Comment:
I revisited this PR, and took a look with fresh eyes and yes the current logic is definitely incorrect for the case where a *non* range read (readFully, readTail perform range reads) is performed.
For the range-reads we don't really care about the current position for the purpose of tracking in the retryable input stream. But for normal seeking based reads we definitely do!
I think the way to solve this is to pass a supplier of the current position to the retryable input stream. That supplier would have a reference to `this` and the stream provider would be a function which accepts a position. Upon retries the stream provider would open a new connection and open a stream that begins with the position that the position supplier (which is guaranteed to be the correct position to start the stream from) returns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "ookumuso (via GitHub)" <gi...@apache.org>.
ookumuso commented on PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2334900818
> > The S3 team (@ookumuso) just published what they have developed internally and is now used in Amazon EMR, Athena, GlueETL distributions of Iceberg: #11052, is there a way we can see how we could collaborate on this?
>
> Sorry for the late reply, Absolutely @jackye1995 ! So it seems like the overlap between this and #11052 is primarily the retrying of the reading of the input stream. #11052 also includes some configurable properties and different configurations for the S3Client.
>
> For the retrying of the input stream, I think the key thing I was trying to achieve was adding the RetryableInputStream primitive so that other FileIOs like GCS/Azure can also use that. How do we feel about that @jackye1995 @ookumuso ? And then the other PR we can talk about the configuration/default updates?
No worries and thanks for following up! I am okay with using the RetryableInputStream as long as we can we can close this out soonish since there seems to be a lot of people hitting the same issue. Have 2 options here:
- I can exclude the InputStream changes entirely from my PR and just keep the s3-retry config changes. Might be good to use the same retry configs that we provide in your RetryableInputStream as well
- I can keep both depending on your timelines but can be replaced by your change, tests are mostly same but might give you some conflicts on InputStream.
What do you think @jackye1995 @amogh-jahagirdar ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1725982688
##########
aws/src/test/java/org/apache/iceberg/aws/s3/TestFuzzyS3InputStream.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.s3;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.spy;
+
+import dev.failsafe.FailsafeException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketTimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import javax.net.ssl.SSLException;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import software.amazon.awssdk.awscore.exception.AwsServiceException;
+import software.amazon.awssdk.core.ResponseInputStream;
+import software.amazon.awssdk.core.exception.SdkClientException;
+import software.amazon.awssdk.core.sync.RequestBody;
+import software.amazon.awssdk.core.sync.ResponseTransformer;
+import software.amazon.awssdk.services.s3.S3Client;
+import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
+import software.amazon.awssdk.services.s3.model.CreateBucketResponse;
+import software.amazon.awssdk.services.s3.model.GetObjectRequest;
+import software.amazon.awssdk.services.s3.model.GetObjectResponse;
+import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
+import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
+import software.amazon.awssdk.services.s3.model.PutObjectRequest;
+import software.amazon.awssdk.services.s3.model.PutObjectResponse;
+
+public class TestFuzzyS3InputStream extends TestS3InputStream {
Review Comment:
I saw @xiaoxuandev also had the same tests in https://github.com/apache/iceberg/pull/8221 so I'm marking her as co-author here as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1725989680
##########
aws/src/test/java/org/apache/iceberg/aws/s3/TestFuzzyS3InputStream.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.s3;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.spy;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketTimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import javax.net.ssl.SSLException;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import software.amazon.awssdk.awscore.exception.AwsServiceException;
+import software.amazon.awssdk.core.ResponseInputStream;
+import software.amazon.awssdk.core.exception.SdkClientException;
+import software.amazon.awssdk.core.sync.RequestBody;
+import software.amazon.awssdk.core.sync.ResponseTransformer;
+import software.amazon.awssdk.services.s3.S3Client;
+import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
+import software.amazon.awssdk.services.s3.model.CreateBucketResponse;
+import software.amazon.awssdk.services.s3.model.GetObjectRequest;
+import software.amazon.awssdk.services.s3.model.GetObjectResponse;
+import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
+import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
+import software.amazon.awssdk.services.s3.model.PutObjectRequest;
+import software.amazon.awssdk.services.s3.model.PutObjectResponse;
+
+public class TestFuzzyS3InputStream extends TestS3InputStream {
+
+ private static final int DATA_SIZE = 100;
+ private static final int SEEK_SIZE = 4;
+ private static final int SEEK_NEW_POSITION = 25;
+
+ @ParameterizedTest
+ @MethodSource("retryableExceptions")
+ public void testReadWithFuzzyStreamRetrySucceed(IOException exception) throws Exception {
+ testRead(fuzzyStreamClient(new AtomicInteger(3), exception), DATA_SIZE);
+ }
+
+ @ParameterizedTest
+ @MethodSource("retryableExceptions")
+ public void testReadWithFuzzyStreamExhaustedRetries(IOException exception) {
+ assertThatThrownBy(
+ () -> testRead(fuzzyStreamClient(new AtomicInteger(5), exception), DATA_SIZE))
+ .isInstanceOf(exception.getClass())
+ .hasMessage(exception.getMessage());
+ }
+
+ @ParameterizedTest
+ @MethodSource("nonRetryableExceptions")
+ public void testReadWithFuzzyStreamNonRetryableException(IOException exception) {
+ assertThatThrownBy(
+ () -> testRead(fuzzyStreamClient(new AtomicInteger(3), exception), DATA_SIZE))
+ .isInstanceOf(exception.getClass())
+ .hasMessage(exception.getMessage());
+ }
+
+ @Override
+ protected void testRead(S3Client s3, int dataSize) throws Exception {
+ testRead(s3, DATA_SIZE, 4, SEEK_SIZE, SEEK_NEW_POSITION);
+ }
+
+ private static Stream<Arguments> retryableExceptions() {
+ return Stream.of(
+ Arguments.of(
+ new SocketTimeoutException("socket timeout exception"),
+ new SSLException("some ssl exception")));
+ }
+
+ private static Stream<Arguments> nonRetryableExceptions() {
+ return Stream.of(Arguments.of(new IOException("some generic non-retryable IO exception")));
+ }
+
+ private S3ClientWrapper fuzzyStreamClient(AtomicInteger counter, IOException failure) {
+ S3ClientWrapper fuzzyClient = spy(new S3ClientWrapper(s3Client()));
+ doAnswer(
+ invocation ->
+ new FuzzyResponseInputStream(invocation.callRealMethod(), counter, failure))
+ .when(fuzzyClient)
+ .getObject(any(GetObjectRequest.class), any(ResponseTransformer.class));
+ return fuzzyClient;
+ }
+
+ /** Wrapper for S3 client, used to mock the final class DefaultS3Client */
+ public static class S3ClientWrapper implements S3Client {
+
+ private final S3Client delegate;
+
+ public S3ClientWrapper(S3Client delegate) {
+ this.delegate = delegate;
+ }
+
+ @Override
+ public String serviceName() {
+ return delegate.serviceName();
+ }
+
+ @Override
+ public void close() {
+ delegate.close();
+ }
+
+ @Override
+ public <ReturnT> ReturnT getObject(
+ GetObjectRequest getObjectRequest,
+ ResponseTransformer<GetObjectResponse, ReturnT> responseTransformer)
+ throws AwsServiceException, SdkClientException {
+ return delegate.getObject(getObjectRequest, responseTransformer);
Review Comment:
I was going to track the start ranges that were requested on each getObject and assert that the positions being retried are as expected but this ends up really complicating the tests for what I think is not too much additional value because actually the `testRead` we are inheriting verifies the contents actually being read in each `readAndCheck`. If there's a retry that's being performed to an incorrect position, this would mean that the contents would be incorrect and tests would fail and we know that the retry path is being exercised given the presence of the fuzzy stream client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624927489
##########
gradle/libs.versions.toml:
##########
@@ -95,7 +96,7 @@ antlr-antlr4 = { module = "org.antlr:antlr4", version.ref = "antlr" }
antlr-runtime = { module = "org.antlr:antlr4-runtime", version.ref = "antlr" }
arrow-memory-netty = { module = "org.apache.arrow:arrow-memory-netty", version.ref = "arrow" }
arrow-vector = { module = "org.apache.arrow:arrow-vector", version.ref = "arrow" }
-avro-avro = { module = "org.apache.avro:avro", version.ref = "avro" }
+avro-avro = { module = "org.apache.avro:avro", version.ref = "avro"}
Review Comment:
Shouldn't have changed, I'll undo this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2322224653
The S3 team (@ookumuso) just published what they have developed internally and is now used in Amazon EMR, Athena, GlueETL distributions: #11052, is there a way we can see how we could collaborate on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Retry S3InputStream reads [iceberg]
Posted by "SandeepSinghGahir (via GitHub)" <gi...@apache.org>.
SandeepSinghGahir commented on PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2361969290
@amogh-jahagirdar Any tentative timeline on merging of this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Retry S3InputStream reads [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar merged PR #10433:
URL: https://github.com/apache/iceberg/pull/10433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1762163236
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java:
##########
@@ -178,18 +183,23 @@ private void positionStream() throws IOException {
}
private void openStream() throws IOException {
- GetObjectRequest.Builder requestBuilder =
- GetObjectRequest.builder()
- .bucket(location.bucket())
- .key(location.key())
- .range(String.format("bytes=%s-", pos));
-
- S3RequestUtil.configureEncryption(s3FileIOProperties, requestBuilder);
-
closeStream();
try {
- stream = s3.getObject(requestBuilder.build(), ResponseTransformer.toInputStream());
+ stream =
+ RetryableInputStream.builderFor(
+ rangeStart -> {
+ GetObjectRequest.Builder requestBuilder =
+ GetObjectRequest.builder()
+ .bucket(location.bucket())
+ .key(location.key())
+ .range(String.format("bytes=%s-", rangeStart));
+ S3RequestUtil.configureEncryption(s3FileIOProperties, requestBuilder);
+ return s3.getObject(
+ requestBuilder.build(), ResponseTransformer.toInputStream());
+ },
+ () -> pos)
Review Comment:
Discussed with @danielcweeks, we can just simplify this to remove the extra position supplier parameter to the builderFor API and just pass in the pos as part of the stream supplier
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java:
##########
@@ -139,7 +140,11 @@ private InputStream readRange(String range) {
S3RequestUtil.configureEncryption(s3FileIOProperties, requestBuilder);
- return s3.getObject(requestBuilder.build(), ResponseTransformer.toInputStream());
+ stream =
+ RetryableInputStream.builderFor(
+ () -> s3.getObject(requestBuilder.build(), ResponseTransformer.toInputStream()))
Review Comment:
Discussed with @danielcweeks , for the range read cases since we're using IOUtil readFully/readRemaining which will read the range in a buffered manner. On retries we would read from the beginning position but the internal stream tracking in readFully/readRemaining would not reset to the right position in the buffer to read so that's still an issue.
What we can do is to just not retry the RangeReadable methods for now since they're not actually exercised anywhere. Down the line, we could just use FailSafe and retry on the whole method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1725989680
##########
aws/src/test/java/org/apache/iceberg/aws/s3/TestFuzzyS3InputStream.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.s3;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.spy;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketTimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import javax.net.ssl.SSLException;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import software.amazon.awssdk.awscore.exception.AwsServiceException;
+import software.amazon.awssdk.core.ResponseInputStream;
+import software.amazon.awssdk.core.exception.SdkClientException;
+import software.amazon.awssdk.core.sync.RequestBody;
+import software.amazon.awssdk.core.sync.ResponseTransformer;
+import software.amazon.awssdk.services.s3.S3Client;
+import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
+import software.amazon.awssdk.services.s3.model.CreateBucketResponse;
+import software.amazon.awssdk.services.s3.model.GetObjectRequest;
+import software.amazon.awssdk.services.s3.model.GetObjectResponse;
+import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
+import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
+import software.amazon.awssdk.services.s3.model.PutObjectRequest;
+import software.amazon.awssdk.services.s3.model.PutObjectResponse;
+
+public class TestFuzzyS3InputStream extends TestS3InputStream {
+
+ private static final int DATA_SIZE = 100;
+ private static final int SEEK_SIZE = 4;
+ private static final int SEEK_NEW_POSITION = 25;
+
+ @ParameterizedTest
+ @MethodSource("retryableExceptions")
+ public void testReadWithFuzzyStreamRetrySucceed(IOException exception) throws Exception {
+ testRead(fuzzyStreamClient(new AtomicInteger(3), exception), DATA_SIZE);
+ }
+
+ @ParameterizedTest
+ @MethodSource("retryableExceptions")
+ public void testReadWithFuzzyStreamExhaustedRetries(IOException exception) {
+ assertThatThrownBy(
+ () -> testRead(fuzzyStreamClient(new AtomicInteger(5), exception), DATA_SIZE))
+ .isInstanceOf(exception.getClass())
+ .hasMessage(exception.getMessage());
+ }
+
+ @ParameterizedTest
+ @MethodSource("nonRetryableExceptions")
+ public void testReadWithFuzzyStreamNonRetryableException(IOException exception) {
+ assertThatThrownBy(
+ () -> testRead(fuzzyStreamClient(new AtomicInteger(3), exception), DATA_SIZE))
+ .isInstanceOf(exception.getClass())
+ .hasMessage(exception.getMessage());
+ }
+
+ @Override
+ protected void testRead(S3Client s3, int dataSize) throws Exception {
+ testRead(s3, DATA_SIZE, 4, SEEK_SIZE, SEEK_NEW_POSITION);
+ }
+
+ private static Stream<Arguments> retryableExceptions() {
+ return Stream.of(
+ Arguments.of(
+ new SocketTimeoutException("socket timeout exception"),
+ new SSLException("some ssl exception")));
+ }
+
+ private static Stream<Arguments> nonRetryableExceptions() {
+ return Stream.of(Arguments.of(new IOException("some generic non-retryable IO exception")));
+ }
+
+ private S3ClientWrapper fuzzyStreamClient(AtomicInteger counter, IOException failure) {
+ S3ClientWrapper fuzzyClient = spy(new S3ClientWrapper(s3Client()));
+ doAnswer(
+ invocation ->
+ new FuzzyResponseInputStream(invocation.callRealMethod(), counter, failure))
+ .when(fuzzyClient)
+ .getObject(any(GetObjectRequest.class), any(ResponseTransformer.class));
+ return fuzzyClient;
+ }
+
+ /** Wrapper for S3 client, used to mock the final class DefaultS3Client */
+ public static class S3ClientWrapper implements S3Client {
+
+ private final S3Client delegate;
+
+ public S3ClientWrapper(S3Client delegate) {
+ this.delegate = delegate;
+ }
+
+ @Override
+ public String serviceName() {
+ return delegate.serviceName();
+ }
+
+ @Override
+ public void close() {
+ delegate.close();
+ }
+
+ @Override
+ public <ReturnT> ReturnT getObject(
+ GetObjectRequest getObjectRequest,
+ ResponseTransformer<GetObjectResponse, ReturnT> responseTransformer)
+ throws AwsServiceException, SdkClientException {
+ return delegate.getObject(getObjectRequest, responseTransformer);
Review Comment:
I was going to track the start ranges that were requested on each getObject and assert that the positions being retried are as expected but this ends up really complicating the tests for what I think is not too much additional value because actually the `testRead` we are inheriting verifies the contents actually being read in each `readAndCheck`. If there's a retry that's being performed to an incorrect position, this would mean that the contents would be incorrect and tests would fail and we know that the retry path is being exercised given the presence of the fuzzy stream client.
cc @nastra @danielcweeks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "edgarRd (via GitHub)" <gi...@apache.org>.
edgarRd commented on PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2321827281
@amogh-jahagirdar Any reason why not handling retries in `S3OutputStream` as well? We've internally seen issues with `UncheckedIOExceptions` during `putObject` for manifest objects. I believe https://github.com/apache/iceberg/pull/8221 aimed at addressing both input and output streams.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1726966943
##########
gradle/libs.versions.toml:
##########
@@ -37,6 +37,7 @@ delta-standalone = "3.1.0"
delta-spark = "3.2.0"
esotericsoftware-kryo = "4.0.3"
errorprone-annotations = "2.27.0"
+failsafe = "3.3.2"
Review Comment:
I like this a lot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Retry S3InputStream reads [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#issuecomment-2371767216
Thanks for the reviews @danielcweeks! Merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624919847
##########
aws/src/test/java/org/apache/iceberg/aws/s3/TestFuzzyS3InputStream.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.s3;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.spy;
+
+import dev.failsafe.FailsafeException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketTimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import javax.net.ssl.SSLException;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import software.amazon.awssdk.awscore.exception.AwsServiceException;
+import software.amazon.awssdk.core.ResponseInputStream;
+import software.amazon.awssdk.core.exception.SdkClientException;
+import software.amazon.awssdk.core.sync.RequestBody;
+import software.amazon.awssdk.core.sync.ResponseTransformer;
+import software.amazon.awssdk.services.s3.S3Client;
+import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
+import software.amazon.awssdk.services.s3.model.CreateBucketResponse;
+import software.amazon.awssdk.services.s3.model.GetObjectRequest;
+import software.amazon.awssdk.services.s3.model.GetObjectResponse;
+import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
+import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
+import software.amazon.awssdk.services.s3.model.PutObjectRequest;
+import software.amazon.awssdk.services.s3.model.PutObjectResponse;
+
+public class TestFuzzyS3InputStream extends TestS3InputStream {
+
+ private static final int DATA_SIZE = 3 * 1024 * 1024;
+
+ @ParameterizedTest
+ @MethodSource("retryableExceptions")
+ public void testReadWithFuzzyStreamRetrySucceed(IOException exception) throws Exception {
+ testRead(
+ fuzzyStreamClient(new AtomicInteger(3), exception), new S3FileIOProperties(), DATA_SIZE);
+ }
Review Comment:
This test takes way too long. That's primarily because it internally tests retrying on just read() (the non-buffered reads) which means every byte read will fail and be retried 2 times with a 500 ms in between. So essentially that's a second per byte.
I think what we can do is modularize further and only do buffered read tests + with a much smaller data size test the per byte read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1624930096
##########
gradle/libs.versions.toml:
##########
@@ -37,6 +37,7 @@ delta-standalone = "3.1.0"
delta-spark = "3.2.0"
esotericsoftware-kryo = "4.0.3"
errorprone-annotations = "2.27.0"
+failsafe = "3.3.2"
Review Comment:
This dependency is quite nice in that it's 0 dependency itself, has Apache licensing and I think there's more use cases in Iceberg to leverage it. For example, I think a lot of the complex logic in Tasks can be simplified.
Furthermore, there's some custom retry logic in JDBC connector which we couldn't use tasks for, but now we could use Failsafe. Wonder what others think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Retry S3InputStream reads [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1771783667
##########
core/src/main/java/org/apache/iceberg/io/RetryableInputStream.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import dev.failsafe.Failsafe;
+import dev.failsafe.FailsafeException;
+import dev.failsafe.RetryPolicy;
+import dev.failsafe.RetryPolicyBuilder;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketException;
+import java.net.SocketTimeoutException;
+import java.time.Duration;
+import java.util.List;
+import java.util.function.Supplier;
+import javax.net.ssl.SSLException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+/**
+ * RetryableInputStream wraps over an underlying InputStream and retries failures encountered when
+ * reading through the stream. On retries, the underlying streams will be reinitialized.
+ */
Review Comment:
Removing RetryableInputStream so we don't need this anymoe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, AWS: Add RetryableInputStream and use that in S3InputStream [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10433:
URL: https://github.com/apache/iceberg/pull/10433#discussion_r1747861096
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java:
##########
@@ -92,13 +93,13 @@ public void seek(long newPos) {
public int read() throws IOException {
Preconditions.checkState(!closed, "Cannot read: already closed");
positionStream();
-
+ int bytesRead = stream.read();
Review Comment:
It's actually also required for correctness because when we retry the input stream we retry from the current position. Previously in this method we advance the position before actually reading the single byte. If we left it as is, upon retry we would read from the position 1 byte ahead of where we needed to re-open the stream from. It's not a problem when there's no retries because if stream.read() fails, it would just fail.
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java:
##########
@@ -92,13 +93,13 @@ public void seek(long newPos) {
public int read() throws IOException {
Preconditions.checkState(!closed, "Cannot read: already closed");
positionStream();
-
+ int bytesRead = stream.read();
Review Comment:
It's actually also required for correctness because when we retry the input stream we retry from the current position. Previously in this method we advance the position before actually reading the single byte. If we left it as is, upon retry we would read from the position 1 byte ahead of where we needed to re-open the stream from. Note the current implementation is not a problem when there's no retries because if stream.read() fails, it would just fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org