You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sqd (via GitHub)" <gi...@apache.org> on 2026/07/28 20:40:53 UTC
[PR] Core, AWS: Fix REST catalog hanging on permission errors [iceberg]
sqd opened a new pull request, #17400:
URL: https://github.com/apache/iceberg/pull/17400
Refreshing table or view metadata retries any read failure up to 20 times with exponential backoff (about 90 seconds) unless it is a NotFoundException. A permission error is permanent, so retrying can never succeed and only delays surfacing the real failure.
Behind a REST catalog, the server holds the loadTable request open for the whole retry loop, so clients hit their socket read timeout first and report SocketTimeoutException while the actual cause (for example an HDFS "Permission denied" on the metadata file) is only visible in server logs. Stuck requests also pin server worker threads, causing timeouts for unrelated tables.
Translate access-denied errors into ForbiddenException at the read boundary - HadoopFileIO on AccessControlException and S3FileIO on S3Exception with status 403 - and stop metadata refresh retries on ForbiddenException. RESTCatalogAdapter already maps ForbiddenException to HTTP 403 and REST clients map 403 back to ForbiddenException, so callers now receive the underlying permission error immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Fix REST catalog hanging on permission errors [iceberg]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] closed pull request #17400: Core, AWS: Fix REST catalog hanging on permission errors
URL: https://github.com/apache/iceberg/pull/17400
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Fix REST catalog hanging on permission errors [iceberg]
Posted by "xndai (via GitHub)" <gi...@apache.org>.
xndai commented on code in PR #17400:
URL: https://github.com/apache/iceberg/pull/17400#discussion_r3670602062
##########
aws/src/test/java/org/apache/iceberg/aws/s3/TestS3InputStream.java:
##########
@@ -44,22 +48,40 @@ public final class TestS3InputStream {
@BeforeEach
void before() {
- when(s3Client.getObject(any(GetObjectRequest.class), any(ResponseTransformer.class)))
- .thenReturn(inputStream);
s3InputStream = new S3InputStream(s3Client, mock());
}
@Test
void testReadFullyClosesTheStream() throws IOException {
+ when(s3Client.getObject(any(GetObjectRequest.class), any(ResponseTransformer.class)))
+ .thenReturn(inputStream);
+
s3InputStream.readFully(0, new byte[0]);
verify(inputStream).close();
}
@Test
void testReadTailClosesTheStream() throws IOException {
+ when(s3Client.getObject(any(GetObjectRequest.class), any(ResponseTransformer.class)))
+ .thenReturn(inputStream);
+
s3InputStream.readTail(new byte[0], 0, 0);
verify(inputStream).close();
}
+
+ @Test
+ void testAccessDeniedTranslatedToForbiddenException() {
+ when(s3Client.getObject(any(GetObjectRequest.class), any(ResponseTransformer.class)))
+ .thenThrow(
+ S3Exception.builder()
+ .statusCode(HttpStatusCode.FORBIDDEN)
+ .message("Access Denied")
+ .build());
+
+ assertThatThrownBy(() -> s3InputStream.readFully(0, new byte[0]))
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("Access Denied");
Review Comment:
Should we validate message content here? The exception message could change in the future.
##########
core/src/test/java/org/apache/iceberg/TestBaseMetastoreTableOperations.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.io.FileIO;
+import org.junit.jupiter.api.Test;
+
+public class TestBaseMetastoreTableOperations {
Review Comment:
Class name is very general. But you are only testing the retries on transient error and un-recoverable errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Fix REST catalog hanging on permission errors [iceberg]
Posted by "xndai (via GitHub)" <gi...@apache.org>.
xndai commented on code in PR #17400:
URL: https://github.com/apache/iceberg/pull/17400#discussion_r3754582773
##########
core/src/test/java/org/apache/iceberg/TestBaseMetastoreTableOperations.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.io.FileIO;
+import org.junit.jupiter.api.Test;
+
+public class TestBaseMetastoreTableOperations {
Review Comment:
LGTM. Thanks for fixing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Fix REST catalog hanging on permission errors [iceberg]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on PR #17400:
URL: https://github.com/apache/iceberg/pull/17400#issuecomment-6729212623
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Fix REST catalog hanging on permission errors [iceberg]
Posted by "sqd (via GitHub)" <gi...@apache.org>.
sqd commented on PR #17400:
URL: https://github.com/apache/iceberg/pull/17400#issuecomment-5591060897
@nastra @amogh-jahagirdar Would you mind taking a look at this PR please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Fix REST catalog hanging on permission errors [iceberg]
Posted by "sqd (via GitHub)" <gi...@apache.org>.
sqd commented on code in PR #17400:
URL: https://github.com/apache/iceberg/pull/17400#discussion_r3674941750
##########
aws/src/test/java/org/apache/iceberg/aws/s3/TestS3InputStream.java:
##########
@@ -44,22 +48,40 @@ public final class TestS3InputStream {
@BeforeEach
void before() {
- when(s3Client.getObject(any(GetObjectRequest.class), any(ResponseTransformer.class)))
- .thenReturn(inputStream);
s3InputStream = new S3InputStream(s3Client, mock());
}
@Test
void testReadFullyClosesTheStream() throws IOException {
+ when(s3Client.getObject(any(GetObjectRequest.class), any(ResponseTransformer.class)))
+ .thenReturn(inputStream);
+
s3InputStream.readFully(0, new byte[0]);
verify(inputStream).close();
}
@Test
void testReadTailClosesTheStream() throws IOException {
+ when(s3Client.getObject(any(GetObjectRequest.class), any(ResponseTransformer.class)))
+ .thenReturn(inputStream);
+
s3InputStream.readTail(new byte[0], 0, 0);
verify(inputStream).close();
}
+
+ @Test
+ void testAccessDeniedTranslatedToForbiddenException() {
+ when(s3Client.getObject(any(GetObjectRequest.class), any(ResponseTransformer.class)))
+ .thenThrow(
+ S3Exception.builder()
+ .statusCode(HttpStatusCode.FORBIDDEN)
+ .message("Access Denied")
+ .build());
+
+ assertThatThrownBy(() -> s3InputStream.readFully(0, new byte[0]))
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("Access Denied");
Review Comment:
This is not testing what exact words are returned from awssdk. This is testing that whatever the library returns will get propagated so the error is clearer to the client. I extrapolated this to a const to make the intention clearer.
##########
core/src/test/java/org/apache/iceberg/TestBaseMetastoreTableOperations.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.io.FileIO;
+import org.junit.jupiter.api.Test;
+
+public class TestBaseMetastoreTableOperations {
Review Comment:
Good point. I'll change this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Fix REST catalog hanging on permission errors [iceberg]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on PR #17400:
URL: https://github.com/apache/iceberg/pull/17400#issuecomment-6551078423
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org