You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "xxubai (via GitHub)" <gi...@apache.org> on 2025/06/17 08:22:19 UTC
[I] [Bug]: Cannot expire historic files without upperbounds [amoro]
xxubai opened a new issue, #3622:
URL: https://github.com/apache/amoro/issues/3622
### What happened?
In som cases, the pos-delete doesn't include any metrics. Data expiring cannot extract timestamp which don't include upperbounds map
### Affects Versions
master
### What table formats are you seeing the problem on?
Iceberg
### What engines are you seeing the problem on?
AMS
### How to reproduce
_No response_
### Relevant log output
```shell
java.lang.IllegalStateException: Connection pool shut down │
│ at org.apache.http.util.Asserts.check(Asserts.java:34) ~[httpcore-4.4.13.jar:4.4.13] │
│ at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) ~[httpclient-4.5.13.jar:4.5.13] │
│ at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:75) ~[apache-client-2.24.12.jar:?] │
│ at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$InstrumentedHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:57) ~[apache-client-2.24.12.jar:?] │
│ at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) ~[httpclient-4.5.13.jar:4.5.13] │
│ at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13] │
│ at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13] │
│ at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13] │
│ at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13] │
│ at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) ~[apache-client-2.24.12.jar:?] │
│ at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254) ~[apache-client-2.24.12.jar:?] │
│ at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104) ~[apache-client-2.24.12.jar:?] │
│ at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231) ~[apache-client-2.24.12.jar:?] │
│ at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228) ~[apache-client-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:99) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:224) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[sdk-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53) ~[aws-core-2.24.12.jar:?] │
│ at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(DefaultS3Client.java:6319) ~[s3-2.24.12.jar:?] │
│ at org.apache.iceberg.aws.s3.BaseS3File.getObjectMetadata(BaseS3File.java:85) ~[iceberg-aws-1.7.2.jar:?] │
│ at org.apache.iceberg.aws.s3.S3InputFile.getLength(S3InputFile.java:77) ~[iceberg-aws-1.7.2.jar:?] │
│ at org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:102) ~[iceberg-core-1.7.2.jar:?] │
│ at org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:77) ~[iceberg-core-1.7.2.jar:?] │
│ at org.apache.iceberg.io.CloseableIterable$7$1.<init>(CloseableIterable.java:188) ~[iceberg-api-1.7.2.jar:?] │
│ at org.apache.iceberg.io.CloseableIterable$7.iterator(CloseableIterable.java:187) ~[iceberg-api-1.7.2.jar:?] │
│ at org.apache.iceberg.io.CloseableIterable$7$1.<init>(CloseableIterable.java:188) ~[iceberg-api-1.7.2.jar:?] │
│ at org.apache.iceberg.io.CloseableIterable$7.iterator(CloseableIterable.java:187) ~[iceberg-api-1.7.2.jar:?] │
│ at org.apache.iceberg.io.CloseableIterable$ConcatCloseableIterable$ConcatCloseableIterator.hasNext(CloseableIterable.java:257) ~[iceberg-api-1.7.2.jar:?] │
│ at java.lang.Iterable.forEach(Iterable.java:74) ~[?:?] │
│ at org.apache.amoro.server.utils.IcebergTableUtil.getAllManifestFiles(IcebergTableUtil.java:213) ~[amoro-ams-0.8.0-incubating.jar:0.8.0-incubating] │
│ at org.apache.amoro.server.optimizing.maintainer.IcebergTableMaintainer.getValidMetadataFiles(IcebergTableMaintainer.java:589) ~[amoro-ams-0.8.0-incubating.jar:0.8.0-incubating]
```
### Anything else
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Cannot expire historic files without upperbounds [amoro]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai closed issue #3622: [Bug]: Cannot expire historic files without upperbounds
URL: https://github.com/apache/amoro/issues/3622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org