You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "dmytroi11i (via GitHub)" <gi...@apache.org> on 2025/03/19 15:33:28 UTC
[I] Commit fails because S3 checksum does match when calling PutObject [iceberg]
dmytroi11i opened a new issue, #12577:
URL: https://github.com/apache/iceberg/issues/12577
### Query engine
**Iceberg Version**: 1.6.0, 1.7.0
**Spark Version**: 3.3
**Java Version**: 1.8
### Question
Hi, Iceberg community!
**Iceberg Version**: 1.6.0, 1.7.0
**Spark Version**: 3.3
**Java Version**: 1.8
I'm attempting to enable the Glue Data Catalog to store Iceberg tables using Spark. However, I'm encountering an issue where a simple job fails when Spark tries to write a new manifest file to S3. Based on the stack trace, it seems to be a potential bug in AWS SDK v2.
1. Is there a way to disable the checksum feature in Iceberg?
2. I've also attempted to rebuild iceberg-spark-runtime-3.3.2.12 for Spark 3.3 and Java 8, but it was unsuccessful. Could you provide any insights or suggestions on this process?
3. Does Iceberg 1.7 support Java 8? If so, are there any specific configurations or settings required to ensure compatibility?
Any guidance or suggestions would be greatly appreciated.
Thank you!
`software.amazon.awssdk.core.exception.RetryableException: Data read has a different checksum than expected. Was 0xd41d8cd98f00b204e9800998ecf8427e, but expected 0x3c7b372cfb40259fce2f731345069344. This commonly means that the data was corrupted between the client and service. Note: Despite this error, the upload still completed and was persisted in S3.
at software.amazon.awssdk.core.exception.RetryableException$BuilderImpl.build(RetryableException.java:99)
at software.amazon.awssdk.core.exception.RetryableException.create(RetryableException.java:35)
at software.amazon.awssdk.services.s3.internal.checksums.ChecksumsEnabledValidator.validatePutObjectChecksum(ChecksumsEnabledValidator.java:173)
at software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor.afterUnmarshalling(SyncChecksumValidationInterceptor.java:95)
at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.lambda$afterUnmarshalling$11(ExecutionInterceptorChain.java:156)
at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.reverseForEach(ExecutionInterceptorChain.java:220)
at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.afterUnmarshalling(ExecutionInterceptorChain.java:156)
at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$runAfterUnmarshallingInterceptors$5(BaseClientHandler.java:174)
at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$composeResponseFunctions$9(BaseClientHandler.java:337)
at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:283)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:38)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.executeRequest(RetryableStage2.java:93)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:56)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:10595)
at org.apache.iceberg.aws.s3.S3OutputStream.completeUploads(S3OutputStream.java:438)
at org.apache.iceberg.aws.s3.S3OutputStream.close(S3OutputStream.java:265)
at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:320)
at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149)
at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233)
at org.apache.iceberg.TableMetadataParser.$closeResource(TableMetadataParser.java:132)
at org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:132)
at org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:115)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:163)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadataIfRequired(BaseMetastoreTableOperations.java:153)
at org.apache.iceberg.aws.glue.GlueTableOperations.doCommit(GlueTableOperations.java:155)
at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:128)
at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:201)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Commit fails because S3 checksum does match when calling PutObject [iceberg]
Posted by "kien-truong (via GitHub)" <gi...@apache.org>.
kien-truong commented on issue #12577:
URL: https://github.com/apache/iceberg/issues/12577#issuecomment-2739203400
BTW `0xd41d8cd98f00b204e9800998ecf8427e` is a special value, it's the MD5 of an empty string, which indicates that something else has read the response before the Iceberg plugin does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Commit fails because S3 checksum does match when calling PutObject [iceberg]
Posted by "dmytroi11i (via GitHub)" <gi...@apache.org>.
dmytroi11i commented on issue #12577:
URL: https://github.com/apache/iceberg/issues/12577#issuecomment-2740324032
Hi, @kien-truong,
Thank you for pointing me in the right direction. I hadn't noticed that it could be an empty file. I added the Spark configuration `spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem`, and it worked. I believe, in this case, Spark uses Hadoop's S3A connector, which does not rely on the AWS SDK's `execution.interceptors`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Commit fails because S3 checksum does match when calling PutObject [iceberg]
Posted by "kien-truong (via GitHub)" <gi...@apache.org>.
kien-truong commented on issue #12577:
URL: https://github.com/apache/iceberg/issues/12577#issuecomment-2739178676
I've encountered this before when using Iceberg in combination with an outdate version of `msk-config-providers` that use AWS Java SDK version 2.18.8.
This version of AWS Java SDK use `execution.interceptors`, which conflicts with the AWS Java SDK used by Iceberg, because it intercepted and read the S3 response before Iceberg plugin, leading to checksum error.
You can solve this by rebuilding `msk-config-providers` trunk with a more modern AWS Java SDK version that does not use `execution.interceptors` (`>2.18.20`)
https://github.com/aws-samples/msk-config-providers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Commit fails because S3 checksum does match when calling PutObject [iceberg]
Posted by "dmytroi11i (via GitHub)" <gi...@apache.org>.
dmytroi11i closed issue #12577: Commit fails because S3 checksum does match when calling PutObject
URL: https://github.com/apache/iceberg/issues/12577
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org