You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ochanism (via GitHub)" <gi...@apache.org> on 2025/10/29 06:53:06 UTC

[I] Iceberg 1.10.0: Unexpected S3 checksum validation despite s3.checksum-enabled=false [iceberg]

ochanism opened a new issue, #14439:
URL: https://github.com/apache/iceberg/issues/14439

   ### Apache Iceberg version
   
   1.10.0 (latest release)
   
   ### Query engine
   
   None
   
   ### Please describe the bug 🐞
   
   Starting in Iceberg 1.10.0, checksum validation began to be enforced for S3 PutObject operations. This happens even when the Iceberg config `s3.checksum-enabled` is set to its default of false.
   
   At first glance this looks like an Iceberg bug, but it actually stems from an AWS SDK policy change: the AWS client’s `AWS_REQUEST_CHECKSUM_CALCULATION` default was changed to `WHEN_SUPPORTED`. With that default, the client automatically performs checksum calculation whenever the service supports it, even if the caller didn’t explicitly enable checksums.
   
   Proposal:
   When initializing the S3 client in Iceberg, set `AWS_REQUEST_CHECKSUM_CALCULATION` to `WHEN_REQUIRED` so that checksums are only calculated when strictly required. Then allow users to control checksum usage via Iceberg’s `s3.checksum-enabled` setting.
   
   This way:
    - Users who want checksums can enable them explicitly via s3.checksum-enabled.
    - Users who keep the default (false) won’t incur checksum calculation/validation unexpectedly after upgrading to 1.10.0.
   
   What do you think about this approach?
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Iceberg 1.10.0: Unexpected S3 checksum validation despite s3.checksum-enabled=false [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14439:
URL: https://github.com/apache/iceberg/issues/14439#issuecomment-6181465721

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Iceberg 1.10.0: Unexpected S3 checksum validation despite s3.checksum-enabled=false [iceberg]

Posted by "ochanism (via GitHub)" <gi...@apache.org>.
ochanism commented on issue #14439:
URL: https://github.com/apache/iceberg/issues/14439#issuecomment-3461039665

   @ebyhr 
   That setting is for enabling legacy MD5; it’s not an option to turn checksum functionality on or off.
   Right now, the only way to disable checksums is to use the AWS client’s AWS_REQUEST_CHECKSUM_CALCULATION option.
   Checksum behavior cannot be controlled via Iceberg config.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Iceberg 1.10.0: Unexpected S3 checksum validation despite s3.checksum-enabled=false [iceberg]

Posted by "kamijin-fanta (via GitHub)" <gi...@apache.org>.
kamijin-fanta commented on issue #14439:
URL: https://github.com/apache/iceberg/issues/14439#issuecomment-3561096219

   Like @ochanism, I’ve also been investigating the behavior with S3-compatible object storage. In conclusion, setting `client.legacy-md5-plugin-enabled` has been very effective in my case.
   
   With this enabled, the `Content-MD5` header is included again as before, and in some requests the `Content-Encoding: aws-chunked` header is removed so that they become plain (non-chunked) requests. It also seems that specifying `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` has limited effect, because some of the S3 APIs that Iceberg calls require checksum calculation at the endpoint level anyway.
   
   Of course, the behavior will likely vary depending on the storage implementation. However, I’m leaving this here in case it helps others resolve similar issues.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Iceberg 1.10.0: Unexpected S3 checksum validation despite s3.checksum-enabled=false [iceberg]

Posted by "patricklucas (via GitHub)" <gi...@apache.org>.
patricklucas commented on issue #14439:
URL: https://github.com/apache/iceberg/issues/14439#issuecomment-4266629034

   I also lost a bit of time to this the past two days, attempting to write to GCS.
   
   For me, setting the `client.legacy-md5-plugin-enabled` property has no effect; only setting the `AWS_REQUEST_CHECKSUM_CALCULATION` env var allows PUTs to succeed.
   
   Is there any workaround for this that doesn't involve asking users to set environment variables?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Iceberg 1.10.0: Unexpected S3 checksum validation despite s3.checksum-enabled=false [iceberg]

Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on issue #14439:
URL: https://github.com/apache/iceberg/issues/14439#issuecomment-3712129356

   we moved to having checksum validation in hadoop 3.4.3 as follows
   https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java#L217
   
   ```
       if (parameters.isMd5HeaderEnabled()) {
         LOG.debug("MD5 header enabled");
         builder.addPlugin(LegacyMd5Plugin.create());
       }
   
       //when to calculate request checksums.
       final RequestChecksumCalculation checksumCalculation =
           parameters.isChecksumCalculationEnabled()
               ? RequestChecksumCalculation.WHEN_SUPPORTED
               : RequestChecksumCalculation.WHEN_REQUIRED;
       LOG.debug("Using checksum calculation policy: {}", checksumCalculation);
       builder.requestChecksumCalculation(checksumCalculation);
   
       // response checksum validation. Slow, even with CRC32 checksums.
       final ResponseChecksumValidation checksumValidation;
       checksumValidation = parameters.isChecksumValidationEnabled()
           ? ResponseChecksumValidation.WHEN_SUPPORTED
           : ResponseChecksumValidation.WHEN_REQUIRED;
       LOG.debug("Using checksum validation policy: {}", checksumValidation);
       builder.responseChecksumValidation(checksumValidation);
   ```
   with defaults of: MD5 enabled; checksum validation down to WHEN_REQUIRED
   
   this works with all external stores tested (dell, gcs). Enabling the MD5 plugin did cause problems with S3-express, but the SDK team came up with a workaround (https://github.com/aws/aws-sdk-java-v2/issues/6459). 
   
   to summarise, on AWS SDK 2.35.4+
   * restoring md5 plugin works everywhere, aws and non-aws
   * changing request and response checkums to when required works everywhere
   
   Given the connections to all production systems are via TLS, adding extra checksums seems, well, needless. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Iceberg 1.10.0: Unexpected S3 checksum validation despite s3.checksum-enabled=false [iceberg]

Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on issue #14439:
URL: https://github.com/apache/iceberg/issues/14439#issuecomment-3461006677

   What if you enable `client.legacy-md5-plugin-enabled` property? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Iceberg 1.10.0: Unexpected S3 checksum validation despite s3.checksum-enabled=false [iceberg]

Posted by "patricklucas (via GitHub)" <gi...@apache.org>.
patricklucas commented on issue #14439:
URL: https://github.com/apache/iceberg/issues/14439#issuecomment-4266666004

   Downgrading to the last pre-2.30 release of the AWS Java SDK (2.29.52) works for me for the moment, but a proper solution in iceberg to configure this correctly would definitely be ideal.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Iceberg 1.10.0: Unexpected S3 checksum validation despite s3.checksum-enabled=false [iceberg]

Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on issue #14439:
URL: https://github.com/apache/iceberg/issues/14439#issuecomment-4267089860

   @patricklucas use the s3a connector with the documented settings?
   https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/third_party_stores.html
   
   overall this sdk change was pretty traumatic, and it did eventually get the statement from the sdk team "we don't test on third party stores". Which means projects that do upgrade frequently are the first place to hit these problems.
   
   
   @kamijin-fanta : which requests were being chunked? this matters because the s3 catalog signer doesn't sign chunked requests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Iceberg 1.10.0: Unexpected S3 checksum validation despite s3.checksum-enabled=false [iceberg]

Posted by "patricklucas (via GitHub)" <gi...@apache.org>.
patricklucas commented on issue #14439:
URL: https://github.com/apache/iceberg/issues/14439#issuecomment-6187908633

   This issue still has PRs #17177 and #17178 open.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org