[jira] [Created] (JAMES-4137) Upgrade S3 SDK to 2.30.x onward version

["Tran Hong Quan (Jira)" <se...@james.apache.org>]

Tran Hong Quan created JAMES-4137:
-------------------------------------

             Summary: Upgrade S3 SDK to 2.30.x onward version
                 Key: JAMES-4137
                 URL: https://issues.apache.org/jira/browse/JAMES-4137
             Project: James Server
          Issue Type: Improvement
          Components: Blob
            Reporter: Tran Hong Quan


In S3 SDK version 2.30.0, AWS team introduced a breaking change in the integrity check in the driver:
- Checksum is always required by default by AWS SDK >= 2.30.x
- AWS SDK >= 2.30.x does not support MD5 checksum anymore (which 3rd party S3 storages rely on), but relies on CRC32 checksum.
cf: [https://github.com/aws/aws-sdk-java-v2/discussions/5802]

This breaks the compatibility with 3rd party object storage(s) that still rely on MD5 checksum. We failed to upgrade the SDK in the past: [https://github.com/apache/james-project/pull/2638.]

It seems after receiving a lot of complaints from the community (Apache Hadoop, Apache Iceberg, Apache Spark...), AWS has agreed to have an option to maintain backward compatibility with S3-compatible storages. cf [https://github.com/aws/aws-sdk-java-v2/pull/6055.|https://github.com/aws/aws-sdk-java-v2/pull/6055]

For example, Apache Iceberg is starting to adopt the backward compatibility option: [https://github.com/apache/iceberg/pull/12264.]

Let's try to adopt and test this on our side, with care.

 

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org