[I] [object_store] Support adding MD5 checksum headers to validate object integrity [arrow-rs]

["tzembo (via GitHub)" <gi...@apache.org>]

tzembo opened a new issue, #6914:
URL: https://github.com/apache/arrow-rs/issues/6914

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   I'd like verify the integrity of uploaded objects (using some kind of checksum) across all three cloud providers.
   
   Currently, the S3 implementation allows setting an AmazonS3 configuration value that attaches a `x-amz-checksum-sha256` header to PUT requests against the store. However:
   - This is only available for AWS (and SHA256).
       - AWS supports other checksum algorithms (MD5, SHA1, CRC32, CRC32C)
       - Azure supports MD5 and CRC64
       - GCP supports MD5 and CRC32C
   - This requires another pass to calculate the checksum value, which the user of this library may already have computed in another context.
   
   **Describe the solution you'd like**
   
   I'm proposing that we add a `Checksum` attribute which specifies a `ChecksumAlgorithm` enum. The value for this attribute would be the base64-encoded checksum value. For now, MD5 can be the only supported checksum algorithm (which all three cloud providers support via the `Content-MD5` header). The value for this algorithm is a base64-encoded 128-bit digest.
   
   ```
   pub enum Attribute {
       ...
       /// Provides a checksum used to verify object data integrity
       Checksum(ChecksumAlgorithm),
   }
   
   pub enum ChecksumAlgorithm {
       MD5,
   }
   ```
   
   **Describe alternatives you've considered**
   
   I considered implementing this for more checksum algorithms, but I'm starting with MD5 because it's the only one supported by all three cloud providers. In the future, we could extend this to support additional checksum algorithms (e.g. CRC32C). However:
   - stores that do not support a particular error would need to return an error
   - stores would ideally be able to "report" their supported checksum algorithms (for usability reasons)
   - (AWS) we'd need to figure out how a SHA-256 checksum provided via attributes interacts with the one provided via config
   
   I considered calling the attribute `ContentMD5` but decided to make it a bit more generic to support additional checksums in the future.
   
   **Additional context**
   
   I can put up a PR for this issue.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [object_store] Support adding MD5 checksum headers to validate object integrity [arrow-rs]

["jstlaurent (via GitHub)" <gi...@apache.org>]

jstlaurent commented on issue #6914:
URL: https://github.com/apache/arrow-rs/issues/6914#issuecomment-2598977301

   This would also be supported by Cloudflare R2 buckets, through their [S3-compatible API](https://developers.cloudflare.com/r2/api/s3/api/). That would be fantastic, since that's the only checksum R2 supports.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org