You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/11/03 23:06:54 UTC

[GitHub] [pulsar] rivernate opened a new issue #8439: Include MD5 hash for server side validation when offloading ledgers

rivernate opened a new issue #8439:
URL: https://github.com/apache/pulsar/issues/8439


   **Is your enhancement request related to a problem? Please describe.**
   It doesn't appear that we currently include the `Content-MD5` when we upload the ledgers using JCloud. While we haven't seen any corruption from offloading so far, it would be nice to send the `Content-MD5` so that services that support validation could check the integrity of the upload
   
   **Describe the solution you'd like**
   Looking at the code here: 
   
   https://github.com/apache/pulsar/blob/ec2de258ce39b38311e92bab6882c198070d8975/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreManagedLedgerOffloader.java#L176
   
   It should be straight forward to add the `Content-MD5` to the ContentMetaData. I have checked that JCould supports this here:
   https://github.com/apache/jclouds/blob/94f09325ba1442ed64fc0a0eba7fe8b336f621ce/core/src/main/java/org/jclouds/io/MutableContentMetadata.java#L35
   
   And AWS supports the validation of the individual parts of the Multipart upload, see:
   https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html
   
   The biggest downside I can see is that we would need to buffer the InputStream in the broker to calculate the MD5Hash
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] rivernate commented on issue #8439: Include MD5 hash for server side validation when offloading ledgers

Posted by GitBox <gi...@apache.org>.
rivernate commented on issue #8439:
URL: https://github.com/apache/pulsar/issues/8439#issuecomment-725649124


   I'd be happy to work on it. It may be a while before I have time to do it though.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on issue #8439: Include MD5 hash for server side validation when offloading ledgers

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #8439:
URL: https://github.com/apache/pulsar/issues/8439#issuecomment-722777111


   @rivernate Are you interested in implementing this feature? I think we can add a config in the broker.conf to enable or disable the Content-MD5 for the ledger offload.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org