You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@libcloud.apache.org by GitBox <gi...@apache.org> on 2020/01/06 12:01:41 UTC
[GitHub] [libcloud] jfharden opened a new issue #1401: Uploading to s3
encrypted with a custom KMS key fails
jfharden opened a new issue #1401: Uploading to s3 encrypted with a custom KMS key fails
URL: https://github.com/apache/libcloud/issues/1401
## Summary
When uploading files to s3 where the bucket has server side encryption with a custom KMS key the upload fails
## Detailed Information
Python 3.7
Libcloud: 2.8.0
OS: Amazon Linux 2
Libcloud is inspecting the etag expecting it to be the md5sum of the uploaded data object but this is not always the case (where using a custom KMS key, or doing a multi-part upload): https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html
> Objects created by the PUT Object, POST Object, or Copy operation, or through the AWS Management Console, and are encrypted by SSE-C or SSE-KMS, have ETags that are not an MD5 digest of their object data.
>
> If an object is created by either the Multipart Upload or Part Copy operation, the ETag is not an MD5 digest, regardless of the method of encryption.
the s3 storage driver is comparing the etag to the hash calculated locally on the streamed file (https://github.com/apache/libcloud/blob/trunk/libcloud/storage/drivers/s3.py#L850) but where the etag is not an md5 hash of the file this will always fail
I've included a stack trace below from cassandra-medusa (https://github.com/thelastpickle/cassandra-medusa) which led to me investigating this problem
```
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/medusa/backup.py", line 274, in main
cassandra, node_backup, storage, differential_mode, config)
File "/usr/local/lib/python3.7/site-packages/medusa/backup.py", line 320, in do_backup
num_files = backup_snapshots(storage, manifest, node_backup, node_backup_cache, snapshot)
File "/usr/local/lib/python3.7/site-packages/medusa/backup.py", line 388, in backup_snapshots
manifest_objects = storage.storage_driver.upload_blobs(needs_backup, dst_path)
File "/usr/local/lib/python3.7/site-packages/medusa/storage/s3_storage.py", line 95, in upload_blobs
multi_part_upload_threshold=int(self.config.multi_part_upload_threshold),
File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py", line 87, in upload_blobs
return job.execute(list(src))
File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py", line 51, in execute
return list(executor.map(self.with_storage, iterables))
File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 598, in result_iterator
yield fs.pop().result()
File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 435, in result
return self.__get_result()
File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/lib64/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py", line 60, in with_storage
return self.func(self.storage, connection, iterable)
File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py", line 83, in <lambda>
storage, connection, src_file, dest, bucket, multi_part_upload_threshold
File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py", line 119, in __upload_file
obj = _upload_single_part(connection, src, bucket, full_object_name)
File "/usr/local/lib/python3.7/site-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/local/lib/python3.7/site-packages/retrying.py", line 212, in call
raise attempt.get()
File "/usr/local/lib/python3.7/site-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/local/lib/python3.7/site-packages/six.py", line 696, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py", line 127, in _upload_single_part
os.fspath(src), container=bucket, object_name=object_name
File "/usr/local/lib/python3.7/site-packages/libcloud/storage/drivers/s3.py", line 492, in upload_object
storage_class=ex_storage_class)
File "/usr/local/lib/python3.7/site-packages/libcloud/storage/drivers/s3.py", line 854, in _put_object
object_name=object_name, driver=self)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] Kami commented on issue #1401: Uploading to s3 encrypted
with a custom KMS key fails
Posted by GitBox <gi...@apache.org>.
Kami commented on issue #1401: Uploading to s3 encrypted with a custom KMS key fails
URL: https://github.com/apache/libcloud/issues/1401#issuecomment-573993526
@jfharden Thanks again for reporting this.
#1406 which should fix this issue has been merged. Can you please confirm?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] Kami commented on issue #1401: Uploading to s3 encrypted
with a custom KMS key fails
Posted by GitBox <gi...@apache.org>.
Kami commented on issue #1401: Uploading to s3 encrypted with a custom KMS key fails
URL: https://github.com/apache/libcloud/issues/1401#issuecomment-572870832
Thanks for reporting this issue.
It looks like we may need to add some logic to skip comparing of E-tag header if the value is not an MD5 hash.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] Kami closed issue #1401: Uploading to s3 encrypted with
a custom KMS key fails
Posted by GitBox <gi...@apache.org>.
Kami closed issue #1401: Uploading to s3 encrypted with a custom KMS key fails
URL: https://github.com/apache/libcloud/issues/1401
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] Kami commented on issue #1401: Uploading to s3 encrypted
with a custom KMS key fails
Posted by GitBox <gi...@apache.org>.
Kami commented on issue #1401: Uploading to s3 encrypted with a custom KMS key fails
URL: https://github.com/apache/libcloud/issues/1401#issuecomment-572873471
I was able to reproduce this issue using ``upload_object``, but not ``upload_object_via_stream`` method.
This makes sense since the upload_object_via_stream utilizes multi part upload which returns ETag which doesn't contain MD5 digest.
For now the easiest workaround would be to simply pass ``verify_hash=False`` argument to the ``upload_object`` method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] Kami commented on issue #1401: Uploading to s3 encrypted
with a custom KMS key fails
Posted by GitBox <gi...@apache.org>.
Kami commented on issue #1401: Uploading to s3 encrypted with a custom KMS key fails
URL: https://github.com/apache/libcloud/issues/1401#issuecomment-572883655
#1406 should fix this.
I verified / tested the change locally, but it would be good if you can confirm it as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services