You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@libcloud.apache.org by GitBox <gi...@apache.org> on 2020/01/26 23:15:09 UTC

[GitHub] [libcloud] wabson opened a new issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket

wabson opened a new issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket
URL: https://github.com/apache/libcloud/issues/1417
 
 
   ## Summary
   
   When running on Ubuntu it is not possible to upload an Excel macro-enabled file with the suffix `.xlsm` to a Google Storage bucket, the upload fails with an `InvalidCredsError` exception raised after calling `upload_object_via_stream()`
   
   ## Detailed Information
   
   I first noticed this in a Django app running on Heroku, and have since reproduced the issue with libcloud only, on standalone Ubuntu 16.04 and 18.04 AMIs. The issue is triggered on Python2 and Python3 but does not occur on my OSX dev enviroment nor in testing on Amazon Linux. It does not reproduce when uploading to S3 and seems specific to GCP.
   
   Steps to reproduce:
   
   Create a bucket in Google Cloud Storage and in Storage Settings add an Access key for your account, under Interoperability.
   
   Install `apache-libcloud==2.8.0` via `pip`
   
   Attempt to upload a test file to the bucket as in the following example
   
   ```
   from io import BytesIO
   from libcloud.storage.types import Provider
   from libcloud.storage.providers import get_driver
   
   cls = get_driver(Provider.GOOGLE_STORAGE)
   driver = cls('my-access-key', 'my-secret')
   container = driver.get_container(container_name='my-test-bucket')
   obj = driver.upload_object_via_stream(iterator=BytesIO(b'hello'), container=container, object_name='test1.xlsm')
   ```
   
   This results in the following stack trace
   
   ```
   Traceback (most recent call last):
     File "upload.py", line 9, in <module>
       obj = driver.upload_object_via_stream(iterator=BytesIO(b'hello'), container=container, object_name='test1.xlsm')
     File "/home/ubuntu/.local/lib/python3.6/site-packages/libcloud/storage/drivers/s3.py", line 698, in upload_object_via_stream
       storage_class=ex_storage_class)
     File "/home/ubuntu/.local/lib/python3.6/site-packages/libcloud/storage/drivers/s3.py", line 842, in _put_object
       headers=headers, file_path=file_path, stream=stream)
     File "/home/ubuntu/.local/lib/python3.6/site-packages/libcloud/storage/base.py", line 640, in _upload_object
       response.parse_error()
     File "/home/ubuntu/.local/lib/python3.6/site-packages/libcloud/storage/drivers/s3.py", line 123, in parse_error
       raise InvalidCredsError(self.body)
     File "/home/ubuntu/.local/lib/python3.6/site-packages/libcloud/common/base.py", line 293, in body
       return self.response.body
     File "/home/ubuntu/.local/lib/python3.6/site-packages/libcloud/common/base.py", line 286, in response
       self.parse_error()
     File "/home/ubuntu/.local/lib/python3.6/site-packages/libcloud/storage/drivers/s3.py", line 123, in parse_error
       raise InvalidCredsError(self.body)
   libcloud.common.types.InvalidCredsError: b"<?xml version='1.0' encoding='UTF-8'?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message><StringToSign>PUT\n\napplication/vnd.ms-excel.sheet.macroEnabled.12\nSun, 26 Jan 2020 22:45:12 GMT\nx-goog-storage-class:STANDARD\n/hasler-race-entry-test/test1.xlsm</StringToSign></Error>"
   ```
   
   Changing the file extension in the object name to something more well-known such as `.xlsx` allows the operation to complete just fine, so I know the access credentials are working OK and it seems to be something to do with the mimetype.
   
   I notice that if I log in to the GCP Storage UI and upload a `.xlsm` file via the browser, it shows as having the type `application/vnd.ms-excel.sheet.macroenabled.12`, that is there is a small difference in case, I wonder if this is significant.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [libcloud] Kami commented on issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket

Posted by GitBox <gi...@apache.org>.

Kami commented on issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket
URL: https://github.com/apache/libcloud/issues/1417#issuecomment-578721952
 
 
   I tried to dig in and find the root cause, but I was unable to find one.
   
   This issue only appears if all the following conditions are met:
   
   1. Ineroperability authentication method is used
   2. Object name ends with ``.xlsm`` (e.g. ``test1.xlsm``)
   3. Content-Type header is ``application/vnd.ms-excel.sheet.macroEnabled.12``
   
   If any of those three things are not met, it will work fine.
   
   The signature itself looks fine, so I wonder if it's some bug / edge case when using Google Storage API in Interoparability mode?
   
   Having said that, probably the best course of action / workaround would be to simply use non-compatibility method with service account credentials.
   
   That's the preferred method in any case.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [libcloud] Kami closed issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket

Posted by GitBox <gi...@apache.org>.

Kami closed issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket
URL: https://github.com/apache/libcloud/issues/1417
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [libcloud] Kami edited a comment on issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket

Posted by GitBox <gi...@apache.org>.

Kami edited a comment on issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket
URL: https://github.com/apache/libcloud/issues/1417#issuecomment-578721952

I tried to dig in and find the root cause, but I was unable to find one.

This issue only appears if all the following conditions are met:

1. Ineroperability authentication method is used
2. Object name ends with ``.xlsm`` (e.g. ``test1.xlsm``)
3. Content-Type header is ``application/vnd.ms-excel.sheet.macroEnabled.12``

If any of those three things are not met, it will work fine.

What makes it look even more bizarre is that it works if explicitly specify ``extra['content_type'] = 'application/vnd.ms-excel.sheet.macroEnabled.12' even though the actual signature it calculates it will be exactly the same (I verified all the values which are used to generate the signature are the same, except the ``Date`` one).

The signature itself looks fine, so I wonder if it's some bug / edge case when using Google Storage API in Interoparability mode?

Having said that, probably the best course of action / workaround would be to simply use non-compatibility method with service account credentials.

That's the preferred method in any case.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [libcloud] Kami commented on issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket

Posted by GitBox <gi...@apache.org>.

Kami commented on issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket
URL: https://github.com/apache/libcloud/issues/1417#issuecomment-578711601
 
 
   EDIT: I was able to reproduce the issue using interoperability credentials + using ``application/vnd.ms-excel.sheet.macroEnabled.12`` Content-Type value.
   
   If I use regular GCP service account credentials, it works fine.
   
   I assume it's something related to how we calculate the signature when interoperability mode is used. It's likely that some special values (e.g. ``.`` are not correctly handled / escaped).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [libcloud] Kami edited a comment on issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket

Posted by GitBox <gi...@apache.org>.

Kami edited a comment on issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket
URL: https://github.com/apache/libcloud/issues/1417#issuecomment-578709089
 
 
   Thanks for reporting this.
   
   I tried to reproduce it with the code you provided and it works fine for me using the latest trunk version.
   
   It could indeed be something with mime type on your system though. On mine, it determines mime to be ``application/vnd.ms-excel.sheet.macroEnabled.12`` and it works fine.
   
   What kind of mime type is returned on your environment? Checking the output of ``LIBCLOUD_DEBUG=/dev/stderr`` might also provide some clues.
   
   Having said that, it also works for me if I set ``Content-Type`` header to arbitrary value - the upload still succeeds.
   
   ---
   
   EDIT: If the issue is indeed related to content type you could try specifying a custom one using ``extra['content_type']`` value.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [libcloud] Kami commented on issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket

Posted by GitBox <gi...@apache.org>.

Kami commented on issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket
URL: https://github.com/apache/libcloud/issues/1417#issuecomment-578709089
 
 
   Thanks for reporting this.
   
   I tried to reproduce it with the code you provided and it works fine for me using the latest trunk version.
   
   It could indeed be something with mime type on your system though. On mine, it determines mime to be ``application/vnd.ms-excel.sheet.macroEnabled.12`` and it works fine.
   
   What kind of mime type is returned on your environment?
   
   Having said that, it also works for me if I set ``Content-Type`` header to arbitrary value - the upload still succeeds.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [libcloud] Kami commented on issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket

Posted by GitBox <gi...@apache.org>.

Kami commented on issue #1417: InvalidCredsError when uploading .xlsm file to Google Storage bucket
URL: https://github.com/apache/libcloud/issues/1417#issuecomment-578724941
 
 
   EDIT: Never mind, you were correct - the issue is indeed related to using interoperability API and lower casing the content type value.
   
   I verified it works correctly (both, with the interoperability API and the regular one) if the content-type value is left as is (and not lower cased). I will go ahead and open a PR with fix.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services