You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@libcloud.apache.org by GitBox <gi...@apache.org> on 2020/12/17 19:39:27 UTC
[GitHub] [libcloud] sdcubber opened a new issue #1530: Object metadata keys are lowercased when uploading to GCS
sdcubber opened a new issue #1530:
URL: https://github.com/apache/libcloud/issues/1530
## Summary
Mixed-case metadata is lowercased when using libcloud to upload objects to Google Cloud Storage.
## Detailed Information
I'm using libcloud to upload files to a Google Cloud Storage bucket together with object metadata.
In the process, the keys in my metadata dict are being lowercased. This is not due to Google Cloud Storage, which does support mixed case metadata.
The issue can be reproduced following the example from the docs:
```
from libcloud.storage.types import Provider
from libcloud.storage.providers import get_driver
cls = get_driver(Provider.GOOGLE_STORAGE)
driver = cls('SA-EMAIL', './SA.json') # provide service account credentials here
FILE_PATH = '/home/user/file'
extra = {'meta_data': {'camelCase': 'foo'}}
# Upload with metadata
with open(FILE_PATH, 'rb') as iterator:
obj = driver.upload_object_via_stream(iterator=iterator,
container=container,
object_name='file',
extra=extra)
```
The file uploads succesfully, but resulting metadata will look as follows:
Where `camelCase` has been turned into `camelcase`.
I'm using apache-libcloud==3.0.0 with Python 3.8 on Linux (the python:3.8 Docker image).
Provide detailed information of your bug report.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on issue #1530: Object metadata keys are lowercased when uploading to GCS
Posted by GitBox <gi...@apache.org>.
Kami commented on issue #1530:
URL: https://github.com/apache/libcloud/issues/1530#issuecomment-748482513
I also verified we correctly pass headers as provided by the user directly to the API endpoint.
Per S3 API docs, it's indeed limitation of the S3 API - https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html:
> ...
User-defined metadata is a set of key-value pairs. Amazon S3 stores user-defined metadata keys in lowercase.
...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on issue #1530: Object metadata keys are lowercased when uploading to GCS
Posted by GitBox <gi...@apache.org>.
Kami commented on issue #1530:
URL: https://github.com/apache/libcloud/issues/1530#issuecomment-748481148
I just had a look - I assume that's limitation of the S3 API.
I verified and S3 API lower cases all the metadata keys (also if you create metadata items through the web interface).
Google Storage driver is based on the S3 one and utilizes S3 compatible API so it's likely S3 API limitation and not much we can do besides implementing native Google Storage API (this would likely take much more work).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] sdcubber commented on issue #1530: Object metadata keys are lowercased when uploading to GCS
Posted by GitBox <gi...@apache.org>.
sdcubber commented on issue #1530:
URL: https://github.com/apache/libcloud/issues/1530#issuecomment-748571356
Alright, clear. Thanks for sorting this out!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on issue #1530: Object metadata keys are lowercased when uploading to GCS
Posted by GitBox <gi...@apache.org>.
Kami commented on issue #1530:
URL: https://github.com/apache/libcloud/issues/1530#issuecomment-748484450
For now I added this note to the docs - be322400028ee9d27e1935dc1b15aa3f6c28cf16. Hope it helps.
Per that note - even if we ever support Google Storage JSON API, it's probably still better to not rely on mixed casing in case cross provider compatibility is desired.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] stale[bot] commented on issue #1530: Object metadata keys are lowercased when uploading to GCS
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on issue #1530:
URL: https://github.com/apache/libcloud/issues/1530#issuecomment-868936527
Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically marking is as stale. If this issue is not relevant or applicable anymore (problem has been fixed in a new version or similar), please close the issue or let us know so we can close it. On the contrary, if the issue is still relevant, there is nothing you need to do, but if you have any additional details or context which would help us when working on this issue, please include it as a comment to this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on issue #1530: Object metadata keys are lowercased when uploading to GCS
Posted by GitBox <gi...@apache.org>.
Kami commented on issue #1530:
URL: https://github.com/apache/libcloud/issues/1530#issuecomment-748450644
Thanks for reporting this.
I assume this was done for some kind of cross-provider compatibility, but I'm not sure.
I think changing it should be fairly straight forward, but the change is backward incompatible which means it would need to be document in upgrade notes and we would also need to have a way to revert back to the old behavior (or perhaps have it as an opt-in instead of opt-out).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami edited a comment on issue #1530: Object metadata keys are lowercased when uploading to GCS
Posted by GitBox <gi...@apache.org>.
Kami edited a comment on issue #1530:
URL: https://github.com/apache/libcloud/issues/1530#issuecomment-748482513
I also verified we correctly pass headers as provided by the user directly to the API endpoint.
Per S3 API docs, it's indeed limitation of the S3 API - https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html:
> ...
User-defined metadata is a set of key-value pairs. Amazon S3 stores user-defined metadata keys in lowercase.
...
And here are a couple of more links for reference::
* https://stackoverflow.com/questions/59627280/setting-custom-metadata-to-an-object-in-google-cloud-storage
* https://stackoverflow.com/questions/56852315/gcp-https-load-balancers-will-convert-http-1-1-header-names-to-lowercase-coul
* https://groups.google.com/forum/#!msg/google-appengine/rS_2v6bTaVo/pCRQcYE7DAAJ
In short, since metadata in our implementation is sent as part of the HTTP request headers, using only lower case value is safer in any case.
And as said above - it appears to work via the console since that likely utilizes native Google Storage JSON API and not the S3 compatible XML one.
So maybe for now we just document this as a limitation in our Google Storage driver docs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org