[jira] [Commented] (BEAM-12714) FileSystem.create does not pass full Metadata to GCS

["Kenneth Knowles (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-12714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476382#comment-17476382 ] 

Kenneth Knowles commented on BEAM-12714:
----------------------------------------

You are certainly correct about not passing the full metadata: https://github.com/apache/beam/blob/c345fe14e581e7675c56f9beeeb8917d1c4e7376/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java#L138

Note that there are some issues with setting the fields, which you can read about at https://cloud.google.com/storage/docs/transcoding#content-type_vs_content-encoding

GCS does its best to decompress for you, which cannot be disabled and causes problems. There's a host of issues filed about it: BEAM-1016, BEAM-1874, BEAM-7411, BEAM-8168, 

The TL;DR is that GCS sends the wrong file size for transcoded files. The headers give the compressed size and then too many bytes come across the wire. The Java SDK has trial-and-error code to survive this, while the Python SDK does not. I believe the intention here is to avoid writing files in a way that will break downstream consumers. That said, I'm open to change here if there is a good solution. Philosophically, I do believe it setting the metadata in the best way possible, and I do believe the current status is a compromise, not ideal.

> FileSystem.create does not pass full Metadata to GCS
> ----------------------------------------------------
>
>                 Key: BEAM-12714
>                 URL: https://issues.apache.org/jira/browse/BEAM-12714
>             Project: Beam
>          Issue Type: Bug
>          Components: io-py-gcp
>    Affects Versions: 2.31.0
>            Reporter: Andreas Bergmeier
>            Priority: P3
>
> When I call *FileSystem.create(foo, mime_type="text/plain", compression_type=CompressionTypes.BZIP2)* I would expect the following metadata set on GCS Blob:
> bq. content_type: text/plain
> bq. content_encoding: application/bzip2
> What we currently get is:
> bq. content_type: application/x-bz2
> Looking at the code it seems to be a problem with GcsIO interface, which does not seem to expose *content_type* and *content_encoding* properly.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)