You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Chamikara Jayalath (JIRA)" <ji...@apache.org> on 2019/01/30 22:16:00 UTC
[jira] [Comment Edited] (BEAM-6552) Upgrade bigdataoss_gcsio
dependency
[ https://issues.apache.org/jira/browse/BEAM-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16756611#comment-16756611 ]
Chamikara Jayalath edited comment on BEAM-6552 at 1/30/19 10:15 PM:
--------------------------------------------------------------------
I believe this can potentially affect all released Beam Java SDK versions when using decompressive transcoding ("Content-Encoding: gzip") [1] for files that are bigger than 2.17 GiB (Integer.MAX_VALUE bytes).
Fix to gcsio library is here: [https://github.com/GoogleCloudPlatform/bigdata-interop/commit/a711b8f86384726e9e149a161eab1c347a0a9512]
This is expected to be released soon. I'll update the dependency after that.
[1] [https://cloud.google.com/storage/docs/transcoding]
was (Author: chamikara):
I believe this can potentially affect all released Beam Java SDK versions when using decompression transcoding ("Content-Encoding: gzip") [1] for files that are bigger than 2.17 GiB (Integer.MAX_VALUE bytes).
Fix to gcsio library is here: [https://github.com/GoogleCloudPlatform/bigdata-interop/commit/a711b8f86384726e9e149a161eab1c347a0a9512]
This is expected to be released soon. I'll update the dependency after that.
[1] https://cloud.google.com/storage/docs/transcoding
> Upgrade bigdataoss_gcsio dependency
> -----------------------------------
>
> Key: BEAM-6552
> URL: https://issues.apache.org/jira/browse/BEAM-6552
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp, runner-dataflow
> Reporter: Chamikara Jayalath
> Assignee: Chamikara Jayalath
> Priority: Blocker
> Fix For: 2.10.0
>
>
> Current version (1.9.0) possibly has very rare data duplication issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)