[jira] [Commented] (BEAM-55) Allow users to compress FileBasedSink output files

["Florian Scharinger (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-55?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15541262#comment-15541262 ] 

Florian Scharinger commented on BEAM-55:
----------------------------------------

Hi Daniel,

At the time when I raised this, I was under the impression that codecs like Snappy are not supported. We have changed our system design significantly so that we do not need to read uncompressed Avro files from GCS. Having said that, we will need to produce compressed text files for another part of our system in the future, so Jeffrey's contribution will be very useful.

Cheers,
Florian


> Allow users to compress FileBasedSink output files
> --------------------------------------------------
>
>                 Key: BEAM-55
>                 URL: https://issues.apache.org/jira/browse/BEAM-55
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Daniel Halperin
>            Priority: Minor
>
> FileBasedSink (also TextIO.Write, AvroIO.Write, etc). does not have an option for compressing its output.
> In general, we discourage compression because it limits or blocks scalably reading from a file in parallel. However, users may want it -- so we should support the option (with appropriate warnings).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)