You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Flink Developer <de...@protonmail.com> on 2018/11/04 23:29:37 UTC

Recommendation for BucketingSink to GZIP compress files to S3

Hi, what is the recommended method for using BucketingSink and compressing files using GZIP before it is uploaded to S3?

I read that one way is to extend the StreamWriterBase class and wrap the stream using GZIPOutputStream. Is there an Flink example for this? If so, what would be the proper way to override the methods (open(), write(), close(), flush())?

Thank you.