You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Henry Kim <he...@resonate.com> on 2016/11/02 18:46:27 UTC

HDFS Connector Compression?

Is it possible to add compression to the HDFS Connector out of the box? Or does it require code change?


Thanks

Henry Kim

Re: HDFS Connector Compression?

Posted by Gwen Shapira <gw...@confluent.io>.

Documentation patches are welcome :)

On Mon, Nov 7, 2016 at 9:06 AM, Henry Kim <he...@resonate.com> wrote:
> Nevermind, it seems snappy is supported by default for ParquetFormat. It was not obvious with the fixed extension.
>
>
> private final static String EXTENSION = ".parquet";
>
> final Schema avroSchema = avroData.fromConnectSchema(record.valueSchema());
> CompressionCodecName compressionCodecName = CompressionCodecName.SNAPPY;
>
> final ParquetWriter<GenericRecord> writer =
>     new AvroParquetWriter<>(path, avroSchema, compressionCodecName, blockSize, pageSize);
>
>
>
> ________________________________
> From: Henry Kim
> Sent: Wednesday, November 2, 2016 2:46:27 PM
> To: users@kafka.apache.org
> Subject: HDFS Connector Compression?
>
>
> Is it possible to add compression to the HDFS Connector out of the box? Or does it require code change?
>
>
> Thanks
>
> Henry Kim



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: HDFS Connector Compression?

Posted by Henry Kim <he...@resonate.com>.

Nevermind, it seems snappy is supported by default for ParquetFormat. It was not obvious with the fixed extension.


private final static String EXTENSION = ".parquet";

final Schema avroSchema = avroData.fromConnectSchema(record.valueSchema());
CompressionCodecName compressionCodecName = CompressionCodecName.SNAPPY;

final ParquetWriter<GenericRecord> writer =
    new AvroParquetWriter<>(path, avroSchema, compressionCodecName, blockSize, pageSize);



________________________________
From: Henry Kim
Sent: Wednesday, November 2, 2016 2:46:27 PM
To: users@kafka.apache.org
Subject: HDFS Connector Compression?


Is it possible to add compression to the HDFS Connector out of the box? Or does it require code change?


Thanks

Henry Kim