[GitHub] peferron commented on issue #7039: Kafka Ingestion Spec to allow kafka topic with gzip compression

[GitBox <gi...@apache.org>]

peferron commented on issue #7039: Kafka Ingestion Spec to allow kafka topic with gzip compression
URL: https://github.com/apache/incubator-druid/issues/7039#issuecomment-461913177
 
 
   Do you mean that the Kafka messages themselves are compressed with gzip, either via a producer or broker configuration? If that's the case, then Druid can automatically handle decompression, just like most Kafka consumers. Try completely removing your `indexSpec` and giving it another shot.
   
   `dimensionCompression` and `metricCompression` inside the `indexSpec` do not refer to how Druid decompresses messages from Kafka—they refer to how Druid compresses dimension and metric columns *after* consuming messages from Kafka. Decompression speed is usually critical there, which is why the doc lists `LZ4` and `LZF` as possible formats, but not `gzip`, which would be too slow.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org