You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "funguy-tech (via GitHub)" <gi...@apache.org> on 2024/09/14 08:04:10 UTC
[I] Add support for Kinesis Compression (druid)
funguy-tech opened a new issue, #17062:
URL: https://github.com/apache/druid/issues/17062
### Description
Placeholder Feature Request for an upcoming PR.
This proposal is to bring support for common compression formats already implemented in Druid’s code base (zstd, gzip, etc) to Kinesis streams.
Compression would be exposed via an optional configuration parameter in the Kinesis ioConfig, ‘compressionFormat’, that when enabled will perform decompression of records at the point of record collection.
### Motivation
Unlike Kafka, Kinesis by default does not offer much opportunity for compression out of the box. Because of this, it is a common usage pattern for Kinesis customers to compress/decompress their own data across the wire.
Given that Druid already has internal concepts for compression in various popular formats (zstd, gzip, etc), it would be useful for high throughput customers to have the ability to compress data across the wire.
Our team (a fleet of enterprise Druid clusters at petabyte scale) has seen Kinesis cost reduction to the tune of 50-80% by implementing a custom build of Druid with Kinesis decompression capabilities with little to no discernible impact on ingestion overhead.
PR forthcoming in a few days, but I wanted to open this feature request for community discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Add support for Kinesis Compression (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #17062: Add support for Kinesis Compression
URL: https://github.com/apache/druid/issues/17062
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Add support for Kinesis Compression (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on issue #17062:
URL: https://github.com/apache/druid/issues/17062#issuecomment-2352093732
Looking forward to the PR. This will be a very useful capability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Add support for Kinesis Compression (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17062:
URL: https://github.com/apache/druid/issues/17062#issuecomment-2998365440
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Add support for Kinesis Compression (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17062:
URL: https://github.com/apache/druid/issues/17062#issuecomment-3100093690
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org