[GitHub] [druid] FrankChen021 commented on issue #10566: Delete data from Kafka after ingestion

[GitBox <gi...@apache.org>]

FrankChen021 commented on issue #10566:
URL: https://github.com/apache/druid/issues/10566#issuecomment-725189863


   We don't manually delete kafka records by offsets as you did but  I think I can share you something we do to avoid such a problem.
   
   1st, it's better to set shorter log retention period for such a large topic to avoid disk shortage. For example, the `log.retention.hours` in our kafka clusters are usually set to 12 hours for high traffic topics however the default configuration is 168 hours(7 days).
   
   2nd, it's also better to enable `LZ4` compression and a reasonable `ling.ms` at producer side which would both greatly benefit the network traffic, and disk storage at kafka broker side.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org