[jira] [Created] (KAFKA-6346) Consolidate multiple background async log operations

["Guozhang Wang (JIRA)" <ji...@apache.org>]

Guozhang Wang created KAFKA-6346:
------------------------------------

             Summary: Consolidate multiple background async log operations
                 Key: KAFKA-6346
                 URL: https://issues.apache.org/jira/browse/KAFKA-6346
             Project: Kafka
          Issue Type: Improvement
          Components: core
            Reporter: Guozhang Wang


On Kafka broker's log managers, today we trigger a lot of repeating and one-time async background tasks, including:

1. log deletion based on retention policy (repeating).
  1.2. async log segment deletion triggered from 1 above (one-time)
2. log deletion based on replica stopping / log directory swapping (repeating)
3. log compaction, i.e. so-called "log cleaner thread" (repeating)
4. dirty log flushing
5. log recovery offsets checkpointing.
6. log starting offsets checkpointing.

I suspect all these background tasks are silently eating our CPU idle time unnecessarily. We should consider 1) benchmark their impacts on CPU cost and 2) if impact is high, consider consolidating some of these tasks into fewer async background threads.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)