[jira] [Created] (KAFKA-3015) Improve JBOD data balancing

["Jay Kreps (JIRA)" <ji...@apache.org>]

Jay Kreps created KAFKA-3015:
--------------------------------

             Summary: Improve JBOD data balancing
                 Key: KAFKA-3015
                 URL: https://issues.apache.org/jira/browse/KAFKA-3015
             Project: Kafka
          Issue Type: Improvement
            Reporter: Jay Kreps


When running with multiple data directories (i.e. JBOD) we currently place partitions entirely within one data directory. This tends to lead to poor balancing across disks as some topics have more throughput/retention and not all disks get data from all topics. You can't fix this problem with smarter partition placement strategies because ultimately you don't know when a partition is created when or how heavily it will be used (this is a subtle point, and the tendency is to try to think of some more sophisticated way to place partitions based on current data size but this is actually exceptionally dangerous and can lead to much worse imbalance when creating many partitions at once as they would all go to the disk with the least data). We don't support online rebalancing across directories/disks so this imbalance is a big problem and limits the usefulness of this configuration. Implementing online rebalancing of data across disks without downtime is actually quite hard and requires lots of I/O since you have to actually rewrite full partitions of data.

An alternative would be to place each partition in *all* directories/drives and round-robin *segments* within the partition across the directories. So the layout would be something like:
  drive-a/mytopic-0/
      0000000.data
      0000000.index
      0024680.data
      0024680.index
  drive-a/mytopic-0/
      0012345.data
      0012345.index
      0036912.data
      0036912.index

This is a little harder to implement than the current approach but not very hard, and it is a lot easier than implementing online data balancing across disks while retaining the current approach. I think this could easily be done in a backwards compatible way.

I think the balancing you would get from this in most cases would be good enough to make JBOD the default configuration. Thoughts?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)