[jira] [Updated] (KAFKA-188) Support multiple data directories

["Jay Kreps (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jay Kreps updated KAFKA-188:
----------------------------

    Attachment: KAFKA-188.patch

Okay, took a shot at this. Attached is a draft patch. It needs some tests specific to the log selection.

There is a hitch, which I discuss below.

The basic change is
1. Change KafkaConfig.logDir to logDirectories and take a CSV of paths (I still support the old setting as a fallback)
2. Move the whole clean shutdown marker file thing into the LogManager. I feel it should have always been inside LogManager since it is an implementation detail of log recovery. Now there is one such file per log directory.
3. Create new logs in the directories in a round-robin fashion. To do this I keep a counter in LogManager that controls the index of the dir in which we will next create a new log. Initialize this to a random dir index. Each time we create a new index use the dir this points at and then increment it.

The hitch is the high watermark file. Currently we keep it in log.dir. But what directory should we keep it in when there are multiple log directories? I hackily just use the first. However this creates a dependency on the order of the log dirs in the config, which is not ideal. If we sort them and use the dir that is alphabetically first then if we add new directories that will mess it up (whereas if we hadn't sorted we could have tacked them on the end).

Some options:
0. What I currently do, just use the first directory for the hwmark file.
1. Add a new configuration property, metadataDir and use this for the highwatermark file and the clean shutdown marker and any future persistent thing. Downside of this is that it requires a new config parameter the user has to set up.
2. Require a top level directory for all log directories. e.g. 
   all_logs/
      log_dir_1/
        my_topic-1/
        my_topic-2
      log_dir_2/
        ...
Obviously since the goal is to support multiple independently mounted disks and you might not control where they mount to you might have to use soft links. From a previous life I am remembering lots of pain related to java and softlinks.

I would like to get people's feedback on this.
                
> Support multiple data directories
> ---------------------------------
>
>                 Key: KAFKA-188
>                 URL: https://issues.apache.org/jira/browse/KAFKA-188
>             Project: Kafka
>          Issue Type: New Feature
>            Reporter: Jay Kreps
>         Attachments: KAFKA-188.patch
>
>
> Currently we allow only a single data directory. This means that a multi-disk configuration needs to be a RAID array or LVM volume or something like that to be mounted as a single directory.
> For a high-throughput low-reliability configuration this would mean RAID0 striping. Common wisdom in Hadoop land has it that a JBOD setup that just mounts each disk as a separate directory and does application-level balancing over these results in about 30% write-improvement. For example see this claim here:
>   http://old.nabble.com/Re%3A-RAID-vs.-JBOD-p21466110.html
> It is not clear to me why this would be the case--it seems the RAID controller should be able to balance writes as well as the application so it may depend on the details of the setup.
> Nonetheless this would be really easy to implement, all you need to do is add multiple data directories and balance partition creation over these disks.
> One problem this might cause is if a particular topic is much larger than the others it might unbalance the load across the disks. The partition->disk assignment policy should probably attempt to evenly spread each topic to avoid this, rather than just trying keep the number of partitions balanced between disks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira