Speeding up Kafka Start cold startup time

[Stephane Maarek <st...@simplemachines.com.au>]

Hi,

 

We have our Kafka Brokers in AWS backed by st1 EBS drives. These are optimised for throughput

On a kafka restart (warm restart), if some of the appropriate data is in the pagecache, everything goes well and the broker boots in just a few seconds. (1000 partitions)

On a cold start (new EC2 instance which just got the volume attached), nothing is in the page cache and the first boot can take up to 10 minutes. 

 

Question: what does the kafka broker do and read on start? I would assume index files and so on, but not the actual segments, does it?

 

If so, would it be worth considering a KIP that would allow to optimise kafka files this way:
Kafka index files, or any small files, would be placed on a faster drive (say a gp2 SSD drive)
Kafka segment files, or any huge files, would be placed on a throughput optimised drive. 
 

I am not sure how the KIP would shape up, or the code change, but:
Is that a possible strategy to drastically improve the broker start-up time? 
Would that also improve shutdown time?
 

Thanks,

Stephane