Slow 'rolled new log segment' ?

[Stephen Powis <sp...@salesforce.com>]

Hey!

I'm trying to determine why I had a weird hiccup in publishing to my Kafka
cluster.  I'm running Kafka 0.10.0.1.  Here's a graph of my producer times:

<http://i.imgur.com/s2RfrbH.png>

You can see at around ~10:08 we had a large spike in kafka's internal
publishing, waiting for the leader.  Our producers are configured to
timeout write requests after 150ms.  So during this bump, lots of producers
timed out.

I first dug into my GC logs, and see nothing relevant or taking over 0.00X
seconds around the same time frame.

Digging thru server logs on the leader, at roughly the same time, I've
found the following:

[2016-09-16 10:08:02,508] INFO Rolled new log segment for 'EventStream-27'
> in 743 ms. (kafka.log.Log)
> [2016-09-16 10:08:34,727] INFO Rolled new log segment for 'EventStream-33'
> in 16 ms. (kafka.log.Log)
> [2016-09-16 10:09:01,485] INFO Rolled new log segment for 'EventStream-15'
> in 14 ms. (kafka.log.Log)
>

Reviewing logs going backwards and forwards, it seems like most new log
segments take ~15ms or less.

Curious if anyone knows any other potential causes or things to review that
might cause this issue?  Do you guys think the new log segment could cause
this issue?  Any ideas what would cause this new segment to take so long to
get created?

Thanks for the help!
Stephen