Uneven disk usage in Kafka 0.8.1.1

[Yury Ruchin <yu...@gmail.com>]

Hi,

With Kafka 0.8.1.1, I'm continuously running into the issue with no disk
space remaining. I'm observing that on the problematic brokers partitions
are distributed across several identical disks very unevenly. For example,
provided the fact that partitions have similar size, I see 6 partitions
being written to one disk, while the other disks get only 1-3 partitions
each.

What could cause that? Is that possible to somehow affect the partition
distribution?

Thanks,
Yury

Re: Uneven disk usage in Kafka 0.8.1.1

[Gwen Shapira <gs...@cloudera.com>]

Hi,

LogManager.nextLogDir() has the logic for choosing which directory to use.
The documentation of the method says:
  /**
   * Choose the next directory in which to create a log. Currently this is
done
   * by calculating the number of partitions in each directory and then
choosing the
   * data directory with the fewest partitions.
   */

A possible explanation for what you are seeing is that some of the
directories were added recently and simply did not fill up yet. Kafka
doesn't attempt to rebalance logs across disks or directories (although
this could be a nice feature request, at least for a utility).

Gwen




On Thu, Dec 25, 2014 at 1:30 AM, Yury Ruchin <yu...@gmail.com> wrote:

> I also noticed that some disks do not contain any partitions which is even
> more strange. Why does broker not use them, while other disks are
> overloaded?
>
> 2014-12-25 12:07 GMT+03:00 Yury Ruchin <yu...@gmail.com>:
>
> > Hi,
> >
> > With Kafka 0.8.1.1, I'm continuously running into the issue with no disk
> > space remaining. I'm observing that on the problematic brokers partitions
> > are distributed across several identical disks very unevenly. For
> example,
> > provided the fact that partitions have similar size, I see 6 partitions
> > being written to one disk, while the other disks get only 1-3 partitions
> > each.
> >
> > What could cause that? Is that possible to somehow affect the partition
> > distribution?
> >
> > Thanks,
> > Yury
> >
>

Re: Uneven disk usage in Kafka 0.8.1.1

[Yury Ruchin <yu...@gmail.com>]

I also noticed that some disks do not contain any partitions which is even
more strange. Why does broker not use them, while other disks are
overloaded?

2014-12-25 12:07 GMT+03:00 Yury Ruchin <yu...@gmail.com>:

> Hi,
>
> With Kafka 0.8.1.1, I'm continuously running into the issue with no disk
> space remaining. I'm observing that on the problematic brokers partitions
> are distributed across several identical disks very unevenly. For example,
> provided the fact that partitions have similar size, I see 6 partitions
> being written to one disk, while the other disks get only 1-3 partitions
> each.
>
> What could cause that? Is that possible to somehow affect the partition
> distribution?
>
> Thanks,
> Yury
>