Kafka replication throttling

[Marcos Juarez Lopez <mj...@gmail.com>]

Hi,

We have several Kafka clusters in production, and we've had to reassign
replication a few times now in production.  Some of our topic/partitions
are pretty large, up to 32 partitions per topic, and 16GB per partition, so
adding a new broker and/or repairing a broker that had been down for some
time turns out to be a major undertaking.

Today, when we attempt to replicate a single partition, it pegs the disk
IO, and uses a significant chunk of the 10Gbps interface for a good ~5
minutes.  This is causing problems for our downstream consumers, which rely
on having a consistent stream of realtime data being sent to them.

Is there a way to throttle Kafka replication between nodes, so that instead
of it going full blast, it will replicate at a fixed rate in megabytes or
activities/batches per second?  Or maybe is this planned for a future
release, maybe 0.9?

Thanks,

Marcos Juarez

Re: Kafka replication throttling

[Marcos Juarez Lopez <mj...@gmail.com>]

Thanks for your response Jun.

JIRA has been filed (see link below).  Please let me know if I should add
more details/context:

https://issues.apache.org/jira/browse/KAFKA-1464

Thanks,

Marcos Juarez



On Wed, May 21, 2014 at 8:40 AM, Jun Rao <ju...@gmail.com> wrote:

> We don't have such throttling right now. Could you file a jira?
>
> Thanks,
>
> Jun
>
>
> On Tue, May 20, 2014 at 10:29 PM, Marcos Juarez Lopez <mjuarez@gmail.com
> >wrote:
>
> > Hi,
> >
> > We have several Kafka clusters in production, and we've had to reassign
> > replication a few times now in production.  Some of our topic/partitions
> > are pretty large, up to 32 partitions per topic, and 16GB per partition,
> so
> > adding a new broker and/or repairing a broker that had been down for some
> > time turns out to be a major undertaking.
> >
> > Today, when we attempt to replicate a single partition, it pegs the disk
> > IO, and uses a significant chunk of the 10Gbps interface for a good ~5
> > minutes.  This is causing problems for our downstream consumers, which
> rely
> > on having a consistent stream of realtime data being sent to them.
> >
> > Is there a way to throttle Kafka replication between nodes, so that
> instead
> > of it going full blast, it will replicate at a fixed rate in megabytes or
> > activities/batches per second?  Or maybe is this planned for a future
> > release, maybe 0.9?
> >
> > Thanks,
> >
> > Marcos Juarez
> >
>

Re: Kafka replication throttling

[Jun Rao <ju...@gmail.com>]

We don't have such throttling right now. Could you file a jira?

Thanks,

Jun


On Tue, May 20, 2014 at 10:29 PM, Marcos Juarez Lopez <mj...@gmail.com>wrote:

> Hi,
>
> We have several Kafka clusters in production, and we've had to reassign
> replication a few times now in production.  Some of our topic/partitions
> are pretty large, up to 32 partitions per topic, and 16GB per partition, so
> adding a new broker and/or repairing a broker that had been down for some
> time turns out to be a major undertaking.
>
> Today, when we attempt to replicate a single partition, it pegs the disk
> IO, and uses a significant chunk of the 10Gbps interface for a good ~5
> minutes.  This is causing problems for our downstream consumers, which rely
> on having a consistent stream of realtime data being sent to them.
>
> Is there a way to throttle Kafka replication between nodes, so that instead
> of it going full blast, it will replicate at a fixed rate in megabytes or
> activities/batches per second?  Or maybe is this planned for a future
> release, maybe 0.9?
>
> Thanks,
>
> Marcos Juarez
>