Questions re: partition reassignment

[Mathias Söderberg <ma...@gmail.com>]

Good day,

We're currently running a Kafka cluster in our staging environment, testing
everything out and so far it runs fairly smoothly.

One thing that we're currently looking into is scaling a cluster that is
already up 'n serving data, and I have some questions regarding moving
partitions between brokers.

My current interpretation is that if I start a partition reassignment, for
the sake of simplicity let's assume it's just for a single partition, the
new leader will first become a follower of the current leader, and when it
has caught up it'll transfer leadership over to itself?
If so, is Kafka only concerned about data that has been flushed to disk? So
if there's data that hasn't been flushed to disk yet it'll be discarded
when the new leader gains leadership?

What happens if neither of the new ARs are part of the previous AR set? I
suppose that they'll all be followers and when the new leader has caught up
it'll assume leadership?

I've played around with controlled shutdown and noticed that the broker
that's shutting down is truncating logs prior to actually shutting down,
which I don't really understand? I cannot really tell whether it does this
for all topics or just a subset of the ones that it's AR for as we have
quite a lot of topics, but I figured I'd just throw it out there, as I feel
like it could be explained by the answers to the questions above.

Should mention that we're running v0.8.1.1 with mostly stock configuration,
and that our producers are currently using ACK = 1 as well.

Best regards,
Mathias

Re: Questions re: partition reassignment

[Neha Narkhede <ne...@gmail.com>]

Okay, so just to clarify, if I have a partition where the leader is broker
0, the ISR is [0, 1] and I make a partition reassignment with a new AR list
of [1, 0], broker 1 won't take over leadership? I was under the impression
that the "preferred" replica would become the leader, and that that would
be the first replica in the AR list?

This should work, though I'm not sure we intended to cover this case. I
guess if you reassign partitions and then run preferred replica election,
it should move the leader to the new preferred replica. Feel free to give
it a try and file a JIRA if it doesn't work.

Alright, so if replica.lag.max.messages is set to the default of 4000, I
could potentially lose ~4000 when the leadership is transferred? (Assuming
it's a very high throughput topic/partition). Or what steps are actually
carried out when the leadership is transferred?

Yes and the guarantees that the producer gets depends on the number of acks
configured on the producer.

Thanks,
Neha

On Mon, Sep 22, 2014 at 1:32 PM, Mathias Söderberg <
mathias.soederberg@gmail.com> wrote:

> >
> > Partition reassignment will not move the leader unless the old leader is
> >
> not part of the new set of replicas.
> > Even when it does move the leader, it waits until the new replicas enter
> > the ISR.
>
>
> Okay, so just to clarify, if I have a partition where the leader is broker
> 0, the ISR is [0, 1] and I make a partition reassignment with a new AR list
> of [1, 0], broker 1 won't take over leadership? I was under the impression
> that the "preferred" replica would become the leader, and that that would
> be the first replica in the AR list?
>
>
> > No. Kafka only waits until a replica enters the ISR set for it to be a
> > candidate for leadership.
>
>
> Alright, so if replica.lag.max.messages is set to the default of 4000, I
> could potentially lose ~4000 when the leadership is transferred? (Assuming
> it's a very high throughput topic/partition). Or what steps are actually
> carried out when the leadership is transferred?
>
>
> > can you share the logs? Is this truncation during the shutdown?
>
>
> I'll see what I can dig up tomorrow, but it appears as if the broker that's
> shutting down is resetting logs to some offset, which happens while it's
> removing replica fetchers.
>
> Best regards,
> Mathias
>
> On 22 September 2014 17:45, Neha Narkhede <ne...@gmail.com> wrote:
>
> > My current interpretation is that if I start a partition reassignment,
> for
> > the sake of simplicity let's assume it's just for a single partition, the
> > new leader will first become a follower of the current leader, and when
> it
> > has caught up it'll transfer leadership over to itself?
> >
> > Partition reassignment will not move the leader unless the old leader is
> > not part of the new set of replicas.
> > Even when it does move the leader, it waits until the new replicas enter
> > the ISR.
> >
> > If so, is Kafka only concerned about data that has been flushed to disk?
> So
> > if there's data that hasn't been flushed to disk yet it'll be discarded
> > when the new leader gains leadership?
> >
> > No. Kafka only waits until a replica enters the ISR set for it to be a
> > candidate for leadership.
> >
> > What happens if neither of the new ARs are part of the previous AR set? I
> > suppose that they'll all be followers and when the new leader has caught
> up
> > it'll assume leadership?
> >
> > Yes, this is the only time the leader is moved as part of partition
> > reassignment.
> >
> > I've played around with controlled shutdown and noticed that the broker
> > that's shutting down is truncating logs prior to actually shutting down,
> > which I don't really understand?
> >
> > can you share the logs? Is this truncation during the shutdown?
> >
> > On Sun, Sep 21, 2014 at 2:32 PM, Mathias Söderberg <
> > mathias.soederberg@gmail.com> wrote:
> >
> > > Good day,
> > >
> > > We're currently running a Kafka cluster in our staging environment,
> > testing
> > > everything out and so far it runs fairly smoothly.
> > >
> > > One thing that we're currently looking into is scaling a cluster that
> is
> > > already up 'n serving data, and I have some questions regarding moving
> > > partitions between brokers.
> > >
> > > My current interpretation is that if I start a partition reassignment,
> > for
> > > the sake of simplicity let's assume it's just for a single partition,
> the
> > > new leader will first become a follower of the current leader, and when
> > it
> > > has caught up it'll transfer leadership over to itself?
> > > If so, is Kafka only concerned about data that has been flushed to
> disk?
> > So
> > > if there's data that hasn't been flushed to disk yet it'll be discarded
> > > when the new leader gains leadership?
> > >
> > > What happens if neither of the new ARs are part of the previous AR
> set? I
> > > suppose that they'll all be followers and when the new leader has
> caught
> > up
> > > it'll assume leadership?
> > >
> > > I've played around with controlled shutdown and noticed that the broker
> > > that's shutting down is truncating logs prior to actually shutting
> down,
> > > which I don't really understand? I cannot really tell whether it does
> > this
> > > for all topics or just a subset of the ones that it's AR for as we have
> > > quite a lot of topics, but I figured I'd just throw it out there, as I
> > feel
> > > like it could be explained by the answers to the questions above.
> > >
> > > Should mention that we're running v0.8.1.1 with mostly stock
> > configuration,
> > > and that our producers are currently using ACK = 1 as well.
> > >
> > > Best regards,
> > > Mathias
> > >
> >
>

Re: Questions re: partition reassignment

[Mathias Söderberg <ma...@gmail.com>]

>
> Partition reassignment will not move the leader unless the old leader is
>
not part of the new set of replicas.
> Even when it does move the leader, it waits until the new replicas enter
> the ISR.


Okay, so just to clarify, if I have a partition where the leader is broker
0, the ISR is [0, 1] and I make a partition reassignment with a new AR list
of [1, 0], broker 1 won't take over leadership? I was under the impression
that the "preferred" replica would become the leader, and that that would
be the first replica in the AR list?


> No. Kafka only waits until a replica enters the ISR set for it to be a
> candidate for leadership.


Alright, so if replica.lag.max.messages is set to the default of 4000, I
could potentially lose ~4000 when the leadership is transferred? (Assuming
it's a very high throughput topic/partition). Or what steps are actually
carried out when the leadership is transferred?


> can you share the logs? Is this truncation during the shutdown?


I'll see what I can dig up tomorrow, but it appears as if the broker that's
shutting down is resetting logs to some offset, which happens while it's
removing replica fetchers.

Best regards,
Mathias

On 22 September 2014 17:45, Neha Narkhede <ne...@gmail.com> wrote:

> My current interpretation is that if I start a partition reassignment, for
> the sake of simplicity let's assume it's just for a single partition, the
> new leader will first become a follower of the current leader, and when it
> has caught up it'll transfer leadership over to itself?
>
> Partition reassignment will not move the leader unless the old leader is
> not part of the new set of replicas.
> Even when it does move the leader, it waits until the new replicas enter
> the ISR.
>
> If so, is Kafka only concerned about data that has been flushed to disk? So
> if there's data that hasn't been flushed to disk yet it'll be discarded
> when the new leader gains leadership?
>
> No. Kafka only waits until a replica enters the ISR set for it to be a
> candidate for leadership.
>
> What happens if neither of the new ARs are part of the previous AR set? I
> suppose that they'll all be followers and when the new leader has caught up
> it'll assume leadership?
>
> Yes, this is the only time the leader is moved as part of partition
> reassignment.
>
> I've played around with controlled shutdown and noticed that the broker
> that's shutting down is truncating logs prior to actually shutting down,
> which I don't really understand?
>
> can you share the logs? Is this truncation during the shutdown?
>
> On Sun, Sep 21, 2014 at 2:32 PM, Mathias Söderberg <
> mathias.soederberg@gmail.com> wrote:
>
> > Good day,
> >
> > We're currently running a Kafka cluster in our staging environment,
> testing
> > everything out and so far it runs fairly smoothly.
> >
> > One thing that we're currently looking into is scaling a cluster that is
> > already up 'n serving data, and I have some questions regarding moving
> > partitions between brokers.
> >
> > My current interpretation is that if I start a partition reassignment,
> for
> > the sake of simplicity let's assume it's just for a single partition, the
> > new leader will first become a follower of the current leader, and when
> it
> > has caught up it'll transfer leadership over to itself?
> > If so, is Kafka only concerned about data that has been flushed to disk?
> So
> > if there's data that hasn't been flushed to disk yet it'll be discarded
> > when the new leader gains leadership?
> >
> > What happens if neither of the new ARs are part of the previous AR set? I
> > suppose that they'll all be followers and when the new leader has caught
> up
> > it'll assume leadership?
> >
> > I've played around with controlled shutdown and noticed that the broker
> > that's shutting down is truncating logs prior to actually shutting down,
> > which I don't really understand? I cannot really tell whether it does
> this
> > for all topics or just a subset of the ones that it's AR for as we have
> > quite a lot of topics, but I figured I'd just throw it out there, as I
> feel
> > like it could be explained by the answers to the questions above.
> >
> > Should mention that we're running v0.8.1.1 with mostly stock
> configuration,
> > and that our producers are currently using ACK = 1 as well.
> >
> > Best regards,
> > Mathias
> >
>

Re: Questions re: partition reassignment

[Neha Narkhede <ne...@gmail.com>]

My current interpretation is that if I start a partition reassignment, for
the sake of simplicity let's assume it's just for a single partition, the
new leader will first become a follower of the current leader, and when it
has caught up it'll transfer leadership over to itself?

Partition reassignment will not move the leader unless the old leader is
not part of the new set of replicas.
Even when it does move the leader, it waits until the new replicas enter
the ISR.

If so, is Kafka only concerned about data that has been flushed to disk? So
if there's data that hasn't been flushed to disk yet it'll be discarded
when the new leader gains leadership?

No. Kafka only waits until a replica enters the ISR set for it to be a
candidate for leadership.

What happens if neither of the new ARs are part of the previous AR set? I
suppose that they'll all be followers and when the new leader has caught up
it'll assume leadership?

Yes, this is the only time the leader is moved as part of partition
reassignment.

I've played around with controlled shutdown and noticed that the broker
that's shutting down is truncating logs prior to actually shutting down,
which I don't really understand?

can you share the logs? Is this truncation during the shutdown?

On Sun, Sep 21, 2014 at 2:32 PM, Mathias Söderberg <
mathias.soederberg@gmail.com> wrote:

> Good day,
>
> We're currently running a Kafka cluster in our staging environment, testing
> everything out and so far it runs fairly smoothly.
>
> One thing that we're currently looking into is scaling a cluster that is
> already up 'n serving data, and I have some questions regarding moving
> partitions between brokers.
>
> My current interpretation is that if I start a partition reassignment, for
> the sake of simplicity let's assume it's just for a single partition, the
> new leader will first become a follower of the current leader, and when it
> has caught up it'll transfer leadership over to itself?
> If so, is Kafka only concerned about data that has been flushed to disk? So
> if there's data that hasn't been flushed to disk yet it'll be discarded
> when the new leader gains leadership?
>
> What happens if neither of the new ARs are part of the previous AR set? I
> suppose that they'll all be followers and when the new leader has caught up
> it'll assume leadership?
>
> I've played around with controlled shutdown and noticed that the broker
> that's shutting down is truncating logs prior to actually shutting down,
> which I don't really understand? I cannot really tell whether it does this
> for all topics or just a subset of the ones that it's AR for as we have
> quite a lot of topics, but I figured I'd just throw it out there, as I feel
> like it could be explained by the answers to the questions above.
>
> Should mention that we're running v0.8.1.1 with mostly stock configuration,
> and that our producers are currently using ACK = 1 as well.
>
> Best regards,
> Mathias
>