You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yury Ruchin <yu...@gmail.com> on 2014/06/18 17:33:19 UTC
Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail
Hi folks,
In my project, we want to perform to update our active Kafka 0.8 cluster to
Kafka 0.8.1.1 without downtime and losing any data. The process (after
reading http://kafka.apache.org/documentation.html#upgrade) looks to me
like this. For each broker in turn:
1. Bring the broker down.
2. Update Kafka to 0.8.1.1 on the broker node.
3. Start the broker.
4. Run preferred-replica-election script to restore broker's leadership for
respective partitions.
5. Wait for the the preferred replica election to complete.
I deem step#5 necessary since preferred replica election is an asynchronous
process. There is a slim chance that bringing other brokers down before the
election is complete would result in all replicas down for some partitions,
so a portion of the incoming data stream would be lost. Is my understanding
of the process correct?
Re: Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail
Posted by Jun Rao <ju...@gmail.com>.
If broker 1 is down in step 4, the producer will get a broken socket error
immediately. If broker 1 is up in step 4 and just the leader is moved
(e.g., due to preferred leader balancing), the producer will get an error
after the timeout specified in the producer request.
Thanks,
Jun
On Mon, Jun 30, 2014 at 7:40 AM, Yury Ruchin <yu...@gmail.com> wrote:
> Hi Michal,
>
> Thanks for the perfect links. They really help. Now it looks like with
> request.required.acks=1 (let alone 0) messages can be lost in the case I
> described. The aphyr's article, seemingly, describes a more tricky case
> than I have.
>
> I'm still not sure on Kafka behavior in case of request.required.acks=-1.
> With this setting in effect, the scenario will turn into this:
>
> 1. Producer sends message A to the leader.
> 2. Leader stores the message, followers fetch it. Everyone's in sync.
> 3. Producer sends message B to the leader. Followers haven't still fetched
> the message and lag by 1 message: the B is still only on the broker 1. The
> produce request for message B sits in the purgatory waiting for
> acknowledgement from the followers.
> 4. The leader goes down.
> 5. Followers cannot fetch B anymore, since its only owner is down. Yet some
> of the replicas needs to take over the leader responsibility. Say, broker 2
> now becomes the leader, 3 is the follower.
> 6. Producer sends message C to the leader (broker 2). Follower fetches it.
> 7. Broker 1 goes back online and starts following broker 2.
>
> What happens with message B that was sitting in the 1's purgatory when it
> went down? Will an error "unable to send message B" be returned to the
> producer immediately after broker 1 shutdown?
>
>
> 2014-06-30 17:41 GMT+04:00 Michal Michalski <michal.michalski@boxever.com
> >:
>
> > Hi Yury,
> >
> > If I understand correctly, the case you're describing is equivalent to
> the
> > leader re-election (in terms of data consistency). In that case messages
> > can be lost depending on your "acks" setting:
> >
> > https://kafka.apache.org/documentation.html
> > see: request.required.acks:
> > E.g. "only messages that were written to the now-dead leader but not yet
> > replicated will be lost)." for acks=1
> >
> > More info on that:
> > http://aphyr.com/posts/293-call-me-maybe-kafka
> >
> > However, I'd be happy if someone with more Kafka experience confirmed my
> > understanding of that issue.
> >
> >
> > Kind regards,
> > Michał Michalski,
> > michal.michalski@boxever.com
> >
> >
> > On 30 June 2014 14:34, Yury Ruchin <yu...@gmail.com> wrote:
> >
> > > Thanks!
> > >
> > > I'm also trying to understand how replicas will catch up once the
> leader
> > > goes down. Say, we have 3 brokers with IDs 1, 2, 3. The leader is
> broker
> > 1.
> > > Followers are 2 and 3. Consider the following scenario assuming that
> all
> > > messages fall into the same partition:
> > >
> > > 1. Producer sends message A to the leader.
> > > 2. Leader stores the message, followers fetch it. Everyone's in sync.
> > > 3. Producer sends message B to the leader. Followers haven't still
> > fetched
> > > the message and lag by 1 message: the B is still only on the broker 1.
> > > 4. I bring the leader down.
> > > 5. Followers cannot fetch B anymore, since its only owner is down. Yet
> > some
> > > of the replicas needs to take over the leader responsibility. Say,
> > broker 2
> > > now becomes the leader, 3 is the follower.
> > > 6. Producer sends message C to the leader (broker 2). Follower fetches
> > it.
> > >
> > > I don't quite understand the state of the log on replicas 2 and 3 after
> > > step#6. It looks like the log will have a gap in it. The expected log
> > state
> > > is ["A", "B", "C"]. But brokers 2 and 3 didn't have a chance to fetch
> > "B",
> > > so their log looks like ["A", "C"]. Will Kafka try to fill the gap in
> the
> > > background once broker 1 started over?
> > >
> > >
> > > 2014-06-18 19:59 GMT+04:00 Neha Narkhede <ne...@gmail.com>:
> > >
> > > > You don't gain much by running #4 between broker bounces. Running it
> > > after
> > > > the cluster is upgraded will be sufficient.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Wed, Jun 18, 2014 at 8:33 AM, Yury Ruchin <yu...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > In my project, we want to perform to update our active Kafka 0.8
> > > cluster
> > > > to
> > > > > Kafka 0.8.1.1 without downtime and losing any data. The process
> > (after
> > > > > reading http://kafka.apache.org/documentation.html#upgrade) looks
> to
> > > me
> > > > > like this. For each broker in turn:
> > > > >
> > > > > 1. Bring the broker down.
> > > > > 2. Update Kafka to 0.8.1.1 on the broker node.
> > > > > 3. Start the broker.
> > > > > 4. Run preferred-replica-election script to restore broker's
> > leadership
> > > > for
> > > > > respective partitions.
> > > > > 5. Wait for the the preferred replica election to complete.
> > > > >
> > > > > I deem step#5 necessary since preferred replica election is an
> > > > asynchronous
> > > > > process. There is a slim chance that bringing other brokers down
> > before
> > > > the
> > > > > election is complete would result in all replicas down for some
> > > > partitions,
> > > > > so a portion of the incoming data stream would be lost. Is my
> > > > understanding
> > > > > of the process correct?
> > > > >
> > > >
> > >
> >
>
Re: Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail
Posted by Yury Ruchin <yu...@gmail.com>.
Hi Michal,
Thanks for the perfect links. They really help. Now it looks like with
request.required.acks=1 (let alone 0) messages can be lost in the case I
described. The aphyr's article, seemingly, describes a more tricky case
than I have.
I'm still not sure on Kafka behavior in case of request.required.acks=-1.
With this setting in effect, the scenario will turn into this:
1. Producer sends message A to the leader.
2. Leader stores the message, followers fetch it. Everyone's in sync.
3. Producer sends message B to the leader. Followers haven't still fetched
the message and lag by 1 message: the B is still only on the broker 1. The
produce request for message B sits in the purgatory waiting for
acknowledgement from the followers.
4. The leader goes down.
5. Followers cannot fetch B anymore, since its only owner is down. Yet some
of the replicas needs to take over the leader responsibility. Say, broker 2
now becomes the leader, 3 is the follower.
6. Producer sends message C to the leader (broker 2). Follower fetches it.
7. Broker 1 goes back online and starts following broker 2.
What happens with message B that was sitting in the 1's purgatory when it
went down? Will an error "unable to send message B" be returned to the
producer immediately after broker 1 shutdown?
2014-06-30 17:41 GMT+04:00 Michal Michalski <mi...@boxever.com>:
> Hi Yury,
>
> If I understand correctly, the case you're describing is equivalent to the
> leader re-election (in terms of data consistency). In that case messages
> can be lost depending on your "acks" setting:
>
> https://kafka.apache.org/documentation.html
> see: request.required.acks:
> E.g. "only messages that were written to the now-dead leader but not yet
> replicated will be lost)." for acks=1
>
> More info on that:
> http://aphyr.com/posts/293-call-me-maybe-kafka
>
> However, I'd be happy if someone with more Kafka experience confirmed my
> understanding of that issue.
>
>
> Kind regards,
> Michał Michalski,
> michal.michalski@boxever.com
>
>
> On 30 June 2014 14:34, Yury Ruchin <yu...@gmail.com> wrote:
>
> > Thanks!
> >
> > I'm also trying to understand how replicas will catch up once the leader
> > goes down. Say, we have 3 brokers with IDs 1, 2, 3. The leader is broker
> 1.
> > Followers are 2 and 3. Consider the following scenario assuming that all
> > messages fall into the same partition:
> >
> > 1. Producer sends message A to the leader.
> > 2. Leader stores the message, followers fetch it. Everyone's in sync.
> > 3. Producer sends message B to the leader. Followers haven't still
> fetched
> > the message and lag by 1 message: the B is still only on the broker 1.
> > 4. I bring the leader down.
> > 5. Followers cannot fetch B anymore, since its only owner is down. Yet
> some
> > of the replicas needs to take over the leader responsibility. Say,
> broker 2
> > now becomes the leader, 3 is the follower.
> > 6. Producer sends message C to the leader (broker 2). Follower fetches
> it.
> >
> > I don't quite understand the state of the log on replicas 2 and 3 after
> > step#6. It looks like the log will have a gap in it. The expected log
> state
> > is ["A", "B", "C"]. But brokers 2 and 3 didn't have a chance to fetch
> "B",
> > so their log looks like ["A", "C"]. Will Kafka try to fill the gap in the
> > background once broker 1 started over?
> >
> >
> > 2014-06-18 19:59 GMT+04:00 Neha Narkhede <ne...@gmail.com>:
> >
> > > You don't gain much by running #4 between broker bounces. Running it
> > after
> > > the cluster is upgraded will be sufficient.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Wed, Jun 18, 2014 at 8:33 AM, Yury Ruchin <yu...@gmail.com>
> > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > In my project, we want to perform to update our active Kafka 0.8
> > cluster
> > > to
> > > > Kafka 0.8.1.1 without downtime and losing any data. The process
> (after
> > > > reading http://kafka.apache.org/documentation.html#upgrade) looks to
> > me
> > > > like this. For each broker in turn:
> > > >
> > > > 1. Bring the broker down.
> > > > 2. Update Kafka to 0.8.1.1 on the broker node.
> > > > 3. Start the broker.
> > > > 4. Run preferred-replica-election script to restore broker's
> leadership
> > > for
> > > > respective partitions.
> > > > 5. Wait for the the preferred replica election to complete.
> > > >
> > > > I deem step#5 necessary since preferred replica election is an
> > > asynchronous
> > > > process. There is a slim chance that bringing other brokers down
> before
> > > the
> > > > election is complete would result in all replicas down for some
> > > partitions,
> > > > so a portion of the incoming data stream would be lost. Is my
> > > understanding
> > > > of the process correct?
> > > >
> > >
> >
>
Re: Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail
Posted by Michal Michalski <mi...@boxever.com>.
Hi Yury,
If I understand correctly, the case you're describing is equivalent to the
leader re-election (in terms of data consistency). In that case messages
can be lost depending on your "acks" setting:
https://kafka.apache.org/documentation.html
see: request.required.acks:
E.g. "only messages that were written to the now-dead leader but not yet
replicated will be lost)." for acks=1
More info on that:
http://aphyr.com/posts/293-call-me-maybe-kafka
However, I'd be happy if someone with more Kafka experience confirmed my
understanding of that issue.
Kind regards,
Michał Michalski,
michal.michalski@boxever.com
On 30 June 2014 14:34, Yury Ruchin <yu...@gmail.com> wrote:
> Thanks!
>
> I'm also trying to understand how replicas will catch up once the leader
> goes down. Say, we have 3 brokers with IDs 1, 2, 3. The leader is broker 1.
> Followers are 2 and 3. Consider the following scenario assuming that all
> messages fall into the same partition:
>
> 1. Producer sends message A to the leader.
> 2. Leader stores the message, followers fetch it. Everyone's in sync.
> 3. Producer sends message B to the leader. Followers haven't still fetched
> the message and lag by 1 message: the B is still only on the broker 1.
> 4. I bring the leader down.
> 5. Followers cannot fetch B anymore, since its only owner is down. Yet some
> of the replicas needs to take over the leader responsibility. Say, broker 2
> now becomes the leader, 3 is the follower.
> 6. Producer sends message C to the leader (broker 2). Follower fetches it.
>
> I don't quite understand the state of the log on replicas 2 and 3 after
> step#6. It looks like the log will have a gap in it. The expected log state
> is ["A", "B", "C"]. But brokers 2 and 3 didn't have a chance to fetch "B",
> so their log looks like ["A", "C"]. Will Kafka try to fill the gap in the
> background once broker 1 started over?
>
>
> 2014-06-18 19:59 GMT+04:00 Neha Narkhede <ne...@gmail.com>:
>
> > You don't gain much by running #4 between broker bounces. Running it
> after
> > the cluster is upgraded will be sufficient.
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, Jun 18, 2014 at 8:33 AM, Yury Ruchin <yu...@gmail.com>
> > wrote:
> >
> > > Hi folks,
> > >
> > > In my project, we want to perform to update our active Kafka 0.8
> cluster
> > to
> > > Kafka 0.8.1.1 without downtime and losing any data. The process (after
> > > reading http://kafka.apache.org/documentation.html#upgrade) looks to
> me
> > > like this. For each broker in turn:
> > >
> > > 1. Bring the broker down.
> > > 2. Update Kafka to 0.8.1.1 on the broker node.
> > > 3. Start the broker.
> > > 4. Run preferred-replica-election script to restore broker's leadership
> > for
> > > respective partitions.
> > > 5. Wait for the the preferred replica election to complete.
> > >
> > > I deem step#5 necessary since preferred replica election is an
> > asynchronous
> > > process. There is a slim chance that bringing other brokers down before
> > the
> > > election is complete would result in all replicas down for some
> > partitions,
> > > so a portion of the incoming data stream would be lost. Is my
> > understanding
> > > of the process correct?
> > >
> >
>
Re: Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail
Posted by Yury Ruchin <yu...@gmail.com>.
Thanks!
I'm also trying to understand how replicas will catch up once the leader
goes down. Say, we have 3 brokers with IDs 1, 2, 3. The leader is broker 1.
Followers are 2 and 3. Consider the following scenario assuming that all
messages fall into the same partition:
1. Producer sends message A to the leader.
2. Leader stores the message, followers fetch it. Everyone's in sync.
3. Producer sends message B to the leader. Followers haven't still fetched
the message and lag by 1 message: the B is still only on the broker 1.
4. I bring the leader down.
5. Followers cannot fetch B anymore, since its only owner is down. Yet some
of the replicas needs to take over the leader responsibility. Say, broker 2
now becomes the leader, 3 is the follower.
6. Producer sends message C to the leader (broker 2). Follower fetches it.
I don't quite understand the state of the log on replicas 2 and 3 after
step#6. It looks like the log will have a gap in it. The expected log state
is ["A", "B", "C"]. But brokers 2 and 3 didn't have a chance to fetch "B",
so their log looks like ["A", "C"]. Will Kafka try to fill the gap in the
background once broker 1 started over?
2014-06-18 19:59 GMT+04:00 Neha Narkhede <ne...@gmail.com>:
> You don't gain much by running #4 between broker bounces. Running it after
> the cluster is upgraded will be sufficient.
>
> Thanks,
> Neha
>
>
> On Wed, Jun 18, 2014 at 8:33 AM, Yury Ruchin <yu...@gmail.com>
> wrote:
>
> > Hi folks,
> >
> > In my project, we want to perform to update our active Kafka 0.8 cluster
> to
> > Kafka 0.8.1.1 without downtime and losing any data. The process (after
> > reading http://kafka.apache.org/documentation.html#upgrade) looks to me
> > like this. For each broker in turn:
> >
> > 1. Bring the broker down.
> > 2. Update Kafka to 0.8.1.1 on the broker node.
> > 3. Start the broker.
> > 4. Run preferred-replica-election script to restore broker's leadership
> for
> > respective partitions.
> > 5. Wait for the the preferred replica election to complete.
> >
> > I deem step#5 necessary since preferred replica election is an
> asynchronous
> > process. There is a slim chance that bringing other brokers down before
> the
> > election is complete would result in all replicas down for some
> partitions,
> > so a portion of the incoming data stream would be lost. Is my
> understanding
> > of the process correct?
> >
>
Re: Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail
Posted by Neha Narkhede <ne...@gmail.com>.
You don't gain much by running #4 between broker bounces. Running it after
the cluster is upgraded will be sufficient.
Thanks,
Neha
On Wed, Jun 18, 2014 at 8:33 AM, Yury Ruchin <yu...@gmail.com> wrote:
> Hi folks,
>
> In my project, we want to perform to update our active Kafka 0.8 cluster to
> Kafka 0.8.1.1 without downtime and losing any data. The process (after
> reading http://kafka.apache.org/documentation.html#upgrade) looks to me
> like this. For each broker in turn:
>
> 1. Bring the broker down.
> 2. Update Kafka to 0.8.1.1 on the broker node.
> 3. Start the broker.
> 4. Run preferred-replica-election script to restore broker's leadership for
> respective partitions.
> 5. Wait for the the preferred replica election to complete.
>
> I deem step#5 necessary since preferred replica election is an asynchronous
> process. There is a slim chance that bringing other brokers down before the
> election is complete would result in all replicas down for some partitions,
> so a portion of the incoming data stream would be lost. Is my understanding
> of the process correct?
>