You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Colin McCabe <cm...@apache.org> on 2020/10/05 16:02:19 UTC
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
On Mon, Sep 28, 2020, at 11:41, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the reply. A few more comments below.
>
> 62.
> 62.1 controller.listener.names: So, is this used for the controller or the
> broker trying to connect to the controller?
>
Hi Jun,
It's used by both. The broker tries to connect to controllers by using the first listener in this list. The controller uses the list to determine which listeners it should bind to.
>
> 62.2 If we want to take the approach to share the configs that are common
> between the broker and the controller, should we share the id too?
>
On nodes where we are running both the controller and broker roles, we need two IDs: one for the controller, and one for the broker. So we can't share the same configuration.
>
> 62.3 We added some configs in KIP-595 prefixed with "quorum" and we plan to
> add some controller specific configs prefixed with "controller". KIP-630
> plans to add some other controller specific configs with no prefix. Should
> we standardize all controller specific configs with the same prefix?
>
That's a good question. As Jose said elsewhere in the thread, let's use "metadata" as the prefix.
>
> 70. Could you explain the impact of process.roles a bit more? For example,
> if process.roles=controller, does the node still maintain metadata cache as
> described in KIP-630? Do we still return the host/port for those nodes in
> the metadata response?
>
No, the node does not need any broker data structures if it is not a broker. Controllers also don't handle MetadataRequests. Clients will still go to the brokers for those.
best,
Colin
> Thanks,
>
> Jun
>
> On Mon, Sep 28, 2020 at 9:24 AM Colin McCabe <cm...@apache.org> wrote:
>
> > On Fri, Sep 25, 2020, at 17:35, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply.
> > >
> > > 62. Thinking about this more, I am wondering what's our overall strategy
> > > for configs shared between the broker and the controller. For example,
> > both
> > > the broker and the controller have to define listeners. So both need
> > > configs like listeners/advertised.listeners. Both the new controller and
> > > the broker replicate data. So both need to define some replication
> > related
> > > configurations (replica.fetch.min.bytes, replica.fetch.wait.max.ms,
> > etc).
> > > Should we let the new controller share those configs with brokers or
> > should
> > > we define new configs just for the controller? It seems that we want to
> > > apply the same strategy consistently for all shared configs?
> > >
> >
> > Hi Jun,
> >
> > That's a good question. I think that we should share as many
> > configurations as possible. There will be a few cases where this isn't
> > possible, and we need to create a new configuration key that is
> > controller-only, but I think that will be relatively rare.
> >
> > >
> > > 63. If a node has process.roles set to controller, does one still need to
> > > set broker.id on this node?
> > >
> >
> > No, broker.id does not need to be set in that case.
> >
> > best,
> > Colin
> >
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > > On Fri, Sep 25, 2020 at 2:17 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Fri, Sep 25, 2020, at 10:17, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > 60. Yes, I think you are right. We probably need the controller id
> > when a
> > > > > broker starts up. A broker only stores the Raft leader id in the
> > metadata
> > > > > file. To do the initial fetch to the Raft leader, it needs to know
> > the
> > > > > host/port associated with the leader id.
> > > > >
> > > > > 62. It seems there are 2 parts to this : (1) which listener a client
> > > > should
> > > > > use to initiate a connection to the controller and (2) which listener
> > > > > should a controller use to accept client requests. For (1), at any
> > point
> > > > of
> > > > > time, a client only needs to use one listener. I think
> > > > > controller.listener.name is meant for the client.
> > > >
> > > > Hi Jun,
> > > >
> > > > controller.listener.names is also used by the controllers. In the case
> > > > where we have a broker and a controller in the same JVM, we have a
> > single
> > > > config file. Then we need to know which listeners belong to the
> > controller
> > > > and which belong to the broker component. That's why it's a list.
> > > >
> > > > > So, a single value seems
> > > > > to make more sense. Currently, we don't have a configuration for
> > (2). We
> > > > > could add a new one for that and support a list. I am wondering how
> > > > useful
> > > > > it will be. One example that I can think of is that we can reject
> > > > > non-controller related requests if accepted on controller-only
> > listeners.
> > > > > However, we already support separate authentication for the
> > controller
> > > > > listener. So, not sure how useful it is.
> > > >
> > > > The controller always has a separate listener and does not share
> > listeners
> > > > with the broker. The main reason for this is to avoid giving clients
> > the
> > > > ability to launch a denial-of-service attack on the controller by
> > flooding
> > > > a broker port. A lot of times, these attacks are made unintentionally
> > by
> > > > poorly coded or configured clients. Additionally, broker ports can
> > also be
> > > > very busy with large fetch requests, and so on. It's just a bad
> > > > configuration in general to try to overload the same port for the
> > > > controller and broker, and we don't want to allow it. It would be a
> > > > regression to go from the current system where control requests are
> > safely
> > > > isolated on a separate port, to one where they're not. It also makes
> > the
> > > > code and configuration a lot messier.
> > > >
> > > > >
> > > > > 63. (a) I think most users won't know controller.id defaults to
> > > > broker.id +
> > > > > 3000. So, it can be confusing for them to set up controller.connect.
> > If
> > > > > this is truly needed, it seems that it's less confusing to make
> > > > > controller.id required.
> > > > > (b) I am still trying to understand if we truly need to expose a
> > > > > controller.id. What if we only expose broker.id and let
> > > > controller.connect
> > > > > just use broker.id? What will be missing?
> > > >
> > > > The controller has a separate ID from the broker, so knowing broker.id
> > is
> > > > not helpful here.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, Sep 24, 2020 at 10:55 PM Colin McCabe <cm...@apache.org>
> > > > wrote:
> > > > >
> > > > > > On Thu, Sep 24, 2020, at 16:24, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the reply and the updated KIP. A few more comments
> > below.
> > > > > > >
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > >
> > > > > > > 53. It seems that you already incorporated the changes in
> > KIP-516.
> > > > With
> > > > > > > topic ids, we don't need to wait for the topic's data to be
> > deleted
> > > > > > before
> > > > > > > removing the topic metadata. If the topic is recreated, we can
> > still
> > > > > > delete
> > > > > > > the data properly based on the topic id. So, it seems that we can
> > > > remove
> > > > > > > TopicRecord.Deleting.
> > > > > > >
> > > > > >
> > > > > > Thanks for the reply. What I was thinking of doing here was using
> > > > topic
> > > > > > IDs internally, but still using names externally. So the topic
> > UUIDs
> > > > are
> > > > > > only for the purpose of associating topics with partitions -- from
> > the
> > > > > > user's point of view, topics are still identified by names.
> > > > > >
> > > > > > You're right that KIP-516 will simplify things, but I'm not sure
> > when
> > > > that
> > > > > > will land, so I wanted to avoid blocking the initial
> > implementation of
> > > > this
> > > > > > KIP on it.
> > > > > >
> > > > > > >
> > > > > > > 55. It seems to me that the current behavior where we favor the
> > > > current
> > > > > > > broker registration is better. This is because uncontrolled
> > broker
> > > > > > shutdown
> > > > > > > is rare and its impact is less severe since one just needs to
> > wait
> > > > for
> > > > > > the
> > > > > > > session timeout before restarting the broker. If a mis-configured
> > > > broker
> > > > > > > replaces an existing broker, the consequence is more severe
> > since it
> > > > can
> > > > > > > cause the leader to be unavailable, a replica to be out of ISR,
> > and
> > > > add
> > > > > > > more load on the leaders etc.
> > > > > > >
> > > > > >
> > > > > > Hmm, that's a good point. Let me check this a bit more before I
> > change
> > > > > > it, though.
> > > > > >
> > > > > > >
> > > > > > > 60. controller.connect=0@controller0.example.com:9093,
> > > > > > > 1@controller1.example.com:9093,2@controller2.example.com : Do we
> > > > need to
> > > > > > > include the controller id before @? It seems that the host/port
> > is
> > > > enough
> > > > > > > for establishing the connection. It would also be useful to make
> > it
> > > > clear
> > > > > > > that controller.connect replaces quorum.voters in KIP-595.
> > > > > > >
> > > > > >
> > > > > > I discussed this with Jason earlier, and he felt that the
> > controller
> > > > IDs
> > > > > > were needed in this configuration key. It is certainly needed when
> > > > > > configuring the controllers themselves, since they need to know
> > each
> > > > > > others' IDs.
> > > > > >
> > > > > > >
> > > > > > > 61. I am not sure that we need both controller.listeners and
> > > > > > > controller.connect.security.protocol since the former implies the
> > > > > > security
> > > > > > > protocol. The reason that we have both
> > inter.broker.listener.name
> > > > and
> > > > > > > inter.broker.security.protocol is because we had the latter
> > first and
> > > > > > later
> > > > > > > realized that the former is more general.
> > > > > > >
> > > > > >
> > > > > > That's a good point. I've removed this from the KIP.
> > > > > >
> > > > > > >
> > > > > > > 62. I am still not sure that you need controller.listeners to be
> > a
> > > > list.
> > > > > > > All listeners are already defined in listeners. To migrate from
> > > > plaintext
> > > > > > > to SSL, one can configure listeners to have both plaintext and
> > SSL.
> > > > After
> > > > > > > that, one can just change controller.listeners from plaintext to
> > SSL.
> > > > > > This
> > > > > > > is similar to how to change the listener for inter broker
> > > > connections.
> > > > > > > Also, controller.listener.name may be a more accurate name?
> > > > > > >
> > > > > >
> > > > > > The issue that I see here is that if you are running with the
> > > > controller
> > > > > > and broker in the same JVM, if you define a few listeners in
> > > > "listeners"
> > > > > > they will get used as regular broker listeners, unless you put
> > them in
> > > > > > controller.listeners. Therefore, controller.listeners needs to be
> > a
> > > > list.
> > > > > >
> > > > > > controller.listener.names does sound like a better name, though...
> > I've
> > > > > > updated it to that.
> > > > > >
> > > > > > >
> > > > > > > 63. Regarding controller.id, I am trying to understand whether
> > it's
> > > > a
> > > > > > > required configuration or an optional one. From the KIP, it
> > sounds
> > > > like
> > > > > > > controller.id is optional. Then, if this is unset, it's not
> > clear
> > > > how
> > > > > > the
> > > > > > > user will obtain the controller id for setting
> > controller.connect.
> > > > > > >
> > > > > >
> > > > > > If you specify broker.id but not controller.id, then controller.id
> > > > will
> > > > > > just be set to broker.id + 3000. This was intended to make some
> > > > > > configurations a little bit more concise to write. You still do
> > have
> > > > to
> > > > > > know the controller IDs when configuring the brokers, though. If
> > this
> > > > > > seems confusing then I can just make it mandatory.
> > > > > >
> > > > > > >
> > > > > > > 64. KIP-516 adds a flag in LeaderAndIsrRequest to indicate
> > whether it
> > > > > > > includes all partitions or not. This will be used to clean up
> > strayed
> > > > > > logs.
> > > > > > > I was thinking how we will do the same thing once the controller
> > > > moves to
> > > > > > > Raft based. One way to do that is on broker startup, it gets the
> > HWM
> > > > in
> > > > > > the
> > > > > > > metadata log from the Raft leader and waits until its metadata
> > > > catches up
> > > > > > > to HWM. At that point, any local log not seen in the metadata
> > can be
> > > > > > > removed. Since the Fetch response returns the HWM, there seems
> > to be
> > > > > > enough
> > > > > > > APIs to achieve this.
> > > > > > >
> > > > > >
> > > > > > That's a very good point. I added a note about this under Broker
> > > > Startup.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 24, 2020 at 1:07 PM Colin McCabe <co...@cmccabe.xyz>
> > > > wrote:
> > > > > > >
> > > > > > > > On Mon, Sep 21, 2020, at 18:13, Jun Rao wrote:
> > > > > > > > > Hi, Colin,
> > > > > > > > >
> > > > > > > > > Sorry for the late reply. A few more comments below.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > Thanks for taking another look.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 50. Configurations
> > > > > > > > > 50.1 controller.listeners: It seems that a controller just
> > needs
> > > > one
> > > > > > > > > listener. Why do we need to have a list here? Also, could you
> > > > > > provide an
> > > > > > > > > example of how this is set and what's its relationship with
> > > > existing
> > > > > > > > > configs such as "security.inter.broker.protocol" and "
> > > > > > > > > inter.broker.listener.name"?
> > > > > > > > >
> > > > > > > >
> > > > > > > > I agree that most administrators will want to run with only one
> > > > > > controller
> > > > > > > > listener. However, just as with brokers, it is nice to have
> > the
> > > > > > option to
> > > > > > > > expose multiple ports.
> > > > > > > >
> > > > > > > > One reason why you might want multiple ports is if you were
> > doing a
> > > > > > > > migration from plaintext to SSL. You could add new SSL
> > listeners
> > > > but
> > > > > > > > continue to expose the PLAINTEXT listeners. Then you could
> > add the
> > > > > > new SSL
> > > > > > > > controller config to each broker and roll each broker. Then
> > you
> > > > could
> > > > > > > > migrate from PLAINTEXT to SSL with no downtime.
> > > > > > > >
> > > > > > > > Here's an example configuration for the controller:
> > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com
> > > > > > > > controller.listeners=CONTROLLER
> > > > > > > > listeners=CONTROLLER://controller0.example.com:9093
> > > > > > > > listener.security.protocol.map=CONTROLLER:SSL
> > > > > > > >
> > > > > > > > Here's an example configuration for the broker:
> > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com
> > > > > > > > controller.connect.security.protocol=SSL
> > > > > > > >
> > > > > > > > security.inter.broker.protocol or inter.broker.listener.name
> > do
> > > > not
> > > > > > > > affect how the broker communicates with the controller. Those
> > > > > > > > configurations relate to how brokers communicate with each
> > other,
> > > > but
> > > > > > the
> > > > > > > > controller is not a broker.
> > > > > > > >
> > > > > > > > (Note that I just added controller.connect.security.protocol
> > to the
> > > > > > KIP --
> > > > > > > > I had forgotten to put it in earlier)
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 50.2 registration.heartbeat.interval.ms and
> > > > > > > > registration.lease.timeout.ms.
> > > > > > > > > Should we match their default value with the corresponding
> > > > default
> > > > > > for
> > > > > > > > ZK?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Fair enough. I'll set them to the values of the
> > > > > > zookeeper.sync.time.ms
> > > > > > > > and zookeeper.connection.timeout.ms configurations. I do
> > think we
> > > > > > should
> > > > > > > > experiment later on to see what works well here, but the ZK
> > values
> > > > are
> > > > > > at
> > > > > > > > least a starting point.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 50.3 controller.connect: Could you provide an example? I am
> > > > > > wondering how
> > > > > > > > > it differs from quorum.voters=1@kafka-1:9092, 2@kafka-2
> > :9092,
> > > > > > 3@kafka-3
> > > > > > > > > :9092.
> > > > > > > > >
> > > > > > > >
> > > > > > > > controller.connect is intended to be the new name for
> > > > quorum.voters.
> > > > > > > > During the vote for KIP-595, we sort of agreed to defer the
> > > > discussion
> > > > > > > > about what this configuration should be called. I proposed
> > this
> > > > new
> > > > > > name
> > > > > > > > because it makes it clear what the configuration is for (how to
> > > > > > connect to
> > > > > > > > the controllers).
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 50.4 controller.id: I am still not sure how this is being
> > used.
> > > > > > Could
> > > > > > > > you
> > > > > > > > > explain this in more detail?
> > > > > > > > >
> > > > > > > >
> > > > > > > > The controller ID needs to be set on each controller. It also
> > > > appears
> > > > > > in
> > > > > > > > controller.connect as the thing before the "at" sign. Its
> > > > function is
> > > > > > > > pretty similar to broker.id.
> > > > > > > >
> > > > > > > > Broker IDs and controller IDs exist in the same ID space, so
> > you
> > > > can't
> > > > > > > > have a broker and a controller that use the same ID.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 51. BrokerHeartbeat: It seems a bit wasteful to include
> > > > Listeners in
> > > > > > > > every
> > > > > > > > > heartbeat request since it typically doesn't change. Could we
> > > > make
> > > > > > that
> > > > > > > > an
> > > > > > > > > optional field?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Ok.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 52. KIP-584 adds a new ZK node /features. Should we add a
> > > > > > corresponding
> > > > > > > > > metadata record?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Good point. I added FeatureLevelRecord.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 53. TopicRecord and DeleteTopic: Both DeleteTopic and
> > > > > > > > TopicRecord.Deleting
> > > > > > > > > indicate topic deletion. Could we outline the flow when each
> > > > will be
> > > > > > set?
> > > > > > > > > In particular, which one indicates the intention to delete
> > and
> > > > which
> > > > > > one
> > > > > > > > > indicates the completion of the deletion.
> > > > > > > > >
> > > > > > > >
> > > > > > > > TopicRecord.Deleting is set when we intend to delete the topic.
> > > > > > > >
> > > > > > > > DeleteTopic removes the topic completely. I will rename
> > > > DeleteTopic to
> > > > > > > > RemoveTopic to make this clearer.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 54. "The controller can generate a new broker epoch by using
> > the
> > > > > > latest
> > > > > > > > log
> > > > > > > > > offset." Which offset is that? Is it the offset of the
> > metadata
> > > > > > topic for
> > > > > > > > > the corresponding BrokerRecord? Is it guaranteed to be
> > different
> > > > on
> > > > > > each
> > > > > > > > > broker restart?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Yes.... a new broker epoch implies that there has been a new
> > record
> > > > > > > > created in the metadata log. Therefore the last committed
> > offset
> > > > must
> > > > > > be
> > > > > > > > different.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 55. "Thereafter, it may lose subsequent conflicts if its
> > broker
> > > > > > epoch is
> > > > > > > > > stale. (See KIP-380 for some background on broker epoch.)
> > The
> > > > > > reason
> > > > > > > > for
> > > > > > > > > favoring new processes is to accommodate the common case
> > where a
> > > > > > process
> > > > > > > > is
> > > > > > > > > killed with kill -9 and then restarted. " Are you saying
> > that if
> > > > > > there is
> > > > > > > > > an active broker registered in the controller, a new broker
> > > > heartbeat
> > > > > > > > > request with the INITIAL state will fence the current broker
> > > > > > session? Not
> > > > > > > > > sure about this. For example, this will allow a broker with
> > > > > > incorrectly
> > > > > > > > > configured broker id to kill an existing valid broker.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Yes, a new broker with an incorrectly configured broker id
> > could
> > > > fence
> > > > > > an
> > > > > > > > existing valid broker.
> > > > > > > >
> > > > > > > > The goal of the conflict resolution process described here is
> > to
> > > > avoid
> > > > > > > > having two brokers go back and forth in claiming the broker id.
> > > > > > Choosing
> > > > > > > > the newest was just an easy way to do that.
> > > > > > > >
> > > > > > > > Choosing the oldest is another possibility, and more inline
> > with
> > > > what
> > > > > > > > ZooKeeper does today. The problem with that is that it would
> > not
> > > > be
> > > > > > > > possible to re-create the broker if the old instance died
> > suddenly,
> > > > > > except
> > > > > > > > by waiting for the full lease timeout.
> > > > > > > >
> > > > > > > > It might be possible to do a little better here by breaking the
> > > > lease
> > > > > > if
> > > > > > > > the TCP connection from the broker drops. But that requires
> > > > crossing
> > > > > > some
> > > > > > > > layers of abstraction in the Kafka networking stack. There's
> > also
> > > > the
> > > > > > > > possibility of false negatives or positives. For example, TCP
> > > > > > connections
> > > > > > > > sometimes just drop even if the node is still there. Or
> > sometimes
> > > > the
> > > > > > > > networking stack keeps a TCP connection alive for a while when
> > the
> > > > > > other
> > > > > > > > end is gone.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 56. kafka-storage.sh:
> > > > > > > > > 56.1 In the info mode, what other information does it show in
> > > > > > addition to
> > > > > > > > > "kip.500.mode=enabled"?
> > > > > > > > >
> > > > > > > >
> > > > > > > > I think it should show a list of all the storage directories,
> > and
> > > > > > whether
> > > > > > > > each one is formatted. In addition, it should show whether
> > > > > > kip.500.mode is
> > > > > > > > enabled, and what the cluster id is.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 56.2 Should the format mode take the config file as the
> > input too
> > > > > > like
> > > > > > > > the
> > > > > > > > > info mode?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Yes, that's a good idea.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Thu, Sep 17, 2020 at 4:12 PM Colin McCabe <
> > cmccabe@apache.org
> > > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Unmesh,
> > > > > > > > > >
> > > > > > > > > > That's a fair point. I have moved the lease duration to
> > the
> > > > broker
> > > > > > > > > > heartbeat response. That way lease durations can be
> > changed
> > > > just
> > > > > > be
> > > > > > > > > > reconfiguring the controllers.
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > On Wed, Sep 16, 2020, at 07:40, Unmesh Joshi wrote:
> > > > > > > > > > > Thanks Colin, the changes look good to me. One small
> > thing.
> > > > > > > > > > > registration.lease.timeout.ms is the configuration on
> > the
> > > > > > controller
> > > > > > > > > > side.
> > > > > > > > > > > It will be good to comment on how brokers know about it,
> > to
> > > > be
> > > > > > able
> > > > > > > > to
> > > > > > > > > > > send LeaseDurationMs
> > > > > > > > > > > in the heartbeat request,
> > > > > > > > > > > or else it can be added in the heartbeat response for
> > > > brokers to
> > > > > > know
> > > > > > > > > > about
> > > > > > > > > > > it.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Unmesh
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Sep 11, 2020 at 10:32 PM Colin McCabe <
> > > > > > cmccabe@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > >
> > > > > > > > > > > > I think you're right that we should use a duration here
> > > > rather
> > > > > > > > than a
> > > > > > > > > > > > time. As you said, the clock on the controller will
> > > > probably
> > > > > > not
> > > > > > > > > > match the
> > > > > > > > > > > > one on the broker. I have updated the KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > > > It's important to keep in mind that messages may be
> > > > > > delayed in
> > > > > > > > the
> > > > > > > > > > > > > > network, or arrive out of order. When this
> > happens, we
> > > > > > will
> > > > > > > > use
> > > > > > > > > > the
> > > > > > > > > > > > start
> > > > > > > > > > > > > > time specified in the request to determine if the
> > > > request
> > > > > > is
> > > > > > > > stale.
> > > > > > > > > > > > > I am assuming that there will be a single TCP
> > connection
> > > > > > > > maintained
> > > > > > > > > > > > between
> > > > > > > > > > > > > broker and active controller. So, there won't be any
> > out
> > > > of
> > > > > > order
> > > > > > > > > > > > requests?
> > > > > > > > > > > > > There will be a scenario of broker GC pause, which
> > might
> > > > > > cause
> > > > > > > > > > connection
> > > > > > > > > > > > > timeout and broker might need to reestablish the
> > > > connection.
> > > > > > If
> > > > > > > > the
> > > > > > > > > > pause
> > > > > > > > > > > > > is too long, lease will expire and the heartbeat sent
> > > > after
> > > > > > the
> > > > > > > > pause
> > > > > > > > > > > > will
> > > > > > > > > > > > > be treated as a new registration (similar to restart
> > > > case),
> > > > > > and
> > > > > > > > a new
> > > > > > > > > > > > epoch
> > > > > > > > > > > > > number will be assigned to the broker.
> > > > > > > > > > > >
> > > > > > > > > > > > I agree with the end of this paragraph, but not with
> > the
> > > > start
> > > > > > :)
> > > > > > > > > > > >
> > > > > > > > > > > > There can be out-of-order requests, since the broker
> > will
> > > > > > simply
> > > > > > > > use a
> > > > > > > > > > new
> > > > > > > > > > > > TCP connection if the old one has problems. This can
> > > > happen
> > > > > > for a
> > > > > > > > > > variety
> > > > > > > > > > > > of reasons. I don't think GC pauses are the most
> > common
> > > > > > reason for
> > > > > > > > > > this to
> > > > > > > > > > > > happen. It's more common to see issues issues in the
> > > > network
> > > > > > > > itself
> > > > > > > > > > that
> > > > > > > > > > > > result connections getting dropped from time to time.
> > > > > > > > > > > >
> > > > > > > > > > > > So we have to assume that messages may arrive out of
> > > > order, and
> > > > > > > > > > possibly
> > > > > > > > > > > > be delayed. I added a note that heartbeat requests
> > should
> > > > be
> > > > > > > > ignored
> > > > > > > > > > if
> > > > > > > > > > > > the metadata log offset they contain is smaller than a
> > > > previous
> > > > > > > > > > heartbeat.
> > > > > > > > > > > >
> > > > > > > > > > > > > When the active controller fails, the new active
> > > > controller
> > > > > > > > needs to
> > > > > > > > > > be
> > > > > > > > > > > > > sure that it considers all the known brokers as alive
> > > > till
> > > > > > the
> > > > > > > > lease
> > > > > > > > > > > > > expiration interval. Because
> > > > registration.lease.timeout.ms,
> > > > > > is
> > > > > > > > > > > > configured
> > > > > > > > > > > > > on the controller, the new active controller will
> > extend
> > > > all
> > > > > > the
> > > > > > > > > > leases
> > > > > > > > > > > > by
> > > > > > > > > > > > > registration.lease.timeout.ms. I see that it won't
> > need
> > > > last
> > > > > > > > > > heartbeat
> > > > > > > > > > > > > time.
> > > > > > > > > > > >
> > > > > > > > > > > > Agreed.
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat, Sep 5, 2020 at 1:28 AM Colin McCabe <
> > > > > > cmccabe@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > Colin wrote:
> > > > > > > > > > > > > > > > The reason for including LeaseStartTimeMs in
> > the
> > > > > > request
> > > > > > > > is to
> > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > that the time required to communicate with the
> > > > > > controller
> > > > > > > > gets
> > > > > > > > > > > > > > included in
> > > > > > > > > > > > > > > > the lease time. Since requests can
> > potentially be
> > > > > > delayed
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > for a long time, this is important.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Aug 31, 2020, at 05:58, Unmesh Joshi wrote:
> > > > > > > > > > > > > > > The network time will be added anyway, because
> > the
> > > > lease
> > > > > > > > timer
> > > > > > > > > > on the
> > > > > > > > > > > > > > > active controller will start only after the
> > heartbeat
> > > > > > request
> > > > > > > > > > > > reaches the
> > > > > > > > > > > > > > > server.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If the start time is not specified in the request,
> > > > then the
> > > > > > > > network
> > > > > > > > > > > > time
> > > > > > > > > > > > > > is excluded from the heartbeat time.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Here's an example:
> > > > > > > > > > > > > > Let's say broker A sends a heartbeat at time 100,
> > and
> > > > it
> > > > > > > > arrives
> > > > > > > > > > on the
> > > > > > > > > > > > > > controller at time 200, and the lease duration is
> > 1000.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The controller looks at the start time in the
> > request,
> > > > > > which is
> > > > > > > > > > 100,
> > > > > > > > > > > > and
> > > > > > > > > > > > > > adds 1000 to it, getting 1100. On the other hand,
> > if
> > > > start
> > > > > > > > time
> > > > > > > > > > is not
> > > > > > > > > > > > > > specified in the request, then the expiration will
> > be
> > > > at
> > > > > > time
> > > > > > > > 1200.
> > > > > > > > > > > > > > That is what I mean by "the network time is
> > included
> > > > in the
> > > > > > > > > > expiration
> > > > > > > > > > > > > > time."
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > And I think, some assumption about network round
> > trip
> > > > > > time is
> > > > > > > > > > > > > > > needed anyway to decide on the frequency of the
> > > > > > heartbeat (
> > > > > > > > > > > > > > > registration.heartbeat.interval.ms), and lease
> > > > timeout (
> > > > > > > > > > > > > > > registration.lease.timeout.ms). So I think just
> > > > having a
> > > > > > > > > > leaseTTL
> > > > > > > > > > > > in the
> > > > > > > > > > > > > > > request is easier to understand and implement.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It's important to keep in mind that messages may be
> > > > > > delayed in
> > > > > > > > the
> > > > > > > > > > > > > > network, or arrive out of order. When this
> > happens, we
> > > > > > will
> > > > > > > > use
> > > > > > > > > > the
> > > > > > > > > > > > start
> > > > > > > > > > > > > > time specified in the request to determine if the
> > > > request
> > > > > > is
> > > > > > > > stale.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on the
> > > > controller
> > > > > > side
> > > > > > > > > > should
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > reset in the case of controller failover. The
> > > > > > alternative
> > > > > > > > > > would
> > > > > > > > > > > > be to
> > > > > > > > > > > > > > > > track the lease as hard state rather than soft
> > > > state,
> > > > > > but I
> > > > > > > > > > think
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > is not really needed, and would result in more
> > log
> > > > > > entries.
> > > > > > > > > > > > > > > My interpretation of the mention of BrokerRecord
> > in
> > > > the
> > > > > > KIP
> > > > > > > > was
> > > > > > > > > > that
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > record exists in the Raft log.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > BrokerRecord does exist in the Raft log, but does
> > not
> > > > > > include
> > > > > > > > the
> > > > > > > > > > last
> > > > > > > > > > > > > > heartbeat time.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > By soft state, do you mean the broker
> > > > > > > > > > > > > > > records exist only on the active leader and will
> > not
> > > > be
> > > > > > > > > > replicated
> > > > > > > > > > > > in the
> > > > > > > > > > > > > > > raft log? If the live brokers list is maintained
> > > > only on
> > > > > > the
> > > > > > > > > > active
> > > > > > > > > > > > > > > controller (raft leader), then, in case of leader
> > > > > > failure,
> > > > > > > > there
> > > > > > > > > > > > will be
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > window where the new leader does not know about
> > the
> > > > live
> > > > > > > > brokers,
> > > > > > > > > > > > till
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > brokers establish the leases again.
> > > > > > > > > > > > > > > I think it will be safer to have leases as a hard
> > > > state
> > > > > > > > managed
> > > > > > > > > > by
> > > > > > > > > > > > > > standard
> > > > > > > > > > > > > > > Raft replication.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Leases are short, so the need to re-establish them
> > > > after a
> > > > > > > > > > controller
> > > > > > > > > > > > > > failover doesn't seem like a big problem. But
> > this is
> > > > > > > > something
> > > > > > > > > > we can
> > > > > > > > > > > > > > tweak if it becomes an issue. One option would be
> > to
> > > > have
> > > > > > a
> > > > > > > > > > separate
> > > > > > > > > > > > log
> > > > > > > > > > > > > > which is only used by the controller nodes for this
> > > > (since,
> > > > > > > > after
> > > > > > > > > > all,
> > > > > > > > > > > > > > brokers don't care about registration renewals).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Or am I misunderstanding something? (I assume
> > that
> > > > with
> > > > > > soft
> > > > > > > > > > state,
> > > > > > > > > > > > you
> > > > > > > > > > > > > > > mean something like zookeeper local sessions
> > > > > > > > > > > > > > >
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1147
> > > > .)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Our code is single threaded as well. I think
> > it
> > > > makes
> > > > > > > > sense
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > controller, since otherwise locking becomes
> > very
> > > > messy.
> > > > > > > > I'm
> > > > > > > > > > not
> > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > understand your question about duplicate
> > broker ID
> > > > > > > > detection,
> > > > > > > > > > > > though.
> > > > > > > > > > > > > > > There's a section in the KIP about this -- is
> > there a
> > > > > > detail
> > > > > > > > we
> > > > > > > > > > > > should
> > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is an implementation detail that doesn't need
> > to
> > > > be
> > > > > > in the
> > > > > > > > > > KIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I assumed broker leases are implemented as a hard
> > > > state.
> > > > > > In
> > > > > > > > that
> > > > > > > > > > > > case, to
> > > > > > > > > > > > > > > check for broker id conflict, we need to check
> > the
> > > > broker
> > > > > > > > ids at
> > > > > > > > > > two
> > > > > > > > > > > > > > places
> > > > > > > > > > > > > > > 1. Pending broker registrations (which are yet
> > to be
> > > > > > > > committed)
> > > > > > > > > > 2.
> > > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > committed broker registrations.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Aug 31, 2020 at 5:42 PM Colin McCabe <
> > > > > > > > cmccabe@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Sat, Aug 29, 2020, at 01:12, Unmesh Joshi
> > wrote:
> > > > > > > > > > > > > > > > > >>>Can you repeat your questions about broker
> > > > leases?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >>>>The LeaseStartTimeMs is expected to be
> > the
> > > > > > broker's
> > > > > > > > > > > > > > > > > 'System.currentTimeMillis()' at the point of
> > the
> > > > > > > > request. The
> > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > controller will add its lease period to this
> > in
> > > > order
> > > > > > > > >>>>to
> > > > > > > > > > > > compute
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > LeaseEndTimeMs.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think the use of LeaseStartTimeMs and
> > > > > > LeaseEndTimeMs
> > > > > > > > in the
> > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > confusing. Monotonic Clock
> > (System.nanoTime) on
> > > > the
> > > > > > > > active
> > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > should be used to track leases.
> > > > > > > > > > > > > > > > > (For example,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1616https://github.com/etcd-io/etcd/pull/6888/commits/e7f4010ccaf28b6ce64fe514d25a4b2fa459d114
> > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Then we will not need LeaseStartTimeMs?
> > > > > > > > > > > > > > > > > Instead of LeaseStartTimeMs, can we call it
> > > > > > LeaseTTL? The
> > > > > > > > > > active
> > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > can then calculate LeaseEndTime =
> > > > System.nanoTime() +
> > > > > > > > > > LeaseTTL.
> > > > > > > > > > > > > > > > > In this case we might just drop
> > LeaseEndTimeMs
> > > > from
> > > > > > the
> > > > > > > > > > > > response, as
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > broker already knows about the TTL and can
> > send
> > > > > > > > heartbeats at
> > > > > > > > > > > > some
> > > > > > > > > > > > > > > > fraction
> > > > > > > > > > > > > > > > > of TTL, say every TTL/4 milliseconds.(elapsed
> > > > time
> > > > > > on the
> > > > > > > > > > broker
> > > > > > > > > > > > > > measured
> > > > > > > > > > > > > > > > > by System.nanoTime)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I agree that the monotonic clock is probably a
> > > > better
> > > > > > idea
> > > > > > > > > > here.
> > > > > > > > > > > > It is
> > > > > > > > > > > > > > > > good to be robust against wall clock changes,
> > > > although
> > > > > > I
> > > > > > > > think
> > > > > > > > > > a
> > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > which had them might suffer other issues. I
> > will
> > > > > > change
> > > > > > > > it to
> > > > > > > > > > > > specify
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > monotonic clock.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The reason for including LeaseStartTimeMs in
> > the
> > > > > > request
> > > > > > > > is to
> > > > > > > > > > > > ensure
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > the time required to communicate with the
> > > > controller
> > > > > > gets
> > > > > > > > > > included
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > lease time. Since requests can potentially be
> > > > delayed
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > network
> > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > long time, this is important.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I have a prototype built to demonstrate this
> > as
> > > > > > > > following:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The Kip631Controller itself depends on a
> > > > Consensus
> > > > > > > > module, to
> > > > > > > > > > > > > > demonstrate
> > > > > > > > > > > > > > > > > how possible interactions with the consensus
> > > > module
> > > > > > will
> > > > > > > > look
> > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > (The Consensus can be pluggable really,
> > with an
> > > > API
> > > > > > to
> > > > > > > > allow
> > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > replicated log upto HighWaterMark)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It has an implementation of LeaseTracker
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/LeaderLeaseTracker.scala
> > > > > > > > > > > > > > > > > to demonstrate LeaseTracker's interaction
> > with
> > > > the
> > > > > > > > consensus
> > > > > > > > > > > > module.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The implementation has the following aspects:
> > > > > > > > > > > > > > > > > 1. The lease tracking happens only on the
> > active
> > > > > > > > controller
> > > > > > > > > > (raft
> > > > > > > > > > > > > > > > > leader)
> > > > > > > > > > > > > > > > > 2. Once the lease expires, it needs to
> > propose
> > > > and
> > > > > > > > commit a
> > > > > > > > > > > > > > FenceBroker
> > > > > > > > > > > > > > > > > record for that lease.
> > > > > > > > > > > > > > > > > 3. In case of active controller failure, the
> > > > lease
> > > > > > will
> > > > > > > > be
> > > > > > > > > > > > tracked by
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > newly raft leader. The new raft leader
> > starts the
> > > > > > lease
> > > > > > > > timer
> > > > > > > > > > > > again,
> > > > > > > > > > > > > > (as
> > > > > > > > > > > > > > > > > implemented in onBecomingLeader method of
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > in effect extending the lease by the time
> > spent
> > > > in
> > > > > > the
> > > > > > > > leader
> > > > > > > > > > > > > > election
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > whatever time was elapsed on the old leader.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on the
> > > > controller
> > > > > > side
> > > > > > > > > > should
> > > > > > > > > > > > be
> > > > > > > > > > > > > > reset
> > > > > > > > > > > > > > > > in the case of controller failover. The
> > > > alternative
> > > > > > would
> > > > > > > > be
> > > > > > > > > > to
> > > > > > > > > > > > track
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > lease as hard state rather than soft state,
> > but I
> > > > think
> > > > > > > > that
> > > > > > > > > > is not
> > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > needed, and would result in more log entries.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There are working tests for this
> > implementation
> > > > here.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > and an end to end test here
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/ProducerConsumerKIP500Test.scala
> > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >>'m not sure what you mean by
> > "de-duplication
> > > > of the
> > > > > > > > > > broker."
> > > > > > > > > > > > Can
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > Apologies for using the confusing term
> > > > > > deduplication. I
> > > > > > > > meant
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > id
> > > > > > > > > > > > > > > > > conflict.
> > > > > > > > > > > > > > > > > As you can see in the prototype
> > handleRequest of
> > > > > > > > > > KIP631Controller
> > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > the duplicate broker id needs to be detected
> > > > before
> > > > > > the
> > > > > > > > > > > > BrokerRecord
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > submitted to the raft module.
> > > > > > > > > > > > > > > > > Also as implemented in the prototype, the
> > > > > > > > KIP631Controller is
> > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > threaded, handling requests one at a time.
> > (an
> > > > > > example of
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://martinfowler.com/articles/patterns-of-distributed-systems/singular-update-queue.html
> > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Our code is single threaded as well. I think
> > it
> > > > makes
> > > > > > > > sense
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > controller, since otherwise locking becomes
> > very
> > > > messy.
> > > > > > > > I'm
> > > > > > > > > > not
> > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > understand your question about duplicate
> > broker ID
> > > > > > > > detection,
> > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > There's a section in the KIP about this -- is
> > > > there a
> > > > > > > > detail we
> > > > > > > > > > > > should
> > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 10:49 AM Colin
> > McCabe <
> > > > > > > > > > colin@cmccabe.xyz
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2020, at 19:36, Unmesh
> > Joshi
> > > > wrote:
> > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There were a few of questions I had..
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for the response.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. Were my comments on the broker lease
> > > > > > > > implementation
> > > > > > > > > > (and
> > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > prototype) appropriate and do we need to
> > > > change
> > > > > > the
> > > > > > > > KIP
> > > > > > > > > > > > > > > > > > > description accordingly?.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Can you repeat your questions about broker
> > > > leases?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2. How will broker epochs be generated?
> > I am
> > > > > > > > assuming it
> > > > > > > > > > can
> > > > > > > > > > > > be
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > committed log offset (like zxid?)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > There isn't any need to use a log offset.
> > We
> > > > can
> > > > > > just
> > > > > > > > > > look at
> > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > in-memory hash table and see what the
> > latest
> > > > > > number is,
> > > > > > > > > > and add
> > > > > > > > > > > > > > one, to
> > > > > > > > > > > > > > > > > > generate a new broker epoch.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3. How will producer registration
> > happen? I
> > > > am
> > > > > > > > assuming
> > > > > > > > > > it
> > > > > > > > > > > > > > should be
> > > > > > > > > > > > > > > > > > > similar to broker registration, with a
> > > > similar
> > > > > > way to
> > > > > > > > > > > > generate
> > > > > > > > > > > > > > > > producer
> > > > > > > > > > > > > > > > > > id.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For the EOS stuff, we will need a few new
> > RPCs
> > > > to
> > > > > > the
> > > > > > > > > > > > controller.
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > we should do that in a follow-on KIP,
> > though,
> > > > since
> > > > > > > > this
> > > > > > > > > > one is
> > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > pretty big.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 4. Because we expose Raft log to all the
> > > > > > brokers, any
> > > > > > > > > > > > > > de-duplication
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > broker needs to happen before the
> > requests
> > > > are
> > > > > > > > proposed
> > > > > > > > > > to
> > > > > > > > > > > > Raft.
> > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > the controller needs to be single
> > threaded,
> > > > and
> > > > > > > > should do
> > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > against the in-process or pending
> > requests
> > > > and
> > > > > > the
> > > > > > > > final
> > > > > > > > > > > > state. I
> > > > > > > > > > > > > > > > read a
> > > > > > > > > > > > > > > > > > > mention of this, in the responses in this
> > > > > > > > thread.Will it
> > > > > > > > > > be
> > > > > > > > > > > > > > useful to
> > > > > > > > > > > > > > > > > > > mention this in the KIP?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'm not sure what you mean by
> > "de-duplication
> > > > of
> > > > > > the
> > > > > > > > > > broker."
> > > > > > > > > > > > Can
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 4:50 AM Colin
> > McCabe
> > > > <
> > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I'm thinking of calling a vote on
> > KIP-631
> > > > on
> > > > > > > > Monday.
> > > > > > > > > > Let
> > > > > > > > > > > > me
> > > > > > > > > > > > > > know
> > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > there's any more comments I should
> > address
> > > > > > before I
> > > > > > > > > > start
> > > > > > > > > > > > the
> > > > > > > > > > > > > > vote.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020, at 05:39, Unmesh
> > > > Joshi
> > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >>Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > >>Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > Thanks. I will be adding more to the
> > > > > > prototype
> > > > > > > > and
> > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > and collaborate.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020 at 12:28 AM
> > Colin
> > > > > > McCabe <
> > > > > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Jose,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > That'a s good point that I hadn't
> > > > > > considered.
> > > > > > > > It's
> > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > a separate leader change message,
> > as
> > > > you
> > > > > > > > mentioned.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2020, at 11:56, Jose
> > > > Garcia
> > > > > > > > Sancio
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Very cool prototype!
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The KIP proposes a record called
> > > > > > IsrChange
> > > > > > > > which
> > > > > > > > > > > > > > includes the
> > > > > > > > > > > > > > > > > > > > > > > partition, topic, isr, leader and
> > > > leader
> > > > > > > > epoch.
> > > > > > > > > > > > During
> > > > > > > > > > > > > > normal
> > > > > > > > > > > > > > > > > > > > > > > operation ISR changes do not
> > result
> > > > in
> > > > > > leader
> > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > Similarly,
> > > > > > > > > > > > > > > > > > > > > > > leader changes do not necessarily
> > > > > > involve ISR
> > > > > > > > > > > > changes.
> > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > implementation that uses ZK
> > modeled
> > > > them
> > > > > > > > together
> > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > > > > > 1. All of this information is
> > stored
> > > > in
> > > > > > one
> > > > > > > > > > znode.
> > > > > > > > > > > > > > > > > > > > > > > 2. ZK's optimistic lock requires
> > > > that you
> > > > > > > > specify
> > > > > > > > > > > > the new
> > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > > > > 3. The change to that znode was
> > being
> > > > > > > > performed
> > > > > > > > > > by
> > > > > > > > > > > > both
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > and the leader.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > None of these reasons are true in
> > > > > > KIP-500.
> > > > > > > > Have
> > > > > > > > > > we
> > > > > > > > > > > > > > considered
> > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > two different records? For
> > example
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. IsrChange record which
> > includes
> > > > topic,
> > > > > > > > > > partition,
> > > > > > > > > > > > isr
> > > > > > > > > > > > > > > > > > > > > > > 2. LeaderChange record which
> > includes
> > > > > > topic,
> > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > leader epoch.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I suspect that making this change
> > > > will
> > > > > > also
> > > > > > > > > > require
> > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > message AlterIsrRequest
> > introduced in
> > > > > > > > KIP-497:
> > > > > > > > > > Add
> > > > > > > > > > > > > > > > inter-broker
> > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > alter ISR.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > -Jose
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Coln. Ignore my previous question about ConfigRecord.ResourceType
having to be a string -- I now see that
org.apache.kafka.common.config.ConfigResource defines the types of
configs for an int8.
I do have a question about how the broker will connect to the
controller. The KIP says that controller.listener.names is "required
if this process is a KIP-500 controller" and "Despite the similar
name, note that this is different from the "control plane listener"
introduced by KIP-291. The "control plane listener" is used on
brokers, not on controllers." This leads me to believe that
controller.listener.names is not used on brokers. But I believe it
will be required, otherwise the only information that the broker will
have is a list of hosts and ports (from controller.connect). I
believe the broker will require a value in controller.listener.names
and then the broker will take that value (the first in the list?) and
convert that to a security protocol via
listener.security.protocol.map. Also, if the security protocol is
SASL_{PLAINTEXT,SSL}, what config will defne the SASL mechanism that
the broker should use?
Ron
On Wed, Oct 28, 2020 at 1:29 PM Ron Dagostino <rn...@gmail.com> wrote:
>
> HI again, Colin. I just noticed that both ConfigRecord and
> AccessControlRecord have a ResourceType of type int8. I thought that
> config resources are in the set {topics, clients, users, brokers} and
> ACL resource types are a different set as defined by the
> org.apache.kafka.common.resource.ResourceType enum. Does
> ConfigRecord.ResourceType need to be a String?
>
> Ron
>
> On Sun, Oct 25, 2020 at 6:04 AM Ron Dagostino <rn...@gmail.com> wrote:
> >
> > Hi Colin and Jun.
> >
> > Regarding these issues:
> >
> > 83.1 It seems that the broker can transition from FENCED to RUNNING
> > without registering for a new broker epoch. I am not sure how this
> > works. Once the controller fences a broker, there is no need for the
> > controller to keep the boker epoch around. So, if the fenced broker's
> > heartbeat request with the existing broker epoch will be rejected,
> > leading the broker back to the FENCED state again.; 104.
> > REGISTERING(1) : It says "Otherwise, the broker moves into the FENCED
> > state.". It seems this should be RUNNING?
> >
> > When would/could a broker re-register -- i.e. send
> > BrokerRegistrationRequest again once it receives a
> > BrokerRegistrationResponse containing no error and its broker epoch?
> > The text states that "Once the period has elapsed, if the broker has
> > not renewed its registration via a heartbeat, it must re-register."
> > But the broker state machine only mentions any type of
> > registration-related event in the REGISTERING state ("While in this
> > state, the broker tries to register with the active controller");
> > there is no other broker state in the text that mentions the
> > possibility of re-registering, and the broker state machine has no
> > transition back to the REGISTERING state.
> >
> > Also, the text now states that there are "three broker registration
> > states: unregistered, registered but fenced, and registered and
> > active." It would be good to map these onto the formal broker state
> > machine so we know which "registration states" a broker can be in for
> > each state within its broker state machine. It is not clear if there
> > is a way for a broker to go backwards into the "unregistered" broker
> > registration state. I suspect it can only flip-flop between
> > registered but fenced/registered and active as the broker flip-flops
> > between ACTIVE and FENCED, and this would imply that a broker is never
> > strictly required to re-register -- though the option isn't precluded.
> >
> > Does a broker JVM keep it's assigned broker epoch throughout the life
> > of the JVM? The BrokerRegistrationRequest includes a place for the
> > broker to specify its current broker epoch, but that would only be
> > useful if the broker is re-registering. If a broker were to
> > re-register, the data in the request might seem to imply that it could
> > do so to specify dynamic changes to its features or endpoints, but
> > those dynamic changes happen centrally, so that doesn't seem to be a
> > valid reason to re-register. So I do not yet see a reason for
> > re-registering despite the text "if the broker has not renewed its
> > registration via a heartbeat, it must re-register."
> >
> > It feels to me that a broker would keep its epoch throughout the life
> > of its JVM and it would never re-register, and the controller would
> > remember/maintain the broker epoch when it fences a broker; the broker
> > would continue to try sending heartbeat requests while it is fenced,
> > and it would continue to do so until the process is killed via an
> > external signal. If the controller eventually does respond with the
> > broker's next state then that next state will either be ACTIVE
> > (meaning communication has been restored; the return broker epoch will
> > be the same one that the broker JVM has had throughout its lifetime
> > and that it provided in the heartbeat request); or the next state will
> > be PENDING_CONTROLLED_SHUTDOWN if some other JVM process has since
> > started with the same broker ID.
> >
> > I hope that helps the discussion. Thanks for the great questions,
> > Jun, and your hard work and responses, Colin.
> >
> > Ron
> >
> >
> >
> >
> >
> > On Sat, Oct 24, 2020 at 4:08 AM Tom Bentley <tb...@redhat.com> wrote:
> > >
> > > Hi Colin,
> > >
> > > Which error code in particular though? Because so far as I'm aware there's
> > > no existing error code which really captures this situation and creating a
> > > new one would not be backward compatible.
> > >
> > > Cheers,
> > >
> > > Tom
> > >
> > > On Sat, Oct 24, 2020 at 12:20 AM Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply. A few more comments.
> > > >
> > > > 55. There is still text that favors new broker registration. "When a broker
> > > > first starts up, when it is in the INITIAL state, it will always "win"
> > > > broker ID conflicts. However, once it is granted a lease, it transitions
> > > > out of the INITIAL state. Thereafter, it may lose subsequent conflicts if
> > > > its broker epoch is stale. (See KIP-380 for some background on broker
> > > > epoch.) The reason for favoring new processes is to accommodate the common
> > > > case where a process is killed with kill -9 and then restarted. We want it
> > > > to be able to reclaim its old ID quickly in this case."
> > > >
> > > > 80.1 Sounds good. Could you document that listeners is a required config
> > > > now? It would also be useful to annotate other required configs. For
> > > > example, controller.connect should be required.
> > > >
> > > > 80.2 Could you list all deprecated existing configs? Another one is
> > > > control.plane.listener.name since the controller no longer sends
> > > > LeaderAndIsr, UpdateMetadata and StopReplica requests.
> > > >
> > > > 83.1 It seems that the broker can transition from FENCED to RUNNING without
> > > > registering for a new broker epoch. I am not sure how this works. Once the
> > > > controller fences a broker, there is no need for the controller to keep the
> > > > boker epoch around. So, if the fenced broker's heartbeat request with the
> > > > existing broker epoch will be rejected, leading the broker back to the
> > > > FENCED state again.
> > > >
> > > > 83.5 Good point on KIP-590. Then should we expose the controller for
> > > > debugging purposes? If not, we should deprecate the controllerID field in
> > > > MetadataResponse?
> > > >
> > > > 90. We rejected the shared ID with just one reason "This is not a good idea
> > > > because NetworkClient assumes a single ID space. So if there is both a
> > > > controller 1 and a broker 1, we don't have a way of picking the "right"
> > > > one." This doesn't seem to be a strong reason. For example, we could
> > > > address the NetworkClient issue with the node type as you pointed out or
> > > > using the negative value of a broker ID as the controller ID.
> > > >
> > > > 100. In KIP-589
> > > > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > >,
> > > > the broker reports all offline replicas due to a disk failure to the
> > > > controller. It seems this information needs to be persisted to the metadata
> > > > log. Do we have a corresponding record for that?
> > > >
> > > > 101. Currently, StopReplica request has 2 modes, without deletion and with
> > > > deletion. The former is used for controlled shutdown and handling disk
> > > > failure, and causes the follower to stop. The latter is for topic deletion
> > > > and partition reassignment, and causes the replica to be deleted. Since we
> > > > are deprecating StopReplica, could we document what triggers the stopping
> > > > of a follower and the deleting of a replica now?
> > > >
> > > > 102. Should we include the metadata topic in the MetadataResponse? If so,
> > > > when it will be included and what will the metadata response look like?
> > > >
> > > > 103. "The active controller assigns the broker a new broker epoch, based on
> > > > the latest committed offset in the log." This seems inaccurate since the
> > > > latest committed offset doesn't always advance on every log append.
> > > >
> > > > 104. REGISTERING(1) : It says "Otherwise, the broker moves into the FENCED
> > > > state.". It seems this should be RUNNING?
> > > >
> > > > 105. RUNNING: Should we require the broker to catch up to the metadata log
> > > > to get into this state?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > >
> > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <cm...@apache.org> wrote:
> > > >
> > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > > > Hi Colin,
> > > > > >
> > > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > > > > >
> > > > > > > > I have some questions about what happens to a broker when it
> > > > becomes
> > > > > > > > fenced (e.g. because it can't send a heartbeat request to keep its
> > > > > > > > lease). The KIP says "When a broker is fenced, it cannot process
> > > > any
> > > > > > > > client requests. This prevents brokers which are not receiving
> > > > > > > > metadata updates or that are not receiving and processing them fast
> > > > > > > > enough from causing issues to clients." And in the description of
> > > > the
> > > > > > > > FENCED(4) state it likewise says "While in this state, the broker
> > > > > does
> > > > > > > > not respond to client requests." It makes sense that a fenced
> > > > broker
> > > > > > > > should not accept producer requests -- I assume any such requests
> > > > > > > > would result in NotLeaderOrFollowerException. But what about
> > > > KIP-392
> > > > > > > > (fetch from follower) consumer requests? It is conceivable that
> > > > > these
> > > > > > > > could continue. Related to that, would a fenced broker continue to
> > > > > > > > fetch data for partitions where it thinks it is a follower? Even
> > > > if
> > > > > > > > it rejects consumer requests it might still continue to fetch as a
> > > > > > > > follower. Might it be helpful to clarify both decisions here?
> > > > > > >
> > > > > > > Hi Ron,
> > > > > > >
> > > > > > > Good question. I think a fenced broker should continue to fetch on
> > > > > > > partitions it was already fetching before it was fenced, unless it
> > > > > hits a
> > > > > > > problem. At that point it won't be able to continue, since it
> > > > doesn't
> > > > > have
> > > > > > > the new metadata. For example, it won't know about leadership
> > > > changes
> > > > > in
> > > > > > > the partitions it's fetching. The rationale for continuing to fetch
> > > > > is to
> > > > > > > try to avoid disruptions as much as possible.
> > > > > > >
> > > > > > > I don't think fenced brokers should accept client requests. The
> > > > issue
> > > > > is
> > > > > > > that the fenced broker may or may not have any data it is supposed to
> > > > > > > have. It may or may not have applied any configuration changes, etc.
> > > > > that
> > > > > > > it is supposed to have applied. So it could get pretty confusing,
> > > > and
> > > > > also
> > > > > > > potentially waste the client's time.
> > > > > > >
> > > > > > >
> > > > > > When fenced, how would the broker reply to a client which did make a
> > > > > > request?
> > > > > >
> > > > >
> > > > > Hi Tom,
> > > > >
> > > > > The broker will respond with a retryable error in that case. Once the
> > > > > client has re-fetched its metadata, it will no longer see the fenced
> > > > broker
> > > > > as part of the cluster. I added a note to the KIP.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Tom
> > > > > >
> > > > >
> > > >
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Ron Dagostino <rn...@gmail.com>.
HI again, Colin. I just noticed that both ConfigRecord and
AccessControlRecord have a ResourceType of type int8. I thought that
config resources are in the set {topics, clients, users, brokers} and
ACL resource types are a different set as defined by the
org.apache.kafka.common.resource.ResourceType enum. Does
ConfigRecord.ResourceType need to be a String?
Ron
On Sun, Oct 25, 2020 at 6:04 AM Ron Dagostino <rn...@gmail.com> wrote:
>
> Hi Colin and Jun.
>
> Regarding these issues:
>
> 83.1 It seems that the broker can transition from FENCED to RUNNING
> without registering for a new broker epoch. I am not sure how this
> works. Once the controller fences a broker, there is no need for the
> controller to keep the boker epoch around. So, if the fenced broker's
> heartbeat request with the existing broker epoch will be rejected,
> leading the broker back to the FENCED state again.; 104.
> REGISTERING(1) : It says "Otherwise, the broker moves into the FENCED
> state.". It seems this should be RUNNING?
>
> When would/could a broker re-register -- i.e. send
> BrokerRegistrationRequest again once it receives a
> BrokerRegistrationResponse containing no error and its broker epoch?
> The text states that "Once the period has elapsed, if the broker has
> not renewed its registration via a heartbeat, it must re-register."
> But the broker state machine only mentions any type of
> registration-related event in the REGISTERING state ("While in this
> state, the broker tries to register with the active controller");
> there is no other broker state in the text that mentions the
> possibility of re-registering, and the broker state machine has no
> transition back to the REGISTERING state.
>
> Also, the text now states that there are "three broker registration
> states: unregistered, registered but fenced, and registered and
> active." It would be good to map these onto the formal broker state
> machine so we know which "registration states" a broker can be in for
> each state within its broker state machine. It is not clear if there
> is a way for a broker to go backwards into the "unregistered" broker
> registration state. I suspect it can only flip-flop between
> registered but fenced/registered and active as the broker flip-flops
> between ACTIVE and FENCED, and this would imply that a broker is never
> strictly required to re-register -- though the option isn't precluded.
>
> Does a broker JVM keep it's assigned broker epoch throughout the life
> of the JVM? The BrokerRegistrationRequest includes a place for the
> broker to specify its current broker epoch, but that would only be
> useful if the broker is re-registering. If a broker were to
> re-register, the data in the request might seem to imply that it could
> do so to specify dynamic changes to its features or endpoints, but
> those dynamic changes happen centrally, so that doesn't seem to be a
> valid reason to re-register. So I do not yet see a reason for
> re-registering despite the text "if the broker has not renewed its
> registration via a heartbeat, it must re-register."
>
> It feels to me that a broker would keep its epoch throughout the life
> of its JVM and it would never re-register, and the controller would
> remember/maintain the broker epoch when it fences a broker; the broker
> would continue to try sending heartbeat requests while it is fenced,
> and it would continue to do so until the process is killed via an
> external signal. If the controller eventually does respond with the
> broker's next state then that next state will either be ACTIVE
> (meaning communication has been restored; the return broker epoch will
> be the same one that the broker JVM has had throughout its lifetime
> and that it provided in the heartbeat request); or the next state will
> be PENDING_CONTROLLED_SHUTDOWN if some other JVM process has since
> started with the same broker ID.
>
> I hope that helps the discussion. Thanks for the great questions,
> Jun, and your hard work and responses, Colin.
>
> Ron
>
>
>
>
>
> On Sat, Oct 24, 2020 at 4:08 AM Tom Bentley <tb...@redhat.com> wrote:
> >
> > Hi Colin,
> >
> > Which error code in particular though? Because so far as I'm aware there's
> > no existing error code which really captures this situation and creating a
> > new one would not be backward compatible.
> >
> > Cheers,
> >
> > Tom
> >
> > On Sat, Oct 24, 2020 at 12:20 AM Jun Rao <ju...@confluent.io> wrote:
> >
> > > Hi, Colin,
> > >
> > > Thanks for the reply. A few more comments.
> > >
> > > 55. There is still text that favors new broker registration. "When a broker
> > > first starts up, when it is in the INITIAL state, it will always "win"
> > > broker ID conflicts. However, once it is granted a lease, it transitions
> > > out of the INITIAL state. Thereafter, it may lose subsequent conflicts if
> > > its broker epoch is stale. (See KIP-380 for some background on broker
> > > epoch.) The reason for favoring new processes is to accommodate the common
> > > case where a process is killed with kill -9 and then restarted. We want it
> > > to be able to reclaim its old ID quickly in this case."
> > >
> > > 80.1 Sounds good. Could you document that listeners is a required config
> > > now? It would also be useful to annotate other required configs. For
> > > example, controller.connect should be required.
> > >
> > > 80.2 Could you list all deprecated existing configs? Another one is
> > > control.plane.listener.name since the controller no longer sends
> > > LeaderAndIsr, UpdateMetadata and StopReplica requests.
> > >
> > > 83.1 It seems that the broker can transition from FENCED to RUNNING without
> > > registering for a new broker epoch. I am not sure how this works. Once the
> > > controller fences a broker, there is no need for the controller to keep the
> > > boker epoch around. So, if the fenced broker's heartbeat request with the
> > > existing broker epoch will be rejected, leading the broker back to the
> > > FENCED state again.
> > >
> > > 83.5 Good point on KIP-590. Then should we expose the controller for
> > > debugging purposes? If not, we should deprecate the controllerID field in
> > > MetadataResponse?
> > >
> > > 90. We rejected the shared ID with just one reason "This is not a good idea
> > > because NetworkClient assumes a single ID space. So if there is both a
> > > controller 1 and a broker 1, we don't have a way of picking the "right"
> > > one." This doesn't seem to be a strong reason. For example, we could
> > > address the NetworkClient issue with the node type as you pointed out or
> > > using the negative value of a broker ID as the controller ID.
> > >
> > > 100. In KIP-589
> > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > >,
> > > the broker reports all offline replicas due to a disk failure to the
> > > controller. It seems this information needs to be persisted to the metadata
> > > log. Do we have a corresponding record for that?
> > >
> > > 101. Currently, StopReplica request has 2 modes, without deletion and with
> > > deletion. The former is used for controlled shutdown and handling disk
> > > failure, and causes the follower to stop. The latter is for topic deletion
> > > and partition reassignment, and causes the replica to be deleted. Since we
> > > are deprecating StopReplica, could we document what triggers the stopping
> > > of a follower and the deleting of a replica now?
> > >
> > > 102. Should we include the metadata topic in the MetadataResponse? If so,
> > > when it will be included and what will the metadata response look like?
> > >
> > > 103. "The active controller assigns the broker a new broker epoch, based on
> > > the latest committed offset in the log." This seems inaccurate since the
> > > latest committed offset doesn't always advance on every log append.
> > >
> > > 104. REGISTERING(1) : It says "Otherwise, the broker moves into the FENCED
> > > state.". It seems this should be RUNNING?
> > >
> > > 105. RUNNING: Should we require the broker to catch up to the metadata log
> > > to get into this state?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > > Hi Colin,
> > > > >
> > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > > > >
> > > > > > > I have some questions about what happens to a broker when it
> > > becomes
> > > > > > > fenced (e.g. because it can't send a heartbeat request to keep its
> > > > > > > lease). The KIP says "When a broker is fenced, it cannot process
> > > any
> > > > > > > client requests. This prevents brokers which are not receiving
> > > > > > > metadata updates or that are not receiving and processing them fast
> > > > > > > enough from causing issues to clients." And in the description of
> > > the
> > > > > > > FENCED(4) state it likewise says "While in this state, the broker
> > > > does
> > > > > > > not respond to client requests." It makes sense that a fenced
> > > broker
> > > > > > > should not accept producer requests -- I assume any such requests
> > > > > > > would result in NotLeaderOrFollowerException. But what about
> > > KIP-392
> > > > > > > (fetch from follower) consumer requests? It is conceivable that
> > > > these
> > > > > > > could continue. Related to that, would a fenced broker continue to
> > > > > > > fetch data for partitions where it thinks it is a follower? Even
> > > if
> > > > > > > it rejects consumer requests it might still continue to fetch as a
> > > > > > > follower. Might it be helpful to clarify both decisions here?
> > > > > >
> > > > > > Hi Ron,
> > > > > >
> > > > > > Good question. I think a fenced broker should continue to fetch on
> > > > > > partitions it was already fetching before it was fenced, unless it
> > > > hits a
> > > > > > problem. At that point it won't be able to continue, since it
> > > doesn't
> > > > have
> > > > > > the new metadata. For example, it won't know about leadership
> > > changes
> > > > in
> > > > > > the partitions it's fetching. The rationale for continuing to fetch
> > > > is to
> > > > > > try to avoid disruptions as much as possible.
> > > > > >
> > > > > > I don't think fenced brokers should accept client requests. The
> > > issue
> > > > is
> > > > > > that the fenced broker may or may not have any data it is supposed to
> > > > > > have. It may or may not have applied any configuration changes, etc.
> > > > that
> > > > > > it is supposed to have applied. So it could get pretty confusing,
> > > and
> > > > also
> > > > > > potentially waste the client's time.
> > > > > >
> > > > > >
> > > > > When fenced, how would the broker reply to a client which did make a
> > > > > request?
> > > > >
> > > >
> > > > Hi Tom,
> > > >
> > > > The broker will respond with a retryable error in that case. Once the
> > > > client has re-fetched its metadata, it will no longer see the fenced
> > > broker
> > > > as part of the cluster. I added a note to the KIP.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Tom
> > > > >
> > > >
> > >
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Colin and Jun.
Regarding these issues:
83.1 It seems that the broker can transition from FENCED to RUNNING
without registering for a new broker epoch. I am not sure how this
works. Once the controller fences a broker, there is no need for the
controller to keep the boker epoch around. So, if the fenced broker's
heartbeat request with the existing broker epoch will be rejected,
leading the broker back to the FENCED state again.; 104.
REGISTERING(1) : It says "Otherwise, the broker moves into the FENCED
state.". It seems this should be RUNNING?
When would/could a broker re-register -- i.e. send
BrokerRegistrationRequest again once it receives a
BrokerRegistrationResponse containing no error and its broker epoch?
The text states that "Once the period has elapsed, if the broker has
not renewed its registration via a heartbeat, it must re-register."
But the broker state machine only mentions any type of
registration-related event in the REGISTERING state ("While in this
state, the broker tries to register with the active controller");
there is no other broker state in the text that mentions the
possibility of re-registering, and the broker state machine has no
transition back to the REGISTERING state.
Also, the text now states that there are "three broker registration
states: unregistered, registered but fenced, and registered and
active." It would be good to map these onto the formal broker state
machine so we know which "registration states" a broker can be in for
each state within its broker state machine. It is not clear if there
is a way for a broker to go backwards into the "unregistered" broker
registration state. I suspect it can only flip-flop between
registered but fenced/registered and active as the broker flip-flops
between ACTIVE and FENCED, and this would imply that a broker is never
strictly required to re-register -- though the option isn't precluded.
Does a broker JVM keep it's assigned broker epoch throughout the life
of the JVM? The BrokerRegistrationRequest includes a place for the
broker to specify its current broker epoch, but that would only be
useful if the broker is re-registering. If a broker were to
re-register, the data in the request might seem to imply that it could
do so to specify dynamic changes to its features or endpoints, but
those dynamic changes happen centrally, so that doesn't seem to be a
valid reason to re-register. So I do not yet see a reason for
re-registering despite the text "if the broker has not renewed its
registration via a heartbeat, it must re-register."
It feels to me that a broker would keep its epoch throughout the life
of its JVM and it would never re-register, and the controller would
remember/maintain the broker epoch when it fences a broker; the broker
would continue to try sending heartbeat requests while it is fenced,
and it would continue to do so until the process is killed via an
external signal. If the controller eventually does respond with the
broker's next state then that next state will either be ACTIVE
(meaning communication has been restored; the return broker epoch will
be the same one that the broker JVM has had throughout its lifetime
and that it provided in the heartbeat request); or the next state will
be PENDING_CONTROLLED_SHUTDOWN if some other JVM process has since
started with the same broker ID.
I hope that helps the discussion. Thanks for the great questions,
Jun, and your hard work and responses, Colin.
Ron
On Sat, Oct 24, 2020 at 4:08 AM Tom Bentley <tb...@redhat.com> wrote:
>
> Hi Colin,
>
> Which error code in particular though? Because so far as I'm aware there's
> no existing error code which really captures this situation and creating a
> new one would not be backward compatible.
>
> Cheers,
>
> Tom
>
> On Sat, Oct 24, 2020 at 12:20 AM Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Colin,
> >
> > Thanks for the reply. A few more comments.
> >
> > 55. There is still text that favors new broker registration. "When a broker
> > first starts up, when it is in the INITIAL state, it will always "win"
> > broker ID conflicts. However, once it is granted a lease, it transitions
> > out of the INITIAL state. Thereafter, it may lose subsequent conflicts if
> > its broker epoch is stale. (See KIP-380 for some background on broker
> > epoch.) The reason for favoring new processes is to accommodate the common
> > case where a process is killed with kill -9 and then restarted. We want it
> > to be able to reclaim its old ID quickly in this case."
> >
> > 80.1 Sounds good. Could you document that listeners is a required config
> > now? It would also be useful to annotate other required configs. For
> > example, controller.connect should be required.
> >
> > 80.2 Could you list all deprecated existing configs? Another one is
> > control.plane.listener.name since the controller no longer sends
> > LeaderAndIsr, UpdateMetadata and StopReplica requests.
> >
> > 83.1 It seems that the broker can transition from FENCED to RUNNING without
> > registering for a new broker epoch. I am not sure how this works. Once the
> > controller fences a broker, there is no need for the controller to keep the
> > boker epoch around. So, if the fenced broker's heartbeat request with the
> > existing broker epoch will be rejected, leading the broker back to the
> > FENCED state again.
> >
> > 83.5 Good point on KIP-590. Then should we expose the controller for
> > debugging purposes? If not, we should deprecate the controllerID field in
> > MetadataResponse?
> >
> > 90. We rejected the shared ID with just one reason "This is not a good idea
> > because NetworkClient assumes a single ID space. So if there is both a
> > controller 1 and a broker 1, we don't have a way of picking the "right"
> > one." This doesn't seem to be a strong reason. For example, we could
> > address the NetworkClient issue with the node type as you pointed out or
> > using the negative value of a broker ID as the controller ID.
> >
> > 100. In KIP-589
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > >,
> > the broker reports all offline replicas due to a disk failure to the
> > controller. It seems this information needs to be persisted to the metadata
> > log. Do we have a corresponding record for that?
> >
> > 101. Currently, StopReplica request has 2 modes, without deletion and with
> > deletion. The former is used for controlled shutdown and handling disk
> > failure, and causes the follower to stop. The latter is for topic deletion
> > and partition reassignment, and causes the replica to be deleted. Since we
> > are deprecating StopReplica, could we document what triggers the stopping
> > of a follower and the deleting of a replica now?
> >
> > 102. Should we include the metadata topic in the MetadataResponse? If so,
> > when it will be included and what will the metadata response look like?
> >
> > 103. "The active controller assigns the broker a new broker epoch, based on
> > the latest committed offset in the log." This seems inaccurate since the
> > latest committed offset doesn't always advance on every log append.
> >
> > 104. REGISTERING(1) : It says "Otherwise, the broker moves into the FENCED
> > state.". It seems this should be RUNNING?
> >
> > 105. RUNNING: Should we require the broker to catch up to the metadata log
> > to get into this state?
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > Hi Colin,
> > > >
> > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > > >
> > > > > > I have some questions about what happens to a broker when it
> > becomes
> > > > > > fenced (e.g. because it can't send a heartbeat request to keep its
> > > > > > lease). The KIP says "When a broker is fenced, it cannot process
> > any
> > > > > > client requests. This prevents brokers which are not receiving
> > > > > > metadata updates or that are not receiving and processing them fast
> > > > > > enough from causing issues to clients." And in the description of
> > the
> > > > > > FENCED(4) state it likewise says "While in this state, the broker
> > > does
> > > > > > not respond to client requests." It makes sense that a fenced
> > broker
> > > > > > should not accept producer requests -- I assume any such requests
> > > > > > would result in NotLeaderOrFollowerException. But what about
> > KIP-392
> > > > > > (fetch from follower) consumer requests? It is conceivable that
> > > these
> > > > > > could continue. Related to that, would a fenced broker continue to
> > > > > > fetch data for partitions where it thinks it is a follower? Even
> > if
> > > > > > it rejects consumer requests it might still continue to fetch as a
> > > > > > follower. Might it be helpful to clarify both decisions here?
> > > > >
> > > > > Hi Ron,
> > > > >
> > > > > Good question. I think a fenced broker should continue to fetch on
> > > > > partitions it was already fetching before it was fenced, unless it
> > > hits a
> > > > > problem. At that point it won't be able to continue, since it
> > doesn't
> > > have
> > > > > the new metadata. For example, it won't know about leadership
> > changes
> > > in
> > > > > the partitions it's fetching. The rationale for continuing to fetch
> > > is to
> > > > > try to avoid disruptions as much as possible.
> > > > >
> > > > > I don't think fenced brokers should accept client requests. The
> > issue
> > > is
> > > > > that the fenced broker may or may not have any data it is supposed to
> > > > > have. It may or may not have applied any configuration changes, etc.
> > > that
> > > > > it is supposed to have applied. So it could get pretty confusing,
> > and
> > > also
> > > > > potentially waste the client's time.
> > > > >
> > > > >
> > > > When fenced, how would the broker reply to a client which did make a
> > > > request?
> > > >
> > >
> > > Hi Tom,
> > >
> > > The broker will respond with a retryable error in that case. Once the
> > > client has re-fetched its metadata, it will no longer see the fenced
> > broker
> > > as part of the cluster. I added a note to the KIP.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > Thanks,
> > > >
> > > > Tom
> > > >
> > >
> >
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Tom Bentley <tb...@redhat.com>.
Hi Colin,
Which error code in particular though? Because so far as I'm aware there's
no existing error code which really captures this situation and creating a
new one would not be backward compatible.
Cheers,
Tom
On Sat, Oct 24, 2020 at 12:20 AM Jun Rao <ju...@confluent.io> wrote:
> Hi, Colin,
>
> Thanks for the reply. A few more comments.
>
> 55. There is still text that favors new broker registration. "When a broker
> first starts up, when it is in the INITIAL state, it will always "win"
> broker ID conflicts. However, once it is granted a lease, it transitions
> out of the INITIAL state. Thereafter, it may lose subsequent conflicts if
> its broker epoch is stale. (See KIP-380 for some background on broker
> epoch.) The reason for favoring new processes is to accommodate the common
> case where a process is killed with kill -9 and then restarted. We want it
> to be able to reclaim its old ID quickly in this case."
>
> 80.1 Sounds good. Could you document that listeners is a required config
> now? It would also be useful to annotate other required configs. For
> example, controller.connect should be required.
>
> 80.2 Could you list all deprecated existing configs? Another one is
> control.plane.listener.name since the controller no longer sends
> LeaderAndIsr, UpdateMetadata and StopReplica requests.
>
> 83.1 It seems that the broker can transition from FENCED to RUNNING without
> registering for a new broker epoch. I am not sure how this works. Once the
> controller fences a broker, there is no need for the controller to keep the
> boker epoch around. So, if the fenced broker's heartbeat request with the
> existing broker epoch will be rejected, leading the broker back to the
> FENCED state again.
>
> 83.5 Good point on KIP-590. Then should we expose the controller for
> debugging purposes? If not, we should deprecate the controllerID field in
> MetadataResponse?
>
> 90. We rejected the shared ID with just one reason "This is not a good idea
> because NetworkClient assumes a single ID space. So if there is both a
> controller 1 and a broker 1, we don't have a way of picking the "right"
> one." This doesn't seem to be a strong reason. For example, we could
> address the NetworkClient issue with the node type as you pointed out or
> using the negative value of a broker ID as the controller ID.
>
> 100. In KIP-589
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> >,
> the broker reports all offline replicas due to a disk failure to the
> controller. It seems this information needs to be persisted to the metadata
> log. Do we have a corresponding record for that?
>
> 101. Currently, StopReplica request has 2 modes, without deletion and with
> deletion. The former is used for controlled shutdown and handling disk
> failure, and causes the follower to stop. The latter is for topic deletion
> and partition reassignment, and causes the replica to be deleted. Since we
> are deprecating StopReplica, could we document what triggers the stopping
> of a follower and the deleting of a replica now?
>
> 102. Should we include the metadata topic in the MetadataResponse? If so,
> when it will be included and what will the metadata response look like?
>
> 103. "The active controller assigns the broker a new broker epoch, based on
> the latest committed offset in the log." This seems inaccurate since the
> latest committed offset doesn't always advance on every log append.
>
> 104. REGISTERING(1) : It says "Otherwise, the broker moves into the FENCED
> state.". It seems this should be RUNNING?
>
> 105. RUNNING: Should we require the broker to catch up to the metadata log
> to get into this state?
>
> Thanks,
>
> Jun
>
>
>
> On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <cm...@apache.org> wrote:
>
> > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > Hi Colin,
> > >
> > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > >
> > > > > I have some questions about what happens to a broker when it
> becomes
> > > > > fenced (e.g. because it can't send a heartbeat request to keep its
> > > > > lease). The KIP says "When a broker is fenced, it cannot process
> any
> > > > > client requests. This prevents brokers which are not receiving
> > > > > metadata updates or that are not receiving and processing them fast
> > > > > enough from causing issues to clients." And in the description of
> the
> > > > > FENCED(4) state it likewise says "While in this state, the broker
> > does
> > > > > not respond to client requests." It makes sense that a fenced
> broker
> > > > > should not accept producer requests -- I assume any such requests
> > > > > would result in NotLeaderOrFollowerException. But what about
> KIP-392
> > > > > (fetch from follower) consumer requests? It is conceivable that
> > these
> > > > > could continue. Related to that, would a fenced broker continue to
> > > > > fetch data for partitions where it thinks it is a follower? Even
> if
> > > > > it rejects consumer requests it might still continue to fetch as a
> > > > > follower. Might it be helpful to clarify both decisions here?
> > > >
> > > > Hi Ron,
> > > >
> > > > Good question. I think a fenced broker should continue to fetch on
> > > > partitions it was already fetching before it was fenced, unless it
> > hits a
> > > > problem. At that point it won't be able to continue, since it
> doesn't
> > have
> > > > the new metadata. For example, it won't know about leadership
> changes
> > in
> > > > the partitions it's fetching. The rationale for continuing to fetch
> > is to
> > > > try to avoid disruptions as much as possible.
> > > >
> > > > I don't think fenced brokers should accept client requests. The
> issue
> > is
> > > > that the fenced broker may or may not have any data it is supposed to
> > > > have. It may or may not have applied any configuration changes, etc.
> > that
> > > > it is supposed to have applied. So it could get pretty confusing,
> and
> > also
> > > > potentially waste the client's time.
> > > >
> > > >
> > > When fenced, how would the broker reply to a client which did make a
> > > request?
> > >
> >
> > Hi Tom,
> >
> > The broker will respond with a retryable error in that case. Once the
> > client has re-fetched its metadata, it will no longer see the fenced
> broker
> > as part of the cluster. I added a note to the KIP.
> >
> > best,
> > Colin
> >
> > >
> > > Thanks,
> > >
> > > Tom
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
Hi Gwen,
Thanks for taking a look. Yes, I do think unregisterBroker would be a better name than decommissionBroker. Let's do that, and also rename the associated RPC.
cheers,
Colin
On Sat, Feb 6, 2021, at 21:13, Gwen Shapira wrote:
> I realize that I'm very very late to a pretty big party, but I was
> just looking at the new code (super readable, btw), and noticed that
> the inverse API for registerBroker() is called decomissionBroker().
> It would be way clearer that we are undoing a registration is we
> rename it to unregisterBroker() (and the AdminClient API that matches,
> to avoid more confusion).
>
> Any objections?
>
> On Fri, Dec 18, 2020 at 10:08 AM Jun Rao <ju...@confluent.io> wrote:
> >
> > Hi, Colin,
> >
> > Thanks for the reply. The KIP looks good to me now. Thanks for your
> > diligence.
> >
> > Jun
> >
> > On Thu, Dec 17, 2020 at 5:15 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Thu, Dec 17, 2020, at 17:02, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply. Sounds good. One more comment.
> > > >
> > > > 231. Currently, we have sasl.mechanism.inter.broker.protocol for inter
> > > > broker communication. It seems that we need a similar config for
> > > specifying
> > > > the sasl mechanism for the communication between the broker and the
> > > > controller.
> > > >
> > >
> > > Hi Jun,
> > >
> > > Yeah... sounds like we could use a new configuration key for this. I
> > > added sasl.mechanism.controller.protocol for this.
> > >
> > > regards,
> > > Colin
> > >
> > > > Jun
> > > >
> > > > On Thu, Dec 17, 2020 at 2:29 PM Colin McCabe <cm...@apache.org> wrote:
> > > >
> > > > > On Thu, Dec 17, 2020, at 10:19, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the reply.
> > > > > >
> > > > > > 211. Hmm, I still don't see a clear benefit of registering a broker
> > > > > before
> > > > > > recovery. It's possible for the recovery to take time. However,
> > > during
> > > > > the
> > > > > > recovery mode, it seems the broker will still be in the fenced mode
> > > and
> > > > > > won't be able to do work for the clients. So, registering and
> > > > > heartbeating
> > > > > > early seems to only add unnecessary overhead. For your point on topic
> > > > > > creation, I thought we now allow replicas to be created on
> > > > > > unregistered brokers.
> > > > > >
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > Thanks again for the reviews.
> > > > >
> > > > > Topics cannot be created on unregistered brokers. They can be created
> > > on
> > > > > registered but fenced brokers. So for that reason I think it makes
> > > sense
> > > > > to register as early as possible.
> > > > >
> > > > > > 230. Currently, we do have a ControllerId field in MetadataResponse.
> > > In
> > > > > the
> > > > > > early discussion, I thought that we want to expose the controller for
> > > > > > debugging purposes, but not used by the client library.
> > > > > >
> > > > >
> > > > > The current plan is that we will expose the controller node ID, but the
> > > > > controller will not be included in the list of nodes in the metadata
> > > > > response.
> > > > >
> > > > > It's not really possible to include the controller in that list of
> > > nodes
> > > > > because the controller may not share the same set of listeners as the
> > > > > broker. So, for example, maybe the controller endpoint is using a
> > > > > different type of security than the broker. So while we could pass
> > > back a
> > > > > hostname and port, the client would have no way to connect since it
> > > doesn't
> > > > > know what security settings to use.
> > > > >
> > > > > regards,
> > > > > Colin
> > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, Dec 16, 2020 at 9:13 PM Colin McCabe <cm...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > On Wed, Dec 16, 2020, at 18:13, Jun Rao wrote:
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the reply. Just a couple of more comments.
> > > > > > > >
> > > > > > > > 211. Currently, the broker only registers itself in ZK after log
> > > > > > > recovery.
> > > > > > > > Is there any benefit to change that? As you mentioned, the broker
> > > > > can't
> > > > > > > do
> > > > > > > > much before completing log recovery.
> > > > > > > >
> > > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > Previously, it wasn't possible to register in ZK without
> > > immediately
> > > > > > > getting added to the MetadataResponse. So I think that's the main
> > > > > reason
> > > > > > > why registration was delayed until after log recovery. Since that
> > > > > > > constraint doesn't exist any more, there seems to be no reason to
> > > delay
> > > > > > > registration.
> > > > > > >
> > > > > > > I think delaying registration would have some major downsides. If
> > > log
> > > > > > > recovery takes a while, that's a longer window during which someone
> > > > > else
> > > > > > > could register a broker with the same ID. Having parts of the
> > > cluster
> > > > > > > missing for a while gives up some of the benefit of separating
> > > > > registration
> > > > > > > from fencing. For example, if a broker somehow gets unregistered
> > > and
> > > > > we
> > > > > > > want to re-register it, but we have to wait for a 10 minute log
> > > > > recovery to
> > > > > > > do that, that could be a window during which topics can't be
> > > created
> > > > > that
> > > > > > > need to be on that broker, and so forth.
> > > > > > >
> > > > > > > > 230. Regarding MetadataResponse, there is a slight awkwardness.
> > > We
> > > > > return
> > > > > > > > rack for each node. However, if that node is for the controller,
> > > the
> > > > > rack
> > > > > > > > field is not really relevant. Should we clean it up here or in
> > > > > another
> > > > > > > KIP
> > > > > > > > like KIP-700?
> > > > > > >
> > > > > > > Oh, controllers don't appear in the MetadataResponses returned to
> > > > > clients,
> > > > > > > since clients can't access them. I should have been more clear
> > > about
> > > > > that
> > > > > > > in the KIP-- I added a sentence to "Networking" describing this.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Wed, Dec 16, 2020 at 4:23 PM Colin McCabe <cmccabe@apache.org
> > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Wed, Dec 16, 2020, at 13:40, Jun Rao wrote:
> > > > > > > > > > Hi, Colin,
> > > > > > > > > >
> > > > > > > > > > Thanks for the reply. A few follow up comments.
> > > > > > > > > >
> > > > > > > > > > 211. When does the broker send the BrokerRegistration
> > > request to
> > > > > the
> > > > > > > > > > controller? Is it after the recovery phase? If so, at that
> > > > > point, the
> > > > > > > > > > broker has already caught up on the metadata (in order to
> > > clean
> > > > > up
> > > > > > > > > deleted
> > > > > > > > > > partitions). Then, it seems we don't need the ShouldFence
> > > field
> > > > > > > > > > in BrokerHeartbeatRequest?
> > > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > Thanks again for the reviews.
> > > > > > > > >
> > > > > > > > > The broker sends the registration request as soon as it starts
> > > > > up. It
> > > > > > > > > cannot wait until the recovery phase is over since sometimes
> > > log
> > > > > > > recovery
> > > > > > > > > can take quite a long time.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 213. kafka-cluster.sh
> > > > > > > > > > 213.1 For the decommision example, should the command take a
> > > > > broker
> > > > > > > id?
> > > > > > > > >
> > > > > > > > > Yes, the example should have a broker id. Fixed.
> > > > > > > > >
> > > > > > > > > > 213.2 Existing tools use the "--" command line option (e.g.
> > > > > > > kafka-topics
> > > > > > > > > > --list --topic test). Should we follow the same convention
> > > > > > > > > > for kafka-cluster.sh (and kafka-storage.sh)?
> > > > > > > > >
> > > > > > > > > Hmm. I don't think argparse4j supports using double dashes to
> > > > > identify
> > > > > > > > > subcommands. I think it might be confusing as well, since the
> > > > > > > subcommand
> > > > > > > > > must come first, unlike a plain old argument which can be
> > > anywhere
> > > > > on
> > > > > > > the
> > > > > > > > > command line.
> > > > > > > > >
> > > > > > > > > > 213.3 Should we add an admin api for broker decommission so
> > > that
> > > > > > > this can
> > > > > > > > > > be done programmatically?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes, there is an admin client API for decommissioning as well.
> > > > > > > > >
> > > > > > > > > > 220. DecommissionBroker: "NOT_CONTROLLER if the node that the
> > > > > > > request was
> > > > > > > > > > sent to is not the controller". I thought the clients never
> > > send
> > > > > a
> > > > > > > > > request
> > > > > > > > > > to the controller directly now and the broker will forward
> > > it to
> > > > > the
> > > > > > > > > > controller?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > If the controller moved recently, it's possible that the broker
> > > > > could
> > > > > > > send
> > > > > > > > > to a controller that has just recently become inactive. In
> > > that
> > > > > case
> > > > > > > > > NOT_CONTROLLER would be returned. (A standby controller
> > > returns
> > > > > > > > > NOT_CONTROLLER for most APIs).
> > > > > > > > >
> > > > > > > > > > 221. Could we add the required ACL for the new requests?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Good point. I added the required ACL for each new RPC.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Wed, Dec 16, 2020 at 11:38 AM Colin McCabe <
> > > > > cmccabe@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > On Wed, Dec 16, 2020, at 09:59, Jun Rao wrote:
> > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the reply. A few more comments below.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Jun,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the comments.
> > > > > > > > > > >
> > > > > > > > > > > > 206. "RemoveTopic is the last step, that scrubs all
> > > metadata
> > > > > > > about
> > > > > > > > > the
> > > > > > > > > > > > topic. In order to get to that last step, the topic data
> > > > > needs
> > > > > > > to
> > > > > > > > > > > removed
> > > > > > > > > > > > from all brokers (after each broker notices that the
> > > topic is
> > > > > > > being
> > > > > > > > > > > > deleted)." Currently, this is done based on the response
> > > of
> > > > > > > > > > > > StopReplicaRequest. Since the controller no longer sends
> > > this
> > > > > > > > > request,
> > > > > > > > > > > how
> > > > > > > > > > > > does the controller know that the data for the deleted
> > > topic
> > > > > has
> > > > > > > > > > > > been removed in the brokers?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > That's a good point... definitely an oversight.
> > > > > > > > > > >
> > > > > > > > > > > It seems complex to force the controller to track when log
> > > > > > > directories
> > > > > > > > > > > have been deleted. Let's just assume that KIP-516 has been
> > > > > > > > > implemented,
> > > > > > > > > > > and track them by UUID. Then we can just have a single
> > > topic
> > > > > > > deletion
> > > > > > > > > > > record.
> > > > > > > > > > >
> > > > > > > > > > > I added a section on "topic identifiers" describing this.
> > > > > > > > > > >
> > > > > > > > > > > > 210. Thanks for the explanation. Sounds good to me.
> > > > > > > > > > > >
> > > > > > > > > > > > 211. I still don't see an example when ShouldFence is
> > > set to
> > > > > > > true in
> > > > > > > > > > > > BrokerHeartbeatReques. Could we add one?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > It is sent to true when the broker is first starting up and
> > > > > > > doesn't yet
> > > > > > > > > > > want to be unfenced. I added a longer explanation of this
> > > in
> > > > > the
> > > > > > > > > "Broker
> > > > > > > > > > > Leases" section.
> > > > > > > > > > >
> > > > > > > > > > > > 213. The KIP now allows replicas to be assigned on
> > > brokers
> > > > > that
> > > > > > > are
> > > > > > > > > > > fenced,
> > > > > > > > > > > > which is an improvement. How do we permanently remove a
> > > > > broker
> > > > > > > (e.g.
> > > > > > > > > > > > cluster shrinking) to prevent it from being used for
> > > future
> > > > > > > replica
> > > > > > > > > > > > assignments?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > This is a fair point. I will create a kafka-cluster.sh
> > > script
> > > > > > > which
> > > > > > > > > can
> > > > > > > > > > > do this, plus a DecommissionBrokerRequest.
> > > > > > > > > > >
> > > > > > > > > > > As a bonus the kafka-cluster.sh script can help people
> > > find the
> > > > > > > > > cluster ID
> > > > > > > > > > > of brokers, something that people have wanted a tool for
> > > in the
> > > > > > > past.
> > > > > > > > > > >
> > > > > > > > > > > > 214. "Currently, when a node is down, all of its ZK
> > > > > registration
> > > > > > > > > > > > information is gone. But we need this information in
> > > order
> > > > > to
> > > > > > > > > > > understand
> > > > > > > > > > > > things like whether the replicas of a particular
> > > partition
> > > > > are
> > > > > > > > > > > > well-balanced across racks." This is not quite true.
> > > > > Currently,
> > > > > > > even
> > > > > > > > > when
> > > > > > > > > > > > ZK registration is gone, the existing replica assignment
> > > is
> > > > > still
> > > > > > > > > > > available
> > > > > > > > > > > > in the metadata response.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I agree that the existing replica assignment is still
> > > > > available.
> > > > > > > But
> > > > > > > > > that
> > > > > > > > > > > just tells you partition X is on nodes A, B, and C. If you
> > > > > don't
> > > > > > > have
> > > > > > > > > the
> > > > > > > > > > > ZK registration for one or more of A, B, or C then you
> > > don't
> > > > > know
> > > > > > > > > whether
> > > > > > > > > > > we are following the policy of "two replicas on one rack,
> > > one
> > > > > > > replica
> > > > > > > > > on
> > > > > > > > > > > another." Or any other more complex rack placement policy
> > > > > that you
> > > > > > > > > might
> > > > > > > > > > > have.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > > Jun
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Dec 16, 2020 at 8:48 AM Colin McCabe <
> > > > > cmccabe@apache.org
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the reply. A few more follow up comments.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 210. initial.broker.registration.timeout.ms: The
> > > default
> > > > > > > value
> > > > > > > > > is
> > > > > > > > > > > 90sec,
> > > > > > > > > > > > > > which seems long. If a broker fails the registration
> > > > > because
> > > > > > > of
> > > > > > > > > > > incorrect
> > > > > > > > > > > > > > configs, we want to fail faster. In comparison, the
> > > > > defaults
> > > > > > > for
> > > > > > > > > > > > > > zookeeper.connection.timeout.ms is 18 secs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I agree that the initial connection timeout here is
> > > longer
> > > > > than
> > > > > > > > > what we
> > > > > > > > > > > > > had with ZK. The main reason I selected a slightly
> > > longer
> > > > > > > timeout
> > > > > > > > > > > here is
> > > > > > > > > > > > > to handle the case where the controllers and the
> > > brokers
> > > > > are
> > > > > > > > > > > co-located.
> > > > > > > > > > > > > For example, if you have a 3 node cluster and all three
> > > > > nodes
> > > > > > > are
> > > > > > > > > > > > > controllers+brokers, when you first bring up the
> > > cluster,
> > > > > we
> > > > > > > will
> > > > > > > > > have
> > > > > > > > > > > to
> > > > > > > > > > > > > stand up the controller quorum and then handle broker
> > > > > > > > > registrations.
> > > > > > > > > > > > > Although we believe Raft will start up relatively
> > > quickly,
> > > > > it's
> > > > > > > > > good to
> > > > > > > > > > > > > leave some extra margin here.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't think there's a big disadvantage to having a
> > > > > slightly
> > > > > > > > > longer
> > > > > > > > > > > > > timeout here. After all, starting up the brokers with
> > > no
> > > > > > > > > controllers
> > > > > > > > > > > is an
> > > > > > > > > > > > > admin mistake, which we don't expect to see very often.
> > > > > Maybe
> > > > > > > > > let's
> > > > > > > > > > > set it
> > > > > > > > > > > > > to 60 seconds for now and maybe see if we can tweak it
> > > in
> > > > > the
> > > > > > > > > future if
> > > > > > > > > > > > > that turns out to be too long or short?
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 211. "Once they are all moved, the controller
> > > responds
> > > > > to the
> > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > with a nextState of SHUTDOWN." It seems that
> > > nextState
> > > > > is no
> > > > > > > > > longer
> > > > > > > > > > > > > > relevant. Also, could you add a bit more explanation
> > > on
> > > > > when
> > > > > > > > > > > ShouldFence
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > set to true in BrokerHeartbeatRequest?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Good catch. I removed the obsolete section referring
> > > to
> > > > > > > nextState
> > > > > > > > > and
> > > > > > > > > > > > > added a reference to the new boolean. I also added
> > > some
> > > > > more
> > > > > > > > > > > information
> > > > > > > > > > > > > about ShouldFence and about the rationale for
> > > separating
> > > > > > > fencing
> > > > > > > > > from
> > > > > > > > > > > > > registration.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 212. Related to the above, what does the broker do if
> > > > > > > IsFenced is
> > > > > > > > > > > true in
> > > > > > > > > > > > > > the BrokerHeartbeatResponse? Will the broker do the
> > > same
> > > > > > > thing on
> > > > > > > > > > > > > receiving
> > > > > > > > > > > > > > a FenceBrokerRecord from the metadata log?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > The broker only checks this boolean during the startup
> > > > > process.
> > > > > > > > > After
> > > > > > > > > > > the
> > > > > > > > > > > > > startup process is finished, it ignores this boolean.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The broker uses fence / unfence records in the metadata
> > > > > log to
> > > > > > > > > > > determine
> > > > > > > > > > > > > which brokers should appear in its MetadataResponse.
> > > > > > > > > > > > >
> > > > > > > > > > > > > best,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <
> > > > > > > cmccabe@apache.org
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The KIP says that "brokers which are fenced will
> > > not
> > > > > > > appear
> > > > > > > > > in
> > > > > > > > > > > > > > > > MetadataResponses. So clients that have
> > > up-to-date
> > > > > > > metadata
> > > > > > > > > will
> > > > > > > > > > > > > not try
> > > > > > > > > > > > > > > > to contact fenced brokers.", which is fine, but
> > > it
> > > > > > > doesn't
> > > > > > > > > seem
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > elaborate on what happens for clients which try
> > > to
> > > > > > > contact a
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > (using
> > > > > > > > > > > > > > > > stale metadata, for example) and find it in the
> > > > > > > > > > > registered-but-fenced
> > > > > > > > > > > > > > > > state.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have dropped the broker-side fencing from this
> > > > > > > proposal. So
> > > > > > > > > now
> > > > > > > > > > > the
> > > > > > > > > > > > > > > fencing is basically the same as today's fencing:
> > > it's
> > > > > > > > > > > controller-side
> > > > > > > > > > > > > > > only. That means that clients using stale
> > > metadata can
> > > > > > > contact
> > > > > > > > > > > fenced
> > > > > > > > > > > > > > > brokers and communicate with them. The only case
> > > > > where the
> > > > > > > > > broker
> > > > > > > > > > > > > will be
> > > > > > > > > > > > > > > inaccessible is when it hasn't finished starting
> > > yet
> > > > > > > (i.e., has
> > > > > > > > > > > not yet
> > > > > > > > > > > > > > > attained RUNNING state.)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Just like today, we expect the safeguards built
> > > into
> > > > > the
> > > > > > > > > > > replication
> > > > > > > > > > > > > > > protocol to prevent the worst corner cases that
> > > could
> > > > > > > result
> > > > > > > > > from
> > > > > > > > > > > > > this. I
> > > > > > > > > > > > > > > do think we will probably take up this issue later
> > > and
> > > > > > > find a
> > > > > > > > > > > better
> > > > > > > > > > > > > > > solution for client-side metadata, but this KIP is
> > > big
> > > > > > > enough
> > > > > > > > > > > as-is.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > You said previously that the broker will respond
> > > > > with a
> > > > > > > > > retriable
> > > > > > > > > > > > > > > > error, which is again fine, but you didn't
> > > answer my
> > > > > > > question
> > > > > > > > > > > about
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > error code you've chosen for this. I realise that
> > > > > this
> > > > > > > > > doesn't
> > > > > > > > > > > really
> > > > > > > > > > > > > > > > affect the correctness of the behaviour, but I'm
> > > not
> > > > > > > aware
> > > > > > > > > of any
> > > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > error code which is a good fit. So it would be
> > > good
> > > > > to
> > > > > > > > > understand
> > > > > > > > > > > > > about
> > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > you're making this backward compatible for
> > > clients.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Many thanks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <
> > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for the reply. Just a couple of more
> > > > > comments
> > > > > > > > > below.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 210. Since we are deprecating
> > > > > > > > > > > zookeeper.connection.timeout.ms,
> > > > > > > > > > > > > > > should we
> > > > > > > > > > > > > > > > > > add a new config to bound the time for a
> > > broker
> > > > > to
> > > > > > > > > connect
> > > > > > > > > > > to the
> > > > > > > > > > > > > > > > > > controller during starting up?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Good idea. I added
> > > > > > > initial.broker.registration.timeout.ms
> > > > > > > > > for
> > > > > > > > > > > > > this.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 211. BrokerHeartbeat no longer has the state
> > > > > field
> > > > > > > in the
> > > > > > > > > > > > > > > > > request/response.
> > > > > > > > > > > > > > > > > > However, (a) the controller shutdown section
> > > > > still
> > > > > > > has
> > > > > > > > > "In
> > > > > > > > > > > its
> > > > > > > > > > > > > > > periodic
> > > > > > > > > > > > > > > > > > heartbeats, the broker asks the controller
> > > if it
> > > > > can
> > > > > > > > > > > transition
> > > > > > > > > > > > > into
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > SHUTDOWN state. This motivates the
> > > controller to
> > > > > > > move
> > > > > > > > > all
> > > > > > > > > > > of the
> > > > > > > > > > > > > > > leaders
> > > > > > > > > > > > > > > > > > off of that broker. Once they are all
> > > moved, the
> > > > > > > > > controller
> > > > > > > > > > > > > > > responds to
> > > > > > > > > > > > > > > > > > the heartbeat with a nextState of SHUTDOWN.";
> > > > > (2) the
> > > > > > > > > > > > > description of
> > > > > > > > > > > > > > > > > > BrokerHeartbeat still references
> > > currentState and
> > > > > > > > > > > targetState.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks. I've made these sections clearer and
> > > > > removed
> > > > > > > the
> > > > > > > > > > > obsolete
> > > > > > > > > > > > > > > > > references to sending states.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe
> > > <
> > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao
> > > wrote:
> > > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for the update. A few more follow
> > > up
> > > > > > > comments.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks again for the review.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 100. FailedReplicaRecord: Since this is
> > > > > reported
> > > > > > > by
> > > > > > > > > each
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > independently, perhaps we could use a
> > > more
> > > > > > > concise
> > > > > > > > > > > > > representation
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > a top level broker field, an array of
> > > topics,
> > > > > > > which
> > > > > > > > > has
> > > > > > > > > > > an
> > > > > > > > > > > > > array
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > partitions.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The issue is that there is a size limit on
> > > the
> > > > > each
> > > > > > > > > record.
> > > > > > > > > > > > > > > Putting
> > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > of the partitions of a log directory into a
> > > > > single
> > > > > > > > > record
> > > > > > > > > > > would
> > > > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > > > break that in many cases. Still, we can
> > > > > optimize a
> > > > > > > > > bit by
> > > > > > > > > > > > > having
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > array
> > > > > > > > > > > > > > > > > > > of partition IDs, since nearly all the
> > > time, we
> > > > > > > have
> > > > > > > > > more
> > > > > > > > > > > than
> > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > from the
> > > > > > > > > > > > > > > > > > > same topic.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 200. Sounds good. If we remove the
> > > > > broker-side
> > > > > > > > > fencing
> > > > > > > > > > > > > logic, do
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > plan
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > still keep FENCED in broker state? Do we
> > > > > plan to
> > > > > > > > > expose
> > > > > > > > > > > the
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > states
> > > > > > > > > > > > > > > > > > > > through the existing BrokerState metric
> > > and
> > > > > if
> > > > > > > so,
> > > > > > > > > what
> > > > > > > > > > > are
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > values
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > the new states?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > No, we don't need FENCED any more. I have
> > > > > removed
> > > > > > > it
> > > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The new states are very similar to the
> > > current
> > > > > > > ones,
> > > > > > > > > > > actually.
> > > > > > > > > > > > > > > There
> > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > no new states or removed ones. The main
> > > > > change in
> > > > > > > the
> > > > > > > > > > > broker
> > > > > > > > > > > > > state
> > > > > > > > > > > > > > > > > machine
> > > > > > > > > > > > > > > > > > > is that the
> > > RECOVERING_FROM_UNCLEAN_SHUTDOWN
> > > > > state
> > > > > > > has
> > > > > > > > > been
> > > > > > > > > > > > > > > renamed to
> > > > > > > > > > > > > > > > > > > RECOVERY. Also, unlike previously, the
> > > broker
> > > > > will
> > > > > > > > > always
> > > > > > > > > > > pass
> > > > > > > > > > > > > > > through
> > > > > > > > > > > > > > > > > > > RECOVERY (although it may only stay in this
> > > > > state
> > > > > > > for a
> > > > > > > > > > > very
> > > > > > > > > > > > > short
> > > > > > > > > > > > > > > > > amount
> > > > > > > > > > > > > > > > > > > of time).
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 201. This may be fine too. Could we
> > > document
> > > > > what
> > > > > > > > > happens
> > > > > > > > > > > > > when
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > broker.id/controller.id in
> > > > > metadata.properties
> > > > > > > don't
> > > > > > > > > > > match
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > config when the broker starts up?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I added some documentation about this.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 204. There is still "The highest metadata
> > > > > offset
> > > > > > > > > which
> > > > > > > > > > > the
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > has not
> > > > > > > > > > > > > > > > > > > > reached" referenced under
> > > BrokerRegistration.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 206. Is that separate step needed given
> > > > > KIP-516?
> > > > > > > With
> > > > > > > > > > > > > KIP-516 (
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > > > > > > > > > > > > > > ),
> > > > > > > > > > > > > > > > > > > > we don't need to wait for the topic data
> > > to
> > > > > be
> > > > > > > > > removed
> > > > > > > > > > > from
> > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > > > > > before removing the topic metadata. The
> > > > > > > combination
> > > > > > > > > of
> > > > > > > > > > > > > unmatching
> > > > > > > > > > > > > > > > > > > > topicId
> > > > > > > > > > > > > > > > > > > > or the missing topicId from the metadata
> > > is
> > > > > > > enough
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > > clean
> > > > > > > > > > > > > > > > > > > > up deleted topics asynchronously.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It won't be needed once KIP-516 is
> > > adopted, but
> > > > > > > this
> > > > > > > > > hasn't
> > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > implemented yet.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin
> > > McCabe <
> > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao
> > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for the updated KIP. A few
> > > more
> > > > > > > comments
> > > > > > > > > > > below.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks again for the reviews.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 80.2 For deprecated configs, we need
> > > to
> > > > > > > include
> > > > > > > > > > > > > zookeeper.*
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > broker.id.generation.enable.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Added.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 83.1 If a broker is down, does the
> > > > > controller
> > > > > > > > > keep
> > > > > > > > > > > the
> > > > > > > > > > > > > > > previously
> > > > > > > > > > > > > > > > > > > > > > registered broker epoch forever? If
> > > not,
> > > > > how
> > > > > > > long
> > > > > > > > > > > does
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > > > > > > > > it? What does the controller do when
> > > > > > > receiving a
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > > > > > with an unfound broker epoch?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Yes, the controller keeps the previous
> > > > > > > registration
> > > > > > > > > > > > > forever.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Broker heartbeat requests with an
> > > incorrect
> > > > > > > broker
> > > > > > > > > > > epoch
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > rejected
> > > > > > > > > > > > > > > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 100. Have you figured out if we need
> > > to
> > > > > add
> > > > > > > a new
> > > > > > > > > > > record
> > > > > > > > > > > > > > > type for
> > > > > > > > > > > > > > > > > > > > > reporting
> > > > > > > > > > > > > > > > > > > > > > partitions on failed disks?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I added FailedReplicaRecord to reflect
> > > the
> > > > > case
> > > > > > > > > where a
> > > > > > > > > > > > > JBOD
> > > > > > > > > > > > > > > > > directory
> > > > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > > failed, leading to failed replicas.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 102. For debugging purposes,
> > > sometimes
> > > > > it's
> > > > > > > > > useful to
> > > > > > > > > > > > > read
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > > > > > topic using tools like
> > > console-consumer.
> > > > > > > Should
> > > > > > > > > we
> > > > > > > > > > > > > support
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > so,
> > > > > > > > > > > > > > > > > > > > > > how?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > For now, we have the ability to read
> > > the
> > > > > > > metadata
> > > > > > > > > logs
> > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > dump-logs
> > > > > > > > > > > > > > > > > > > > > tool. I think we will come up with
> > > some
> > > > > other
> > > > > > > > > tools
> > > > > > > > > > > in the
> > > > > > > > > > > > > > > future
> > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > get experience.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 200. "brokers which are fenced will
> > > not
> > > > > > > appear in
> > > > > > > > > > > > > > > > > MetadataResponses.
> > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > broker will not respond to these
> > > > > requests--
> > > > > > > > > instead,
> > > > > > > > > > > it
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > > > > > disconnect." If the controller is
> > > > > > > partitioned off
> > > > > > > > > > > from
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > brokers,
> > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > design will cause every broker to
> > > stop
> > > > > > > accepting
> > > > > > > > > new
> > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > > requests. In
> > > > > > > > > > > > > > > > > > > > > > contrast, if ZK is partitioned off,
> > > the
> > > > > > > existing
> > > > > > > > > > > > > behavior is
> > > > > > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > > > > > brokers can continue to work based
> > > on the
> > > > > > > last
> > > > > > > > > known
> > > > > > > > > > > > > > > metadata.
> > > > > > > > > > > > > > > > > So, I
> > > > > > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > sure if we should change the existing
> > > > > > > behavior
> > > > > > > > > > > because
> > > > > > > > > > > > > of the
> > > > > > > > > > > > > > > > > bigger
> > > > > > > > > > > > > > > > > > > > > impact
> > > > > > > > > > > > > > > > > > > > > > in the new one. Another option is to
> > > > > keep the
> > > > > > > > > > > existing
> > > > > > > > > > > > > > > behavior
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > expose
> > > > > > > > > > > > > > > > > > > > > > a metric for fenced brokers so that
> > > the
> > > > > > > operator
> > > > > > > > > > > could be
> > > > > > > > > > > > > > > > > alerted.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I'm skeptical about how well running
> > > > > without ZK
> > > > > > > > > > > currently
> > > > > > > > > > > > > > > works.
> > > > > > > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > > > > > > I will move the broker-side fencing
> > > into a
> > > > > > > > > follow-up
> > > > > > > > > > > KIP.
> > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > > > > > > already pretty large and there is no
> > > hard
> > > > > > > > > dependency on
> > > > > > > > > > > > > this.
> > > > > > > > > > > > > > > > > There
> > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > also be other ways of accomplishing the
> > > > > > > positive
> > > > > > > > > > > effects of
> > > > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > > > > > broker-side fencing, so more
> > > discussion is
> > > > > > > needed.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 201. I read Ron's comment, but I am
> > > > > still not
> > > > > > > > > sure
> > > > > > > > > > > the
> > > > > > > > > > > > > > > benefit of
> > > > > > > > > > > > > > > > > > > keeping
> > > > > > > > > > > > > > > > > > > > > > broker.id and controller.id in
> > > > > > > meta.properties.
> > > > > > > > > It
> > > > > > > > > > > seems
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > we are
> > > > > > > > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > > > > duplicating the same info in two
> > > places
> > > > > and
> > > > > > > have
> > > > > > > > > the
> > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > burden of
> > > > > > > > > > > > > > > > > > > > > > making sure the values in the two
> > > places
> > > > > are
> > > > > > > > > > > consistent.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think the reasoning is that having
> > > > > broker.id
> > > > > > > > > > > protects us
> > > > > > > > > > > > > > > against
> > > > > > > > > > > > > > > > > > > > > accidentally bringing up a broker with
> > > a
> > > > > disk
> > > > > > > from
> > > > > > > > > a
> > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > broker. I
> > > > > > > > > > > > > > > > > > > > > don't feel strongly about this but it
> > > > > seemed
> > > > > > > > > simpler to
> > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 202.
> > > > > controller.connect.security.protocol: Is
> > > > > > > > > this
> > > > > > > > > > > needed
> > > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > > > > > controller.listener.names and
> > > > > > > > > > > > > listener.security.protocol.map
> > > > > > > > > > > > > > > > > imply
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > security protocol already?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > You're right, this isn't needed. I'll
> > > > > remove
> > > > > > > it.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 203.
> > > registration.heartbeat.interval.ms:
> > > > > It
> > > > > > > > > > > defaults to
> > > > > > > > > > > > > 2k.
> > > > > > > > > > > > > > > ZK
> > > > > > > > > > > > > > > > > uses
> > > > > > > > > > > > > > > > > > > 1/3
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > the session timeout for heartbeat.
> > > So,
> > > > > given
> > > > > > > the
> > > > > > > > > > > default
> > > > > > > > > > > > > 18k
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > registration.lease.timeout.ms,
> > > should we
> > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms
> > > to
> > > > > 6k?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 6 seconds seems like a pretty long time
> > > > > between
> > > > > > > > > > > > > heartbeats. It
> > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > useful to know when a broker is missing
> > > > > > > heartbeats,
> > > > > > > > > > > with
> > > > > > > > > > > > > less
> > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > > that. I provisionally set it to 3
> > > seconds
> > > > > (we
> > > > > > > can
> > > > > > > > > > > always
> > > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > > > later...)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I also changed the name of these
> > > > > > > configurations to
> > > > > > > > > "
> > > > > > > > > > > > > > > > > > > > > broker.heartbeat.interval.ms" and "
> > > > > > > > > > > > > > > broker.registration.timeout.ms"
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > > > > > > > to clarify them a bit.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 204. "The highest metadata offset
> > > which
> > > > > the
> > > > > > > > > broker
> > > > > > > > > > > has
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > reached."
> > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > seems this should be "has reached".
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I changed this to "one more than the
> > > > > highest
> > > > > > > > > metadata
> > > > > > > > > > > > > offset
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > broker has reached."
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 205. UnfenceBrokerRecord and
> > > > > > > > > UnregisterBrokerRecord:
> > > > > > > > > > > To
> > > > > > > > > > > > > me,
> > > > > > > > > > > > > > > they
> > > > > > > > > > > > > > > > > > > seem to
> > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > the same. Do we need both?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Unregistration means that the broker
> > > has
> > > > > been
> > > > > > > > > removed
> > > > > > > > > > > from
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > cluster.
> > > > > > > > > > > > > > > > > > > > > That is different than unfencing, which
> > > > > marks
> > > > > > > the
> > > > > > > > > > > broker as
> > > > > > > > > > > > > > > active.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 206. TopicRecord: The Deleting field
> > > is
> > > > > used
> > > > > > > to
> > > > > > > > > > > indicate
> > > > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > being deleted. I am wondering if
> > > this is
> > > > > > > really
> > > > > > > > > > > needed
> > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > > RemoveTopic
> > > > > > > > > > > > > > > > > > > > > > already indicates the same thing.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > RemoveTopic is the last step, that
> > > scrubs
> > > > > all
> > > > > > > > > metadata
> > > > > > > > > > > > > about
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > topic.
> > > > > > > > > > > > > > > > > > > > > In order to get to that last step, the
> > > > > topic
> > > > > > > data
> > > > > > > > > > > needs to
> > > > > > > > > > > > > > > removed
> > > > > > > > > > > > > > > > > > > from all
> > > > > > > > > > > > > > > > > > > > > brokers (after each broker notices
> > > that the
> > > > > > > topic
> > > > > > > > > is
> > > > > > > > > > > being
> > > > > > > > > > > > > > > > > deleted).
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin
> > > > > McCabe <
> > > > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron
> > > > > > > Dagostino
> > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the
> > > updates.
> > > > > It's
> > > > > > > now
> > > > > > > > > > > clear
> > > > > > > > > > > > > to me
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > > > > > > > > > keep their broker epoch for the
> > > life
> > > > > of
> > > > > > > their
> > > > > > > > > > > JVM --
> > > > > > > > > > > > > they
> > > > > > > > > > > > > > > > > > > register
> > > > > > > > > > > > > > > > > > > > > > > > once, get their broker epoch in
> > > the
> > > > > > > > > response, and
> > > > > > > > > > > > > then
> > > > > > > > > > > > > > > never
> > > > > > > > > > > > > > > > > > > > > > > > re-register again. Brokers may
> > > get
> > > > > > > fenced,
> > > > > > > > > but
> > > > > > > > > > > they
> > > > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > > > > > > broker epoch for the life of
> > > their
> > > > > JVM.
> > > > > > > The
> > > > > > > > > > > > > incarnation
> > > > > > > > > > > > > > > ID
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > kept for the life of the JVM but
> > > is
> > > > > > > > > generated by
> > > > > > > > > > > the
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > > > > > > upon startup, and the
> > > combination of
> > > > > the
> > > > > > > two
> > > > > > > > > > > allows
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Controller to
> > > > > > > > > > > > > > > > > > > > > > > > act idempotently if any
> > > > > previously-sent
> > > > > > > > > > > registration
> > > > > > > > > > > > > > > response
> > > > > > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks, Ron. That's a good
> > > summary.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > One thing I wonder about is if it
> > > > > might
> > > > > > > be
> > > > > > > > > > > helpful
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > > > > > > send the Cluster ID as determined
> > > > > from
> > > > > > > its
> > > > > > > > > > > > > > > meta.properties
> > > > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > > > > > > > registration request. Does it
> > > even
> > > > > make
> > > > > > > > > sense
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > > > > > > successfully register and enter
> > > the
> > > > > > > Fenced
> > > > > > > > > state
> > > > > > > > > > > if
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > wrong
> > > > > > > > > > > > > > > > > > > > > > > > Cluster ID?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Yeah, that's a good idea. Let's
> > > have
> > > > > the
> > > > > > > > > broker
> > > > > > > > > > > pass
> > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > > > ID in
> > > > > > > > > > > > > > > > > > > > > > > the registration RPC, and then
> > > > > > > registration can
> > > > > > > > > > > fail
> > > > > > > > > > > > > if the
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The nextMetadatOffset value
> > > that the
> > > > > > > broker
> > > > > > > > > > > > > communicates
> > > > > > > > > > > > > > > > > > > > > > > > in its registration request only
> > > has
> > > > > > > meaning
> > > > > > > > > > > within
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > > > > > cluster, so it feels to me that
> > > the
> > > > > > > > > Controller
> > > > > > > > > > > should
> > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > perform this sanity check.
> > > There is
> > > > > > > > > currently
> > > > > > > > > > > > > (pre-KIP
> > > > > > > > > > > > > > > 500)
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > check
> > > > > > > > > > > > > > > > > > > > > > > > in the broker to make sure its
> > > > > configured
> > > > > > > > > > > cluster ID
> > > > > > > > > > > > > > > matches
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > > > > > > > stored in ZooKeeper, and we will
> > > > > have to
> > > > > > > > > perform
> > > > > > > > > > > this
> > > > > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > > > > > somewhere in the KIP-500 world.
> > > If
> > > > > the
> > > > > > > > > > > Controller
> > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > do it
> > > > > > > > > > > > > > > > > > > > > > > > within the registration request
> > > then
> > > > > the
> > > > > > > > > broker
> > > > > > > > > > > will
> > > > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > > > > > metadata request to the
> > > Controller,
> > > > > > > retrieve
> > > > > > > > > the
> > > > > > > > > > > > > Cluster
> > > > > > > > > > > > > > > ID,
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > perform the check itself. It
> > > feels
> > > > > to me
> > > > > > > > > that it
> > > > > > > > > > > > > might
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > better for
> > > > > > > > > > > > > > > > > > > > > > > > the Controller to just do it, and
> > > > > then
> > > > > > > the
> > > > > > > > > broker
> > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > > > > > > worry about it anymore once it
> > > > > > > successfully
> > > > > > > > > > > > > registers.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I also have a question about the
> > > > > > > broker.id
> > > > > > > > > > > value and
> > > > > > > > > > > > > > > > > > > > > meta.properties.
> > > > > > > > > > > > > > > > > > > > > > > > The KIP now says "In version 0 of
> > > > > > > > > > > meta.properties,
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > > > > > > broker.id field. Version 1
> > > does not
> > > > > > > have
> > > > > > > > > this
> > > > > > > > > > > > > field.
> > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > is no
> > > > > > > > > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > > > > > > > > > needed because we no longer
> > > support
> > > > > > > dynamic
> > > > > > > > > > > broker id
> > > > > > > > > > > > > > > > > > > assignment."
> > > > > > > > > > > > > > > > > > > > > > > > But then there is an example
> > > version
> > > > > 1
> > > > > > > > > > > > > meta.properties
> > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > shows
> > > > > > > > > > > > > > > > > > > > > > > > the broker.id value. I actually
> > > > > wonder
> > > > > > > if
> > > > > > > > > > > maybe the
> > > > > > > > > > > > > > > > > broker.id
> > > > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > > > > would be good to keep in the
> > > version
> > > > > 1
> > > > > > > > > > > > > meta.properties
> > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > currently (pre-KIP 500, version
> > > 0)
> > > > > acts
> > > > > > > as a
> > > > > > > > > > > sanity
> > > > > > > > > > > > > > > check to
> > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > > > sure
> > > > > > > > > > > > > > > > > > > > > > > > the broker is using the correct
> > > log
> > > > > > > > > directory.
> > > > > > > > > > > > > Similarly
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > controller.id value on
> > > controllers
> > > > > -- it
> > > > > > > > > would
> > > > > > > > > > > > > allow the
> > > > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > sanity check for quorum
> > > controllers.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > That's a good point. I will add
> > > > > broker.id
> > > > > > > > > back,
> > > > > > > > > > > and
> > > > > > > > > > > > > also
> > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > > > > controller.id as a possibility.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM
> > > Colin
> > > > > > > McCabe
> > > > > > > > > <
> > > > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10,
> > > > > Jun Rao
> > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the reply. A few
> > > more
> > > > > > > > > comments.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks again for the reply.
> > > Sorry
> > > > > for
> > > > > > > the
> > > > > > > > > long
> > > > > > > > > > > > > > > hiatus. I
> > > > > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > > > > vacation for a while.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 55. There is still text that
> > > > > favors
> > > > > > > new
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > registration.
> > > > > > > > > > > > > > > > > > > > > "When a
> > > > > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > > > > first starts up, when it is
> > > in
> > > > > the
> > > > > > > > > INITIAL
> > > > > > > > > > > > > state, it
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > always
> > > > > > > > > > > > > > > > > > > > > > > "win"
> > > > > > > > > > > > > > > > > > > > > > > > > > broker ID conflicts.
> > > However,
> > > > > once
> > > > > > > it is
> > > > > > > > > > > > > granted a
> > > > > > > > > > > > > > > > > lease, it
> > > > > > > > > > > > > > > > > > > > > > > transitions
> > > > > > > > > > > > > > > > > > > > > > > > > > out of the INITIAL state.
> > > > > > > Thereafter,
> > > > > > > > > it may
> > > > > > > > > > > > > lose
> > > > > > > > > > > > > > > > > subsequent
> > > > > > > > > > > > > > > > > > > > > > > conflicts if
> > > > > > > > > > > > > > > > > > > > > > > > > > its broker epoch is stale.
> > > (See
> > > > > > > KIP-380
> > > > > > > > > for
> > > > > > > > > > > some
> > > > > > > > > > > > > > > > > background
> > > > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > > > > epoch.) The reason for
> > > favoring
> > > > > new
> > > > > > > > > > > processes
> > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > accommodate
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > > > > > > > > > > case where a process is
> > > killed
> > > > > with
> > > > > > > kill
> > > > > > > > > -9
> > > > > > > > > > > and
> > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > > > restarted.
> > > > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > > > want it
> > > > > > > > > > > > > > > > > > > > > > > > > > to be able to reclaim its
> > > old ID
> > > > > > > quickly
> > > > > > > > > in
> > > > > > > > > > > this
> > > > > > > > > > > > > > > case."
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the reminder. I
> > > have
> > > > > > > clarified
> > > > > > > > > the
> > > > > > > > > > > > > language
> > > > > > > > > > > > > > > > > here.
> > > > > > > > > > > > > > > > > > > > > > > Hopefully now it is clear that we
> > > don't
> > > > > > > allow
> > > > > > > > > quick
> > > > > > > > > > > > > re-use
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 80.1 Sounds good. Could you
> > > > > document
> > > > > > > that
> > > > > > > > > > > > > listeners
> > > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > > > > > > > now? It would also be useful
> > > to
> > > > > > > annotate
> > > > > > > > > > > other
> > > > > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > > > configs.
> > > > > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > > > > > > > example, controller.connect
> > > > > should be
> > > > > > > > > > > required.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I added a note specifying that
> > > > > these
> > > > > > > are
> > > > > > > > > > > required.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 80.2 Could you list all
> > > > > deprecated
> > > > > > > > > existing
> > > > > > > > > > > > > configs?
> > > > > > > > > > > > > > > > > Another
> > > > > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > control.plane.listener.name
> > > > > since
> > > > > > > the
> > > > > > > > > > > > > controller no
> > > > > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > > > > sends
> > > > > > > > > > > > > > > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata
> > > and
> > > > > > > > > StopReplica
> > > > > > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I added a section specifying
> > > some
> > > > > > > > > deprecated
> > > > > > > > > > > > > configs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 83.1 It seems that the
> > > broker can
> > > > > > > > > transition
> > > > > > > > > > > from
> > > > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > RUNNING
> > > > > > > > > > > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > > > > > > > > > > registering for a new broker
> > > > > epoch.
> > > > > > > I am
> > > > > > > > > not
> > > > > > > > > > > > > sure how
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > works.
> > > > > > > > > > > > > > > > > > > > > > > Once the
> > > > > > > > > > > > > > > > > > > > > > > > > > controller fences a broker,
> > > > > there is
> > > > > > > no
> > > > > > > > > need
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > > > > > > > > > > > boker epoch around. So, if
> > > the
> > > > > fenced
> > > > > > > > > > > broker's
> > > > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > > > > > > > > existing broker epoch will be
> > > > > > > rejected,
> > > > > > > > > > > leading
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The broker epoch refers to the
> > > > > broker
> > > > > > > > > > > registration.
> > > > > > > > > > > > > > > So we
> > > > > > > > > > > > > > > > > DO
> > > > > > > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > broker epoch around even while the
> > > > > broker
> > > > > > > is
> > > > > > > > > > > fenced.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The broker epoch changes only
> > > when
> > > > > > > there
> > > > > > > > > is a
> > > > > > > > > > > new
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > registration. Fencing or
> > > unfencing the
> > > > > > > broker
> > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 83.5 Good point on KIP-590.
> > > Then
> > > > > > > should
> > > > > > > > > we
> > > > > > > > > > > > > expose the
> > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > debugging purposes? If not,
> > > we
> > > > > should
> > > > > > > > > > > deprecate
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > controllerID
> > > > > > > > > > > > > > > > > > > > > > > field in
> > > > > > > > > > > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I think it's OK to expose it
> > > for
> > > > > now,
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > proviso
> > > > > > > > > > > > > > > > > that it
> > > > > > > > > > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > > > > > > > > > be reachable by clients.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 90. We rejected the shared ID
> > > > > with
> > > > > > > just
> > > > > > > > > one
> > > > > > > > > > > > > reason
> > > > > > > > > > > > > > > "This
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > not a
> > > > > > > > > > > > > > > > > > > > > > > good idea
> > > > > > > > > > > > > > > > > > > > > > > > > > because NetworkClient
> > > assumes a
> > > > > > > single ID
> > > > > > > > > > > > > space. So
> > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > there is
> > > > > > > > > > > > > > > > > > > > > > > both a
> > > > > > > > > > > > > > > > > > > > > > > > > > controller 1 and a broker 1,
> > > we
> > > > > don't
> > > > > > > > > have a
> > > > > > > > > > > way
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > picking
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > "right"
> > > > > > > > > > > > > > > > > > > > > > > > > > one." This doesn't seem to
> > > be a
> > > > > > > strong
> > > > > > > > > > > reason.
> > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > example,
> > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > > > > > > > > > address the NetworkClient
> > > issue
> > > > > with
> > > > > > > the
> > > > > > > > > node
> > > > > > > > > > > > > type
> > > > > > > > > > > > > > > as you
> > > > > > > > > > > > > > > > > > > pointed
> > > > > > > > > > > > > > > > > > > > > > > out or
> > > > > > > > > > > > > > > > > > > > > > > > > > using the negative value of a
> > > > > broker
> > > > > > > ID
> > > > > > > > > as
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > controller ID.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > It would require a lot of code
> > > > > changes
> > > > > > > to
> > > > > > > > > > > support
> > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > types of
> > > > > > > > > > > > > > > > > > > > > > > node IDs. It's not clear to me
> > > that
> > > > > the
> > > > > > > end
> > > > > > > > > result
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > better --
> > > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > > tend to think it would be worse,
> > > since
> > > > > it
> > > > > > > > > would be
> > > > > > > > > > > more
> > > > > > > > > > > > > > > > > complex.
> > > > > > > > > > > > > > > > > > > In a
> > > > > > > > > > > > > > > > > > > > > > > similar vein, using negative
> > > numbers
> > > > > seems
> > > > > > > > > > > dangerous,
> > > > > > > > > > > > > > > since we
> > > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > > > negatives or -1 as "special
> > > values" in
> > > > > many
> > > > > > > > > places.
> > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > example,
> > > > > > > > > > > > > > > > > > > -1
> > > > > > > > > > > > > > > > > > > > > often
> > > > > > > > > > > > > > > > > > > > > > > represents "no such node."
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > One important thing to keep in
> > > > > mind is
> > > > > > > > > that we
> > > > > > > > > > > > > want to
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > transition from a broker and a
> > > > > controller
> > > > > > > being
> > > > > > > > > > > > > co-located
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > them
> > > > > > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > > > > > > > longer being co-located. This is
> > > much
> > > > > > > easier
> > > > > > > > > to do
> > > > > > > > > > > > > when
> > > > > > > > > > > > > > > they
> > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > > > > > > > the broker reports all
> > > offline
> > > > > > > replicas
> > > > > > > > > due
> > > > > > > > > > > to a
> > > > > > > > > > > > > disk
> > > > > > > > > > > > > > > > > > > failure to
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > controller. It seems this
> > > > > information
> > > > > > > > > needs
> > > > > > > > > > > to be
> > > > > > > > > > > > > > > > > persisted
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > > > > > > > > > log. Do we have a
> > > corresponding
> > > > > > > record
> > > > > > > > > for
> > > > > > > > > > > that?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hmm, I have to look into this a
> > > > > little
> > > > > > > bit
> > > > > > > > > > > more.
> > > > > > > > > > > > > We
> > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > > > > > > > > record type.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 101. Currently, StopReplica
> > > > > request
> > > > > > > has 2
> > > > > > > > > > > modes,
> > > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > > > > > > and with
> > > > > > > > > > > > > > > > > > > > > > > > > > deletion. The former is used
> > > for
> > > > > > > > > controlled
> > > > > > > > > > > > > shutdown
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > handling
> > > > > > > > > > > > > > > > > > > > > > > disk
> > > > > > > > > > > > > > > > > > > > > > > > > > failure, and causes the
> > > follower
> > > > > to
> > > > > > > > > stop. The
> > > > > > > > > > > > > latter
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > > > > > > > > > and partition reassignment,
> > > and
> > > > > > > causes
> > > > > > > > > the
> > > > > > > > > > > > > replica
> > > > > > > > > > > > > > > to be
> > > > > > > > > > > > > > > > > > > deleted.
> > > > > > > > > > > > > > > > > > > > > > > Since we
> > > > > > > > > > > > > > > > > > > > > > > > > > are deprecating StopReplica,
> > > > > could we
> > > > > > > > > > > document
> > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > triggers
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > stopping
> > > > > > > > > > > > > > > > > > > > > > > > > > of a follower and the
> > > deleting
> > > > > of a
> > > > > > > > > replica
> > > > > > > > > > > now?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > RemoveTopic triggers
> > > deletion. In
> > > > > > > general
> > > > > > > > > the
> > > > > > > > > > > > > > > > > functionality of
> > > > > > > > > > > > > > > > > > > > > > > StopReplica is subsumed by the
> > > metadata
> > > > > > > > > records.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 102. Should we include the
> > > > > metadata
> > > > > > > > > topic in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > > > > > > If so,
> > > > > > > > > > > > > > > > > > > > > > > > > > when it will be included and
> > > what
> > > > > > > will
> > > > > > > > > the
> > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > response
> > > > > > > > > > > > > > > > > > > look
> > > > > > > > > > > > > > > > > > > > > > > like?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > No, it won't be included in the
> > > > > > > metadata
> > > > > > > > > > > response
> > > > > > > > > > > > > sent
> > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > brokers.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 103. "The active controller
> > > > > assigns
> > > > > > > the
> > > > > > > > > > > broker a
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > epoch,
> > > > > > > > > > > > > > > > > > > > > > > based on
> > > > > > > > > > > > > > > > > > > > > > > > > > the latest committed offset
> > > in
> > > > > the
> > > > > > > log."
> > > > > > > > > This
> > > > > > > > > > > > > seems
> > > > > > > > > > > > > > > > > > > inaccurate
> > > > > > > > > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > latest committed offset
> > > doesn't
> > > > > > > always
> > > > > > > > > > > advance on
> > > > > > > > > > > > > > > every
> > > > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > > > > append.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Given that the new broker epoch
> > > > > won't
> > > > > > > be
> > > > > > > > > > > visible
> > > > > > > > > > > > > until
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > commit
> > > > > > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > > > > happened, I have changed this to
> > > "the
> > > > > next
> > > > > > > > > > > available
> > > > > > > > > > > > > > > offset in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > log"
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 104. REGISTERING(1) : It says
> > > > > > > > > "Otherwise, the
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > moves
> > > > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > > > > > > > > > > > > state.". It seems this
> > > should be
> > > > > > > RUNNING?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 105. RUNNING: Should we
> > > require
> > > > > the
> > > > > > > > > broker to
> > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > to the
> > > > > > > > > > > > > > > > > > > > > > > metadata log
> > > > > > > > > > > > > > > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > For 104 and 105, these sections
> > > > > have
> > > > > > > been
> > > > > > > > > > > reworked.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20
> > > PM
> > > > > Colin
> > > > > > > > > McCabe
> > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020, at
> > > > > 05:51, Tom
> > > > > > > > > Bentley
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at
> > > > > 08:59,
> > > > > > > Ron
> > > > > > > > > > > Dagostino
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks
> > > for the
> > > > > > > hard
> > > > > > > > > work
> > > > > > > > > > > on
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have some questions
> > > > > about
> > > > > > > what
> > > > > > > > > > > happens
> > > > > > > > > > > > > to a
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > when it
> > > > > > > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fenced (e.g. because
> > > it
> > > > > can't
> > > > > > > > > send a
> > > > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > > > request to
> > > > > > > > > > > > > > > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > lease). The KIP says
> > > > > "When a
> > > > > > > > > broker
> > > > > > > > > > > is
> > > > > > > > > > > > > > > fenced,
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > cannot
> > > > > > > > > > > > > > > > > > > > > > > process any
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > client requests.
> > > This
> > > > > > > prevents
> > > > > > > > > > > brokers
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > are not
> > > > > > > > > > > > > > > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > metadata updates or
> > > that
> > > > > are
> > > > > > > not
> > > > > > > > > > > > > receiving
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > processing
> > > > > > > > > > > > > > > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > enough from causing
> > > > > issues to
> > > > > > > > > > > clients."
> > > > > > > > > > > > > And
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > FENCED(4) state it
> > > > > likewise
> > > > > > > says
> > > > > > > > > > > "While
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > not respond to client
> > > > > > > > > requests." It
> > > > > > > > > > > > > makes
> > > > > > > > > > > > > > > sense
> > > > > > > > > > > > > > > > > > > that a
> > > > > > > > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > should not accept
> > > > > producer
> > > > > > > > > requests
> > > > > > > > > > > -- I
> > > > > > > > > > > > > > > assume
> > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > would result in
> > > > > > > > > > > > > NotLeaderOrFollowerException.
> > > > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (fetch from follower)
> > > > > > > consumer
> > > > > > > > > > > > > requests? It
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > conceivable
> > > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > could continue.
> > > Related
> > > > > to
> > > > > > > that,
> > > > > > > > > > > would a
> > > > > > > > > > > > > > > fenced
> > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fetch data for
> > > partitions
> > > > > > > where
> > > > > > > > > it
> > > > > > > > > > > > > thinks it
> > > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > > > follower?
> > > > > > > > > > > > > > > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > it rejects consumer
> > > > > requests
> > > > > > > it
> > > > > > > > > might
> > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > > continue
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > follower. Might it
> > > be
> > > > > > > helpful to
> > > > > > > > > > > clarify
> > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > decisions
> > > > > > > > > > > > > > > > > > > > > > > here?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Good question. I
> > > think a
> > > > > > > fenced
> > > > > > > > > broker
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > partitions it was
> > > already
> > > > > > > fetching
> > > > > > > > > > > before
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > > > fenced,
> > > > > > > > > > > > > > > > > > > > > > > unless it
> > > > > > > > > > > > > > > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > problem. At that
> > > point it
> > > > > > > won't be
> > > > > > > > > > > able to
> > > > > > > > > > > > > > > > > continue,
> > > > > > > > > > > > > > > > > > > > > since it
> > > > > > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the new metadata. For
> > > > > > > example, it
> > > > > > > > > > > won't
> > > > > > > > > > > > > know
> > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > leadership
> > > > > > > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the partitions it's
> > > > > fetching.
> > > > > > > The
> > > > > > > > > > > > > rationale
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > continuing to
> > > > > > > > > > > > > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > try to avoid
> > > disruptions as
> > > > > > > much as
> > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't think fenced
> > > > > brokers
> > > > > > > should
> > > > > > > > > > > accept
> > > > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > > > > > > > > The issue
> > > > > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > that the fenced broker
> > > may
> > > > > or
> > > > > > > may
> > > > > > > > > not
> > > > > > > > > > > have
> > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > data it
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > have. It may or may
> > > not
> > > > > have
> > > > > > > > > applied
> > > > > > > > > > > any
> > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > it is supposed to have
> > > > > applied.
> > > > > > > > > So it
> > > > > > > > > > > > > could
> > > > > > > > > > > > > > > get
> > > > > > > > > > > > > > > > > pretty
> > > > > > > > > > > > > > > > > > > > > > > confusing, and
> > > > > > > > > > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > potentially waste the
> > > > > client's
> > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > When fenced, how would
> > > the
> > > > > broker
> > > > > > > > > reply
> > > > > > > > > > > to a
> > > > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > > which did
> > > > > > > > > > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > The broker will respond
> > > with a
> > > > > > > > > retryable
> > > > > > > > > > > error
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > case.
> > > > > > > > > > > > > > > > > > > > > Once
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > client has re-fetched its
> > > > > > > metadata, it
> > > > > > > > > > > will no
> > > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > > > > > > as part of the cluster. I
> > > > > added a
> > > > > > > > > note to
> > > > > > > > > > > the
> > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
>
>
> --
> Gwen Shapira
> Engineering Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Gwen Shapira <gw...@confluent.io>.
I realize that I'm very very late to a pretty big party, but I was
just looking at the new code (super readable, btw), and noticed that
the inverse API for registerBroker() is called decomissionBroker().
It would be way clearer that we are undoing a registration is we
rename it to unregisterBroker() (and the AdminClient API that matches,
to avoid more confusion).
Any objections?
On Fri, Dec 18, 2020 at 10:08 AM Jun Rao <ju...@confluent.io> wrote:
>
> Hi, Colin,
>
> Thanks for the reply. The KIP looks good to me now. Thanks for your
> diligence.
>
> Jun
>
> On Thu, Dec 17, 2020 at 5:15 PM Colin McCabe <cm...@apache.org> wrote:
>
> > On Thu, Dec 17, 2020, at 17:02, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply. Sounds good. One more comment.
> > >
> > > 231. Currently, we have sasl.mechanism.inter.broker.protocol for inter
> > > broker communication. It seems that we need a similar config for
> > specifying
> > > the sasl mechanism for the communication between the broker and the
> > > controller.
> > >
> >
> > Hi Jun,
> >
> > Yeah... sounds like we could use a new configuration key for this. I
> > added sasl.mechanism.controller.protocol for this.
> >
> > regards,
> > Colin
> >
> > > Jun
> > >
> > > On Thu, Dec 17, 2020 at 2:29 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Thu, Dec 17, 2020, at 10:19, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > 211. Hmm, I still don't see a clear benefit of registering a broker
> > > > before
> > > > > recovery. It's possible for the recovery to take time. However,
> > during
> > > > the
> > > > > recovery mode, it seems the broker will still be in the fenced mode
> > and
> > > > > won't be able to do work for the clients. So, registering and
> > > > heartbeating
> > > > > early seems to only add unnecessary overhead. For your point on topic
> > > > > creation, I thought we now allow replicas to be created on
> > > > > unregistered brokers.
> > > > >
> > > >
> > > > Hi Jun,
> > > >
> > > > Thanks again for the reviews.
> > > >
> > > > Topics cannot be created on unregistered brokers. They can be created
> > on
> > > > registered but fenced brokers. So for that reason I think it makes
> > sense
> > > > to register as early as possible.
> > > >
> > > > > 230. Currently, we do have a ControllerId field in MetadataResponse.
> > In
> > > > the
> > > > > early discussion, I thought that we want to expose the controller for
> > > > > debugging purposes, but not used by the client library.
> > > > >
> > > >
> > > > The current plan is that we will expose the controller node ID, but the
> > > > controller will not be included in the list of nodes in the metadata
> > > > response.
> > > >
> > > > It's not really possible to include the controller in that list of
> > nodes
> > > > because the controller may not share the same set of listeners as the
> > > > broker. So, for example, maybe the controller endpoint is using a
> > > > different type of security than the broker. So while we could pass
> > back a
> > > > hostname and port, the client would have no way to connect since it
> > doesn't
> > > > know what security settings to use.
> > > >
> > > > regards,
> > > > Colin
> > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Dec 16, 2020 at 9:13 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > On Wed, Dec 16, 2020, at 18:13, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the reply. Just a couple of more comments.
> > > > > > >
> > > > > > > 211. Currently, the broker only registers itself in ZK after log
> > > > > > recovery.
> > > > > > > Is there any benefit to change that? As you mentioned, the broker
> > > > can't
> > > > > > do
> > > > > > > much before completing log recovery.
> > > > > > >
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Previously, it wasn't possible to register in ZK without
> > immediately
> > > > > > getting added to the MetadataResponse. So I think that's the main
> > > > reason
> > > > > > why registration was delayed until after log recovery. Since that
> > > > > > constraint doesn't exist any more, there seems to be no reason to
> > delay
> > > > > > registration.
> > > > > >
> > > > > > I think delaying registration would have some major downsides. If
> > log
> > > > > > recovery takes a while, that's a longer window during which someone
> > > > else
> > > > > > could register a broker with the same ID. Having parts of the
> > cluster
> > > > > > missing for a while gives up some of the benefit of separating
> > > > registration
> > > > > > from fencing. For example, if a broker somehow gets unregistered
> > and
> > > > we
> > > > > > want to re-register it, but we have to wait for a 10 minute log
> > > > recovery to
> > > > > > do that, that could be a window during which topics can't be
> > created
> > > > that
> > > > > > need to be on that broker, and so forth.
> > > > > >
> > > > > > > 230. Regarding MetadataResponse, there is a slight awkwardness.
> > We
> > > > return
> > > > > > > rack for each node. However, if that node is for the controller,
> > the
> > > > rack
> > > > > > > field is not really relevant. Should we clean it up here or in
> > > > another
> > > > > > KIP
> > > > > > > like KIP-700?
> > > > > >
> > > > > > Oh, controllers don't appear in the MetadataResponses returned to
> > > > clients,
> > > > > > since clients can't access them. I should have been more clear
> > about
> > > > that
> > > > > > in the KIP-- I added a sentence to "Networking" describing this.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Wed, Dec 16, 2020 at 4:23 PM Colin McCabe <cmccabe@apache.org
> > >
> > > > wrote:
> > > > > > >
> > > > > > > > On Wed, Dec 16, 2020, at 13:40, Jun Rao wrote:
> > > > > > > > > Hi, Colin,
> > > > > > > > >
> > > > > > > > > Thanks for the reply. A few follow up comments.
> > > > > > > > >
> > > > > > > > > 211. When does the broker send the BrokerRegistration
> > request to
> > > > the
> > > > > > > > > controller? Is it after the recovery phase? If so, at that
> > > > point, the
> > > > > > > > > broker has already caught up on the metadata (in order to
> > clean
> > > > up
> > > > > > > > deleted
> > > > > > > > > partitions). Then, it seems we don't need the ShouldFence
> > field
> > > > > > > > > in BrokerHeartbeatRequest?
> > > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > Thanks again for the reviews.
> > > > > > > >
> > > > > > > > The broker sends the registration request as soon as it starts
> > > > up. It
> > > > > > > > cannot wait until the recovery phase is over since sometimes
> > log
> > > > > > recovery
> > > > > > > > can take quite a long time.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 213. kafka-cluster.sh
> > > > > > > > > 213.1 For the decommision example, should the command take a
> > > > broker
> > > > > > id?
> > > > > > > >
> > > > > > > > Yes, the example should have a broker id. Fixed.
> > > > > > > >
> > > > > > > > > 213.2 Existing tools use the "--" command line option (e.g.
> > > > > > kafka-topics
> > > > > > > > > --list --topic test). Should we follow the same convention
> > > > > > > > > for kafka-cluster.sh (and kafka-storage.sh)?
> > > > > > > >
> > > > > > > > Hmm. I don't think argparse4j supports using double dashes to
> > > > identify
> > > > > > > > subcommands. I think it might be confusing as well, since the
> > > > > > subcommand
> > > > > > > > must come first, unlike a plain old argument which can be
> > anywhere
> > > > on
> > > > > > the
> > > > > > > > command line.
> > > > > > > >
> > > > > > > > > 213.3 Should we add an admin api for broker decommission so
> > that
> > > > > > this can
> > > > > > > > > be done programmatically?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Yes, there is an admin client API for decommissioning as well.
> > > > > > > >
> > > > > > > > > 220. DecommissionBroker: "NOT_CONTROLLER if the node that the
> > > > > > request was
> > > > > > > > > sent to is not the controller". I thought the clients never
> > send
> > > > a
> > > > > > > > request
> > > > > > > > > to the controller directly now and the broker will forward
> > it to
> > > > the
> > > > > > > > > controller?
> > > > > > > > >
> > > > > > > >
> > > > > > > > If the controller moved recently, it's possible that the broker
> > > > could
> > > > > > send
> > > > > > > > to a controller that has just recently become inactive. In
> > that
> > > > case
> > > > > > > > NOT_CONTROLLER would be returned. (A standby controller
> > returns
> > > > > > > > NOT_CONTROLLER for most APIs).
> > > > > > > >
> > > > > > > > > 221. Could we add the required ACL for the new requests?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Good point. I added the required ACL for each new RPC.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Wed, Dec 16, 2020 at 11:38 AM Colin McCabe <
> > > > cmccabe@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Wed, Dec 16, 2020, at 09:59, Jun Rao wrote:
> > > > > > > > > > > Hi, Colin,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the reply. A few more comments below.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi Jun,
> > > > > > > > > >
> > > > > > > > > > Thanks for the comments.
> > > > > > > > > >
> > > > > > > > > > > 206. "RemoveTopic is the last step, that scrubs all
> > metadata
> > > > > > about
> > > > > > > > the
> > > > > > > > > > > topic. In order to get to that last step, the topic data
> > > > needs
> > > > > > to
> > > > > > > > > > removed
> > > > > > > > > > > from all brokers (after each broker notices that the
> > topic is
> > > > > > being
> > > > > > > > > > > deleted)." Currently, this is done based on the response
> > of
> > > > > > > > > > > StopReplicaRequest. Since the controller no longer sends
> > this
> > > > > > > > request,
> > > > > > > > > > how
> > > > > > > > > > > does the controller know that the data for the deleted
> > topic
> > > > has
> > > > > > > > > > > been removed in the brokers?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > That's a good point... definitely an oversight.
> > > > > > > > > >
> > > > > > > > > > It seems complex to force the controller to track when log
> > > > > > directories
> > > > > > > > > > have been deleted. Let's just assume that KIP-516 has been
> > > > > > > > implemented,
> > > > > > > > > > and track them by UUID. Then we can just have a single
> > topic
> > > > > > deletion
> > > > > > > > > > record.
> > > > > > > > > >
> > > > > > > > > > I added a section on "topic identifiers" describing this.
> > > > > > > > > >
> > > > > > > > > > > 210. Thanks for the explanation. Sounds good to me.
> > > > > > > > > > >
> > > > > > > > > > > 211. I still don't see an example when ShouldFence is
> > set to
> > > > > > true in
> > > > > > > > > > > BrokerHeartbeatReques. Could we add one?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > It is sent to true when the broker is first starting up and
> > > > > > doesn't yet
> > > > > > > > > > want to be unfenced. I added a longer explanation of this
> > in
> > > > the
> > > > > > > > "Broker
> > > > > > > > > > Leases" section.
> > > > > > > > > >
> > > > > > > > > > > 213. The KIP now allows replicas to be assigned on
> > brokers
> > > > that
> > > > > > are
> > > > > > > > > > fenced,
> > > > > > > > > > > which is an improvement. How do we permanently remove a
> > > > broker
> > > > > > (e.g.
> > > > > > > > > > > cluster shrinking) to prevent it from being used for
> > future
> > > > > > replica
> > > > > > > > > > > assignments?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > This is a fair point. I will create a kafka-cluster.sh
> > script
> > > > > > which
> > > > > > > > can
> > > > > > > > > > do this, plus a DecommissionBrokerRequest.
> > > > > > > > > >
> > > > > > > > > > As a bonus the kafka-cluster.sh script can help people
> > find the
> > > > > > > > cluster ID
> > > > > > > > > > of brokers, something that people have wanted a tool for
> > in the
> > > > > > past.
> > > > > > > > > >
> > > > > > > > > > > 214. "Currently, when a node is down, all of its ZK
> > > > registration
> > > > > > > > > > > information is gone. But we need this information in
> > order
> > > > to
> > > > > > > > > > understand
> > > > > > > > > > > things like whether the replicas of a particular
> > partition
> > > > are
> > > > > > > > > > > well-balanced across racks." This is not quite true.
> > > > Currently,
> > > > > > even
> > > > > > > > when
> > > > > > > > > > > ZK registration is gone, the existing replica assignment
> > is
> > > > still
> > > > > > > > > > available
> > > > > > > > > > > in the metadata response.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I agree that the existing replica assignment is still
> > > > available.
> > > > > > But
> > > > > > > > that
> > > > > > > > > > just tells you partition X is on nodes A, B, and C. If you
> > > > don't
> > > > > > have
> > > > > > > > the
> > > > > > > > > > ZK registration for one or more of A, B, or C then you
> > don't
> > > > know
> > > > > > > > whether
> > > > > > > > > > we are following the policy of "two replicas on one rack,
> > one
> > > > > > replica
> > > > > > > > on
> > > > > > > > > > another." Or any other more complex rack placement policy
> > > > that you
> > > > > > > > might
> > > > > > > > > > have.
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Dec 16, 2020 at 8:48 AM Colin McCabe <
> > > > cmccabe@apache.org
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the reply. A few more follow up comments.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 210. initial.broker.registration.timeout.ms: The
> > default
> > > > > > value
> > > > > > > > is
> > > > > > > > > > 90sec,
> > > > > > > > > > > > > which seems long. If a broker fails the registration
> > > > because
> > > > > > of
> > > > > > > > > > incorrect
> > > > > > > > > > > > > configs, we want to fail faster. In comparison, the
> > > > defaults
> > > > > > for
> > > > > > > > > > > > > zookeeper.connection.timeout.ms is 18 secs.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > >
> > > > > > > > > > > > I agree that the initial connection timeout here is
> > longer
> > > > than
> > > > > > > > what we
> > > > > > > > > > > > had with ZK. The main reason I selected a slightly
> > longer
> > > > > > timeout
> > > > > > > > > > here is
> > > > > > > > > > > > to handle the case where the controllers and the
> > brokers
> > > > are
> > > > > > > > > > co-located.
> > > > > > > > > > > > For example, if you have a 3 node cluster and all three
> > > > nodes
> > > > > > are
> > > > > > > > > > > > controllers+brokers, when you first bring up the
> > cluster,
> > > > we
> > > > > > will
> > > > > > > > have
> > > > > > > > > > to
> > > > > > > > > > > > stand up the controller quorum and then handle broker
> > > > > > > > registrations.
> > > > > > > > > > > > Although we believe Raft will start up relatively
> > quickly,
> > > > it's
> > > > > > > > good to
> > > > > > > > > > > > leave some extra margin here.
> > > > > > > > > > > >
> > > > > > > > > > > > I don't think there's a big disadvantage to having a
> > > > slightly
> > > > > > > > longer
> > > > > > > > > > > > timeout here. After all, starting up the brokers with
> > no
> > > > > > > > controllers
> > > > > > > > > > is an
> > > > > > > > > > > > admin mistake, which we don't expect to see very often.
> > > > Maybe
> > > > > > > > let's
> > > > > > > > > > set it
> > > > > > > > > > > > to 60 seconds for now and maybe see if we can tweak it
> > in
> > > > the
> > > > > > > > future if
> > > > > > > > > > > > that turns out to be too long or short?
> > > > > > > > > > > >
> > > > > > > > > > > > > 211. "Once they are all moved, the controller
> > responds
> > > > to the
> > > > > > > > > > heartbeat
> > > > > > > > > > > > > with a nextState of SHUTDOWN." It seems that
> > nextState
> > > > is no
> > > > > > > > longer
> > > > > > > > > > > > > relevant. Also, could you add a bit more explanation
> > on
> > > > when
> > > > > > > > > > ShouldFence
> > > > > > > > > > > > is
> > > > > > > > > > > > > set to true in BrokerHeartbeatRequest?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Good catch. I removed the obsolete section referring
> > to
> > > > > > nextState
> > > > > > > > and
> > > > > > > > > > > > added a reference to the new boolean. I also added
> > some
> > > > more
> > > > > > > > > > information
> > > > > > > > > > > > about ShouldFence and about the rationale for
> > separating
> > > > > > fencing
> > > > > > > > from
> > > > > > > > > > > > registration.
> > > > > > > > > > > >
> > > > > > > > > > > > > 212. Related to the above, what does the broker do if
> > > > > > IsFenced is
> > > > > > > > > > true in
> > > > > > > > > > > > > the BrokerHeartbeatResponse? Will the broker do the
> > same
> > > > > > thing on
> > > > > > > > > > > > receiving
> > > > > > > > > > > > > a FenceBrokerRecord from the metadata log?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > The broker only checks this boolean during the startup
> > > > process.
> > > > > > > > After
> > > > > > > > > > the
> > > > > > > > > > > > startup process is finished, it ignores this boolean.
> > > > > > > > > > > >
> > > > > > > > > > > > The broker uses fence / unfence records in the metadata
> > > > log to
> > > > > > > > > > determine
> > > > > > > > > > > > which brokers should appear in its MetadataResponse.
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > > > Jun
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <
> > > > > > cmccabe@apache.org
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The KIP says that "brokers which are fenced will
> > not
> > > > > > appear
> > > > > > > > in
> > > > > > > > > > > > > > > MetadataResponses. So clients that have
> > up-to-date
> > > > > > metadata
> > > > > > > > will
> > > > > > > > > > > > not try
> > > > > > > > > > > > > > > to contact fenced brokers.", which is fine, but
> > it
> > > > > > doesn't
> > > > > > > > seem
> > > > > > > > > > to
> > > > > > > > > > > > > > > elaborate on what happens for clients which try
> > to
> > > > > > contact a
> > > > > > > > > > broker
> > > > > > > > > > > > > > (using
> > > > > > > > > > > > > > > stale metadata, for example) and find it in the
> > > > > > > > > > registered-but-fenced
> > > > > > > > > > > > > > > state.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have dropped the broker-side fencing from this
> > > > > > proposal. So
> > > > > > > > now
> > > > > > > > > > the
> > > > > > > > > > > > > > fencing is basically the same as today's fencing:
> > it's
> > > > > > > > > > controller-side
> > > > > > > > > > > > > > only. That means that clients using stale
> > metadata can
> > > > > > contact
> > > > > > > > > > fenced
> > > > > > > > > > > > > > brokers and communicate with them. The only case
> > > > where the
> > > > > > > > broker
> > > > > > > > > > > > will be
> > > > > > > > > > > > > > inaccessible is when it hasn't finished starting
> > yet
> > > > > > (i.e., has
> > > > > > > > > > not yet
> > > > > > > > > > > > > > attained RUNNING state.)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Just like today, we expect the safeguards built
> > into
> > > > the
> > > > > > > > > > replication
> > > > > > > > > > > > > > protocol to prevent the worst corner cases that
> > could
> > > > > > result
> > > > > > > > from
> > > > > > > > > > > > this. I
> > > > > > > > > > > > > > do think we will probably take up this issue later
> > and
> > > > > > find a
> > > > > > > > > > better
> > > > > > > > > > > > > > solution for client-side metadata, but this KIP is
> > big
> > > > > > enough
> > > > > > > > > > as-is.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You said previously that the broker will respond
> > > > with a
> > > > > > > > retriable
> > > > > > > > > > > > > > > error, which is again fine, but you didn't
> > answer my
> > > > > > question
> > > > > > > > > > about
> > > > > > > > > > > > which
> > > > > > > > > > > > > > > error code you've chosen for this. I realise that
> > > > this
> > > > > > > > doesn't
> > > > > > > > > > really
> > > > > > > > > > > > > > > affect the correctness of the behaviour, but I'm
> > not
> > > > > > aware
> > > > > > > > of any
> > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > error code which is a good fit. So it would be
> > good
> > > > to
> > > > > > > > understand
> > > > > > > > > > > > about
> > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > you're making this backward compatible for
> > clients.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Many thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <
> > > > > > > > cmccabe@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for the reply. Just a couple of more
> > > > comments
> > > > > > > > below.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 210. Since we are deprecating
> > > > > > > > > > zookeeper.connection.timeout.ms,
> > > > > > > > > > > > > > should we
> > > > > > > > > > > > > > > > > add a new config to bound the time for a
> > broker
> > > > to
> > > > > > > > connect
> > > > > > > > > > to the
> > > > > > > > > > > > > > > > > controller during starting up?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Good idea. I added
> > > > > > initial.broker.registration.timeout.ms
> > > > > > > > for
> > > > > > > > > > > > this.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 211. BrokerHeartbeat no longer has the state
> > > > field
> > > > > > in the
> > > > > > > > > > > > > > > > request/response.
> > > > > > > > > > > > > > > > > However, (a) the controller shutdown section
> > > > still
> > > > > > has
> > > > > > > > "In
> > > > > > > > > > its
> > > > > > > > > > > > > > periodic
> > > > > > > > > > > > > > > > > heartbeats, the broker asks the controller
> > if it
> > > > can
> > > > > > > > > > transition
> > > > > > > > > > > > into
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > SHUTDOWN state. This motivates the
> > controller to
> > > > > > move
> > > > > > > > all
> > > > > > > > > > of the
> > > > > > > > > > > > > > leaders
> > > > > > > > > > > > > > > > > off of that broker. Once they are all
> > moved, the
> > > > > > > > controller
> > > > > > > > > > > > > > responds to
> > > > > > > > > > > > > > > > > the heartbeat with a nextState of SHUTDOWN.";
> > > > (2) the
> > > > > > > > > > > > description of
> > > > > > > > > > > > > > > > > BrokerHeartbeat still references
> > currentState and
> > > > > > > > > > targetState.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks. I've made these sections clearer and
> > > > removed
> > > > > > the
> > > > > > > > > > obsolete
> > > > > > > > > > > > > > > > references to sending states.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe
> > <
> > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao
> > wrote:
> > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for the update. A few more follow
> > up
> > > > > > comments.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks again for the review.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 100. FailedReplicaRecord: Since this is
> > > > reported
> > > > > > by
> > > > > > > > each
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > independently, perhaps we could use a
> > more
> > > > > > concise
> > > > > > > > > > > > representation
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > a top level broker field, an array of
> > topics,
> > > > > > which
> > > > > > > > has
> > > > > > > > > > an
> > > > > > > > > > > > array
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > partitions.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The issue is that there is a size limit on
> > the
> > > > each
> > > > > > > > record.
> > > > > > > > > > > > > > Putting
> > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > of the partitions of a log directory into a
> > > > single
> > > > > > > > record
> > > > > > > > > > would
> > > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > > break that in many cases. Still, we can
> > > > optimize a
> > > > > > > > bit by
> > > > > > > > > > > > having
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > array
> > > > > > > > > > > > > > > > > > of partition IDs, since nearly all the
> > time, we
> > > > > > have
> > > > > > > > more
> > > > > > > > > > than
> > > > > > > > > > > > one
> > > > > > > > > > > > > > > > from the
> > > > > > > > > > > > > > > > > > same topic.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 200. Sounds good. If we remove the
> > > > broker-side
> > > > > > > > fencing
> > > > > > > > > > > > logic, do
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > plan
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > still keep FENCED in broker state? Do we
> > > > plan to
> > > > > > > > expose
> > > > > > > > > > the
> > > > > > > > > > > > new
> > > > > > > > > > > > > > > > states
> > > > > > > > > > > > > > > > > > > through the existing BrokerState metric
> > and
> > > > if
> > > > > > so,
> > > > > > > > what
> > > > > > > > > > are
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > values
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > the new states?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > No, we don't need FENCED any more. I have
> > > > removed
> > > > > > it
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The new states are very similar to the
> > current
> > > > > > ones,
> > > > > > > > > > actually.
> > > > > > > > > > > > > > There
> > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > no new states or removed ones. The main
> > > > change in
> > > > > > the
> > > > > > > > > > broker
> > > > > > > > > > > > state
> > > > > > > > > > > > > > > > machine
> > > > > > > > > > > > > > > > > > is that the
> > RECOVERING_FROM_UNCLEAN_SHUTDOWN
> > > > state
> > > > > > has
> > > > > > > > been
> > > > > > > > > > > > > > renamed to
> > > > > > > > > > > > > > > > > > RECOVERY. Also, unlike previously, the
> > broker
> > > > will
> > > > > > > > always
> > > > > > > > > > pass
> > > > > > > > > > > > > > through
> > > > > > > > > > > > > > > > > > RECOVERY (although it may only stay in this
> > > > state
> > > > > > for a
> > > > > > > > > > very
> > > > > > > > > > > > short
> > > > > > > > > > > > > > > > amount
> > > > > > > > > > > > > > > > > > of time).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 201. This may be fine too. Could we
> > document
> > > > what
> > > > > > > > happens
> > > > > > > > > > > > when
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > broker.id/controller.id in
> > > > metadata.properties
> > > > > > don't
> > > > > > > > > > match
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > config when the broker starts up?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I added some documentation about this.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 204. There is still "The highest metadata
> > > > offset
> > > > > > > > which
> > > > > > > > > > the
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > has not
> > > > > > > > > > > > > > > > > > > reached" referenced under
> > BrokerRegistration.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 206. Is that separate step needed given
> > > > KIP-516?
> > > > > > With
> > > > > > > > > > > > KIP-516 (
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > > > > > > > > > > > > > ),
> > > > > > > > > > > > > > > > > > > we don't need to wait for the topic data
> > to
> > > > be
> > > > > > > > removed
> > > > > > > > > > from
> > > > > > > > > > > > all
> > > > > > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > > > > before removing the topic metadata. The
> > > > > > combination
> > > > > > > > of
> > > > > > > > > > > > unmatching
> > > > > > > > > > > > > > > > > > > topicId
> > > > > > > > > > > > > > > > > > > or the missing topicId from the metadata
> > is
> > > > > > enough
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > clean
> > > > > > > > > > > > > > > > > > > up deleted topics asynchronously.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It won't be needed once KIP-516 is
> > adopted, but
> > > > > > this
> > > > > > > > hasn't
> > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > implemented yet.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin
> > McCabe <
> > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao
> > > > wrote:
> > > > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for the updated KIP. A few
> > more
> > > > > > comments
> > > > > > > > > > below.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks again for the reviews.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 80.2 For deprecated configs, we need
> > to
> > > > > > include
> > > > > > > > > > > > zookeeper.*
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > broker.id.generation.enable.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Added.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 83.1 If a broker is down, does the
> > > > controller
> > > > > > > > keep
> > > > > > > > > > the
> > > > > > > > > > > > > > previously
> > > > > > > > > > > > > > > > > > > > > registered broker epoch forever? If
> > not,
> > > > how
> > > > > > long
> > > > > > > > > > does
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > > > > > > > it? What does the controller do when
> > > > > > receiving a
> > > > > > > > > > broker
> > > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > > > > with an unfound broker epoch?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Yes, the controller keeps the previous
> > > > > > registration
> > > > > > > > > > > > forever.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Broker heartbeat requests with an
> > incorrect
> > > > > > broker
> > > > > > > > > > epoch
> > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > rejected
> > > > > > > > > > > > > > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 100. Have you figured out if we need
> > to
> > > > add
> > > > > > a new
> > > > > > > > > > record
> > > > > > > > > > > > > > type for
> > > > > > > > > > > > > > > > > > > > reporting
> > > > > > > > > > > > > > > > > > > > > partitions on failed disks?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I added FailedReplicaRecord to reflect
> > the
> > > > case
> > > > > > > > where a
> > > > > > > > > > > > JBOD
> > > > > > > > > > > > > > > > directory
> > > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > failed, leading to failed replicas.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 102. For debugging purposes,
> > sometimes
> > > > it's
> > > > > > > > useful to
> > > > > > > > > > > > read
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > > > > topic using tools like
> > console-consumer.
> > > > > > Should
> > > > > > > > we
> > > > > > > > > > > > support
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > so,
> > > > > > > > > > > > > > > > > > > > > how?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > For now, we have the ability to read
> > the
> > > > > > metadata
> > > > > > > > logs
> > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > dump-logs
> > > > > > > > > > > > > > > > > > > > tool. I think we will come up with
> > some
> > > > other
> > > > > > > > tools
> > > > > > > > > > in the
> > > > > > > > > > > > > > future
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > get experience.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 200. "brokers which are fenced will
> > not
> > > > > > appear in
> > > > > > > > > > > > > > > > MetadataResponses.
> > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > broker will not respond to these
> > > > requests--
> > > > > > > > instead,
> > > > > > > > > > it
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > > > > disconnect." If the controller is
> > > > > > partitioned off
> > > > > > > > > > from
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > brokers,
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > design will cause every broker to
> > stop
> > > > > > accepting
> > > > > > > > new
> > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > requests. In
> > > > > > > > > > > > > > > > > > > > > contrast, if ZK is partitioned off,
> > the
> > > > > > existing
> > > > > > > > > > > > behavior is
> > > > > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > > > > brokers can continue to work based
> > on the
> > > > > > last
> > > > > > > > known
> > > > > > > > > > > > > > metadata.
> > > > > > > > > > > > > > > > So, I
> > > > > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > sure if we should change the existing
> > > > > > behavior
> > > > > > > > > > because
> > > > > > > > > > > > of the
> > > > > > > > > > > > > > > > bigger
> > > > > > > > > > > > > > > > > > > > impact
> > > > > > > > > > > > > > > > > > > > > in the new one. Another option is to
> > > > keep the
> > > > > > > > > > existing
> > > > > > > > > > > > > > behavior
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > expose
> > > > > > > > > > > > > > > > > > > > > a metric for fenced brokers so that
> > the
> > > > > > operator
> > > > > > > > > > could be
> > > > > > > > > > > > > > > > alerted.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I'm skeptical about how well running
> > > > without ZK
> > > > > > > > > > currently
> > > > > > > > > > > > > > works.
> > > > > > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > > > > > I will move the broker-side fencing
> > into a
> > > > > > > > follow-up
> > > > > > > > > > KIP.
> > > > > > > > > > > > This
> > > > > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > > > > > already pretty large and there is no
> > hard
> > > > > > > > dependency on
> > > > > > > > > > > > this.
> > > > > > > > > > > > > > > > There
> > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > also be other ways of accomplishing the
> > > > > > positive
> > > > > > > > > > effects of
> > > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > > > > broker-side fencing, so more
> > discussion is
> > > > > > needed.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 201. I read Ron's comment, but I am
> > > > still not
> > > > > > > > sure
> > > > > > > > > > the
> > > > > > > > > > > > > > benefit of
> > > > > > > > > > > > > > > > > > keeping
> > > > > > > > > > > > > > > > > > > > > broker.id and controller.id in
> > > > > > meta.properties.
> > > > > > > > It
> > > > > > > > > > seems
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > we are
> > > > > > > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > > > duplicating the same info in two
> > places
> > > > and
> > > > > > have
> > > > > > > > the
> > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > burden of
> > > > > > > > > > > > > > > > > > > > > making sure the values in the two
> > places
> > > > are
> > > > > > > > > > consistent.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think the reasoning is that having
> > > > broker.id
> > > > > > > > > > protects us
> > > > > > > > > > > > > > against
> > > > > > > > > > > > > > > > > > > > accidentally bringing up a broker with
> > a
> > > > disk
> > > > > > from
> > > > > > > > a
> > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > broker. I
> > > > > > > > > > > > > > > > > > > > don't feel strongly about this but it
> > > > seemed
> > > > > > > > simpler to
> > > > > > > > > > > > keep
> > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 202.
> > > > controller.connect.security.protocol: Is
> > > > > > > > this
> > > > > > > > > > needed
> > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > > > > controller.listener.names and
> > > > > > > > > > > > listener.security.protocol.map
> > > > > > > > > > > > > > > > imply
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > security protocol already?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > You're right, this isn't needed. I'll
> > > > remove
> > > > > > it.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 203.
> > registration.heartbeat.interval.ms:
> > > > It
> > > > > > > > > > defaults to
> > > > > > > > > > > > 2k.
> > > > > > > > > > > > > > ZK
> > > > > > > > > > > > > > > > uses
> > > > > > > > > > > > > > > > > > 1/3
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > the session timeout for heartbeat.
> > So,
> > > > given
> > > > > > the
> > > > > > > > > > default
> > > > > > > > > > > > 18k
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > registration.lease.timeout.ms,
> > should we
> > > > > > default
> > > > > > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms
> > to
> > > > 6k?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 6 seconds seems like a pretty long time
> > > > between
> > > > > > > > > > > > heartbeats. It
> > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > useful to know when a broker is missing
> > > > > > heartbeats,
> > > > > > > > > > with
> > > > > > > > > > > > less
> > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > that. I provisionally set it to 3
> > seconds
> > > > (we
> > > > > > can
> > > > > > > > > > always
> > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > > later...)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I also changed the name of these
> > > > > > configurations to
> > > > > > > > "
> > > > > > > > > > > > > > > > > > > > broker.heartbeat.interval.ms" and "
> > > > > > > > > > > > > > broker.registration.timeout.ms"
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > > > > > > to clarify them a bit.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 204. "The highest metadata offset
> > which
> > > > the
> > > > > > > > broker
> > > > > > > > > > has
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > > reached."
> > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > seems this should be "has reached".
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I changed this to "one more than the
> > > > highest
> > > > > > > > metadata
> > > > > > > > > > > > offset
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > broker has reached."
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 205. UnfenceBrokerRecord and
> > > > > > > > UnregisterBrokerRecord:
> > > > > > > > > > To
> > > > > > > > > > > > me,
> > > > > > > > > > > > > > they
> > > > > > > > > > > > > > > > > > seem to
> > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > the same. Do we need both?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Unregistration means that the broker
> > has
> > > > been
> > > > > > > > removed
> > > > > > > > > > from
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > cluster.
> > > > > > > > > > > > > > > > > > > > That is different than unfencing, which
> > > > marks
> > > > > > the
> > > > > > > > > > broker as
> > > > > > > > > > > > > > active.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 206. TopicRecord: The Deleting field
> > is
> > > > used
> > > > > > to
> > > > > > > > > > indicate
> > > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > being deleted. I am wondering if
> > this is
> > > > > > really
> > > > > > > > > > needed
> > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > RemoveTopic
> > > > > > > > > > > > > > > > > > > > > already indicates the same thing.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > RemoveTopic is the last step, that
> > scrubs
> > > > all
> > > > > > > > metadata
> > > > > > > > > > > > about
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > topic.
> > > > > > > > > > > > > > > > > > > > In order to get to that last step, the
> > > > topic
> > > > > > data
> > > > > > > > > > needs to
> > > > > > > > > > > > > > removed
> > > > > > > > > > > > > > > > > > from all
> > > > > > > > > > > > > > > > > > > > brokers (after each broker notices
> > that the
> > > > > > topic
> > > > > > > > is
> > > > > > > > > > being
> > > > > > > > > > > > > > > > deleted).
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin
> > > > McCabe <
> > > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron
> > > > > > Dagostino
> > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the
> > updates.
> > > > It's
> > > > > > now
> > > > > > > > > > clear
> > > > > > > > > > > > to me
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > > > > > > > > keep their broker epoch for the
> > life
> > > > of
> > > > > > their
> > > > > > > > > > JVM --
> > > > > > > > > > > > they
> > > > > > > > > > > > > > > > > > register
> > > > > > > > > > > > > > > > > > > > > > > once, get their broker epoch in
> > the
> > > > > > > > response, and
> > > > > > > > > > > > then
> > > > > > > > > > > > > > never
> > > > > > > > > > > > > > > > > > > > > > > re-register again. Brokers may
> > get
> > > > > > fenced,
> > > > > > > > but
> > > > > > > > > > they
> > > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > > > > > broker epoch for the life of
> > their
> > > > JVM.
> > > > > > The
> > > > > > > > > > > > incarnation
> > > > > > > > > > > > > > ID
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > kept for the life of the JVM but
> > is
> > > > > > > > generated by
> > > > > > > > > > the
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > > > > > upon startup, and the
> > combination of
> > > > the
> > > > > > two
> > > > > > > > > > allows
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Controller to
> > > > > > > > > > > > > > > > > > > > > > > act idempotently if any
> > > > previously-sent
> > > > > > > > > > registration
> > > > > > > > > > > > > > response
> > > > > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks, Ron. That's a good
> > summary.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > One thing I wonder about is if it
> > > > might
> > > > > > be
> > > > > > > > > > helpful
> > > > > > > > > > > > for
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > > > > > send the Cluster ID as determined
> > > > from
> > > > > > its
> > > > > > > > > > > > > > meta.properties
> > > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > > > > > > registration request. Does it
> > even
> > > > make
> > > > > > > > sense
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > > > > > successfully register and enter
> > the
> > > > > > Fenced
> > > > > > > > state
> > > > > > > > > > if
> > > > > > > > > > > > it
> > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > wrong
> > > > > > > > > > > > > > > > > > > > > > > Cluster ID?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Yeah, that's a good idea. Let's
> > have
> > > > the
> > > > > > > > broker
> > > > > > > > > > pass
> > > > > > > > > > > > its
> > > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > > ID in
> > > > > > > > > > > > > > > > > > > > > > the registration RPC, and then
> > > > > > registration can
> > > > > > > > > > fail
> > > > > > > > > > > > if the
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The nextMetadatOffset value
> > that the
> > > > > > broker
> > > > > > > > > > > > communicates
> > > > > > > > > > > > > > > > > > > > > > > in its registration request only
> > has
> > > > > > meaning
> > > > > > > > > > within
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > > > > cluster, so it feels to me that
> > the
> > > > > > > > Controller
> > > > > > > > > > should
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > perform this sanity check.
> > There is
> > > > > > > > currently
> > > > > > > > > > > > (pre-KIP
> > > > > > > > > > > > > > 500)
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > check
> > > > > > > > > > > > > > > > > > > > > > > in the broker to make sure its
> > > > configured
> > > > > > > > > > cluster ID
> > > > > > > > > > > > > > matches
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > > > > > > stored in ZooKeeper, and we will
> > > > have to
> > > > > > > > perform
> > > > > > > > > > this
> > > > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > > > > somewhere in the KIP-500 world.
> > If
> > > > the
> > > > > > > > > > Controller
> > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > do it
> > > > > > > > > > > > > > > > > > > > > > > within the registration request
> > then
> > > > the
> > > > > > > > broker
> > > > > > > > > > will
> > > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > > > > metadata request to the
> > Controller,
> > > > > > retrieve
> > > > > > > > the
> > > > > > > > > > > > Cluster
> > > > > > > > > > > > > > ID,
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > perform the check itself. It
> > feels
> > > > to me
> > > > > > > > that it
> > > > > > > > > > > > might
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > better for
> > > > > > > > > > > > > > > > > > > > > > > the Controller to just do it, and
> > > > then
> > > > > > the
> > > > > > > > broker
> > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > > > > > worry about it anymore once it
> > > > > > successfully
> > > > > > > > > > > > registers.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I also have a question about the
> > > > > > broker.id
> > > > > > > > > > value and
> > > > > > > > > > > > > > > > > > > > meta.properties.
> > > > > > > > > > > > > > > > > > > > > > > The KIP now says "In version 0 of
> > > > > > > > > > meta.properties,
> > > > > > > > > > > > there
> > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > > > > > broker.id field. Version 1
> > does not
> > > > > > have
> > > > > > > > this
> > > > > > > > > > > > field.
> > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > is no
> > > > > > > > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > > > > > > > > needed because we no longer
> > support
> > > > > > dynamic
> > > > > > > > > > broker id
> > > > > > > > > > > > > > > > > > assignment."
> > > > > > > > > > > > > > > > > > > > > > > But then there is an example
> > version
> > > > 1
> > > > > > > > > > > > meta.properties
> > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > shows
> > > > > > > > > > > > > > > > > > > > > > > the broker.id value. I actually
> > > > wonder
> > > > > > if
> > > > > > > > > > maybe the
> > > > > > > > > > > > > > > > broker.id
> > > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > > > would be good to keep in the
> > version
> > > > 1
> > > > > > > > > > > > meta.properties
> > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > currently (pre-KIP 500, version
> > 0)
> > > > acts
> > > > > > as a
> > > > > > > > > > sanity
> > > > > > > > > > > > > > check to
> > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > > sure
> > > > > > > > > > > > > > > > > > > > > > > the broker is using the correct
> > log
> > > > > > > > directory.
> > > > > > > > > > > > Similarly
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > controller.id value on
> > controllers
> > > > -- it
> > > > > > > > would
> > > > > > > > > > > > allow the
> > > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > sanity check for quorum
> > controllers.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > That's a good point. I will add
> > > > broker.id
> > > > > > > > back,
> > > > > > > > > > and
> > > > > > > > > > > > also
> > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > > > controller.id as a possibility.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM
> > Colin
> > > > > > McCabe
> > > > > > > > <
> > > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10,
> > > > Jun Rao
> > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the reply. A few
> > more
> > > > > > > > comments.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks again for the reply.
> > Sorry
> > > > for
> > > > > > the
> > > > > > > > long
> > > > > > > > > > > > > > hiatus. I
> > > > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > > > vacation for a while.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 55. There is still text that
> > > > favors
> > > > > > new
> > > > > > > > > > broker
> > > > > > > > > > > > > > > > registration.
> > > > > > > > > > > > > > > > > > > > "When a
> > > > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > > > first starts up, when it is
> > in
> > > > the
> > > > > > > > INITIAL
> > > > > > > > > > > > state, it
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > always
> > > > > > > > > > > > > > > > > > > > > > "win"
> > > > > > > > > > > > > > > > > > > > > > > > > broker ID conflicts.
> > However,
> > > > once
> > > > > > it is
> > > > > > > > > > > > granted a
> > > > > > > > > > > > > > > > lease, it
> > > > > > > > > > > > > > > > > > > > > > transitions
> > > > > > > > > > > > > > > > > > > > > > > > > out of the INITIAL state.
> > > > > > Thereafter,
> > > > > > > > it may
> > > > > > > > > > > > lose
> > > > > > > > > > > > > > > > subsequent
> > > > > > > > > > > > > > > > > > > > > > conflicts if
> > > > > > > > > > > > > > > > > > > > > > > > > its broker epoch is stale.
> > (See
> > > > > > KIP-380
> > > > > > > > for
> > > > > > > > > > some
> > > > > > > > > > > > > > > > background
> > > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > > > epoch.) The reason for
> > favoring
> > > > new
> > > > > > > > > > processes
> > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > accommodate
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > > > > > > > > > case where a process is
> > killed
> > > > with
> > > > > > kill
> > > > > > > > -9
> > > > > > > > > > and
> > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > > restarted.
> > > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > > want it
> > > > > > > > > > > > > > > > > > > > > > > > > to be able to reclaim its
> > old ID
> > > > > > quickly
> > > > > > > > in
> > > > > > > > > > this
> > > > > > > > > > > > > > case."
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for the reminder. I
> > have
> > > > > > clarified
> > > > > > > > the
> > > > > > > > > > > > language
> > > > > > > > > > > > > > > > here.
> > > > > > > > > > > > > > > > > > > > > > Hopefully now it is clear that we
> > don't
> > > > > > allow
> > > > > > > > quick
> > > > > > > > > > > > re-use
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 80.1 Sounds good. Could you
> > > > document
> > > > > > that
> > > > > > > > > > > > listeners
> > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > > > > > > now? It would also be useful
> > to
> > > > > > annotate
> > > > > > > > > > other
> > > > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > > configs.
> > > > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > > > > > > example, controller.connect
> > > > should be
> > > > > > > > > > required.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I added a note specifying that
> > > > these
> > > > > > are
> > > > > > > > > > required.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 80.2 Could you list all
> > > > deprecated
> > > > > > > > existing
> > > > > > > > > > > > configs?
> > > > > > > > > > > > > > > > Another
> > > > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > control.plane.listener.name
> > > > since
> > > > > > the
> > > > > > > > > > > > controller no
> > > > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > > > sends
> > > > > > > > > > > > > > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata
> > and
> > > > > > > > StopReplica
> > > > > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I added a section specifying
> > some
> > > > > > > > deprecated
> > > > > > > > > > > > configs.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 83.1 It seems that the
> > broker can
> > > > > > > > transition
> > > > > > > > > > from
> > > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > RUNNING
> > > > > > > > > > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > > > > > > > > > registering for a new broker
> > > > epoch.
> > > > > > I am
> > > > > > > > not
> > > > > > > > > > > > sure how
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > works.
> > > > > > > > > > > > > > > > > > > > > > Once the
> > > > > > > > > > > > > > > > > > > > > > > > > controller fences a broker,
> > > > there is
> > > > > > no
> > > > > > > > need
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > > > > > > > > > > boker epoch around. So, if
> > the
> > > > fenced
> > > > > > > > > > broker's
> > > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > > > > > > > existing broker epoch will be
> > > > > > rejected,
> > > > > > > > > > leading
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The broker epoch refers to the
> > > > broker
> > > > > > > > > > registration.
> > > > > > > > > > > > > > So we
> > > > > > > > > > > > > > > > DO
> > > > > > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > broker epoch around even while the
> > > > broker
> > > > > > is
> > > > > > > > > > fenced.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The broker epoch changes only
> > when
> > > > > > there
> > > > > > > > is a
> > > > > > > > > > new
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > registration. Fencing or
> > unfencing the
> > > > > > broker
> > > > > > > > > > doesn't
> > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 83.5 Good point on KIP-590.
> > Then
> > > > > > should
> > > > > > > > we
> > > > > > > > > > > > expose the
> > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > debugging purposes? If not,
> > we
> > > > should
> > > > > > > > > > deprecate
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > controllerID
> > > > > > > > > > > > > > > > > > > > > > field in
> > > > > > > > > > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think it's OK to expose it
> > for
> > > > now,
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > > proviso
> > > > > > > > > > > > > > > > that it
> > > > > > > > > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > > > > > > > > be reachable by clients.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 90. We rejected the shared ID
> > > > with
> > > > > > just
> > > > > > > > one
> > > > > > > > > > > > reason
> > > > > > > > > > > > > > "This
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > not a
> > > > > > > > > > > > > > > > > > > > > > good idea
> > > > > > > > > > > > > > > > > > > > > > > > > because NetworkClient
> > assumes a
> > > > > > single ID
> > > > > > > > > > > > space. So
> > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > there is
> > > > > > > > > > > > > > > > > > > > > > both a
> > > > > > > > > > > > > > > > > > > > > > > > > controller 1 and a broker 1,
> > we
> > > > don't
> > > > > > > > have a
> > > > > > > > > > way
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > picking
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > "right"
> > > > > > > > > > > > > > > > > > > > > > > > > one." This doesn't seem to
> > be a
> > > > > > strong
> > > > > > > > > > reason.
> > > > > > > > > > > > For
> > > > > > > > > > > > > > > > example,
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > > > > > > > > address the NetworkClient
> > issue
> > > > with
> > > > > > the
> > > > > > > > node
> > > > > > > > > > > > type
> > > > > > > > > > > > > > as you
> > > > > > > > > > > > > > > > > > pointed
> > > > > > > > > > > > > > > > > > > > > > out or
> > > > > > > > > > > > > > > > > > > > > > > > > using the negative value of a
> > > > broker
> > > > > > ID
> > > > > > > > as
> > > > > > > > > > the
> > > > > > > > > > > > > > > > controller ID.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > It would require a lot of code
> > > > changes
> > > > > > to
> > > > > > > > > > support
> > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > types of
> > > > > > > > > > > > > > > > > > > > > > node IDs. It's not clear to me
> > that
> > > > the
> > > > > > end
> > > > > > > > result
> > > > > > > > > > > > would
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > better --
> > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > tend to think it would be worse,
> > since
> > > > it
> > > > > > > > would be
> > > > > > > > > > more
> > > > > > > > > > > > > > > > complex.
> > > > > > > > > > > > > > > > > > In a
> > > > > > > > > > > > > > > > > > > > > > similar vein, using negative
> > numbers
> > > > seems
> > > > > > > > > > dangerous,
> > > > > > > > > > > > > > since we
> > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > > negatives or -1 as "special
> > values" in
> > > > many
> > > > > > > > places.
> > > > > > > > > > > > For
> > > > > > > > > > > > > > > > example,
> > > > > > > > > > > > > > > > > > -1
> > > > > > > > > > > > > > > > > > > > often
> > > > > > > > > > > > > > > > > > > > > > represents "no such node."
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > One important thing to keep in
> > > > mind is
> > > > > > > > that we
> > > > > > > > > > > > want to
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > transition from a broker and a
> > > > controller
> > > > > > being
> > > > > > > > > > > > co-located
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > them
> > > > > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > > > > > > longer being co-located. This is
> > much
> > > > > > easier
> > > > > > > > to do
> > > > > > > > > > > > when
> > > > > > > > > > > > > > they
> > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > > > > > > the broker reports all
> > offline
> > > > > > replicas
> > > > > > > > due
> > > > > > > > > > to a
> > > > > > > > > > > > disk
> > > > > > > > > > > > > > > > > > failure to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > controller. It seems this
> > > > information
> > > > > > > > needs
> > > > > > > > > > to be
> > > > > > > > > > > > > > > > persisted
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > > > > > > > > log. Do we have a
> > corresponding
> > > > > > record
> > > > > > > > for
> > > > > > > > > > that?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hmm, I have to look into this a
> > > > little
> > > > > > bit
> > > > > > > > > > more.
> > > > > > > > > > > > We
> > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > > > > > > > record type.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 101. Currently, StopReplica
> > > > request
> > > > > > has 2
> > > > > > > > > > modes,
> > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > > > > > and with
> > > > > > > > > > > > > > > > > > > > > > > > > deletion. The former is used
> > for
> > > > > > > > controlled
> > > > > > > > > > > > shutdown
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > handling
> > > > > > > > > > > > > > > > > > > > > > disk
> > > > > > > > > > > > > > > > > > > > > > > > > failure, and causes the
> > follower
> > > > to
> > > > > > > > stop. The
> > > > > > > > > > > > latter
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > > > > > > > > and partition reassignment,
> > and
> > > > > > causes
> > > > > > > > the
> > > > > > > > > > > > replica
> > > > > > > > > > > > > > to be
> > > > > > > > > > > > > > > > > > deleted.
> > > > > > > > > > > > > > > > > > > > > > Since we
> > > > > > > > > > > > > > > > > > > > > > > > > are deprecating StopReplica,
> > > > could we
> > > > > > > > > > document
> > > > > > > > > > > > what
> > > > > > > > > > > > > > > > triggers
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > stopping
> > > > > > > > > > > > > > > > > > > > > > > > > of a follower and the
> > deleting
> > > > of a
> > > > > > > > replica
> > > > > > > > > > now?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > RemoveTopic triggers
> > deletion. In
> > > > > > general
> > > > > > > > the
> > > > > > > > > > > > > > > > functionality of
> > > > > > > > > > > > > > > > > > > > > > StopReplica is subsumed by the
> > metadata
> > > > > > > > records.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 102. Should we include the
> > > > metadata
> > > > > > > > topic in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > > > > > If so,
> > > > > > > > > > > > > > > > > > > > > > > > > when it will be included and
> > what
> > > > > > will
> > > > > > > > the
> > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > response
> > > > > > > > > > > > > > > > > > look
> > > > > > > > > > > > > > > > > > > > > > like?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > No, it won't be included in the
> > > > > > metadata
> > > > > > > > > > response
> > > > > > > > > > > > sent
> > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > brokers.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 103. "The active controller
> > > > assigns
> > > > > > the
> > > > > > > > > > broker a
> > > > > > > > > > > > new
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > epoch,
> > > > > > > > > > > > > > > > > > > > > > based on
> > > > > > > > > > > > > > > > > > > > > > > > > the latest committed offset
> > in
> > > > the
> > > > > > log."
> > > > > > > > This
> > > > > > > > > > > > seems
> > > > > > > > > > > > > > > > > > inaccurate
> > > > > > > > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > latest committed offset
> > doesn't
> > > > > > always
> > > > > > > > > > advance on
> > > > > > > > > > > > > > every
> > > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > > > append.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Given that the new broker epoch
> > > > won't
> > > > > > be
> > > > > > > > > > visible
> > > > > > > > > > > > until
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > commit
> > > > > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > > > happened, I have changed this to
> > "the
> > > > next
> > > > > > > > > > available
> > > > > > > > > > > > > > offset in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > log"
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 104. REGISTERING(1) : It says
> > > > > > > > "Otherwise, the
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > moves
> > > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > > > > > > > > > > > state.". It seems this
> > should be
> > > > > > RUNNING?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 105. RUNNING: Should we
> > require
> > > > the
> > > > > > > > broker to
> > > > > > > > > > > > catch
> > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > to the
> > > > > > > > > > > > > > > > > > > > > > metadata log
> > > > > > > > > > > > > > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > For 104 and 105, these sections
> > > > have
> > > > > > been
> > > > > > > > > > reworked.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20
> > PM
> > > > Colin
> > > > > > > > McCabe
> > > > > > > > > > <
> > > > > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020, at
> > > > 05:51, Tom
> > > > > > > > Bentley
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at
> > > > 08:59,
> > > > > > Ron
> > > > > > > > > > Dagostino
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks
> > for the
> > > > > > hard
> > > > > > > > work
> > > > > > > > > > on
> > > > > > > > > > > > this
> > > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have some questions
> > > > about
> > > > > > what
> > > > > > > > > > happens
> > > > > > > > > > > > to a
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > when it
> > > > > > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > fenced (e.g. because
> > it
> > > > can't
> > > > > > > > send a
> > > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > > request to
> > > > > > > > > > > > > > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > lease). The KIP says
> > > > "When a
> > > > > > > > broker
> > > > > > > > > > is
> > > > > > > > > > > > > > fenced,
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > cannot
> > > > > > > > > > > > > > > > > > > > > > process any
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > client requests.
> > This
> > > > > > prevents
> > > > > > > > > > brokers
> > > > > > > > > > > > which
> > > > > > > > > > > > > > > > are not
> > > > > > > > > > > > > > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > metadata updates or
> > that
> > > > are
> > > > > > not
> > > > > > > > > > > > receiving
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > processing
> > > > > > > > > > > > > > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > enough from causing
> > > > issues to
> > > > > > > > > > clients."
> > > > > > > > > > > > And
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > FENCED(4) state it
> > > > likewise
> > > > > > says
> > > > > > > > > > "While
> > > > > > > > > > > > in
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > not respond to client
> > > > > > > > requests." It
> > > > > > > > > > > > makes
> > > > > > > > > > > > > > sense
> > > > > > > > > > > > > > > > > > that a
> > > > > > > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > should not accept
> > > > producer
> > > > > > > > requests
> > > > > > > > > > -- I
> > > > > > > > > > > > > > assume
> > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > would result in
> > > > > > > > > > > > NotLeaderOrFollowerException.
> > > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > (fetch from follower)
> > > > > > consumer
> > > > > > > > > > > > requests? It
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > conceivable
> > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > could continue.
> > Related
> > > > to
> > > > > > that,
> > > > > > > > > > would a
> > > > > > > > > > > > > > fenced
> > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > fetch data for
> > partitions
> > > > > > where
> > > > > > > > it
> > > > > > > > > > > > thinks it
> > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > > follower?
> > > > > > > > > > > > > > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > it rejects consumer
> > > > requests
> > > > > > it
> > > > > > > > might
> > > > > > > > > > > > still
> > > > > > > > > > > > > > > > continue
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > follower. Might it
> > be
> > > > > > helpful to
> > > > > > > > > > clarify
> > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > decisions
> > > > > > > > > > > > > > > > > > > > > > here?
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Good question. I
> > think a
> > > > > > fenced
> > > > > > > > broker
> > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > > > > > > > > > > > > > partitions it was
> > already
> > > > > > fetching
> > > > > > > > > > before
> > > > > > > > > > > > it
> > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > > fenced,
> > > > > > > > > > > > > > > > > > > > > > unless it
> > > > > > > > > > > > > > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > problem. At that
> > point it
> > > > > > won't be
> > > > > > > > > > able to
> > > > > > > > > > > > > > > > continue,
> > > > > > > > > > > > > > > > > > > > since it
> > > > > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > > > > > > the new metadata. For
> > > > > > example, it
> > > > > > > > > > won't
> > > > > > > > > > > > know
> > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > leadership
> > > > > > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > > the partitions it's
> > > > fetching.
> > > > > > The
> > > > > > > > > > > > rationale
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > continuing to
> > > > > > > > > > > > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > try to avoid
> > disruptions as
> > > > > > much as
> > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't think fenced
> > > > brokers
> > > > > > should
> > > > > > > > > > accept
> > > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > > > > > > > The issue
> > > > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > > > that the fenced broker
> > may
> > > > or
> > > > > > may
> > > > > > > > not
> > > > > > > > > > have
> > > > > > > > > > > > any
> > > > > > > > > > > > > > > > data it
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > have. It may or may
> > not
> > > > have
> > > > > > > > applied
> > > > > > > > > > any
> > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > > > > > it is supposed to have
> > > > applied.
> > > > > > > > So it
> > > > > > > > > > > > could
> > > > > > > > > > > > > > get
> > > > > > > > > > > > > > > > pretty
> > > > > > > > > > > > > > > > > > > > > > confusing, and
> > > > > > > > > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > > > > > potentially waste the
> > > > client's
> > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > When fenced, how would
> > the
> > > > broker
> > > > > > > > reply
> > > > > > > > > > to a
> > > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > which did
> > > > > > > > > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > The broker will respond
> > with a
> > > > > > > > retryable
> > > > > > > > > > error
> > > > > > > > > > > > in
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > case.
> > > > > > > > > > > > > > > > > > > > Once
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > client has re-fetched its
> > > > > > metadata, it
> > > > > > > > > > will no
> > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > > > > > as part of the cluster. I
> > > > added a
> > > > > > > > note to
> > > > > > > > > > the
> > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
--
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the reply. The KIP looks good to me now. Thanks for your
diligence.
Jun
On Thu, Dec 17, 2020 at 5:15 PM Colin McCabe <cm...@apache.org> wrote:
> On Thu, Dec 17, 2020, at 17:02, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the reply. Sounds good. One more comment.
> >
> > 231. Currently, we have sasl.mechanism.inter.broker.protocol for inter
> > broker communication. It seems that we need a similar config for
> specifying
> > the sasl mechanism for the communication between the broker and the
> > controller.
> >
>
> Hi Jun,
>
> Yeah... sounds like we could use a new configuration key for this. I
> added sasl.mechanism.controller.protocol for this.
>
> regards,
> Colin
>
> > Jun
> >
> > On Thu, Dec 17, 2020 at 2:29 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Thu, Dec 17, 2020, at 10:19, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > 211. Hmm, I still don't see a clear benefit of registering a broker
> > > before
> > > > recovery. It's possible for the recovery to take time. However,
> during
> > > the
> > > > recovery mode, it seems the broker will still be in the fenced mode
> and
> > > > won't be able to do work for the clients. So, registering and
> > > heartbeating
> > > > early seems to only add unnecessary overhead. For your point on topic
> > > > creation, I thought we now allow replicas to be created on
> > > > unregistered brokers.
> > > >
> > >
> > > Hi Jun,
> > >
> > > Thanks again for the reviews.
> > >
> > > Topics cannot be created on unregistered brokers. They can be created
> on
> > > registered but fenced brokers. So for that reason I think it makes
> sense
> > > to register as early as possible.
> > >
> > > > 230. Currently, we do have a ControllerId field in MetadataResponse.
> In
> > > the
> > > > early discussion, I thought that we want to expose the controller for
> > > > debugging purposes, but not used by the client library.
> > > >
> > >
> > > The current plan is that we will expose the controller node ID, but the
> > > controller will not be included in the list of nodes in the metadata
> > > response.
> > >
> > > It's not really possible to include the controller in that list of
> nodes
> > > because the controller may not share the same set of listeners as the
> > > broker. So, for example, maybe the controller endpoint is using a
> > > different type of security than the broker. So while we could pass
> back a
> > > hostname and port, the client would have no way to connect since it
> doesn't
> > > know what security settings to use.
> > >
> > > regards,
> > > Colin
> > >
> > > > Jun
> > > >
> > > > On Wed, Dec 16, 2020 at 9:13 PM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > On Wed, Dec 16, 2020, at 18:13, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the reply. Just a couple of more comments.
> > > > > >
> > > > > > 211. Currently, the broker only registers itself in ZK after log
> > > > > recovery.
> > > > > > Is there any benefit to change that? As you mentioned, the broker
> > > can't
> > > > > do
> > > > > > much before completing log recovery.
> > > > > >
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > Previously, it wasn't possible to register in ZK without
> immediately
> > > > > getting added to the MetadataResponse. So I think that's the main
> > > reason
> > > > > why registration was delayed until after log recovery. Since that
> > > > > constraint doesn't exist any more, there seems to be no reason to
> delay
> > > > > registration.
> > > > >
> > > > > I think delaying registration would have some major downsides. If
> log
> > > > > recovery takes a while, that's a longer window during which someone
> > > else
> > > > > could register a broker with the same ID. Having parts of the
> cluster
> > > > > missing for a while gives up some of the benefit of separating
> > > registration
> > > > > from fencing. For example, if a broker somehow gets unregistered
> and
> > > we
> > > > > want to re-register it, but we have to wait for a 10 minute log
> > > recovery to
> > > > > do that, that could be a window during which topics can't be
> created
> > > that
> > > > > need to be on that broker, and so forth.
> > > > >
> > > > > > 230. Regarding MetadataResponse, there is a slight awkwardness.
> We
> > > return
> > > > > > rack for each node. However, if that node is for the controller,
> the
> > > rack
> > > > > > field is not really relevant. Should we clean it up here or in
> > > another
> > > > > KIP
> > > > > > like KIP-700?
> > > > >
> > > > > Oh, controllers don't appear in the MetadataResponses returned to
> > > clients,
> > > > > since clients can't access them. I should have been more clear
> about
> > > that
> > > > > in the KIP-- I added a sentence to "Networking" describing this.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, Dec 16, 2020 at 4:23 PM Colin McCabe <cmccabe@apache.org
> >
> > > wrote:
> > > > > >
> > > > > > > On Wed, Dec 16, 2020, at 13:40, Jun Rao wrote:
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the reply. A few follow up comments.
> > > > > > > >
> > > > > > > > 211. When does the broker send the BrokerRegistration
> request to
> > > the
> > > > > > > > controller? Is it after the recovery phase? If so, at that
> > > point, the
> > > > > > > > broker has already caught up on the metadata (in order to
> clean
> > > up
> > > > > > > deleted
> > > > > > > > partitions). Then, it seems we don't need the ShouldFence
> field
> > > > > > > > in BrokerHeartbeatRequest?
> > > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > Thanks again for the reviews.
> > > > > > >
> > > > > > > The broker sends the registration request as soon as it starts
> > > up. It
> > > > > > > cannot wait until the recovery phase is over since sometimes
> log
> > > > > recovery
> > > > > > > can take quite a long time.
> > > > > > >
> > > > > > > >
> > > > > > > > 213. kafka-cluster.sh
> > > > > > > > 213.1 For the decommision example, should the command take a
> > > broker
> > > > > id?
> > > > > > >
> > > > > > > Yes, the example should have a broker id. Fixed.
> > > > > > >
> > > > > > > > 213.2 Existing tools use the "--" command line option (e.g.
> > > > > kafka-topics
> > > > > > > > --list --topic test). Should we follow the same convention
> > > > > > > > for kafka-cluster.sh (and kafka-storage.sh)?
> > > > > > >
> > > > > > > Hmm. I don't think argparse4j supports using double dashes to
> > > identify
> > > > > > > subcommands. I think it might be confusing as well, since the
> > > > > subcommand
> > > > > > > must come first, unlike a plain old argument which can be
> anywhere
> > > on
> > > > > the
> > > > > > > command line.
> > > > > > >
> > > > > > > > 213.3 Should we add an admin api for broker decommission so
> that
> > > > > this can
> > > > > > > > be done programmatically?
> > > > > > > >
> > > > > > >
> > > > > > > Yes, there is an admin client API for decommissioning as well.
> > > > > > >
> > > > > > > > 220. DecommissionBroker: "NOT_CONTROLLER if the node that the
> > > > > request was
> > > > > > > > sent to is not the controller". I thought the clients never
> send
> > > a
> > > > > > > request
> > > > > > > > to the controller directly now and the broker will forward
> it to
> > > the
> > > > > > > > controller?
> > > > > > > >
> > > > > > >
> > > > > > > If the controller moved recently, it's possible that the broker
> > > could
> > > > > send
> > > > > > > to a controller that has just recently become inactive. In
> that
> > > case
> > > > > > > NOT_CONTROLLER would be returned. (A standby controller
> returns
> > > > > > > NOT_CONTROLLER for most APIs).
> > > > > > >
> > > > > > > > 221. Could we add the required ACL for the new requests?
> > > > > > > >
> > > > > > >
> > > > > > > Good point. I added the required ACL for each new RPC.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Wed, Dec 16, 2020 at 11:38 AM Colin McCabe <
> > > cmccabe@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > On Wed, Dec 16, 2020, at 09:59, Jun Rao wrote:
> > > > > > > > > > Hi, Colin,
> > > > > > > > > >
> > > > > > > > > > Thanks for the reply. A few more comments below.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > Thanks for the comments.
> > > > > > > > >
> > > > > > > > > > 206. "RemoveTopic is the last step, that scrubs all
> metadata
> > > > > about
> > > > > > > the
> > > > > > > > > > topic. In order to get to that last step, the topic data
> > > needs
> > > > > to
> > > > > > > > > removed
> > > > > > > > > > from all brokers (after each broker notices that the
> topic is
> > > > > being
> > > > > > > > > > deleted)." Currently, this is done based on the response
> of
> > > > > > > > > > StopReplicaRequest. Since the controller no longer sends
> this
> > > > > > > request,
> > > > > > > > > how
> > > > > > > > > > does the controller know that the data for the deleted
> topic
> > > has
> > > > > > > > > > been removed in the brokers?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > That's a good point... definitely an oversight.
> > > > > > > > >
> > > > > > > > > It seems complex to force the controller to track when log
> > > > > directories
> > > > > > > > > have been deleted. Let's just assume that KIP-516 has been
> > > > > > > implemented,
> > > > > > > > > and track them by UUID. Then we can just have a single
> topic
> > > > > deletion
> > > > > > > > > record.
> > > > > > > > >
> > > > > > > > > I added a section on "topic identifiers" describing this.
> > > > > > > > >
> > > > > > > > > > 210. Thanks for the explanation. Sounds good to me.
> > > > > > > > > >
> > > > > > > > > > 211. I still don't see an example when ShouldFence is
> set to
> > > > > true in
> > > > > > > > > > BrokerHeartbeatReques. Could we add one?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > It is sent to true when the broker is first starting up and
> > > > > doesn't yet
> > > > > > > > > want to be unfenced. I added a longer explanation of this
> in
> > > the
> > > > > > > "Broker
> > > > > > > > > Leases" section.
> > > > > > > > >
> > > > > > > > > > 213. The KIP now allows replicas to be assigned on
> brokers
> > > that
> > > > > are
> > > > > > > > > fenced,
> > > > > > > > > > which is an improvement. How do we permanently remove a
> > > broker
> > > > > (e.g.
> > > > > > > > > > cluster shrinking) to prevent it from being used for
> future
> > > > > replica
> > > > > > > > > > assignments?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > This is a fair point. I will create a kafka-cluster.sh
> script
> > > > > which
> > > > > > > can
> > > > > > > > > do this, plus a DecommissionBrokerRequest.
> > > > > > > > >
> > > > > > > > > As a bonus the kafka-cluster.sh script can help people
> find the
> > > > > > > cluster ID
> > > > > > > > > of brokers, something that people have wanted a tool for
> in the
> > > > > past.
> > > > > > > > >
> > > > > > > > > > 214. "Currently, when a node is down, all of its ZK
> > > registration
> > > > > > > > > > information is gone. But we need this information in
> order
> > > to
> > > > > > > > > understand
> > > > > > > > > > things like whether the replicas of a particular
> partition
> > > are
> > > > > > > > > > well-balanced across racks." This is not quite true.
> > > Currently,
> > > > > even
> > > > > > > when
> > > > > > > > > > ZK registration is gone, the existing replica assignment
> is
> > > still
> > > > > > > > > available
> > > > > > > > > > in the metadata response.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I agree that the existing replica assignment is still
> > > available.
> > > > > But
> > > > > > > that
> > > > > > > > > just tells you partition X is on nodes A, B, and C. If you
> > > don't
> > > > > have
> > > > > > > the
> > > > > > > > > ZK registration for one or more of A, B, or C then you
> don't
> > > know
> > > > > > > whether
> > > > > > > > > we are following the policy of "two replicas on one rack,
> one
> > > > > replica
> > > > > > > on
> > > > > > > > > another." Or any other more complex rack placement policy
> > > that you
> > > > > > > might
> > > > > > > > > have.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Wed, Dec 16, 2020 at 8:48 AM Colin McCabe <
> > > cmccabe@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the reply. A few more follow up comments.
> > > > > > > > > > > >
> > > > > > > > > > > > 210. initial.broker.registration.timeout.ms: The
> default
> > > > > value
> > > > > > > is
> > > > > > > > > 90sec,
> > > > > > > > > > > > which seems long. If a broker fails the registration
> > > because
> > > > > of
> > > > > > > > > incorrect
> > > > > > > > > > > > configs, we want to fail faster. In comparison, the
> > > defaults
> > > > > for
> > > > > > > > > > > > zookeeper.connection.timeout.ms is 18 secs.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Jun,
> > > > > > > > > > >
> > > > > > > > > > > I agree that the initial connection timeout here is
> longer
> > > than
> > > > > > > what we
> > > > > > > > > > > had with ZK. The main reason I selected a slightly
> longer
> > > > > timeout
> > > > > > > > > here is
> > > > > > > > > > > to handle the case where the controllers and the
> brokers
> > > are
> > > > > > > > > co-located.
> > > > > > > > > > > For example, if you have a 3 node cluster and all three
> > > nodes
> > > > > are
> > > > > > > > > > > controllers+brokers, when you first bring up the
> cluster,
> > > we
> > > > > will
> > > > > > > have
> > > > > > > > > to
> > > > > > > > > > > stand up the controller quorum and then handle broker
> > > > > > > registrations.
> > > > > > > > > > > Although we believe Raft will start up relatively
> quickly,
> > > it's
> > > > > > > good to
> > > > > > > > > > > leave some extra margin here.
> > > > > > > > > > >
> > > > > > > > > > > I don't think there's a big disadvantage to having a
> > > slightly
> > > > > > > longer
> > > > > > > > > > > timeout here. After all, starting up the brokers with
> no
> > > > > > > controllers
> > > > > > > > > is an
> > > > > > > > > > > admin mistake, which we don't expect to see very often.
> > > Maybe
> > > > > > > let's
> > > > > > > > > set it
> > > > > > > > > > > to 60 seconds for now and maybe see if we can tweak it
> in
> > > the
> > > > > > > future if
> > > > > > > > > > > that turns out to be too long or short?
> > > > > > > > > > >
> > > > > > > > > > > > 211. "Once they are all moved, the controller
> responds
> > > to the
> > > > > > > > > heartbeat
> > > > > > > > > > > > with a nextState of SHUTDOWN." It seems that
> nextState
> > > is no
> > > > > > > longer
> > > > > > > > > > > > relevant. Also, could you add a bit more explanation
> on
> > > when
> > > > > > > > > ShouldFence
> > > > > > > > > > > is
> > > > > > > > > > > > set to true in BrokerHeartbeatRequest?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Good catch. I removed the obsolete section referring
> to
> > > > > nextState
> > > > > > > and
> > > > > > > > > > > added a reference to the new boolean. I also added
> some
> > > more
> > > > > > > > > information
> > > > > > > > > > > about ShouldFence and about the rationale for
> separating
> > > > > fencing
> > > > > > > from
> > > > > > > > > > > registration.
> > > > > > > > > > >
> > > > > > > > > > > > 212. Related to the above, what does the broker do if
> > > > > IsFenced is
> > > > > > > > > true in
> > > > > > > > > > > > the BrokerHeartbeatResponse? Will the broker do the
> same
> > > > > thing on
> > > > > > > > > > > receiving
> > > > > > > > > > > > a FenceBrokerRecord from the metadata log?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The broker only checks this boolean during the startup
> > > process.
> > > > > > > After
> > > > > > > > > the
> > > > > > > > > > > startup process is finished, it ignores this boolean.
> > > > > > > > > > >
> > > > > > > > > > > The broker uses fence / unfence records in the metadata
> > > log to
> > > > > > > > > determine
> > > > > > > > > > > which brokers should appear in its MetadataResponse.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > > Jun
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <
> > > > > cmccabe@apache.org
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The KIP says that "brokers which are fenced will
> not
> > > > > appear
> > > > > > > in
> > > > > > > > > > > > > > MetadataResponses. So clients that have
> up-to-date
> > > > > metadata
> > > > > > > will
> > > > > > > > > > > not try
> > > > > > > > > > > > > > to contact fenced brokers.", which is fine, but
> it
> > > > > doesn't
> > > > > > > seem
> > > > > > > > > to
> > > > > > > > > > > > > > elaborate on what happens for clients which try
> to
> > > > > contact a
> > > > > > > > > broker
> > > > > > > > > > > > > (using
> > > > > > > > > > > > > > stale metadata, for example) and find it in the
> > > > > > > > > registered-but-fenced
> > > > > > > > > > > > > > state.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have dropped the broker-side fencing from this
> > > > > proposal. So
> > > > > > > now
> > > > > > > > > the
> > > > > > > > > > > > > fencing is basically the same as today's fencing:
> it's
> > > > > > > > > controller-side
> > > > > > > > > > > > > only. That means that clients using stale
> metadata can
> > > > > contact
> > > > > > > > > fenced
> > > > > > > > > > > > > brokers and communicate with them. The only case
> > > where the
> > > > > > > broker
> > > > > > > > > > > will be
> > > > > > > > > > > > > inaccessible is when it hasn't finished starting
> yet
> > > > > (i.e., has
> > > > > > > > > not yet
> > > > > > > > > > > > > attained RUNNING state.)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Just like today, we expect the safeguards built
> into
> > > the
> > > > > > > > > replication
> > > > > > > > > > > > > protocol to prevent the worst corner cases that
> could
> > > > > result
> > > > > > > from
> > > > > > > > > > > this. I
> > > > > > > > > > > > > do think we will probably take up this issue later
> and
> > > > > find a
> > > > > > > > > better
> > > > > > > > > > > > > solution for client-side metadata, but this KIP is
> big
> > > > > enough
> > > > > > > > > as-is.
> > > > > > > > > > > > >
> > > > > > > > > > > > > best,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > You said previously that the broker will respond
> > > with a
> > > > > > > retriable
> > > > > > > > > > > > > > error, which is again fine, but you didn't
> answer my
> > > > > question
> > > > > > > > > about
> > > > > > > > > > > which
> > > > > > > > > > > > > > error code you've chosen for this. I realise that
> > > this
> > > > > > > doesn't
> > > > > > > > > really
> > > > > > > > > > > > > > affect the correctness of the behaviour, but I'm
> not
> > > > > aware
> > > > > > > of any
> > > > > > > > > > > > > existing
> > > > > > > > > > > > > > error code which is a good fit. So it would be
> good
> > > to
> > > > > > > understand
> > > > > > > > > > > about
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > you're making this backward compatible for
> clients.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Many thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <
> > > > > > > cmccabe@apache.org
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the reply. Just a couple of more
> > > comments
> > > > > > > below.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 210. Since we are deprecating
> > > > > > > > > zookeeper.connection.timeout.ms,
> > > > > > > > > > > > > should we
> > > > > > > > > > > > > > > > add a new config to bound the time for a
> broker
> > > to
> > > > > > > connect
> > > > > > > > > to the
> > > > > > > > > > > > > > > > controller during starting up?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Good idea. I added
> > > > > initial.broker.registration.timeout.ms
> > > > > > > for
> > > > > > > > > > > this.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 211. BrokerHeartbeat no longer has the state
> > > field
> > > > > in the
> > > > > > > > > > > > > > > request/response.
> > > > > > > > > > > > > > > > However, (a) the controller shutdown section
> > > still
> > > > > has
> > > > > > > "In
> > > > > > > > > its
> > > > > > > > > > > > > periodic
> > > > > > > > > > > > > > > > heartbeats, the broker asks the controller
> if it
> > > can
> > > > > > > > > transition
> > > > > > > > > > > into
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > SHUTDOWN state. This motivates the
> controller to
> > > > > move
> > > > > > > all
> > > > > > > > > of the
> > > > > > > > > > > > > leaders
> > > > > > > > > > > > > > > > off of that broker. Once they are all
> moved, the
> > > > > > > controller
> > > > > > > > > > > > > responds to
> > > > > > > > > > > > > > > > the heartbeat with a nextState of SHUTDOWN.";
> > > (2) the
> > > > > > > > > > > description of
> > > > > > > > > > > > > > > > BrokerHeartbeat still references
> currentState and
> > > > > > > > > targetState.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks. I've made these sections clearer and
> > > removed
> > > > > the
> > > > > > > > > obsolete
> > > > > > > > > > > > > > > references to sending states.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe
> <
> > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao
> wrote:
> > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for the update. A few more follow
> up
> > > > > comments.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks again for the review.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 100. FailedReplicaRecord: Since this is
> > > reported
> > > > > by
> > > > > > > each
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > independently, perhaps we could use a
> more
> > > > > concise
> > > > > > > > > > > representation
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > a top level broker field, an array of
> topics,
> > > > > which
> > > > > > > has
> > > > > > > > > an
> > > > > > > > > > > array
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > partitions.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The issue is that there is a size limit on
> the
> > > each
> > > > > > > record.
> > > > > > > > > > > > > Putting
> > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > of the partitions of a log directory into a
> > > single
> > > > > > > record
> > > > > > > > > would
> > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > break that in many cases. Still, we can
> > > optimize a
> > > > > > > bit by
> > > > > > > > > > > having
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > array
> > > > > > > > > > > > > > > > > of partition IDs, since nearly all the
> time, we
> > > > > have
> > > > > > > more
> > > > > > > > > than
> > > > > > > > > > > one
> > > > > > > > > > > > > > > from the
> > > > > > > > > > > > > > > > > same topic.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 200. Sounds good. If we remove the
> > > broker-side
> > > > > > > fencing
> > > > > > > > > > > logic, do
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > plan
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > still keep FENCED in broker state? Do we
> > > plan to
> > > > > > > expose
> > > > > > > > > the
> > > > > > > > > > > new
> > > > > > > > > > > > > > > states
> > > > > > > > > > > > > > > > > > through the existing BrokerState metric
> and
> > > if
> > > > > so,
> > > > > > > what
> > > > > > > > > are
> > > > > > > > > > > the
> > > > > > > > > > > > > > > values
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > the new states?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > No, we don't need FENCED any more. I have
> > > removed
> > > > > it
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The new states are very similar to the
> current
> > > > > ones,
> > > > > > > > > actually.
> > > > > > > > > > > > > There
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > no new states or removed ones. The main
> > > change in
> > > > > the
> > > > > > > > > broker
> > > > > > > > > > > state
> > > > > > > > > > > > > > > machine
> > > > > > > > > > > > > > > > > is that the
> RECOVERING_FROM_UNCLEAN_SHUTDOWN
> > > state
> > > > > has
> > > > > > > been
> > > > > > > > > > > > > renamed to
> > > > > > > > > > > > > > > > > RECOVERY. Also, unlike previously, the
> broker
> > > will
> > > > > > > always
> > > > > > > > > pass
> > > > > > > > > > > > > through
> > > > > > > > > > > > > > > > > RECOVERY (although it may only stay in this
> > > state
> > > > > for a
> > > > > > > > > very
> > > > > > > > > > > short
> > > > > > > > > > > > > > > amount
> > > > > > > > > > > > > > > > > of time).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 201. This may be fine too. Could we
> document
> > > what
> > > > > > > happens
> > > > > > > > > > > when
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > broker.id/controller.id in
> > > metadata.properties
> > > > > don't
> > > > > > > > > match
> > > > > > > > > > > the
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > config when the broker starts up?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I added some documentation about this.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 204. There is still "The highest metadata
> > > offset
> > > > > > > which
> > > > > > > > > the
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > has not
> > > > > > > > > > > > > > > > > > reached" referenced under
> BrokerRegistration.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 206. Is that separate step needed given
> > > KIP-516?
> > > > > With
> > > > > > > > > > > KIP-516 (
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > > > > > > > > > > > > ),
> > > > > > > > > > > > > > > > > > we don't need to wait for the topic data
> to
> > > be
> > > > > > > removed
> > > > > > > > > from
> > > > > > > > > > > all
> > > > > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > > > before removing the topic metadata. The
> > > > > combination
> > > > > > > of
> > > > > > > > > > > unmatching
> > > > > > > > > > > > > > > > > > topicId
> > > > > > > > > > > > > > > > > > or the missing topicId from the metadata
> is
> > > > > enough
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > clean
> > > > > > > > > > > > > > > > > > up deleted topics asynchronously.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It won't be needed once KIP-516 is
> adopted, but
> > > > > this
> > > > > > > hasn't
> > > > > > > > > > > been
> > > > > > > > > > > > > > > > > implemented yet.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin
> McCabe <
> > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao
> > > wrote:
> > > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for the updated KIP. A few
> more
> > > > > comments
> > > > > > > > > below.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks again for the reviews.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 80.2 For deprecated configs, we need
> to
> > > > > include
> > > > > > > > > > > zookeeper.*
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > broker.id.generation.enable.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Added.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 83.1 If a broker is down, does the
> > > controller
> > > > > > > keep
> > > > > > > > > the
> > > > > > > > > > > > > previously
> > > > > > > > > > > > > > > > > > > > registered broker epoch forever? If
> not,
> > > how
> > > > > long
> > > > > > > > > does
> > > > > > > > > > > the
> > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > > > > > > it? What does the controller do when
> > > > > receiving a
> > > > > > > > > broker
> > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > > > with an unfound broker epoch?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Yes, the controller keeps the previous
> > > > > registration
> > > > > > > > > > > forever.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Broker heartbeat requests with an
> incorrect
> > > > > broker
> > > > > > > > > epoch
> > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > rejected
> > > > > > > > > > > > > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 100. Have you figured out if we need
> to
> > > add
> > > > > a new
> > > > > > > > > record
> > > > > > > > > > > > > type for
> > > > > > > > > > > > > > > > > > > reporting
> > > > > > > > > > > > > > > > > > > > partitions on failed disks?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I added FailedReplicaRecord to reflect
> the
> > > case
> > > > > > > where a
> > > > > > > > > > > JBOD
> > > > > > > > > > > > > > > directory
> > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > failed, leading to failed replicas.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 102. For debugging purposes,
> sometimes
> > > it's
> > > > > > > useful to
> > > > > > > > > > > read
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > > > topic using tools like
> console-consumer.
> > > > > Should
> > > > > > > we
> > > > > > > > > > > support
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > so,
> > > > > > > > > > > > > > > > > > > > how?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > For now, we have the ability to read
> the
> > > > > metadata
> > > > > > > logs
> > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > dump-logs
> > > > > > > > > > > > > > > > > > > tool. I think we will come up with
> some
> > > other
> > > > > > > tools
> > > > > > > > > in the
> > > > > > > > > > > > > future
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > get experience.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 200. "brokers which are fenced will
> not
> > > > > appear in
> > > > > > > > > > > > > > > MetadataResponses.
> > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > broker will not respond to these
> > > requests--
> > > > > > > instead,
> > > > > > > > > it
> > > > > > > > > > > will
> > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > > > disconnect." If the controller is
> > > > > partitioned off
> > > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > > > > brokers,
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > design will cause every broker to
> stop
> > > > > accepting
> > > > > > > new
> > > > > > > > > > > client
> > > > > > > > > > > > > > > > > requests. In
> > > > > > > > > > > > > > > > > > > > contrast, if ZK is partitioned off,
> the
> > > > > existing
> > > > > > > > > > > behavior is
> > > > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > > > brokers can continue to work based
> on the
> > > > > last
> > > > > > > known
> > > > > > > > > > > > > metadata.
> > > > > > > > > > > > > > > So, I
> > > > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > sure if we should change the existing
> > > > > behavior
> > > > > > > > > because
> > > > > > > > > > > of the
> > > > > > > > > > > > > > > bigger
> > > > > > > > > > > > > > > > > > > impact
> > > > > > > > > > > > > > > > > > > > in the new one. Another option is to
> > > keep the
> > > > > > > > > existing
> > > > > > > > > > > > > behavior
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > expose
> > > > > > > > > > > > > > > > > > > > a metric for fenced brokers so that
> the
> > > > > operator
> > > > > > > > > could be
> > > > > > > > > > > > > > > alerted.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I'm skeptical about how well running
> > > without ZK
> > > > > > > > > currently
> > > > > > > > > > > > > works.
> > > > > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > > > > I will move the broker-side fencing
> into a
> > > > > > > follow-up
> > > > > > > > > KIP.
> > > > > > > > > > > This
> > > > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > > > > already pretty large and there is no
> hard
> > > > > > > dependency on
> > > > > > > > > > > this.
> > > > > > > > > > > > > > > There
> > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > also be other ways of accomplishing the
> > > > > positive
> > > > > > > > > effects of
> > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > > > broker-side fencing, so more
> discussion is
> > > > > needed.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 201. I read Ron's comment, but I am
> > > still not
> > > > > > > sure
> > > > > > > > > the
> > > > > > > > > > > > > benefit of
> > > > > > > > > > > > > > > > > keeping
> > > > > > > > > > > > > > > > > > > > broker.id and controller.id in
> > > > > meta.properties.
> > > > > > > It
> > > > > > > > > seems
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > we are
> > > > > > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > > duplicating the same info in two
> places
> > > and
> > > > > have
> > > > > > > the
> > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > burden of
> > > > > > > > > > > > > > > > > > > > making sure the values in the two
> places
> > > are
> > > > > > > > > consistent.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think the reasoning is that having
> > > broker.id
> > > > > > > > > protects us
> > > > > > > > > > > > > against
> > > > > > > > > > > > > > > > > > > accidentally bringing up a broker with
> a
> > > disk
> > > > > from
> > > > > > > a
> > > > > > > > > > > different
> > > > > > > > > > > > > > > > > broker. I
> > > > > > > > > > > > > > > > > > > don't feel strongly about this but it
> > > seemed
> > > > > > > simpler to
> > > > > > > > > > > keep
> > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 202.
> > > controller.connect.security.protocol: Is
> > > > > > > this
> > > > > > > > > needed
> > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > > > controller.listener.names and
> > > > > > > > > > > listener.security.protocol.map
> > > > > > > > > > > > > > > imply
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > security protocol already?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > You're right, this isn't needed. I'll
> > > remove
> > > > > it.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 203.
> registration.heartbeat.interval.ms:
> > > It
> > > > > > > > > defaults to
> > > > > > > > > > > 2k.
> > > > > > > > > > > > > ZK
> > > > > > > > > > > > > > > uses
> > > > > > > > > > > > > > > > > 1/3
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > the session timeout for heartbeat.
> So,
> > > given
> > > > > the
> > > > > > > > > default
> > > > > > > > > > > 18k
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > registration.lease.timeout.ms,
> should we
> > > > > default
> > > > > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms
> to
> > > 6k?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 6 seconds seems like a pretty long time
> > > between
> > > > > > > > > > > heartbeats. It
> > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > useful to know when a broker is missing
> > > > > heartbeats,
> > > > > > > > > with
> > > > > > > > > > > less
> > > > > > > > > > > > > time
> > > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > that. I provisionally set it to 3
> seconds
> > > (we
> > > > > can
> > > > > > > > > always
> > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > later...)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I also changed the name of these
> > > > > configurations to
> > > > > > > "
> > > > > > > > > > > > > > > > > > > broker.heartbeat.interval.ms" and "
> > > > > > > > > > > > > broker.registration.timeout.ms"
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > > > > > to clarify them a bit.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 204. "The highest metadata offset
> which
> > > the
> > > > > > > broker
> > > > > > > > > has
> > > > > > > > > > > not
> > > > > > > > > > > > > > > reached."
> > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > seems this should be "has reached".
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I changed this to "one more than the
> > > highest
> > > > > > > metadata
> > > > > > > > > > > offset
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > broker has reached."
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 205. UnfenceBrokerRecord and
> > > > > > > UnregisterBrokerRecord:
> > > > > > > > > To
> > > > > > > > > > > me,
> > > > > > > > > > > > > they
> > > > > > > > > > > > > > > > > seem to
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > the same. Do we need both?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Unregistration means that the broker
> has
> > > been
> > > > > > > removed
> > > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > > > > cluster.
> > > > > > > > > > > > > > > > > > > That is different than unfencing, which
> > > marks
> > > > > the
> > > > > > > > > broker as
> > > > > > > > > > > > > active.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 206. TopicRecord: The Deleting field
> is
> > > used
> > > > > to
> > > > > > > > > indicate
> > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > being deleted. I am wondering if
> this is
> > > > > really
> > > > > > > > > needed
> > > > > > > > > > > since
> > > > > > > > > > > > > > > > > RemoveTopic
> > > > > > > > > > > > > > > > > > > > already indicates the same thing.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > RemoveTopic is the last step, that
> scrubs
> > > all
> > > > > > > metadata
> > > > > > > > > > > about
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > topic.
> > > > > > > > > > > > > > > > > > > In order to get to that last step, the
> > > topic
> > > > > data
> > > > > > > > > needs to
> > > > > > > > > > > > > removed
> > > > > > > > > > > > > > > > > from all
> > > > > > > > > > > > > > > > > > > brokers (after each broker notices
> that the
> > > > > topic
> > > > > > > is
> > > > > > > > > being
> > > > > > > > > > > > > > > deleted).
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin
> > > McCabe <
> > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron
> > > > > Dagostino
> > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the
> updates.
> > > It's
> > > > > now
> > > > > > > > > clear
> > > > > > > > > > > to me
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > > > > > > > keep their broker epoch for the
> life
> > > of
> > > > > their
> > > > > > > > > JVM --
> > > > > > > > > > > they
> > > > > > > > > > > > > > > > > register
> > > > > > > > > > > > > > > > > > > > > > once, get their broker epoch in
> the
> > > > > > > response, and
> > > > > > > > > > > then
> > > > > > > > > > > > > never
> > > > > > > > > > > > > > > > > > > > > > re-register again. Brokers may
> get
> > > > > fenced,
> > > > > > > but
> > > > > > > > > they
> > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > > > > broker epoch for the life of
> their
> > > JVM.
> > > > > The
> > > > > > > > > > > incarnation
> > > > > > > > > > > > > ID
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > kept for the life of the JVM but
> is
> > > > > > > generated by
> > > > > > > > > the
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > > > > upon startup, and the
> combination of
> > > the
> > > > > two
> > > > > > > > > allows
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Controller to
> > > > > > > > > > > > > > > > > > > > > > act idempotently if any
> > > previously-sent
> > > > > > > > > registration
> > > > > > > > > > > > > response
> > > > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks, Ron. That's a good
> summary.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > One thing I wonder about is if it
> > > might
> > > > > be
> > > > > > > > > helpful
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > > > > send the Cluster ID as determined
> > > from
> > > > > its
> > > > > > > > > > > > > meta.properties
> > > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > > > > > registration request. Does it
> even
> > > make
> > > > > > > sense
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > > > > successfully register and enter
> the
> > > > > Fenced
> > > > > > > state
> > > > > > > > > if
> > > > > > > > > > > it
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > wrong
> > > > > > > > > > > > > > > > > > > > > > Cluster ID?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Yeah, that's a good idea. Let's
> have
> > > the
> > > > > > > broker
> > > > > > > > > pass
> > > > > > > > > > > its
> > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > ID in
> > > > > > > > > > > > > > > > > > > > > the registration RPC, and then
> > > > > registration can
> > > > > > > > > fail
> > > > > > > > > > > if the
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The nextMetadatOffset value
> that the
> > > > > broker
> > > > > > > > > > > communicates
> > > > > > > > > > > > > > > > > > > > > > in its registration request only
> has
> > > > > meaning
> > > > > > > > > within
> > > > > > > > > > > the
> > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > > > cluster, so it feels to me that
> the
> > > > > > > Controller
> > > > > > > > > should
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > perform this sanity check.
> There is
> > > > > > > currently
> > > > > > > > > > > (pre-KIP
> > > > > > > > > > > > > 500)
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > check
> > > > > > > > > > > > > > > > > > > > > > in the broker to make sure its
> > > configured
> > > > > > > > > cluster ID
> > > > > > > > > > > > > matches
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > > > > > stored in ZooKeeper, and we will
> > > have to
> > > > > > > perform
> > > > > > > > > this
> > > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > > > somewhere in the KIP-500 world.
> If
> > > the
> > > > > > > > > Controller
> > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > do it
> > > > > > > > > > > > > > > > > > > > > > within the registration request
> then
> > > the
> > > > > > > broker
> > > > > > > > > will
> > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > > > metadata request to the
> Controller,
> > > > > retrieve
> > > > > > > the
> > > > > > > > > > > Cluster
> > > > > > > > > > > > > ID,
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > perform the check itself. It
> feels
> > > to me
> > > > > > > that it
> > > > > > > > > > > might
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > better for
> > > > > > > > > > > > > > > > > > > > > > the Controller to just do it, and
> > > then
> > > > > the
> > > > > > > broker
> > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > > > > worry about it anymore once it
> > > > > successfully
> > > > > > > > > > > registers.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I also have a question about the
> > > > > broker.id
> > > > > > > > > value and
> > > > > > > > > > > > > > > > > > > meta.properties.
> > > > > > > > > > > > > > > > > > > > > > The KIP now says "In version 0 of
> > > > > > > > > meta.properties,
> > > > > > > > > > > there
> > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > > > > broker.id field. Version 1
> does not
> > > > > have
> > > > > > > this
> > > > > > > > > > > field.
> > > > > > > > > > > > > It
> > > > > > > > > > > > > > > is no
> > > > > > > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > > > > > > > needed because we no longer
> support
> > > > > dynamic
> > > > > > > > > broker id
> > > > > > > > > > > > > > > > > assignment."
> > > > > > > > > > > > > > > > > > > > > > But then there is an example
> version
> > > 1
> > > > > > > > > > > meta.properties
> > > > > > > > > > > > > file
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > shows
> > > > > > > > > > > > > > > > > > > > > > the broker.id value. I actually
> > > wonder
> > > > > if
> > > > > > > > > maybe the
> > > > > > > > > > > > > > > broker.id
> > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > > would be good to keep in the
> version
> > > 1
> > > > > > > > > > > meta.properties
> > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > currently (pre-KIP 500, version
> 0)
> > > acts
> > > > > as a
> > > > > > > > > sanity
> > > > > > > > > > > > > check to
> > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > sure
> > > > > > > > > > > > > > > > > > > > > > the broker is using the correct
> log
> > > > > > > directory.
> > > > > > > > > > > Similarly
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > controller.id value on
> controllers
> > > -- it
> > > > > > > would
> > > > > > > > > > > allow the
> > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > sanity check for quorum
> controllers.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > That's a good point. I will add
> > > broker.id
> > > > > > > back,
> > > > > > > > > and
> > > > > > > > > > > also
> > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > > controller.id as a possibility.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM
> Colin
> > > > > McCabe
> > > > > > > <
> > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10,
> > > Jun Rao
> > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for the reply. A few
> more
> > > > > > > comments.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks again for the reply.
> Sorry
> > > for
> > > > > the
> > > > > > > long
> > > > > > > > > > > > > hiatus. I
> > > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > > vacation for a while.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 55. There is still text that
> > > favors
> > > > > new
> > > > > > > > > broker
> > > > > > > > > > > > > > > registration.
> > > > > > > > > > > > > > > > > > > "When a
> > > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > > first starts up, when it is
> in
> > > the
> > > > > > > INITIAL
> > > > > > > > > > > state, it
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > always
> > > > > > > > > > > > > > > > > > > > > "win"
> > > > > > > > > > > > > > > > > > > > > > > > broker ID conflicts.
> However,
> > > once
> > > > > it is
> > > > > > > > > > > granted a
> > > > > > > > > > > > > > > lease, it
> > > > > > > > > > > > > > > > > > > > > transitions
> > > > > > > > > > > > > > > > > > > > > > > > out of the INITIAL state.
> > > > > Thereafter,
> > > > > > > it may
> > > > > > > > > > > lose
> > > > > > > > > > > > > > > subsequent
> > > > > > > > > > > > > > > > > > > > > conflicts if
> > > > > > > > > > > > > > > > > > > > > > > > its broker epoch is stale.
> (See
> > > > > KIP-380
> > > > > > > for
> > > > > > > > > some
> > > > > > > > > > > > > > > background
> > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > > epoch.) The reason for
> favoring
> > > new
> > > > > > > > > processes
> > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > accommodate
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > > > > > > > > case where a process is
> killed
> > > with
> > > > > kill
> > > > > > > -9
> > > > > > > > > and
> > > > > > > > > > > then
> > > > > > > > > > > > > > > > > restarted.
> > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > want it
> > > > > > > > > > > > > > > > > > > > > > > > to be able to reclaim its
> old ID
> > > > > quickly
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > > > > > case."
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for the reminder. I
> have
> > > > > clarified
> > > > > > > the
> > > > > > > > > > > language
> > > > > > > > > > > > > > > here.
> > > > > > > > > > > > > > > > > > > > > Hopefully now it is clear that we
> don't
> > > > > allow
> > > > > > > quick
> > > > > > > > > > > re-use
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 80.1 Sounds good. Could you
> > > document
> > > > > that
> > > > > > > > > > > listeners
> > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > > > > > now? It would also be useful
> to
> > > > > annotate
> > > > > > > > > other
> > > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > configs.
> > > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > > > > > example, controller.connect
> > > should be
> > > > > > > > > required.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I added a note specifying that
> > > these
> > > > > are
> > > > > > > > > required.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 80.2 Could you list all
> > > deprecated
> > > > > > > existing
> > > > > > > > > > > configs?
> > > > > > > > > > > > > > > Another
> > > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > control.plane.listener.name
> > > since
> > > > > the
> > > > > > > > > > > controller no
> > > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > > sends
> > > > > > > > > > > > > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata
> and
> > > > > > > StopReplica
> > > > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I added a section specifying
> some
> > > > > > > deprecated
> > > > > > > > > > > configs.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 83.1 It seems that the
> broker can
> > > > > > > transition
> > > > > > > > > from
> > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > RUNNING
> > > > > > > > > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > > > > > > > > registering for a new broker
> > > epoch.
> > > > > I am
> > > > > > > not
> > > > > > > > > > > sure how
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > works.
> > > > > > > > > > > > > > > > > > > > > Once the
> > > > > > > > > > > > > > > > > > > > > > > > controller fences a broker,
> > > there is
> > > > > no
> > > > > > > need
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > > > > > > > > > boker epoch around. So, if
> the
> > > fenced
> > > > > > > > > broker's
> > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > > > > > > existing broker epoch will be
> > > > > rejected,
> > > > > > > > > leading
> > > > > > > > > > > the
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The broker epoch refers to the
> > > broker
> > > > > > > > > registration.
> > > > > > > > > > > > > So we
> > > > > > > > > > > > > > > DO
> > > > > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > broker epoch around even while the
> > > broker
> > > > > is
> > > > > > > > > fenced.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The broker epoch changes only
> when
> > > > > there
> > > > > > > is a
> > > > > > > > > new
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > registration. Fencing or
> unfencing the
> > > > > broker
> > > > > > > > > doesn't
> > > > > > > > > > > > > change
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 83.5 Good point on KIP-590.
> Then
> > > > > should
> > > > > > > we
> > > > > > > > > > > expose the
> > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > debugging purposes? If not,
> we
> > > should
> > > > > > > > > deprecate
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > controllerID
> > > > > > > > > > > > > > > > > > > > > field in
> > > > > > > > > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think it's OK to expose it
> for
> > > now,
> > > > > with
> > > > > > > the
> > > > > > > > > > > proviso
> > > > > > > > > > > > > > > that it
> > > > > > > > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > > > > > > > be reachable by clients.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 90. We rejected the shared ID
> > > with
> > > > > just
> > > > > > > one
> > > > > > > > > > > reason
> > > > > > > > > > > > > "This
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > not a
> > > > > > > > > > > > > > > > > > > > > good idea
> > > > > > > > > > > > > > > > > > > > > > > > because NetworkClient
> assumes a
> > > > > single ID
> > > > > > > > > > > space. So
> > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > there is
> > > > > > > > > > > > > > > > > > > > > both a
> > > > > > > > > > > > > > > > > > > > > > > > controller 1 and a broker 1,
> we
> > > don't
> > > > > > > have a
> > > > > > > > > way
> > > > > > > > > > > of
> > > > > > > > > > > > > > > picking
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > "right"
> > > > > > > > > > > > > > > > > > > > > > > > one." This doesn't seem to
> be a
> > > > > strong
> > > > > > > > > reason.
> > > > > > > > > > > For
> > > > > > > > > > > > > > > example,
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > > > > > > > address the NetworkClient
> issue
> > > with
> > > > > the
> > > > > > > node
> > > > > > > > > > > type
> > > > > > > > > > > > > as you
> > > > > > > > > > > > > > > > > pointed
> > > > > > > > > > > > > > > > > > > > > out or
> > > > > > > > > > > > > > > > > > > > > > > > using the negative value of a
> > > broker
> > > > > ID
> > > > > > > as
> > > > > > > > > the
> > > > > > > > > > > > > > > controller ID.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > It would require a lot of code
> > > changes
> > > > > to
> > > > > > > > > support
> > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > types of
> > > > > > > > > > > > > > > > > > > > > node IDs. It's not clear to me
> that
> > > the
> > > > > end
> > > > > > > result
> > > > > > > > > > > would
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > better --
> > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > tend to think it would be worse,
> since
> > > it
> > > > > > > would be
> > > > > > > > > more
> > > > > > > > > > > > > > > complex.
> > > > > > > > > > > > > > > > > In a
> > > > > > > > > > > > > > > > > > > > > similar vein, using negative
> numbers
> > > seems
> > > > > > > > > dangerous,
> > > > > > > > > > > > > since we
> > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > negatives or -1 as "special
> values" in
> > > many
> > > > > > > places.
> > > > > > > > > > > For
> > > > > > > > > > > > > > > example,
> > > > > > > > > > > > > > > > > -1
> > > > > > > > > > > > > > > > > > > often
> > > > > > > > > > > > > > > > > > > > > represents "no such node."
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > One important thing to keep in
> > > mind is
> > > > > > > that we
> > > > > > > > > > > want to
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > transition from a broker and a
> > > controller
> > > > > being
> > > > > > > > > > > co-located
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > them
> > > > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > > > > > longer being co-located. This is
> much
> > > > > easier
> > > > > > > to do
> > > > > > > > > > > when
> > > > > > > > > > > > > they
> > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > > > > > the broker reports all
> offline
> > > > > replicas
> > > > > > > due
> > > > > > > > > to a
> > > > > > > > > > > disk
> > > > > > > > > > > > > > > > > failure to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > controller. It seems this
> > > information
> > > > > > > needs
> > > > > > > > > to be
> > > > > > > > > > > > > > > persisted
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > > > > > > > log. Do we have a
> corresponding
> > > > > record
> > > > > > > for
> > > > > > > > > that?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hmm, I have to look into this a
> > > little
> > > > > bit
> > > > > > > > > more.
> > > > > > > > > > > We
> > > > > > > > > > > > > may
> > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > > > > > > record type.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 101. Currently, StopReplica
> > > request
> > > > > has 2
> > > > > > > > > modes,
> > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > > > > and with
> > > > > > > > > > > > > > > > > > > > > > > > deletion. The former is used
> for
> > > > > > > controlled
> > > > > > > > > > > shutdown
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > handling
> > > > > > > > > > > > > > > > > > > > > disk
> > > > > > > > > > > > > > > > > > > > > > > > failure, and causes the
> follower
> > > to
> > > > > > > stop. The
> > > > > > > > > > > latter
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > > > > > > > and partition reassignment,
> and
> > > > > causes
> > > > > > > the
> > > > > > > > > > > replica
> > > > > > > > > > > > > to be
> > > > > > > > > > > > > > > > > deleted.
> > > > > > > > > > > > > > > > > > > > > Since we
> > > > > > > > > > > > > > > > > > > > > > > > are deprecating StopReplica,
> > > could we
> > > > > > > > > document
> > > > > > > > > > > what
> > > > > > > > > > > > > > > triggers
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > stopping
> > > > > > > > > > > > > > > > > > > > > > > > of a follower and the
> deleting
> > > of a
> > > > > > > replica
> > > > > > > > > now?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > RemoveTopic triggers
> deletion. In
> > > > > general
> > > > > > > the
> > > > > > > > > > > > > > > functionality of
> > > > > > > > > > > > > > > > > > > > > StopReplica is subsumed by the
> metadata
> > > > > > > records.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 102. Should we include the
> > > metadata
> > > > > > > topic in
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > > > > If so,
> > > > > > > > > > > > > > > > > > > > > > > > when it will be included and
> what
> > > > > will
> > > > > > > the
> > > > > > > > > > > metadata
> > > > > > > > > > > > > > > response
> > > > > > > > > > > > > > > > > look
> > > > > > > > > > > > > > > > > > > > > like?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > No, it won't be included in the
> > > > > metadata
> > > > > > > > > response
> > > > > > > > > > > sent
> > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > brokers.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 103. "The active controller
> > > assigns
> > > > > the
> > > > > > > > > broker a
> > > > > > > > > > > new
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > epoch,
> > > > > > > > > > > > > > > > > > > > > based on
> > > > > > > > > > > > > > > > > > > > > > > > the latest committed offset
> in
> > > the
> > > > > log."
> > > > > > > This
> > > > > > > > > > > seems
> > > > > > > > > > > > > > > > > inaccurate
> > > > > > > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > latest committed offset
> doesn't
> > > > > always
> > > > > > > > > advance on
> > > > > > > > > > > > > every
> > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > > append.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Given that the new broker epoch
> > > won't
> > > > > be
> > > > > > > > > visible
> > > > > > > > > > > until
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > commit
> > > > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > > happened, I have changed this to
> "the
> > > next
> > > > > > > > > available
> > > > > > > > > > > > > offset in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > log"
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 104. REGISTERING(1) : It says
> > > > > > > "Otherwise, the
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > moves
> > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > > > > > > > > > > state.". It seems this
> should be
> > > > > RUNNING?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 105. RUNNING: Should we
> require
> > > the
> > > > > > > broker to
> > > > > > > > > > > catch
> > > > > > > > > > > > > up
> > > > > > > > > > > > > > > to the
> > > > > > > > > > > > > > > > > > > > > metadata log
> > > > > > > > > > > > > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > For 104 and 105, these sections
> > > have
> > > > > been
> > > > > > > > > reworked.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20
> PM
> > > Colin
> > > > > > > McCabe
> > > > > > > > > <
> > > > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020, at
> > > 05:51, Tom
> > > > > > > Bentley
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at
> > > 08:59,
> > > > > Ron
> > > > > > > > > Dagostino
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks
> for the
> > > > > hard
> > > > > > > work
> > > > > > > > > on
> > > > > > > > > > > this
> > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > I have some questions
> > > about
> > > > > what
> > > > > > > > > happens
> > > > > > > > > > > to a
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > when it
> > > > > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > > > > > > > fenced (e.g. because
> it
> > > can't
> > > > > > > send a
> > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > request to
> > > > > > > > > > > > > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > > > > > > > > > > > > lease). The KIP says
> > > "When a
> > > > > > > broker
> > > > > > > > > is
> > > > > > > > > > > > > fenced,
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > cannot
> > > > > > > > > > > > > > > > > > > > > process any
> > > > > > > > > > > > > > > > > > > > > > > > > > > > client requests.
> This
> > > > > prevents
> > > > > > > > > brokers
> > > > > > > > > > > which
> > > > > > > > > > > > > > > are not
> > > > > > > > > > > > > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > > > > > > > > > > > > metadata updates or
> that
> > > are
> > > > > not
> > > > > > > > > > > receiving
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > processing
> > > > > > > > > > > > > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > > > > > > > > > > > > enough from causing
> > > issues to
> > > > > > > > > clients."
> > > > > > > > > > > And
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > FENCED(4) state it
> > > likewise
> > > > > says
> > > > > > > > > "While
> > > > > > > > > > > in
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > > > > > > > > not respond to client
> > > > > > > requests." It
> > > > > > > > > > > makes
> > > > > > > > > > > > > sense
> > > > > > > > > > > > > > > > > that a
> > > > > > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > > > > > > > should not accept
> > > producer
> > > > > > > requests
> > > > > > > > > -- I
> > > > > > > > > > > > > assume
> > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > > > > > > > > > > > would result in
> > > > > > > > > > > NotLeaderOrFollowerException.
> > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > > > > > > > > > > > > (fetch from follower)
> > > > > consumer
> > > > > > > > > > > requests? It
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > conceivable
> > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > > > > > > > could continue.
> Related
> > > to
> > > > > that,
> > > > > > > > > would a
> > > > > > > > > > > > > fenced
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > fetch data for
> partitions
> > > > > where
> > > > > > > it
> > > > > > > > > > > thinks it
> > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > follower?
> > > > > > > > > > > > > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > > > > > > > > > > > > it rejects consumer
> > > requests
> > > > > it
> > > > > > > might
> > > > > > > > > > > still
> > > > > > > > > > > > > > > continue
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > follower. Might it
> be
> > > > > helpful to
> > > > > > > > > clarify
> > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > decisions
> > > > > > > > > > > > > > > > > > > > > here?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Good question. I
> think a
> > > > > fenced
> > > > > > > broker
> > > > > > > > > > > should
> > > > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > > > > > > > > > > > > partitions it was
> already
> > > > > fetching
> > > > > > > > > before
> > > > > > > > > > > it
> > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > fenced,
> > > > > > > > > > > > > > > > > > > > > unless it
> > > > > > > > > > > > > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > > > > > > > > > > > > problem. At that
> point it
> > > > > won't be
> > > > > > > > > able to
> > > > > > > > > > > > > > > continue,
> > > > > > > > > > > > > > > > > > > since it
> > > > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > > > > > the new metadata. For
> > > > > example, it
> > > > > > > > > won't
> > > > > > > > > > > know
> > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > leadership
> > > > > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > the partitions it's
> > > fetching.
> > > > > The
> > > > > > > > > > > rationale
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > continuing to
> > > > > > > > > > > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > > > > > > > > > try to avoid
> disruptions as
> > > > > much as
> > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I don't think fenced
> > > brokers
> > > > > should
> > > > > > > > > accept
> > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > > > > > > The issue
> > > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > > that the fenced broker
> may
> > > or
> > > > > may
> > > > > > > not
> > > > > > > > > have
> > > > > > > > > > > any
> > > > > > > > > > > > > > > data it
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > > > > > > > > > > > > have. It may or may
> not
> > > have
> > > > > > > applied
> > > > > > > > > any
> > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > > > > it is supposed to have
> > > applied.
> > > > > > > So it
> > > > > > > > > > > could
> > > > > > > > > > > > > get
> > > > > > > > > > > > > > > pretty
> > > > > > > > > > > > > > > > > > > > > confusing, and
> > > > > > > > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > > > > potentially waste the
> > > client's
> > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > When fenced, how would
> the
> > > broker
> > > > > > > reply
> > > > > > > > > to a
> > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > which did
> > > > > > > > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The broker will respond
> with a
> > > > > > > retryable
> > > > > > > > > error
> > > > > > > > > > > in
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > case.
> > > > > > > > > > > > > > > > > > > Once
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > client has re-fetched its
> > > > > metadata, it
> > > > > > > > > will no
> > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > > > > as part of the cluster. I
> > > added a
> > > > > > > note to
> > > > > > > > > the
> > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Thu, Dec 17, 2020, at 17:02, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the reply. Sounds good. One more comment.
>
> 231. Currently, we have sasl.mechanism.inter.broker.protocol for inter
> broker communication. It seems that we need a similar config for specifying
> the sasl mechanism for the communication between the broker and the
> controller.
>
Hi Jun,
Yeah... sounds like we could use a new configuration key for this. I added sasl.mechanism.controller.protocol for this.
regards,
Colin
> Jun
>
> On Thu, Dec 17, 2020 at 2:29 PM Colin McCabe <cm...@apache.org> wrote:
>
> > On Thu, Dec 17, 2020, at 10:19, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply.
> > >
> > > 211. Hmm, I still don't see a clear benefit of registering a broker
> > before
> > > recovery. It's possible for the recovery to take time. However, during
> > the
> > > recovery mode, it seems the broker will still be in the fenced mode and
> > > won't be able to do work for the clients. So, registering and
> > heartbeating
> > > early seems to only add unnecessary overhead. For your point on topic
> > > creation, I thought we now allow replicas to be created on
> > > unregistered brokers.
> > >
> >
> > Hi Jun,
> >
> > Thanks again for the reviews.
> >
> > Topics cannot be created on unregistered brokers. They can be created on
> > registered but fenced brokers. So for that reason I think it makes sense
> > to register as early as possible.
> >
> > > 230. Currently, we do have a ControllerId field in MetadataResponse. In
> > the
> > > early discussion, I thought that we want to expose the controller for
> > > debugging purposes, but not used by the client library.
> > >
> >
> > The current plan is that we will expose the controller node ID, but the
> > controller will not be included in the list of nodes in the metadata
> > response.
> >
> > It's not really possible to include the controller in that list of nodes
> > because the controller may not share the same set of listeners as the
> > broker. So, for example, maybe the controller endpoint is using a
> > different type of security than the broker. So while we could pass back a
> > hostname and port, the client would have no way to connect since it doesn't
> > know what security settings to use.
> >
> > regards,
> > Colin
> >
> > > Jun
> > >
> > > On Wed, Dec 16, 2020 at 9:13 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Wed, Dec 16, 2020, at 18:13, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the reply. Just a couple of more comments.
> > > > >
> > > > > 211. Currently, the broker only registers itself in ZK after log
> > > > recovery.
> > > > > Is there any benefit to change that? As you mentioned, the broker
> > can't
> > > > do
> > > > > much before completing log recovery.
> > > > >
> > > >
> > > > Hi Jun,
> > > >
> > > > Previously, it wasn't possible to register in ZK without immediately
> > > > getting added to the MetadataResponse. So I think that's the main
> > reason
> > > > why registration was delayed until after log recovery. Since that
> > > > constraint doesn't exist any more, there seems to be no reason to delay
> > > > registration.
> > > >
> > > > I think delaying registration would have some major downsides. If log
> > > > recovery takes a while, that's a longer window during which someone
> > else
> > > > could register a broker with the same ID. Having parts of the cluster
> > > > missing for a while gives up some of the benefit of separating
> > registration
> > > > from fencing. For example, if a broker somehow gets unregistered and
> > we
> > > > want to re-register it, but we have to wait for a 10 minute log
> > recovery to
> > > > do that, that could be a window during which topics can't be created
> > that
> > > > need to be on that broker, and so forth.
> > > >
> > > > > 230. Regarding MetadataResponse, there is a slight awkwardness. We
> > return
> > > > > rack for each node. However, if that node is for the controller, the
> > rack
> > > > > field is not really relevant. Should we clean it up here or in
> > another
> > > > KIP
> > > > > like KIP-700?
> > > >
> > > > Oh, controllers don't appear in the MetadataResponses returned to
> > clients,
> > > > since clients can't access them. I should have been more clear about
> > that
> > > > in the KIP-- I added a sentence to "Networking" describing this.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Dec 16, 2020 at 4:23 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > On Wed, Dec 16, 2020, at 13:40, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the reply. A few follow up comments.
> > > > > > >
> > > > > > > 211. When does the broker send the BrokerRegistration request to
> > the
> > > > > > > controller? Is it after the recovery phase? If so, at that
> > point, the
> > > > > > > broker has already caught up on the metadata (in order to clean
> > up
> > > > > > deleted
> > > > > > > partitions). Then, it seems we don't need the ShouldFence field
> > > > > > > in BrokerHeartbeatRequest?
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Thanks again for the reviews.
> > > > > >
> > > > > > The broker sends the registration request as soon as it starts
> > up. It
> > > > > > cannot wait until the recovery phase is over since sometimes log
> > > > recovery
> > > > > > can take quite a long time.
> > > > > >
> > > > > > >
> > > > > > > 213. kafka-cluster.sh
> > > > > > > 213.1 For the decommision example, should the command take a
> > broker
> > > > id?
> > > > > >
> > > > > > Yes, the example should have a broker id. Fixed.
> > > > > >
> > > > > > > 213.2 Existing tools use the "--" command line option (e.g.
> > > > kafka-topics
> > > > > > > --list --topic test). Should we follow the same convention
> > > > > > > for kafka-cluster.sh (and kafka-storage.sh)?
> > > > > >
> > > > > > Hmm. I don't think argparse4j supports using double dashes to
> > identify
> > > > > > subcommands. I think it might be confusing as well, since the
> > > > subcommand
> > > > > > must come first, unlike a plain old argument which can be anywhere
> > on
> > > > the
> > > > > > command line.
> > > > > >
> > > > > > > 213.3 Should we add an admin api for broker decommission so that
> > > > this can
> > > > > > > be done programmatically?
> > > > > > >
> > > > > >
> > > > > > Yes, there is an admin client API for decommissioning as well.
> > > > > >
> > > > > > > 220. DecommissionBroker: "NOT_CONTROLLER if the node that the
> > > > request was
> > > > > > > sent to is not the controller". I thought the clients never send
> > a
> > > > > > request
> > > > > > > to the controller directly now and the broker will forward it to
> > the
> > > > > > > controller?
> > > > > > >
> > > > > >
> > > > > > If the controller moved recently, it's possible that the broker
> > could
> > > > send
> > > > > > to a controller that has just recently become inactive. In that
> > case
> > > > > > NOT_CONTROLLER would be returned. (A standby controller returns
> > > > > > NOT_CONTROLLER for most APIs).
> > > > > >
> > > > > > > 221. Could we add the required ACL for the new requests?
> > > > > > >
> > > > > >
> > > > > > Good point. I added the required ACL for each new RPC.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Wed, Dec 16, 2020 at 11:38 AM Colin McCabe <
> > cmccabe@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > On Wed, Dec 16, 2020, at 09:59, Jun Rao wrote:
> > > > > > > > > Hi, Colin,
> > > > > > > > >
> > > > > > > > > Thanks for the reply. A few more comments below.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > Thanks for the comments.
> > > > > > > >
> > > > > > > > > 206. "RemoveTopic is the last step, that scrubs all metadata
> > > > about
> > > > > > the
> > > > > > > > > topic. In order to get to that last step, the topic data
> > needs
> > > > to
> > > > > > > > removed
> > > > > > > > > from all brokers (after each broker notices that the topic is
> > > > being
> > > > > > > > > deleted)." Currently, this is done based on the response of
> > > > > > > > > StopReplicaRequest. Since the controller no longer sends this
> > > > > > request,
> > > > > > > > how
> > > > > > > > > does the controller know that the data for the deleted topic
> > has
> > > > > > > > > been removed in the brokers?
> > > > > > > > >
> > > > > > > >
> > > > > > > > That's a good point... definitely an oversight.
> > > > > > > >
> > > > > > > > It seems complex to force the controller to track when log
> > > > directories
> > > > > > > > have been deleted. Let's just assume that KIP-516 has been
> > > > > > implemented,
> > > > > > > > and track them by UUID. Then we can just have a single topic
> > > > deletion
> > > > > > > > record.
> > > > > > > >
> > > > > > > > I added a section on "topic identifiers" describing this.
> > > > > > > >
> > > > > > > > > 210. Thanks for the explanation. Sounds good to me.
> > > > > > > > >
> > > > > > > > > 211. I still don't see an example when ShouldFence is set to
> > > > true in
> > > > > > > > > BrokerHeartbeatReques. Could we add one?
> > > > > > > > >
> > > > > > > >
> > > > > > > > It is sent to true when the broker is first starting up and
> > > > doesn't yet
> > > > > > > > want to be unfenced. I added a longer explanation of this in
> > the
> > > > > > "Broker
> > > > > > > > Leases" section.
> > > > > > > >
> > > > > > > > > 213. The KIP now allows replicas to be assigned on brokers
> > that
> > > > are
> > > > > > > > fenced,
> > > > > > > > > which is an improvement. How do we permanently remove a
> > broker
> > > > (e.g.
> > > > > > > > > cluster shrinking) to prevent it from being used for future
> > > > replica
> > > > > > > > > assignments?
> > > > > > > > >
> > > > > > > >
> > > > > > > > This is a fair point. I will create a kafka-cluster.sh script
> > > > which
> > > > > > can
> > > > > > > > do this, plus a DecommissionBrokerRequest.
> > > > > > > >
> > > > > > > > As a bonus the kafka-cluster.sh script can help people find the
> > > > > > cluster ID
> > > > > > > > of brokers, something that people have wanted a tool for in the
> > > > past.
> > > > > > > >
> > > > > > > > > 214. "Currently, when a node is down, all of its ZK
> > registration
> > > > > > > > > information is gone. But we need this information in order
> > to
> > > > > > > > understand
> > > > > > > > > things like whether the replicas of a particular partition
> > are
> > > > > > > > > well-balanced across racks." This is not quite true.
> > Currently,
> > > > even
> > > > > > when
> > > > > > > > > ZK registration is gone, the existing replica assignment is
> > still
> > > > > > > > available
> > > > > > > > > in the metadata response.
> > > > > > > > >
> > > > > > > >
> > > > > > > > I agree that the existing replica assignment is still
> > available.
> > > > But
> > > > > > that
> > > > > > > > just tells you partition X is on nodes A, B, and C. If you
> > don't
> > > > have
> > > > > > the
> > > > > > > > ZK registration for one or more of A, B, or C then you don't
> > know
> > > > > > whether
> > > > > > > > we are following the policy of "two replicas on one rack, one
> > > > replica
> > > > > > on
> > > > > > > > another." Or any other more complex rack placement policy
> > that you
> > > > > > might
> > > > > > > > have.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Wed, Dec 16, 2020 at 8:48 AM Colin McCabe <
> > cmccabe@apache.org
> > > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> > > > > > > > > > > Hi, Colin,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the reply. A few more follow up comments.
> > > > > > > > > > >
> > > > > > > > > > > 210. initial.broker.registration.timeout.ms: The default
> > > > value
> > > > > > is
> > > > > > > > 90sec,
> > > > > > > > > > > which seems long. If a broker fails the registration
> > because
> > > > of
> > > > > > > > incorrect
> > > > > > > > > > > configs, we want to fail faster. In comparison, the
> > defaults
> > > > for
> > > > > > > > > > > zookeeper.connection.timeout.ms is 18 secs.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi Jun,
> > > > > > > > > >
> > > > > > > > > > I agree that the initial connection timeout here is longer
> > than
> > > > > > what we
> > > > > > > > > > had with ZK. The main reason I selected a slightly longer
> > > > timeout
> > > > > > > > here is
> > > > > > > > > > to handle the case where the controllers and the brokers
> > are
> > > > > > > > co-located.
> > > > > > > > > > For example, if you have a 3 node cluster and all three
> > nodes
> > > > are
> > > > > > > > > > controllers+brokers, when you first bring up the cluster,
> > we
> > > > will
> > > > > > have
> > > > > > > > to
> > > > > > > > > > stand up the controller quorum and then handle broker
> > > > > > registrations.
> > > > > > > > > > Although we believe Raft will start up relatively quickly,
> > it's
> > > > > > good to
> > > > > > > > > > leave some extra margin here.
> > > > > > > > > >
> > > > > > > > > > I don't think there's a big disadvantage to having a
> > slightly
> > > > > > longer
> > > > > > > > > > timeout here. After all, starting up the brokers with no
> > > > > > controllers
> > > > > > > > is an
> > > > > > > > > > admin mistake, which we don't expect to see very often.
> > Maybe
> > > > > > let's
> > > > > > > > set it
> > > > > > > > > > to 60 seconds for now and maybe see if we can tweak it in
> > the
> > > > > > future if
> > > > > > > > > > that turns out to be too long or short?
> > > > > > > > > >
> > > > > > > > > > > 211. "Once they are all moved, the controller responds
> > to the
> > > > > > > > heartbeat
> > > > > > > > > > > with a nextState of SHUTDOWN." It seems that nextState
> > is no
> > > > > > longer
> > > > > > > > > > > relevant. Also, could you add a bit more explanation on
> > when
> > > > > > > > ShouldFence
> > > > > > > > > > is
> > > > > > > > > > > set to true in BrokerHeartbeatRequest?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Good catch. I removed the obsolete section referring to
> > > > nextState
> > > > > > and
> > > > > > > > > > added a reference to the new boolean. I also added some
> > more
> > > > > > > > information
> > > > > > > > > > about ShouldFence and about the rationale for separating
> > > > fencing
> > > > > > from
> > > > > > > > > > registration.
> > > > > > > > > >
> > > > > > > > > > > 212. Related to the above, what does the broker do if
> > > > IsFenced is
> > > > > > > > true in
> > > > > > > > > > > the BrokerHeartbeatResponse? Will the broker do the same
> > > > thing on
> > > > > > > > > > receiving
> > > > > > > > > > > a FenceBrokerRecord from the metadata log?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The broker only checks this boolean during the startup
> > process.
> > > > > > After
> > > > > > > > the
> > > > > > > > > > startup process is finished, it ignores this boolean.
> > > > > > > > > >
> > > > > > > > > > The broker uses fence / unfence records in the metadata
> > log to
> > > > > > > > determine
> > > > > > > > > > which brokers should appear in its MetadataResponse.
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <
> > > > cmccabe@apache.org
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > >
> > > > > > > > > > > > > The KIP says that "brokers which are fenced will not
> > > > appear
> > > > > > in
> > > > > > > > > > > > > MetadataResponses. So clients that have up-to-date
> > > > metadata
> > > > > > will
> > > > > > > > > > not try
> > > > > > > > > > > > > to contact fenced brokers.", which is fine, but it
> > > > doesn't
> > > > > > seem
> > > > > > > > to
> > > > > > > > > > > > > elaborate on what happens for clients which try to
> > > > contact a
> > > > > > > > broker
> > > > > > > > > > > > (using
> > > > > > > > > > > > > stale metadata, for example) and find it in the
> > > > > > > > registered-but-fenced
> > > > > > > > > > > > > state.
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > >
> > > > > > > > > > > > I have dropped the broker-side fencing from this
> > > > proposal. So
> > > > > > now
> > > > > > > > the
> > > > > > > > > > > > fencing is basically the same as today's fencing: it's
> > > > > > > > controller-side
> > > > > > > > > > > > only. That means that clients using stale metadata can
> > > > contact
> > > > > > > > fenced
> > > > > > > > > > > > brokers and communicate with them. The only case
> > where the
> > > > > > broker
> > > > > > > > > > will be
> > > > > > > > > > > > inaccessible is when it hasn't finished starting yet
> > > > (i.e., has
> > > > > > > > not yet
> > > > > > > > > > > > attained RUNNING state.)
> > > > > > > > > > > >
> > > > > > > > > > > > Just like today, we expect the safeguards built into
> > the
> > > > > > > > replication
> > > > > > > > > > > > protocol to prevent the worst corner cases that could
> > > > result
> > > > > > from
> > > > > > > > > > this. I
> > > > > > > > > > > > do think we will probably take up this issue later and
> > > > find a
> > > > > > > > better
> > > > > > > > > > > > solution for client-side metadata, but this KIP is big
> > > > enough
> > > > > > > > as-is.
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > You said previously that the broker will respond
> > with a
> > > > > > retriable
> > > > > > > > > > > > > error, which is again fine, but you didn't answer my
> > > > question
> > > > > > > > about
> > > > > > > > > > which
> > > > > > > > > > > > > error code you've chosen for this. I realise that
> > this
> > > > > > doesn't
> > > > > > > > really
> > > > > > > > > > > > > affect the correctness of the behaviour, but I'm not
> > > > aware
> > > > > > of any
> > > > > > > > > > > > existing
> > > > > > > > > > > > > error code which is a good fit. So it would be good
> > to
> > > > > > understand
> > > > > > > > > > about
> > > > > > > > > > > > how
> > > > > > > > > > > > > you're making this backward compatible for clients.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Many thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Tom
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <
> > > > > > cmccabe@apache.org
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the reply. Just a couple of more
> > comments
> > > > > > below.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 210. Since we are deprecating
> > > > > > > > zookeeper.connection.timeout.ms,
> > > > > > > > > > > > should we
> > > > > > > > > > > > > > > add a new config to bound the time for a broker
> > to
> > > > > > connect
> > > > > > > > to the
> > > > > > > > > > > > > > > controller during starting up?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Good idea. I added
> > > > initial.broker.registration.timeout.ms
> > > > > > for
> > > > > > > > > > this.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 211. BrokerHeartbeat no longer has the state
> > field
> > > > in the
> > > > > > > > > > > > > > request/response.
> > > > > > > > > > > > > > > However, (a) the controller shutdown section
> > still
> > > > has
> > > > > > "In
> > > > > > > > its
> > > > > > > > > > > > periodic
> > > > > > > > > > > > > > > heartbeats, the broker asks the controller if it
> > can
> > > > > > > > transition
> > > > > > > > > > into
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > SHUTDOWN state. This motivates the controller to
> > > > move
> > > > > > all
> > > > > > > > of the
> > > > > > > > > > > > leaders
> > > > > > > > > > > > > > > off of that broker. Once they are all moved, the
> > > > > > controller
> > > > > > > > > > > > responds to
> > > > > > > > > > > > > > > the heartbeat with a nextState of SHUTDOWN.";
> > (2) the
> > > > > > > > > > description of
> > > > > > > > > > > > > > > BrokerHeartbeat still references currentState and
> > > > > > > > targetState.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks. I've made these sections clearer and
> > removed
> > > > the
> > > > > > > > obsolete
> > > > > > > > > > > > > > references to sending states.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <
> > > > > > > > cmccabe@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for the update. A few more follow up
> > > > comments.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks again for the review.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 100. FailedReplicaRecord: Since this is
> > reported
> > > > by
> > > > > > each
> > > > > > > > > > broker
> > > > > > > > > > > > > > > > > independently, perhaps we could use a more
> > > > concise
> > > > > > > > > > representation
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > a top level broker field, an array of topics,
> > > > which
> > > > > > has
> > > > > > > > an
> > > > > > > > > > array
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > partitions.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The issue is that there is a size limit on the
> > each
> > > > > > record.
> > > > > > > > > > > > Putting
> > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > of the partitions of a log directory into a
> > single
> > > > > > record
> > > > > > > > would
> > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > break that in many cases. Still, we can
> > optimize a
> > > > > > bit by
> > > > > > > > > > having
> > > > > > > > > > > > an
> > > > > > > > > > > > > > array
> > > > > > > > > > > > > > > > of partition IDs, since nearly all the time, we
> > > > have
> > > > > > more
> > > > > > > > than
> > > > > > > > > > one
> > > > > > > > > > > > > > from the
> > > > > > > > > > > > > > > > same topic.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 200. Sounds good. If we remove the
> > broker-side
> > > > > > fencing
> > > > > > > > > > logic, do
> > > > > > > > > > > > we
> > > > > > > > > > > > > > plan
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > still keep FENCED in broker state? Do we
> > plan to
> > > > > > expose
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > > > > > states
> > > > > > > > > > > > > > > > > through the existing BrokerState metric and
> > if
> > > > so,
> > > > > > what
> > > > > > > > are
> > > > > > > > > > the
> > > > > > > > > > > > > > values
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > the new states?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > No, we don't need FENCED any more. I have
> > removed
> > > > it
> > > > > > from
> > > > > > > > the
> > > > > > > > > > KIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The new states are very similar to the current
> > > > ones,
> > > > > > > > actually.
> > > > > > > > > > > > There
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > no new states or removed ones. The main
> > change in
> > > > the
> > > > > > > > broker
> > > > > > > > > > state
> > > > > > > > > > > > > > machine
> > > > > > > > > > > > > > > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN
> > state
> > > > has
> > > > > > been
> > > > > > > > > > > > renamed to
> > > > > > > > > > > > > > > > RECOVERY. Also, unlike previously, the broker
> > will
> > > > > > always
> > > > > > > > pass
> > > > > > > > > > > > through
> > > > > > > > > > > > > > > > RECOVERY (although it may only stay in this
> > state
> > > > for a
> > > > > > > > very
> > > > > > > > > > short
> > > > > > > > > > > > > > amount
> > > > > > > > > > > > > > > > of time).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 201. This may be fine too. Could we document
> > what
> > > > > > happens
> > > > > > > > > > when
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > broker.id/controller.id in
> > metadata.properties
> > > > don't
> > > > > > > > match
> > > > > > > > > > the
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > config when the broker starts up?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I added some documentation about this.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 204. There is still "The highest metadata
> > offset
> > > > > > which
> > > > > > > > the
> > > > > > > > > > broker
> > > > > > > > > > > > > > has not
> > > > > > > > > > > > > > > > > reached" referenced under BrokerRegistration.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 206. Is that separate step needed given
> > KIP-516?
> > > > With
> > > > > > > > > > KIP-516 (
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > > > > > > > > > > > ),
> > > > > > > > > > > > > > > > > we don't need to wait for the topic data to
> > be
> > > > > > removed
> > > > > > > > from
> > > > > > > > > > all
> > > > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > > before removing the topic metadata. The
> > > > combination
> > > > > > of
> > > > > > > > > > unmatching
> > > > > > > > > > > > > > > > > topicId
> > > > > > > > > > > > > > > > > or the missing topicId from the metadata is
> > > > enough
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > clean
> > > > > > > > > > > > > > > > > up deleted topics asynchronously.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It won't be needed once KIP-516 is adopted, but
> > > > this
> > > > > > hasn't
> > > > > > > > > > been
> > > > > > > > > > > > > > > > implemented yet.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <
> > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao
> > wrote:
> > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for the updated KIP. A few more
> > > > comments
> > > > > > > > below.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks again for the reviews.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 80.2 For deprecated configs, we need to
> > > > include
> > > > > > > > > > zookeeper.*
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > broker.id.generation.enable.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Added.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 83.1 If a broker is down, does the
> > controller
> > > > > > keep
> > > > > > > > the
> > > > > > > > > > > > previously
> > > > > > > > > > > > > > > > > > > registered broker epoch forever? If not,
> > how
> > > > long
> > > > > > > > does
> > > > > > > > > > the
> > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > > > > > it? What does the controller do when
> > > > receiving a
> > > > > > > > broker
> > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > > with an unfound broker epoch?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Yes, the controller keeps the previous
> > > > registration
> > > > > > > > > > forever.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Broker heartbeat requests with an incorrect
> > > > broker
> > > > > > > > epoch
> > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > rejected
> > > > > > > > > > > > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 100. Have you figured out if we need to
> > add
> > > > a new
> > > > > > > > record
> > > > > > > > > > > > type for
> > > > > > > > > > > > > > > > > > reporting
> > > > > > > > > > > > > > > > > > > partitions on failed disks?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I added FailedReplicaRecord to reflect the
> > case
> > > > > > where a
> > > > > > > > > > JBOD
> > > > > > > > > > > > > > directory
> > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > failed, leading to failed replicas.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 102. For debugging purposes, sometimes
> > it's
> > > > > > useful to
> > > > > > > > > > read
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > > topic using tools like console-consumer.
> > > > Should
> > > > > > we
> > > > > > > > > > support
> > > > > > > > > > > > that
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > so,
> > > > > > > > > > > > > > > > > > > how?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For now, we have the ability to read the
> > > > metadata
> > > > > > logs
> > > > > > > > > > with the
> > > > > > > > > > > > > > > > dump-logs
> > > > > > > > > > > > > > > > > > tool. I think we will come up with some
> > other
> > > > > > tools
> > > > > > > > in the
> > > > > > > > > > > > future
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > get experience.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 200. "brokers which are fenced will not
> > > > appear in
> > > > > > > > > > > > > > MetadataResponses.
> > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > broker will not respond to these
> > requests--
> > > > > > instead,
> > > > > > > > it
> > > > > > > > > > will
> > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > > disconnect." If the controller is
> > > > partitioned off
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > > > brokers,
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > design will cause every broker to stop
> > > > accepting
> > > > > > new
> > > > > > > > > > client
> > > > > > > > > > > > > > > > requests. In
> > > > > > > > > > > > > > > > > > > contrast, if ZK is partitioned off, the
> > > > existing
> > > > > > > > > > behavior is
> > > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > > brokers can continue to work based on the
> > > > last
> > > > > > known
> > > > > > > > > > > > metadata.
> > > > > > > > > > > > > > So, I
> > > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > sure if we should change the existing
> > > > behavior
> > > > > > > > because
> > > > > > > > > > of the
> > > > > > > > > > > > > > bigger
> > > > > > > > > > > > > > > > > > impact
> > > > > > > > > > > > > > > > > > > in the new one. Another option is to
> > keep the
> > > > > > > > existing
> > > > > > > > > > > > behavior
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > expose
> > > > > > > > > > > > > > > > > > > a metric for fenced brokers so that the
> > > > operator
> > > > > > > > could be
> > > > > > > > > > > > > > alerted.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'm skeptical about how well running
> > without ZK
> > > > > > > > currently
> > > > > > > > > > > > works.
> > > > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > > > I will move the broker-side fencing into a
> > > > > > follow-up
> > > > > > > > KIP.
> > > > > > > > > > This
> > > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > > > already pretty large and there is no hard
> > > > > > dependency on
> > > > > > > > > > this.
> > > > > > > > > > > > > > There
> > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > also be other ways of accomplishing the
> > > > positive
> > > > > > > > effects of
> > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > > broker-side fencing, so more discussion is
> > > > needed.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 201. I read Ron's comment, but I am
> > still not
> > > > > > sure
> > > > > > > > the
> > > > > > > > > > > > benefit of
> > > > > > > > > > > > > > > > keeping
> > > > > > > > > > > > > > > > > > > broker.id and controller.id in
> > > > meta.properties.
> > > > > > It
> > > > > > > > seems
> > > > > > > > > > > > that
> > > > > > > > > > > > > > we are
> > > > > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > duplicating the same info in two places
> > and
> > > > have
> > > > > > the
> > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > burden of
> > > > > > > > > > > > > > > > > > > making sure the values in the two places
> > are
> > > > > > > > consistent.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think the reasoning is that having
> > broker.id
> > > > > > > > protects us
> > > > > > > > > > > > against
> > > > > > > > > > > > > > > > > > accidentally bringing up a broker with a
> > disk
> > > > from
> > > > > > a
> > > > > > > > > > different
> > > > > > > > > > > > > > > > broker. I
> > > > > > > > > > > > > > > > > > don't feel strongly about this but it
> > seemed
> > > > > > simpler to
> > > > > > > > > > keep
> > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 202.
> > controller.connect.security.protocol: Is
> > > > > > this
> > > > > > > > needed
> > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > > controller.listener.names and
> > > > > > > > > > listener.security.protocol.map
> > > > > > > > > > > > > > imply
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > security protocol already?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > You're right, this isn't needed. I'll
> > remove
> > > > it.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 203. registration.heartbeat.interval.ms:
> > It
> > > > > > > > defaults to
> > > > > > > > > > 2k.
> > > > > > > > > > > > ZK
> > > > > > > > > > > > > > uses
> > > > > > > > > > > > > > > > 1/3
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > the session timeout for heartbeat. So,
> > given
> > > > the
> > > > > > > > default
> > > > > > > > > > 18k
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > registration.lease.timeout.ms, should we
> > > > default
> > > > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms to
> > 6k?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 6 seconds seems like a pretty long time
> > between
> > > > > > > > > > heartbeats. It
> > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > useful to know when a broker is missing
> > > > heartbeats,
> > > > > > > > with
> > > > > > > > > > less
> > > > > > > > > > > > time
> > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > that. I provisionally set it to 3 seconds
> > (we
> > > > can
> > > > > > > > always
> > > > > > > > > > > > change
> > > > > > > > > > > > > > > > later...)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I also changed the name of these
> > > > configurations to
> > > > > > "
> > > > > > > > > > > > > > > > > > broker.heartbeat.interval.ms" and "
> > > > > > > > > > > > broker.registration.timeout.ms"
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > > > > to clarify them a bit.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 204. "The highest metadata offset which
> > the
> > > > > > broker
> > > > > > > > has
> > > > > > > > > > not
> > > > > > > > > > > > > > reached."
> > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > seems this should be "has reached".
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I changed this to "one more than the
> > highest
> > > > > > metadata
> > > > > > > > > > offset
> > > > > > > > > > > > which
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > broker has reached."
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 205. UnfenceBrokerRecord and
> > > > > > UnregisterBrokerRecord:
> > > > > > > > To
> > > > > > > > > > me,
> > > > > > > > > > > > they
> > > > > > > > > > > > > > > > seem to
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > the same. Do we need both?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Unregistration means that the broker has
> > been
> > > > > > removed
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > > > cluster.
> > > > > > > > > > > > > > > > > > That is different than unfencing, which
> > marks
> > > > the
> > > > > > > > broker as
> > > > > > > > > > > > active.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 206. TopicRecord: The Deleting field is
> > used
> > > > to
> > > > > > > > indicate
> > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > being deleted. I am wondering if this is
> > > > really
> > > > > > > > needed
> > > > > > > > > > since
> > > > > > > > > > > > > > > > RemoveTopic
> > > > > > > > > > > > > > > > > > > already indicates the same thing.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > RemoveTopic is the last step, that scrubs
> > all
> > > > > > metadata
> > > > > > > > > > about
> > > > > > > > > > > > the
> > > > > > > > > > > > > > topic.
> > > > > > > > > > > > > > > > > > In order to get to that last step, the
> > topic
> > > > data
> > > > > > > > needs to
> > > > > > > > > > > > removed
> > > > > > > > > > > > > > > > from all
> > > > > > > > > > > > > > > > > > brokers (after each broker notices that the
> > > > topic
> > > > > > is
> > > > > > > > being
> > > > > > > > > > > > > > deleted).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin
> > McCabe <
> > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron
> > > > Dagostino
> > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the updates.
> > It's
> > > > now
> > > > > > > > clear
> > > > > > > > > > to me
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > > > > > > keep their broker epoch for the life
> > of
> > > > their
> > > > > > > > JVM --
> > > > > > > > > > they
> > > > > > > > > > > > > > > > register
> > > > > > > > > > > > > > > > > > > > > once, get their broker epoch in the
> > > > > > response, and
> > > > > > > > > > then
> > > > > > > > > > > > never
> > > > > > > > > > > > > > > > > > > > > re-register again. Brokers may get
> > > > fenced,
> > > > > > but
> > > > > > > > they
> > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > > > broker epoch for the life of their
> > JVM.
> > > > The
> > > > > > > > > > incarnation
> > > > > > > > > > > > ID
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > kept for the life of the JVM but is
> > > > > > generated by
> > > > > > > > the
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > > > upon startup, and the combination of
> > the
> > > > two
> > > > > > > > allows
> > > > > > > > > > the
> > > > > > > > > > > > > > > > Controller to
> > > > > > > > > > > > > > > > > > > > > act idempotently if any
> > previously-sent
> > > > > > > > registration
> > > > > > > > > > > > response
> > > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks, Ron. That's a good summary.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > One thing I wonder about is if it
> > might
> > > > be
> > > > > > > > helpful
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > > > send the Cluster ID as determined
> > from
> > > > its
> > > > > > > > > > > > meta.properties
> > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > > > > registration request. Does it even
> > make
> > > > > > sense
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > > > successfully register and enter the
> > > > Fenced
> > > > > > state
> > > > > > > > if
> > > > > > > > > > it
> > > > > > > > > > > > has
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > wrong
> > > > > > > > > > > > > > > > > > > > > Cluster ID?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Yeah, that's a good idea. Let's have
> > the
> > > > > > broker
> > > > > > > > pass
> > > > > > > > > > its
> > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > ID in
> > > > > > > > > > > > > > > > > > > > the registration RPC, and then
> > > > registration can
> > > > > > > > fail
> > > > > > > > > > if the
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The nextMetadatOffset value that the
> > > > broker
> > > > > > > > > > communicates
> > > > > > > > > > > > > > > > > > > > > in its registration request only has
> > > > meaning
> > > > > > > > within
> > > > > > > > > > the
> > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > > cluster, so it feels to me that the
> > > > > > Controller
> > > > > > > > should
> > > > > > > > > > > > have
> > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > perform this sanity check. There is
> > > > > > currently
> > > > > > > > > > (pre-KIP
> > > > > > > > > > > > 500)
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > check
> > > > > > > > > > > > > > > > > > > > > in the broker to make sure its
> > configured
> > > > > > > > cluster ID
> > > > > > > > > > > > matches
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > > > > stored in ZooKeeper, and we will
> > have to
> > > > > > perform
> > > > > > > > this
> > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > > somewhere in the KIP-500 world. If
> > the
> > > > > > > > Controller
> > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > do it
> > > > > > > > > > > > > > > > > > > > > within the registration request then
> > the
> > > > > > broker
> > > > > > > > will
> > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > > metadata request to the Controller,
> > > > retrieve
> > > > > > the
> > > > > > > > > > Cluster
> > > > > > > > > > > > ID,
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > perform the check itself. It feels
> > to me
> > > > > > that it
> > > > > > > > > > might
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > better for
> > > > > > > > > > > > > > > > > > > > > the Controller to just do it, and
> > then
> > > > the
> > > > > > broker
> > > > > > > > > > doesn't
> > > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > > > worry about it anymore once it
> > > > successfully
> > > > > > > > > > registers.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I also have a question about the
> > > > broker.id
> > > > > > > > value and
> > > > > > > > > > > > > > > > > > meta.properties.
> > > > > > > > > > > > > > > > > > > > > The KIP now says "In version 0 of
> > > > > > > > meta.properties,
> > > > > > > > > > there
> > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > > > broker.id field. Version 1 does not
> > > > have
> > > > > > this
> > > > > > > > > > field.
> > > > > > > > > > > > It
> > > > > > > > > > > > > > is no
> > > > > > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > > > > > > needed because we no longer support
> > > > dynamic
> > > > > > > > broker id
> > > > > > > > > > > > > > > > assignment."
> > > > > > > > > > > > > > > > > > > > > But then there is an example version
> > 1
> > > > > > > > > > meta.properties
> > > > > > > > > > > > file
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > shows
> > > > > > > > > > > > > > > > > > > > > the broker.id value. I actually
> > wonder
> > > > if
> > > > > > > > maybe the
> > > > > > > > > > > > > > broker.id
> > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > would be good to keep in the version
> > 1
> > > > > > > > > > meta.properties
> > > > > > > > > > > > file
> > > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > currently (pre-KIP 500, version 0)
> > acts
> > > > as a
> > > > > > > > sanity
> > > > > > > > > > > > check to
> > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > sure
> > > > > > > > > > > > > > > > > > > > > the broker is using the correct log
> > > > > > directory.
> > > > > > > > > > Similarly
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > controller.id value on controllers
> > -- it
> > > > > > would
> > > > > > > > > > allow the
> > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > sanity check for quorum controllers.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > That's a good point. I will add
> > broker.id
> > > > > > back,
> > > > > > > > and
> > > > > > > > > > also
> > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > controller.id as a possibility.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin
> > > > McCabe
> > > > > > <
> > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10,
> > Jun Rao
> > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for the reply. A few more
> > > > > > comments.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks again for the reply. Sorry
> > for
> > > > the
> > > > > > long
> > > > > > > > > > > > hiatus. I
> > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > vacation for a while.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 55. There is still text that
> > favors
> > > > new
> > > > > > > > broker
> > > > > > > > > > > > > > registration.
> > > > > > > > > > > > > > > > > > "When a
> > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > first starts up, when it is in
> > the
> > > > > > INITIAL
> > > > > > > > > > state, it
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > > always
> > > > > > > > > > > > > > > > > > > > "win"
> > > > > > > > > > > > > > > > > > > > > > > broker ID conflicts. However,
> > once
> > > > it is
> > > > > > > > > > granted a
> > > > > > > > > > > > > > lease, it
> > > > > > > > > > > > > > > > > > > > transitions
> > > > > > > > > > > > > > > > > > > > > > > out of the INITIAL state.
> > > > Thereafter,
> > > > > > it may
> > > > > > > > > > lose
> > > > > > > > > > > > > > subsequent
> > > > > > > > > > > > > > > > > > > > conflicts if
> > > > > > > > > > > > > > > > > > > > > > > its broker epoch is stale. (See
> > > > KIP-380
> > > > > > for
> > > > > > > > some
> > > > > > > > > > > > > > background
> > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > epoch.) The reason for favoring
> > new
> > > > > > > > processes
> > > > > > > > > > is to
> > > > > > > > > > > > > > > > accommodate
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > > > > > > > case where a process is killed
> > with
> > > > kill
> > > > > > -9
> > > > > > > > and
> > > > > > > > > > then
> > > > > > > > > > > > > > > > restarted.
> > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > want it
> > > > > > > > > > > > > > > > > > > > > > > to be able to reclaim its old ID
> > > > quickly
> > > > > > in
> > > > > > > > this
> > > > > > > > > > > > case."
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for the reminder. I have
> > > > clarified
> > > > > > the
> > > > > > > > > > language
> > > > > > > > > > > > > > here.
> > > > > > > > > > > > > > > > > > > > Hopefully now it is clear that we don't
> > > > allow
> > > > > > quick
> > > > > > > > > > re-use
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 80.1 Sounds good. Could you
> > document
> > > > that
> > > > > > > > > > listeners
> > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > > > > now? It would also be useful to
> > > > annotate
> > > > > > > > other
> > > > > > > > > > > > required
> > > > > > > > > > > > > > > > configs.
> > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > > > > example, controller.connect
> > should be
> > > > > > > > required.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I added a note specifying that
> > these
> > > > are
> > > > > > > > required.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 80.2 Could you list all
> > deprecated
> > > > > > existing
> > > > > > > > > > configs?
> > > > > > > > > > > > > > Another
> > > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > control.plane.listener.name
> > since
> > > > the
> > > > > > > > > > controller no
> > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > sends
> > > > > > > > > > > > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata and
> > > > > > StopReplica
> > > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I added a section specifying some
> > > > > > deprecated
> > > > > > > > > > configs.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 83.1 It seems that the broker can
> > > > > > transition
> > > > > > > > from
> > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > RUNNING
> > > > > > > > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > > > > > > > registering for a new broker
> > epoch.
> > > > I am
> > > > > > not
> > > > > > > > > > sure how
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > works.
> > > > > > > > > > > > > > > > > > > > Once the
> > > > > > > > > > > > > > > > > > > > > > > controller fences a broker,
> > there is
> > > > no
> > > > > > need
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > > > > > > > > boker epoch around. So, if the
> > fenced
> > > > > > > > broker's
> > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > > > > > existing broker epoch will be
> > > > rejected,
> > > > > > > > leading
> > > > > > > > > > the
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The broker epoch refers to the
> > broker
> > > > > > > > registration.
> > > > > > > > > > > > So we
> > > > > > > > > > > > > > DO
> > > > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > broker epoch around even while the
> > broker
> > > > is
> > > > > > > > fenced.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The broker epoch changes only when
> > > > there
> > > > > > is a
> > > > > > > > new
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > registration. Fencing or unfencing the
> > > > broker
> > > > > > > > doesn't
> > > > > > > > > > > > change
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 83.5 Good point on KIP-590. Then
> > > > should
> > > > > > we
> > > > > > > > > > expose the
> > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > debugging purposes? If not, we
> > should
> > > > > > > > deprecate
> > > > > > > > > > the
> > > > > > > > > > > > > > > > controllerID
> > > > > > > > > > > > > > > > > > > > field in
> > > > > > > > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think it's OK to expose it for
> > now,
> > > > with
> > > > > > the
> > > > > > > > > > proviso
> > > > > > > > > > > > > > that it
> > > > > > > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > > > > > > be reachable by clients.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 90. We rejected the shared ID
> > with
> > > > just
> > > > > > one
> > > > > > > > > > reason
> > > > > > > > > > > > "This
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > not a
> > > > > > > > > > > > > > > > > > > > good idea
> > > > > > > > > > > > > > > > > > > > > > > because NetworkClient assumes a
> > > > single ID
> > > > > > > > > > space. So
> > > > > > > > > > > > if
> > > > > > > > > > > > > > > > there is
> > > > > > > > > > > > > > > > > > > > both a
> > > > > > > > > > > > > > > > > > > > > > > controller 1 and a broker 1, we
> > don't
> > > > > > have a
> > > > > > > > way
> > > > > > > > > > of
> > > > > > > > > > > > > > picking
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > "right"
> > > > > > > > > > > > > > > > > > > > > > > one." This doesn't seem to be a
> > > > strong
> > > > > > > > reason.
> > > > > > > > > > For
> > > > > > > > > > > > > > example,
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > > > > > > address the NetworkClient issue
> > with
> > > > the
> > > > > > node
> > > > > > > > > > type
> > > > > > > > > > > > as you
> > > > > > > > > > > > > > > > pointed
> > > > > > > > > > > > > > > > > > > > out or
> > > > > > > > > > > > > > > > > > > > > > > using the negative value of a
> > broker
> > > > ID
> > > > > > as
> > > > > > > > the
> > > > > > > > > > > > > > controller ID.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > It would require a lot of code
> > changes
> > > > to
> > > > > > > > support
> > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > types of
> > > > > > > > > > > > > > > > > > > > node IDs. It's not clear to me that
> > the
> > > > end
> > > > > > result
> > > > > > > > > > would
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > better --
> > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > tend to think it would be worse, since
> > it
> > > > > > would be
> > > > > > > > more
> > > > > > > > > > > > > > complex.
> > > > > > > > > > > > > > > > In a
> > > > > > > > > > > > > > > > > > > > similar vein, using negative numbers
> > seems
> > > > > > > > dangerous,
> > > > > > > > > > > > since we
> > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > negatives or -1 as "special values" in
> > many
> > > > > > places.
> > > > > > > > > > For
> > > > > > > > > > > > > > example,
> > > > > > > > > > > > > > > > -1
> > > > > > > > > > > > > > > > > > often
> > > > > > > > > > > > > > > > > > > > represents "no such node."
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > One important thing to keep in
> > mind is
> > > > > > that we
> > > > > > > > > > want to
> > > > > > > > > > > > be
> > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > transition from a broker and a
> > controller
> > > > being
> > > > > > > > > > co-located
> > > > > > > > > > > > to
> > > > > > > > > > > > > > them
> > > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > > > > longer being co-located. This is much
> > > > easier
> > > > > > to do
> > > > > > > > > > when
> > > > > > > > > > > > they
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > > > > the broker reports all offline
> > > > replicas
> > > > > > due
> > > > > > > > to a
> > > > > > > > > > disk
> > > > > > > > > > > > > > > > failure to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > controller. It seems this
> > information
> > > > > > needs
> > > > > > > > to be
> > > > > > > > > > > > > > persisted
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > > > > > > log. Do we have a corresponding
> > > > record
> > > > > > for
> > > > > > > > that?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hmm, I have to look into this a
> > little
> > > > bit
> > > > > > > > more.
> > > > > > > > > > We
> > > > > > > > > > > > may
> > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > > > > > record type.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 101. Currently, StopReplica
> > request
> > > > has 2
> > > > > > > > modes,
> > > > > > > > > > > > without
> > > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > > > and with
> > > > > > > > > > > > > > > > > > > > > > > deletion. The former is used for
> > > > > > controlled
> > > > > > > > > > shutdown
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > handling
> > > > > > > > > > > > > > > > > > > > disk
> > > > > > > > > > > > > > > > > > > > > > > failure, and causes the follower
> > to
> > > > > > stop. The
> > > > > > > > > > latter
> > > > > > > > > > > > is
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > > > > > > and partition reassignment, and
> > > > causes
> > > > > > the
> > > > > > > > > > replica
> > > > > > > > > > > > to be
> > > > > > > > > > > > > > > > deleted.
> > > > > > > > > > > > > > > > > > > > Since we
> > > > > > > > > > > > > > > > > > > > > > > are deprecating StopReplica,
> > could we
> > > > > > > > document
> > > > > > > > > > what
> > > > > > > > > > > > > > triggers
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > stopping
> > > > > > > > > > > > > > > > > > > > > > > of a follower and the deleting
> > of a
> > > > > > replica
> > > > > > > > now?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > RemoveTopic triggers deletion. In
> > > > general
> > > > > > the
> > > > > > > > > > > > > > functionality of
> > > > > > > > > > > > > > > > > > > > StopReplica is subsumed by the metadata
> > > > > > records.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 102. Should we include the
> > metadata
> > > > > > topic in
> > > > > > > > the
> > > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > > > If so,
> > > > > > > > > > > > > > > > > > > > > > > when it will be included and what
> > > > will
> > > > > > the
> > > > > > > > > > metadata
> > > > > > > > > > > > > > response
> > > > > > > > > > > > > > > > look
> > > > > > > > > > > > > > > > > > > > like?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > No, it won't be included in the
> > > > metadata
> > > > > > > > response
> > > > > > > > > > sent
> > > > > > > > > > > > back
> > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > brokers.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 103. "The active controller
> > assigns
> > > > the
> > > > > > > > broker a
> > > > > > > > > > new
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > epoch,
> > > > > > > > > > > > > > > > > > > > based on
> > > > > > > > > > > > > > > > > > > > > > > the latest committed offset in
> > the
> > > > log."
> > > > > > This
> > > > > > > > > > seems
> > > > > > > > > > > > > > > > inaccurate
> > > > > > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > latest committed offset doesn't
> > > > always
> > > > > > > > advance on
> > > > > > > > > > > > every
> > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > append.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Given that the new broker epoch
> > won't
> > > > be
> > > > > > > > visible
> > > > > > > > > > until
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > commit
> > > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > happened, I have changed this to "the
> > next
> > > > > > > > available
> > > > > > > > > > > > offset in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > log"
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 104. REGISTERING(1) : It says
> > > > > > "Otherwise, the
> > > > > > > > > > broker
> > > > > > > > > > > > > > moves
> > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > > > > > > > > > state.". It seems this should be
> > > > RUNNING?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 105. RUNNING: Should we require
> > the
> > > > > > broker to
> > > > > > > > > > catch
> > > > > > > > > > > > up
> > > > > > > > > > > > > > to the
> > > > > > > > > > > > > > > > > > > > metadata log
> > > > > > > > > > > > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > For 104 and 105, these sections
> > have
> > > > been
> > > > > > > > reworked.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM
> > Colin
> > > > > > McCabe
> > > > > > > > <
> > > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020, at
> > 05:51, Tom
> > > > > > Bentley
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at
> > 08:59,
> > > > Ron
> > > > > > > > Dagostino
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the
> > > > hard
> > > > > > work
> > > > > > > > on
> > > > > > > > > > this
> > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I have some questions
> > about
> > > > what
> > > > > > > > happens
> > > > > > > > > > to a
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > when it
> > > > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > > > > > > fenced (e.g. because it
> > can't
> > > > > > send a
> > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > request to
> > > > > > > > > > > > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > > > > > > > > > > > lease). The KIP says
> > "When a
> > > > > > broker
> > > > > > > > is
> > > > > > > > > > > > fenced,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > cannot
> > > > > > > > > > > > > > > > > > > > process any
> > > > > > > > > > > > > > > > > > > > > > > > > > > client requests. This
> > > > prevents
> > > > > > > > brokers
> > > > > > > > > > which
> > > > > > > > > > > > > > are not
> > > > > > > > > > > > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > > > > > > > > > > > metadata updates or that
> > are
> > > > not
> > > > > > > > > > receiving
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > processing
> > > > > > > > > > > > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > > > > > > > > > > > enough from causing
> > issues to
> > > > > > > > clients."
> > > > > > > > > > And
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > > > > > > > > > > > FENCED(4) state it
> > likewise
> > > > says
> > > > > > > > "While
> > > > > > > > > > in
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > > > > > > > not respond to client
> > > > > > requests." It
> > > > > > > > > > makes
> > > > > > > > > > > > sense
> > > > > > > > > > > > > > > > that a
> > > > > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > > > > > > should not accept
> > producer
> > > > > > requests
> > > > > > > > -- I
> > > > > > > > > > > > assume
> > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > > > > > > > > > > would result in
> > > > > > > > > > NotLeaderOrFollowerException.
> > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > > > > > > > > > > > (fetch from follower)
> > > > consumer
> > > > > > > > > > requests? It
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > conceivable
> > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > > > > > > could continue. Related
> > to
> > > > that,
> > > > > > > > would a
> > > > > > > > > > > > fenced
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > > > > > > > fetch data for partitions
> > > > where
> > > > > > it
> > > > > > > > > > thinks it
> > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > follower?
> > > > > > > > > > > > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > > > > > > > > > > > it rejects consumer
> > requests
> > > > it
> > > > > > might
> > > > > > > > > > still
> > > > > > > > > > > > > > continue
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > > > > > > > > > > > follower. Might it be
> > > > helpful to
> > > > > > > > clarify
> > > > > > > > > > > > both
> > > > > > > > > > > > > > > > decisions
> > > > > > > > > > > > > > > > > > > > here?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Good question. I think a
> > > > fenced
> > > > > > broker
> > > > > > > > > > should
> > > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > > > > > > > > > > > partitions it was already
> > > > fetching
> > > > > > > > before
> > > > > > > > > > it
> > > > > > > > > > > > was
> > > > > > > > > > > > > > > > fenced,
> > > > > > > > > > > > > > > > > > > > unless it
> > > > > > > > > > > > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > > > > > > > > > > > problem. At that point it
> > > > won't be
> > > > > > > > able to
> > > > > > > > > > > > > > continue,
> > > > > > > > > > > > > > > > > > since it
> > > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > > > > the new metadata. For
> > > > example, it
> > > > > > > > won't
> > > > > > > > > > know
> > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > leadership
> > > > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > the partitions it's
> > fetching.
> > > > The
> > > > > > > > > > rationale
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > continuing to
> > > > > > > > > > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > > > > > > > > try to avoid disruptions as
> > > > much as
> > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I don't think fenced
> > brokers
> > > > should
> > > > > > > > accept
> > > > > > > > > > > > client
> > > > > > > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > > > > > The issue
> > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > that the fenced broker may
> > or
> > > > may
> > > > > > not
> > > > > > > > have
> > > > > > > > > > any
> > > > > > > > > > > > > > data it
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > > > > > > > > > > > have. It may or may not
> > have
> > > > > > applied
> > > > > > > > any
> > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > > > it is supposed to have
> > applied.
> > > > > > So it
> > > > > > > > > > could
> > > > > > > > > > > > get
> > > > > > > > > > > > > > pretty
> > > > > > > > > > > > > > > > > > > > confusing, and
> > > > > > > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > > > potentially waste the
> > client's
> > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > When fenced, how would the
> > broker
> > > > > > reply
> > > > > > > > to a
> > > > > > > > > > > > client
> > > > > > > > > > > > > > > > which did
> > > > > > > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The broker will respond with a
> > > > > > retryable
> > > > > > > > error
> > > > > > > > > > in
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > case.
> > > > > > > > > > > > > > > > > > Once
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > client has re-fetched its
> > > > metadata, it
> > > > > > > > will no
> > > > > > > > > > > > longer
> > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > > > as part of the cluster. I
> > added a
> > > > > > note to
> > > > > > > > the
> > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the reply. Sounds good. One more comment.
231. Currently, we have sasl.mechanism.inter.broker.protocol for inter
broker communication. It seems that we need a similar config for specifying
the sasl mechanism for the communication between the broker and the
controller.
Jun
On Thu, Dec 17, 2020 at 2:29 PM Colin McCabe <cm...@apache.org> wrote:
> On Thu, Dec 17, 2020, at 10:19, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the reply.
> >
> > 211. Hmm, I still don't see a clear benefit of registering a broker
> before
> > recovery. It's possible for the recovery to take time. However, during
> the
> > recovery mode, it seems the broker will still be in the fenced mode and
> > won't be able to do work for the clients. So, registering and
> heartbeating
> > early seems to only add unnecessary overhead. For your point on topic
> > creation, I thought we now allow replicas to be created on
> > unregistered brokers.
> >
>
> Hi Jun,
>
> Thanks again for the reviews.
>
> Topics cannot be created on unregistered brokers. They can be created on
> registered but fenced brokers. So for that reason I think it makes sense
> to register as early as possible.
>
> > 230. Currently, we do have a ControllerId field in MetadataResponse. In
> the
> > early discussion, I thought that we want to expose the controller for
> > debugging purposes, but not used by the client library.
> >
>
> The current plan is that we will expose the controller node ID, but the
> controller will not be included in the list of nodes in the metadata
> response.
>
> It's not really possible to include the controller in that list of nodes
> because the controller may not share the same set of listeners as the
> broker. So, for example, maybe the controller endpoint is using a
> different type of security than the broker. So while we could pass back a
> hostname and port, the client would have no way to connect since it doesn't
> know what security settings to use.
>
> regards,
> Colin
>
> > Jun
> >
> > On Wed, Dec 16, 2020 at 9:13 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Wed, Dec 16, 2020, at 18:13, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply. Just a couple of more comments.
> > > >
> > > > 211. Currently, the broker only registers itself in ZK after log
> > > recovery.
> > > > Is there any benefit to change that? As you mentioned, the broker
> can't
> > > do
> > > > much before completing log recovery.
> > > >
> > >
> > > Hi Jun,
> > >
> > > Previously, it wasn't possible to register in ZK without immediately
> > > getting added to the MetadataResponse. So I think that's the main
> reason
> > > why registration was delayed until after log recovery. Since that
> > > constraint doesn't exist any more, there seems to be no reason to delay
> > > registration.
> > >
> > > I think delaying registration would have some major downsides. If log
> > > recovery takes a while, that's a longer window during which someone
> else
> > > could register a broker with the same ID. Having parts of the cluster
> > > missing for a while gives up some of the benefit of separating
> registration
> > > from fencing. For example, if a broker somehow gets unregistered and
> we
> > > want to re-register it, but we have to wait for a 10 minute log
> recovery to
> > > do that, that could be a window during which topics can't be created
> that
> > > need to be on that broker, and so forth.
> > >
> > > > 230. Regarding MetadataResponse, there is a slight awkwardness. We
> return
> > > > rack for each node. However, if that node is for the controller, the
> rack
> > > > field is not really relevant. Should we clean it up here or in
> another
> > > KIP
> > > > like KIP-700?
> > >
> > > Oh, controllers don't appear in the MetadataResponses returned to
> clients,
> > > since clients can't access them. I should have been more clear about
> that
> > > in the KIP-- I added a sentence to "Networking" describing this.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Dec 16, 2020 at 4:23 PM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > On Wed, Dec 16, 2020, at 13:40, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the reply. A few follow up comments.
> > > > > >
> > > > > > 211. When does the broker send the BrokerRegistration request to
> the
> > > > > > controller? Is it after the recovery phase? If so, at that
> point, the
> > > > > > broker has already caught up on the metadata (in order to clean
> up
> > > > > deleted
> > > > > > partitions). Then, it seems we don't need the ShouldFence field
> > > > > > in BrokerHeartbeatRequest?
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > Thanks again for the reviews.
> > > > >
> > > > > The broker sends the registration request as soon as it starts
> up. It
> > > > > cannot wait until the recovery phase is over since sometimes log
> > > recovery
> > > > > can take quite a long time.
> > > > >
> > > > > >
> > > > > > 213. kafka-cluster.sh
> > > > > > 213.1 For the decommision example, should the command take a
> broker
> > > id?
> > > > >
> > > > > Yes, the example should have a broker id. Fixed.
> > > > >
> > > > > > 213.2 Existing tools use the "--" command line option (e.g.
> > > kafka-topics
> > > > > > --list --topic test). Should we follow the same convention
> > > > > > for kafka-cluster.sh (and kafka-storage.sh)?
> > > > >
> > > > > Hmm. I don't think argparse4j supports using double dashes to
> identify
> > > > > subcommands. I think it might be confusing as well, since the
> > > subcommand
> > > > > must come first, unlike a plain old argument which can be anywhere
> on
> > > the
> > > > > command line.
> > > > >
> > > > > > 213.3 Should we add an admin api for broker decommission so that
> > > this can
> > > > > > be done programmatically?
> > > > > >
> > > > >
> > > > > Yes, there is an admin client API for decommissioning as well.
> > > > >
> > > > > > 220. DecommissionBroker: "NOT_CONTROLLER if the node that the
> > > request was
> > > > > > sent to is not the controller". I thought the clients never send
> a
> > > > > request
> > > > > > to the controller directly now and the broker will forward it to
> the
> > > > > > controller?
> > > > > >
> > > > >
> > > > > If the controller moved recently, it's possible that the broker
> could
> > > send
> > > > > to a controller that has just recently become inactive. In that
> case
> > > > > NOT_CONTROLLER would be returned. (A standby controller returns
> > > > > NOT_CONTROLLER for most APIs).
> > > > >
> > > > > > 221. Could we add the required ACL for the new requests?
> > > > > >
> > > > >
> > > > > Good point. I added the required ACL for each new RPC.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, Dec 16, 2020 at 11:38 AM Colin McCabe <
> cmccabe@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > On Wed, Dec 16, 2020, at 09:59, Jun Rao wrote:
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the reply. A few more comments below.
> > > > > > > >
> > > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > Thanks for the comments.
> > > > > > >
> > > > > > > > 206. "RemoveTopic is the last step, that scrubs all metadata
> > > about
> > > > > the
> > > > > > > > topic. In order to get to that last step, the topic data
> needs
> > > to
> > > > > > > removed
> > > > > > > > from all brokers (after each broker notices that the topic is
> > > being
> > > > > > > > deleted)." Currently, this is done based on the response of
> > > > > > > > StopReplicaRequest. Since the controller no longer sends this
> > > > > request,
> > > > > > > how
> > > > > > > > does the controller know that the data for the deleted topic
> has
> > > > > > > > been removed in the brokers?
> > > > > > > >
> > > > > > >
> > > > > > > That's a good point... definitely an oversight.
> > > > > > >
> > > > > > > It seems complex to force the controller to track when log
> > > directories
> > > > > > > have been deleted. Let's just assume that KIP-516 has been
> > > > > implemented,
> > > > > > > and track them by UUID. Then we can just have a single topic
> > > deletion
> > > > > > > record.
> > > > > > >
> > > > > > > I added a section on "topic identifiers" describing this.
> > > > > > >
> > > > > > > > 210. Thanks for the explanation. Sounds good to me.
> > > > > > > >
> > > > > > > > 211. I still don't see an example when ShouldFence is set to
> > > true in
> > > > > > > > BrokerHeartbeatReques. Could we add one?
> > > > > > > >
> > > > > > >
> > > > > > > It is sent to true when the broker is first starting up and
> > > doesn't yet
> > > > > > > want to be unfenced. I added a longer explanation of this in
> the
> > > > > "Broker
> > > > > > > Leases" section.
> > > > > > >
> > > > > > > > 213. The KIP now allows replicas to be assigned on brokers
> that
> > > are
> > > > > > > fenced,
> > > > > > > > which is an improvement. How do we permanently remove a
> broker
> > > (e.g.
> > > > > > > > cluster shrinking) to prevent it from being used for future
> > > replica
> > > > > > > > assignments?
> > > > > > > >
> > > > > > >
> > > > > > > This is a fair point. I will create a kafka-cluster.sh script
> > > which
> > > > > can
> > > > > > > do this, plus a DecommissionBrokerRequest.
> > > > > > >
> > > > > > > As a bonus the kafka-cluster.sh script can help people find the
> > > > > cluster ID
> > > > > > > of brokers, something that people have wanted a tool for in the
> > > past.
> > > > > > >
> > > > > > > > 214. "Currently, when a node is down, all of its ZK
> registration
> > > > > > > > information is gone. But we need this information in order
> to
> > > > > > > understand
> > > > > > > > things like whether the replicas of a particular partition
> are
> > > > > > > > well-balanced across racks." This is not quite true.
> Currently,
> > > even
> > > > > when
> > > > > > > > ZK registration is gone, the existing replica assignment is
> still
> > > > > > > available
> > > > > > > > in the metadata response.
> > > > > > > >
> > > > > > >
> > > > > > > I agree that the existing replica assignment is still
> available.
> > > But
> > > > > that
> > > > > > > just tells you partition X is on nodes A, B, and C. If you
> don't
> > > have
> > > > > the
> > > > > > > ZK registration for one or more of A, B, or C then you don't
> know
> > > > > whether
> > > > > > > we are following the policy of "two replicas on one rack, one
> > > replica
> > > > > on
> > > > > > > another." Or any other more complex rack placement policy
> that you
> > > > > might
> > > > > > > have.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Wed, Dec 16, 2020 at 8:48 AM Colin McCabe <
> cmccabe@apache.org
> > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> > > > > > > > > > Hi, Colin,
> > > > > > > > > >
> > > > > > > > > > Thanks for the reply. A few more follow up comments.
> > > > > > > > > >
> > > > > > > > > > 210. initial.broker.registration.timeout.ms: The default
> > > value
> > > > > is
> > > > > > > 90sec,
> > > > > > > > > > which seems long. If a broker fails the registration
> because
> > > of
> > > > > > > incorrect
> > > > > > > > > > configs, we want to fail faster. In comparison, the
> defaults
> > > for
> > > > > > > > > > zookeeper.connection.timeout.ms is 18 secs.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > I agree that the initial connection timeout here is longer
> than
> > > > > what we
> > > > > > > > > had with ZK. The main reason I selected a slightly longer
> > > timeout
> > > > > > > here is
> > > > > > > > > to handle the case where the controllers and the brokers
> are
> > > > > > > co-located.
> > > > > > > > > For example, if you have a 3 node cluster and all three
> nodes
> > > are
> > > > > > > > > controllers+brokers, when you first bring up the cluster,
> we
> > > will
> > > > > have
> > > > > > > to
> > > > > > > > > stand up the controller quorum and then handle broker
> > > > > registrations.
> > > > > > > > > Although we believe Raft will start up relatively quickly,
> it's
> > > > > good to
> > > > > > > > > leave some extra margin here.
> > > > > > > > >
> > > > > > > > > I don't think there's a big disadvantage to having a
> slightly
> > > > > longer
> > > > > > > > > timeout here. After all, starting up the brokers with no
> > > > > controllers
> > > > > > > is an
> > > > > > > > > admin mistake, which we don't expect to see very often.
> Maybe
> > > > > let's
> > > > > > > set it
> > > > > > > > > to 60 seconds for now and maybe see if we can tweak it in
> the
> > > > > future if
> > > > > > > > > that turns out to be too long or short?
> > > > > > > > >
> > > > > > > > > > 211. "Once they are all moved, the controller responds
> to the
> > > > > > > heartbeat
> > > > > > > > > > with a nextState of SHUTDOWN." It seems that nextState
> is no
> > > > > longer
> > > > > > > > > > relevant. Also, could you add a bit more explanation on
> when
> > > > > > > ShouldFence
> > > > > > > > > is
> > > > > > > > > > set to true in BrokerHeartbeatRequest?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Good catch. I removed the obsolete section referring to
> > > nextState
> > > > > and
> > > > > > > > > added a reference to the new boolean. I also added some
> more
> > > > > > > information
> > > > > > > > > about ShouldFence and about the rationale for separating
> > > fencing
> > > > > from
> > > > > > > > > registration.
> > > > > > > > >
> > > > > > > > > > 212. Related to the above, what does the broker do if
> > > IsFenced is
> > > > > > > true in
> > > > > > > > > > the BrokerHeartbeatResponse? Will the broker do the same
> > > thing on
> > > > > > > > > receiving
> > > > > > > > > > a FenceBrokerRecord from the metadata log?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > The broker only checks this boolean during the startup
> process.
> > > > > After
> > > > > > > the
> > > > > > > > > startup process is finished, it ignores this boolean.
> > > > > > > > >
> > > > > > > > > The broker uses fence / unfence records in the metadata
> log to
> > > > > > > determine
> > > > > > > > > which brokers should appear in its MetadataResponse.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <
> > > cmccabe@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > >
> > > > > > > > > > > > The KIP says that "brokers which are fenced will not
> > > appear
> > > > > in
> > > > > > > > > > > > MetadataResponses. So clients that have up-to-date
> > > metadata
> > > > > will
> > > > > > > > > not try
> > > > > > > > > > > > to contact fenced brokers.", which is fine, but it
> > > doesn't
> > > > > seem
> > > > > > > to
> > > > > > > > > > > > elaborate on what happens for clients which try to
> > > contact a
> > > > > > > broker
> > > > > > > > > > > (using
> > > > > > > > > > > > stale metadata, for example) and find it in the
> > > > > > > registered-but-fenced
> > > > > > > > > > > > state.
> > > > > > > > > > >
> > > > > > > > > > > Hi Tom,
> > > > > > > > > > >
> > > > > > > > > > > I have dropped the broker-side fencing from this
> > > proposal. So
> > > > > now
> > > > > > > the
> > > > > > > > > > > fencing is basically the same as today's fencing: it's
> > > > > > > controller-side
> > > > > > > > > > > only. That means that clients using stale metadata can
> > > contact
> > > > > > > fenced
> > > > > > > > > > > brokers and communicate with them. The only case
> where the
> > > > > broker
> > > > > > > > > will be
> > > > > > > > > > > inaccessible is when it hasn't finished starting yet
> > > (i.e., has
> > > > > > > not yet
> > > > > > > > > > > attained RUNNING state.)
> > > > > > > > > > >
> > > > > > > > > > > Just like today, we expect the safeguards built into
> the
> > > > > > > replication
> > > > > > > > > > > protocol to prevent the worst corner cases that could
> > > result
> > > > > from
> > > > > > > > > this. I
> > > > > > > > > > > do think we will probably take up this issue later and
> > > find a
> > > > > > > better
> > > > > > > > > > > solution for client-side metadata, but this KIP is big
> > > enough
> > > > > > > as-is.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > You said previously that the broker will respond
> with a
> > > > > retriable
> > > > > > > > > > > > error, which is again fine, but you didn't answer my
> > > question
> > > > > > > about
> > > > > > > > > which
> > > > > > > > > > > > error code you've chosen for this. I realise that
> this
> > > > > doesn't
> > > > > > > really
> > > > > > > > > > > > affect the correctness of the behaviour, but I'm not
> > > aware
> > > > > of any
> > > > > > > > > > > existing
> > > > > > > > > > > > error code which is a good fit. So it would be good
> to
> > > > > understand
> > > > > > > > > about
> > > > > > > > > > > how
> > > > > > > > > > > > you're making this backward compatible for clients.
> > > > > > > > > > > >
> > > > > > > > > > > > Many thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Tom
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <
> > > > > cmccabe@apache.org
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the reply. Just a couple of more
> comments
> > > > > below.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 210. Since we are deprecating
> > > > > > > zookeeper.connection.timeout.ms,
> > > > > > > > > > > should we
> > > > > > > > > > > > > > add a new config to bound the time for a broker
> to
> > > > > connect
> > > > > > > to the
> > > > > > > > > > > > > > controller during starting up?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Good idea. I added
> > > initial.broker.registration.timeout.ms
> > > > > for
> > > > > > > > > this.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 211. BrokerHeartbeat no longer has the state
> field
> > > in the
> > > > > > > > > > > > > request/response.
> > > > > > > > > > > > > > However, (a) the controller shutdown section
> still
> > > has
> > > > > "In
> > > > > > > its
> > > > > > > > > > > periodic
> > > > > > > > > > > > > > heartbeats, the broker asks the controller if it
> can
> > > > > > > transition
> > > > > > > > > into
> > > > > > > > > > > the
> > > > > > > > > > > > > > SHUTDOWN state. This motivates the controller to
> > > move
> > > > > all
> > > > > > > of the
> > > > > > > > > > > leaders
> > > > > > > > > > > > > > off of that broker. Once they are all moved, the
> > > > > controller
> > > > > > > > > > > responds to
> > > > > > > > > > > > > > the heartbeat with a nextState of SHUTDOWN.";
> (2) the
> > > > > > > > > description of
> > > > > > > > > > > > > > BrokerHeartbeat still references currentState and
> > > > > > > targetState.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks. I've made these sections clearer and
> removed
> > > the
> > > > > > > obsolete
> > > > > > > > > > > > > references to sending states.
> > > > > > > > > > > > >
> > > > > > > > > > > > > best,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <
> > > > > > > cmccabe@apache.org
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the update. A few more follow up
> > > comments.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks again for the review.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 100. FailedReplicaRecord: Since this is
> reported
> > > by
> > > > > each
> > > > > > > > > broker
> > > > > > > > > > > > > > > > independently, perhaps we could use a more
> > > concise
> > > > > > > > > representation
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > a top level broker field, an array of topics,
> > > which
> > > > > has
> > > > > > > an
> > > > > > > > > array
> > > > > > > > > > > of
> > > > > > > > > > > > > > > > partitions.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The issue is that there is a size limit on the
> each
> > > > > record.
> > > > > > > > > > > Putting
> > > > > > > > > > > > > all
> > > > > > > > > > > > > > > of the partitions of a log directory into a
> single
> > > > > record
> > > > > > > would
> > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > break that in many cases. Still, we can
> optimize a
> > > > > bit by
> > > > > > > > > having
> > > > > > > > > > > an
> > > > > > > > > > > > > array
> > > > > > > > > > > > > > > of partition IDs, since nearly all the time, we
> > > have
> > > > > more
> > > > > > > than
> > > > > > > > > one
> > > > > > > > > > > > > from the
> > > > > > > > > > > > > > > same topic.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 200. Sounds good. If we remove the
> broker-side
> > > > > fencing
> > > > > > > > > logic, do
> > > > > > > > > > > we
> > > > > > > > > > > > > plan
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > still keep FENCED in broker state? Do we
> plan to
> > > > > expose
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > > > > states
> > > > > > > > > > > > > > > > through the existing BrokerState metric and
> if
> > > so,
> > > > > what
> > > > > > > are
> > > > > > > > > the
> > > > > > > > > > > > > values
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the new states?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > No, we don't need FENCED any more. I have
> removed
> > > it
> > > > > from
> > > > > > > the
> > > > > > > > > KIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The new states are very similar to the current
> > > ones,
> > > > > > > actually.
> > > > > > > > > > > There
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > no new states or removed ones. The main
> change in
> > > the
> > > > > > > broker
> > > > > > > > > state
> > > > > > > > > > > > > machine
> > > > > > > > > > > > > > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN
> state
> > > has
> > > > > been
> > > > > > > > > > > renamed to
> > > > > > > > > > > > > > > RECOVERY. Also, unlike previously, the broker
> will
> > > > > always
> > > > > > > pass
> > > > > > > > > > > through
> > > > > > > > > > > > > > > RECOVERY (although it may only stay in this
> state
> > > for a
> > > > > > > very
> > > > > > > > > short
> > > > > > > > > > > > > amount
> > > > > > > > > > > > > > > of time).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 201. This may be fine too. Could we document
> what
> > > > > happens
> > > > > > > > > when
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > broker.id/controller.id in
> metadata.properties
> > > don't
> > > > > > > match
> > > > > > > > > the
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > config when the broker starts up?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I added some documentation about this.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 204. There is still "The highest metadata
> offset
> > > > > which
> > > > > > > the
> > > > > > > > > broker
> > > > > > > > > > > > > has not
> > > > > > > > > > > > > > > > reached" referenced under BrokerRegistration.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 206. Is that separate step needed given
> KIP-516?
> > > With
> > > > > > > > > KIP-516 (
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > > > > > > > > > > ),
> > > > > > > > > > > > > > > > we don't need to wait for the topic data to
> be
> > > > > removed
> > > > > > > from
> > > > > > > > > all
> > > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > before removing the topic metadata. The
> > > combination
> > > > > of
> > > > > > > > > unmatching
> > > > > > > > > > > > > > > > topicId
> > > > > > > > > > > > > > > > or the missing topicId from the metadata is
> > > enough
> > > > > for
> > > > > > > the
> > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > clean
> > > > > > > > > > > > > > > > up deleted topics asynchronously.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It won't be needed once KIP-516 is adopted, but
> > > this
> > > > > hasn't
> > > > > > > > > been
> > > > > > > > > > > > > > > implemented yet.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <
> > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao
> wrote:
> > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for the updated KIP. A few more
> > > comments
> > > > > > > below.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks again for the reviews.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 80.2 For deprecated configs, we need to
> > > include
> > > > > > > > > zookeeper.*
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > broker.id.generation.enable.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Added.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 83.1 If a broker is down, does the
> controller
> > > > > keep
> > > > > > > the
> > > > > > > > > > > previously
> > > > > > > > > > > > > > > > > > registered broker epoch forever? If not,
> how
> > > long
> > > > > > > does
> > > > > > > > > the
> > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > > > > it? What does the controller do when
> > > receiving a
> > > > > > > broker
> > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > with an unfound broker epoch?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Yes, the controller keeps the previous
> > > registration
> > > > > > > > > forever.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Broker heartbeat requests with an incorrect
> > > broker
> > > > > > > epoch
> > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > > > rejected
> > > > > > > > > > > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 100. Have you figured out if we need to
> add
> > > a new
> > > > > > > record
> > > > > > > > > > > type for
> > > > > > > > > > > > > > > > > reporting
> > > > > > > > > > > > > > > > > > partitions on failed disks?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I added FailedReplicaRecord to reflect the
> case
> > > > > where a
> > > > > > > > > JBOD
> > > > > > > > > > > > > directory
> > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > failed, leading to failed replicas.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 102. For debugging purposes, sometimes
> it's
> > > > > useful to
> > > > > > > > > read
> > > > > > > > > > > the
> > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > topic using tools like console-consumer.
> > > Should
> > > > > we
> > > > > > > > > support
> > > > > > > > > > > that
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > so,
> > > > > > > > > > > > > > > > > > how?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For now, we have the ability to read the
> > > metadata
> > > > > logs
> > > > > > > > > with the
> > > > > > > > > > > > > > > dump-logs
> > > > > > > > > > > > > > > > > tool. I think we will come up with some
> other
> > > > > tools
> > > > > > > in the
> > > > > > > > > > > future
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > get experience.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 200. "brokers which are fenced will not
> > > appear in
> > > > > > > > > > > > > MetadataResponses.
> > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > broker will not respond to these
> requests--
> > > > > instead,
> > > > > > > it
> > > > > > > > > will
> > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > disconnect." If the controller is
> > > partitioned off
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > > > brokers,
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > design will cause every broker to stop
> > > accepting
> > > > > new
> > > > > > > > > client
> > > > > > > > > > > > > > > requests. In
> > > > > > > > > > > > > > > > > > contrast, if ZK is partitioned off, the
> > > existing
> > > > > > > > > behavior is
> > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > brokers can continue to work based on the
> > > last
> > > > > known
> > > > > > > > > > > metadata.
> > > > > > > > > > > > > So, I
> > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > sure if we should change the existing
> > > behavior
> > > > > > > because
> > > > > > > > > of the
> > > > > > > > > > > > > bigger
> > > > > > > > > > > > > > > > > impact
> > > > > > > > > > > > > > > > > > in the new one. Another option is to
> keep the
> > > > > > > existing
> > > > > > > > > > > behavior
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > expose
> > > > > > > > > > > > > > > > > > a metric for fenced brokers so that the
> > > operator
> > > > > > > could be
> > > > > > > > > > > > > alerted.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'm skeptical about how well running
> without ZK
> > > > > > > currently
> > > > > > > > > > > works.
> > > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > > I will move the broker-side fencing into a
> > > > > follow-up
> > > > > > > KIP.
> > > > > > > > > This
> > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > > already pretty large and there is no hard
> > > > > dependency on
> > > > > > > > > this.
> > > > > > > > > > > > > There
> > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > also be other ways of accomplishing the
> > > positive
> > > > > > > effects of
> > > > > > > > > > > what
> > > > > > > > > > > > > > > > > broker-side fencing, so more discussion is
> > > needed.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 201. I read Ron's comment, but I am
> still not
> > > > > sure
> > > > > > > the
> > > > > > > > > > > benefit of
> > > > > > > > > > > > > > > keeping
> > > > > > > > > > > > > > > > > > broker.id and controller.id in
> > > meta.properties.
> > > > > It
> > > > > > > seems
> > > > > > > > > > > that
> > > > > > > > > > > > > we are
> > > > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > duplicating the same info in two places
> and
> > > have
> > > > > the
> > > > > > > > > > > additional
> > > > > > > > > > > > > > > burden of
> > > > > > > > > > > > > > > > > > making sure the values in the two places
> are
> > > > > > > consistent.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think the reasoning is that having
> broker.id
> > > > > > > protects us
> > > > > > > > > > > against
> > > > > > > > > > > > > > > > > accidentally bringing up a broker with a
> disk
> > > from
> > > > > a
> > > > > > > > > different
> > > > > > > > > > > > > > > broker. I
> > > > > > > > > > > > > > > > > don't feel strongly about this but it
> seemed
> > > > > simpler to
> > > > > > > > > keep
> > > > > > > > > > > it.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 202.
> controller.connect.security.protocol: Is
> > > > > this
> > > > > > > needed
> > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > controller.listener.names and
> > > > > > > > > listener.security.protocol.map
> > > > > > > > > > > > > imply
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > security protocol already?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > You're right, this isn't needed. I'll
> remove
> > > it.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 203. registration.heartbeat.interval.ms:
> It
> > > > > > > defaults to
> > > > > > > > > 2k.
> > > > > > > > > > > ZK
> > > > > > > > > > > > > uses
> > > > > > > > > > > > > > > 1/3
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > the session timeout for heartbeat. So,
> given
> > > the
> > > > > > > default
> > > > > > > > > 18k
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > registration.lease.timeout.ms, should we
> > > default
> > > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms to
> 6k?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 6 seconds seems like a pretty long time
> between
> > > > > > > > > heartbeats. It
> > > > > > > > > > > > > might
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > useful to know when a broker is missing
> > > heartbeats,
> > > > > > > with
> > > > > > > > > less
> > > > > > > > > > > time
> > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > that. I provisionally set it to 3 seconds
> (we
> > > can
> > > > > > > always
> > > > > > > > > > > change
> > > > > > > > > > > > > > > later...)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I also changed the name of these
> > > configurations to
> > > > > "
> > > > > > > > > > > > > > > > > broker.heartbeat.interval.ms" and "
> > > > > > > > > > > broker.registration.timeout.ms"
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > > > to clarify them a bit.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 204. "The highest metadata offset which
> the
> > > > > broker
> > > > > > > has
> > > > > > > > > not
> > > > > > > > > > > > > reached."
> > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > seems this should be "has reached".
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I changed this to "one more than the
> highest
> > > > > metadata
> > > > > > > > > offset
> > > > > > > > > > > which
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > broker has reached."
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 205. UnfenceBrokerRecord and
> > > > > UnregisterBrokerRecord:
> > > > > > > To
> > > > > > > > > me,
> > > > > > > > > > > they
> > > > > > > > > > > > > > > seem to
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > the same. Do we need both?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Unregistration means that the broker has
> been
> > > > > removed
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > > > cluster.
> > > > > > > > > > > > > > > > > That is different than unfencing, which
> marks
> > > the
> > > > > > > broker as
> > > > > > > > > > > active.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 206. TopicRecord: The Deleting field is
> used
> > > to
> > > > > > > indicate
> > > > > > > > > > > that the
> > > > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > being deleted. I am wondering if this is
> > > really
> > > > > > > needed
> > > > > > > > > since
> > > > > > > > > > > > > > > RemoveTopic
> > > > > > > > > > > > > > > > > > already indicates the same thing.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > RemoveTopic is the last step, that scrubs
> all
> > > > > metadata
> > > > > > > > > about
> > > > > > > > > > > the
> > > > > > > > > > > > > topic.
> > > > > > > > > > > > > > > > > In order to get to that last step, the
> topic
> > > data
> > > > > > > needs to
> > > > > > > > > > > removed
> > > > > > > > > > > > > > > from all
> > > > > > > > > > > > > > > > > brokers (after each broker notices that the
> > > topic
> > > > > is
> > > > > > > being
> > > > > > > > > > > > > deleted).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin
> McCabe <
> > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron
> > > Dagostino
> > > > > wrote:
> > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the updates.
> It's
> > > now
> > > > > > > clear
> > > > > > > > > to me
> > > > > > > > > > > that
> > > > > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > > > > > keep their broker epoch for the life
> of
> > > their
> > > > > > > JVM --
> > > > > > > > > they
> > > > > > > > > > > > > > > register
> > > > > > > > > > > > > > > > > > > > once, get their broker epoch in the
> > > > > response, and
> > > > > > > > > then
> > > > > > > > > > > never
> > > > > > > > > > > > > > > > > > > > re-register again. Brokers may get
> > > fenced,
> > > > > but
> > > > > > > they
> > > > > > > > > > > keep the
> > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > > broker epoch for the life of their
> JVM.
> > > The
> > > > > > > > > incarnation
> > > > > > > > > > > ID
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > kept for the life of the JVM but is
> > > > > generated by
> > > > > > > the
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > > upon startup, and the combination of
> the
> > > two
> > > > > > > allows
> > > > > > > > > the
> > > > > > > > > > > > > > > Controller to
> > > > > > > > > > > > > > > > > > > > act idempotently if any
> previously-sent
> > > > > > > registration
> > > > > > > > > > > response
> > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks, Ron. That's a good summary.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > One thing I wonder about is if it
> might
> > > be
> > > > > > > helpful
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > > send the Cluster ID as determined
> from
> > > its
> > > > > > > > > > > meta.properties
> > > > > > > > > > > > > file
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > > > registration request. Does it even
> make
> > > > > sense
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > > successfully register and enter the
> > > Fenced
> > > > > state
> > > > > > > if
> > > > > > > > > it
> > > > > > > > > > > has
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > wrong
> > > > > > > > > > > > > > > > > > > > Cluster ID?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Yeah, that's a good idea. Let's have
> the
> > > > > broker
> > > > > > > pass
> > > > > > > > > its
> > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > ID in
> > > > > > > > > > > > > > > > > > > the registration RPC, and then
> > > registration can
> > > > > > > fail
> > > > > > > > > if the
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The nextMetadatOffset value that the
> > > broker
> > > > > > > > > communicates
> > > > > > > > > > > > > > > > > > > > in its registration request only has
> > > meaning
> > > > > > > within
> > > > > > > > > the
> > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > cluster, so it feels to me that the
> > > > > Controller
> > > > > > > should
> > > > > > > > > > > have
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > perform this sanity check. There is
> > > > > currently
> > > > > > > > > (pre-KIP
> > > > > > > > > > > 500)
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > check
> > > > > > > > > > > > > > > > > > > > in the broker to make sure its
> configured
> > > > > > > cluster ID
> > > > > > > > > > > matches
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > > > stored in ZooKeeper, and we will
> have to
> > > > > perform
> > > > > > > this
> > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > somewhere in the KIP-500 world. If
> the
> > > > > > > Controller
> > > > > > > > > > > doesn't
> > > > > > > > > > > > > do it
> > > > > > > > > > > > > > > > > > > > within the registration request then
> the
> > > > > broker
> > > > > > > will
> > > > > > > > > > > have to
> > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > metadata request to the Controller,
> > > retrieve
> > > > > the
> > > > > > > > > Cluster
> > > > > > > > > > > ID,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > perform the check itself. It feels
> to me
> > > > > that it
> > > > > > > > > might
> > > > > > > > > > > be
> > > > > > > > > > > > > > > better for
> > > > > > > > > > > > > > > > > > > > the Controller to just do it, and
> then
> > > the
> > > > > broker
> > > > > > > > > doesn't
> > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > > worry about it anymore once it
> > > successfully
> > > > > > > > > registers.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I also have a question about the
> > > broker.id
> > > > > > > value and
> > > > > > > > > > > > > > > > > meta.properties.
> > > > > > > > > > > > > > > > > > > > The KIP now says "In version 0 of
> > > > > > > meta.properties,
> > > > > > > > > there
> > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > > broker.id field. Version 1 does not
> > > have
> > > > > this
> > > > > > > > > field.
> > > > > > > > > > > It
> > > > > > > > > > > > > is no
> > > > > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > > > > > needed because we no longer support
> > > dynamic
> > > > > > > broker id
> > > > > > > > > > > > > > > assignment."
> > > > > > > > > > > > > > > > > > > > But then there is an example version
> 1
> > > > > > > > > meta.properties
> > > > > > > > > > > file
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > shows
> > > > > > > > > > > > > > > > > > > > the broker.id value. I actually
> wonder
> > > if
> > > > > > > maybe the
> > > > > > > > > > > > > broker.id
> > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > would be good to keep in the version
> 1
> > > > > > > > > meta.properties
> > > > > > > > > > > file
> > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > currently (pre-KIP 500, version 0)
> acts
> > > as a
> > > > > > > sanity
> > > > > > > > > > > check to
> > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > sure
> > > > > > > > > > > > > > > > > > > > the broker is using the correct log
> > > > > directory.
> > > > > > > > > Similarly
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > controller.id value on controllers
> -- it
> > > > > would
> > > > > > > > > allow the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > sanity check for quorum controllers.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > That's a good point. I will add
> broker.id
> > > > > back,
> > > > > > > and
> > > > > > > > > also
> > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > controller.id as a possibility.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin
> > > McCabe
> > > > > <
> > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10,
> Jun Rao
> > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for the reply. A few more
> > > > > comments.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks again for the reply. Sorry
> for
> > > the
> > > > > long
> > > > > > > > > > > hiatus. I
> > > > > > > > > > > > > was
> > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > vacation for a while.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 55. There is still text that
> favors
> > > new
> > > > > > > broker
> > > > > > > > > > > > > registration.
> > > > > > > > > > > > > > > > > "When a
> > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > first starts up, when it is in
> the
> > > > > INITIAL
> > > > > > > > > state, it
> > > > > > > > > > > will
> > > > > > > > > > > > > > > always
> > > > > > > > > > > > > > > > > > > "win"
> > > > > > > > > > > > > > > > > > > > > > broker ID conflicts. However,
> once
> > > it is
> > > > > > > > > granted a
> > > > > > > > > > > > > lease, it
> > > > > > > > > > > > > > > > > > > transitions
> > > > > > > > > > > > > > > > > > > > > > out of the INITIAL state.
> > > Thereafter,
> > > > > it may
> > > > > > > > > lose
> > > > > > > > > > > > > subsequent
> > > > > > > > > > > > > > > > > > > conflicts if
> > > > > > > > > > > > > > > > > > > > > > its broker epoch is stale. (See
> > > KIP-380
> > > > > for
> > > > > > > some
> > > > > > > > > > > > > background
> > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > epoch.) The reason for favoring
> new
> > > > > > > processes
> > > > > > > > > is to
> > > > > > > > > > > > > > > accommodate
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > > > > > > case where a process is killed
> with
> > > kill
> > > > > -9
> > > > > > > and
> > > > > > > > > then
> > > > > > > > > > > > > > > restarted.
> > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > want it
> > > > > > > > > > > > > > > > > > > > > > to be able to reclaim its old ID
> > > quickly
> > > > > in
> > > > > > > this
> > > > > > > > > > > case."
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for the reminder. I have
> > > clarified
> > > > > the
> > > > > > > > > language
> > > > > > > > > > > > > here.
> > > > > > > > > > > > > > > > > > > Hopefully now it is clear that we don't
> > > allow
> > > > > quick
> > > > > > > > > re-use
> > > > > > > > > > > of
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 80.1 Sounds good. Could you
> document
> > > that
> > > > > > > > > listeners
> > > > > > > > > > > is a
> > > > > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > > > now? It would also be useful to
> > > annotate
> > > > > > > other
> > > > > > > > > > > required
> > > > > > > > > > > > > > > configs.
> > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > > > example, controller.connect
> should be
> > > > > > > required.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I added a note specifying that
> these
> > > are
> > > > > > > required.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 80.2 Could you list all
> deprecated
> > > > > existing
> > > > > > > > > configs?
> > > > > > > > > > > > > Another
> > > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > control.plane.listener.name
> since
> > > the
> > > > > > > > > controller no
> > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > sends
> > > > > > > > > > > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata and
> > > > > StopReplica
> > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I added a section specifying some
> > > > > deprecated
> > > > > > > > > configs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 83.1 It seems that the broker can
> > > > > transition
> > > > > > > from
> > > > > > > > > > > FENCED
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > RUNNING
> > > > > > > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > > > > > > registering for a new broker
> epoch.
> > > I am
> > > > > not
> > > > > > > > > sure how
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > works.
> > > > > > > > > > > > > > > > > > > Once the
> > > > > > > > > > > > > > > > > > > > > > controller fences a broker,
> there is
> > > no
> > > > > need
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > > > > > > > boker epoch around. So, if the
> fenced
> > > > > > > broker's
> > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > > > > existing broker epoch will be
> > > rejected,
> > > > > > > leading
> > > > > > > > > the
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The broker epoch refers to the
> broker
> > > > > > > registration.
> > > > > > > > > > > So we
> > > > > > > > > > > > > DO
> > > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > broker epoch around even while the
> broker
> > > is
> > > > > > > fenced.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The broker epoch changes only when
> > > there
> > > > > is a
> > > > > > > new
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > registration. Fencing or unfencing the
> > > broker
> > > > > > > doesn't
> > > > > > > > > > > change
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 83.5 Good point on KIP-590. Then
> > > should
> > > > > we
> > > > > > > > > expose the
> > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > debugging purposes? If not, we
> should
> > > > > > > deprecate
> > > > > > > > > the
> > > > > > > > > > > > > > > controllerID
> > > > > > > > > > > > > > > > > > > field in
> > > > > > > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think it's OK to expose it for
> now,
> > > with
> > > > > the
> > > > > > > > > proviso
> > > > > > > > > > > > > that it
> > > > > > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > > > > > be reachable by clients.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 90. We rejected the shared ID
> with
> > > just
> > > > > one
> > > > > > > > > reason
> > > > > > > > > > > "This
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > not a
> > > > > > > > > > > > > > > > > > > good idea
> > > > > > > > > > > > > > > > > > > > > > because NetworkClient assumes a
> > > single ID
> > > > > > > > > space. So
> > > > > > > > > > > if
> > > > > > > > > > > > > > > there is
> > > > > > > > > > > > > > > > > > > both a
> > > > > > > > > > > > > > > > > > > > > > controller 1 and a broker 1, we
> don't
> > > > > have a
> > > > > > > way
> > > > > > > > > of
> > > > > > > > > > > > > picking
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > "right"
> > > > > > > > > > > > > > > > > > > > > > one." This doesn't seem to be a
> > > strong
> > > > > > > reason.
> > > > > > > > > For
> > > > > > > > > > > > > example,
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > > > > > address the NetworkClient issue
> with
> > > the
> > > > > node
> > > > > > > > > type
> > > > > > > > > > > as you
> > > > > > > > > > > > > > > pointed
> > > > > > > > > > > > > > > > > > > out or
> > > > > > > > > > > > > > > > > > > > > > using the negative value of a
> broker
> > > ID
> > > > > as
> > > > > > > the
> > > > > > > > > > > > > controller ID.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > It would require a lot of code
> changes
> > > to
> > > > > > > support
> > > > > > > > > > > multiple
> > > > > > > > > > > > > > > types of
> > > > > > > > > > > > > > > > > > > node IDs. It's not clear to me that
> the
> > > end
> > > > > result
> > > > > > > > > would
> > > > > > > > > > > be
> > > > > > > > > > > > > > > better --
> > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > tend to think it would be worse, since
> it
> > > > > would be
> > > > > > > more
> > > > > > > > > > > > > complex.
> > > > > > > > > > > > > > > In a
> > > > > > > > > > > > > > > > > > > similar vein, using negative numbers
> seems
> > > > > > > dangerous,
> > > > > > > > > > > since we
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > negatives or -1 as "special values" in
> many
> > > > > places.
> > > > > > > > > For
> > > > > > > > > > > > > example,
> > > > > > > > > > > > > > > -1
> > > > > > > > > > > > > > > > > often
> > > > > > > > > > > > > > > > > > > represents "no such node."
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > One important thing to keep in
> mind is
> > > > > that we
> > > > > > > > > want to
> > > > > > > > > > > be
> > > > > > > > > > > > > able
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > transition from a broker and a
> controller
> > > being
> > > > > > > > > co-located
> > > > > > > > > > > to
> > > > > > > > > > > > > them
> > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > > > longer being co-located. This is much
> > > easier
> > > > > to do
> > > > > > > > > when
> > > > > > > > > > > they
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > > > the broker reports all offline
> > > replicas
> > > > > due
> > > > > > > to a
> > > > > > > > > disk
> > > > > > > > > > > > > > > failure to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > controller. It seems this
> information
> > > > > needs
> > > > > > > to be
> > > > > > > > > > > > > persisted
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > > > > > log. Do we have a corresponding
> > > record
> > > > > for
> > > > > > > that?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hmm, I have to look into this a
> little
> > > bit
> > > > > > > more.
> > > > > > > > > We
> > > > > > > > > > > may
> > > > > > > > > > > > > need
> > > > > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > > > > record type.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 101. Currently, StopReplica
> request
> > > has 2
> > > > > > > modes,
> > > > > > > > > > > without
> > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > > and with
> > > > > > > > > > > > > > > > > > > > > > deletion. The former is used for
> > > > > controlled
> > > > > > > > > shutdown
> > > > > > > > > > > and
> > > > > > > > > > > > > > > handling
> > > > > > > > > > > > > > > > > > > disk
> > > > > > > > > > > > > > > > > > > > > > failure, and causes the follower
> to
> > > > > stop. The
> > > > > > > > > latter
> > > > > > > > > > > is
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > > > > > and partition reassignment, and
> > > causes
> > > > > the
> > > > > > > > > replica
> > > > > > > > > > > to be
> > > > > > > > > > > > > > > deleted.
> > > > > > > > > > > > > > > > > > > Since we
> > > > > > > > > > > > > > > > > > > > > > are deprecating StopReplica,
> could we
> > > > > > > document
> > > > > > > > > what
> > > > > > > > > > > > > triggers
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > stopping
> > > > > > > > > > > > > > > > > > > > > > of a follower and the deleting
> of a
> > > > > replica
> > > > > > > now?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > RemoveTopic triggers deletion. In
> > > general
> > > > > the
> > > > > > > > > > > > > functionality of
> > > > > > > > > > > > > > > > > > > StopReplica is subsumed by the metadata
> > > > > records.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 102. Should we include the
> metadata
> > > > > topic in
> > > > > > > the
> > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > > If so,
> > > > > > > > > > > > > > > > > > > > > > when it will be included and what
> > > will
> > > > > the
> > > > > > > > > metadata
> > > > > > > > > > > > > response
> > > > > > > > > > > > > > > look
> > > > > > > > > > > > > > > > > > > like?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > No, it won't be included in the
> > > metadata
> > > > > > > response
> > > > > > > > > sent
> > > > > > > > > > > back
> > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > brokers.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 103. "The active controller
> assigns
> > > the
> > > > > > > broker a
> > > > > > > > > new
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > epoch,
> > > > > > > > > > > > > > > > > > > based on
> > > > > > > > > > > > > > > > > > > > > > the latest committed offset in
> the
> > > log."
> > > > > This
> > > > > > > > > seems
> > > > > > > > > > > > > > > inaccurate
> > > > > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > latest committed offset doesn't
> > > always
> > > > > > > advance on
> > > > > > > > > > > every
> > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > append.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Given that the new broker epoch
> won't
> > > be
> > > > > > > visible
> > > > > > > > > until
> > > > > > > > > > > the
> > > > > > > > > > > > > > > commit
> > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > happened, I have changed this to "the
> next
> > > > > > > available
> > > > > > > > > > > offset in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > log"
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 104. REGISTERING(1) : It says
> > > > > "Otherwise, the
> > > > > > > > > broker
> > > > > > > > > > > > > moves
> > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > > > > > > > > state.". It seems this should be
> > > RUNNING?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 105. RUNNING: Should we require
> the
> > > > > broker to
> > > > > > > > > catch
> > > > > > > > > > > up
> > > > > > > > > > > > > to the
> > > > > > > > > > > > > > > > > > > metadata log
> > > > > > > > > > > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > For 104 and 105, these sections
> have
> > > been
> > > > > > > reworked.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM
> Colin
> > > > > McCabe
> > > > > > > <
> > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020, at
> 05:51, Tom
> > > > > Bentley
> > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at
> 08:59,
> > > Ron
> > > > > > > Dagostino
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the
> > > hard
> > > > > work
> > > > > > > on
> > > > > > > > > this
> > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I have some questions
> about
> > > what
> > > > > > > happens
> > > > > > > > > to a
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > when it
> > > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > > > > > fenced (e.g. because it
> can't
> > > > > send a
> > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > request to
> > > > > > > > > > > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > > > > > > > > > > lease). The KIP says
> "When a
> > > > > broker
> > > > > > > is
> > > > > > > > > > > fenced,
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > cannot
> > > > > > > > > > > > > > > > > > > process any
> > > > > > > > > > > > > > > > > > > > > > > > > > client requests. This
> > > prevents
> > > > > > > brokers
> > > > > > > > > which
> > > > > > > > > > > > > are not
> > > > > > > > > > > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > > > > > > > > > > metadata updates or that
> are
> > > not
> > > > > > > > > receiving
> > > > > > > > > > > and
> > > > > > > > > > > > > > > processing
> > > > > > > > > > > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > > > > > > > > > > enough from causing
> issues to
> > > > > > > clients."
> > > > > > > > > And
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > > > > > > > > > > FENCED(4) state it
> likewise
> > > says
> > > > > > > "While
> > > > > > > > > in
> > > > > > > > > > > this
> > > > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > > > > > > not respond to client
> > > > > requests." It
> > > > > > > > > makes
> > > > > > > > > > > sense
> > > > > > > > > > > > > > > that a
> > > > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > > > > > should not accept
> producer
> > > > > requests
> > > > > > > -- I
> > > > > > > > > > > assume
> > > > > > > > > > > > > any
> > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > > > > > > > > > would result in
> > > > > > > > > NotLeaderOrFollowerException.
> > > > > > > > > > > > > But
> > > > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > > > > > > > > > > (fetch from follower)
> > > consumer
> > > > > > > > > requests? It
> > > > > > > > > > > is
> > > > > > > > > > > > > > > > > conceivable
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > > > > > could continue. Related
> to
> > > that,
> > > > > > > would a
> > > > > > > > > > > fenced
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > > > > > > fetch data for partitions
> > > where
> > > > > it
> > > > > > > > > thinks it
> > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > follower?
> > > > > > > > > > > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > > > > > > > > > > it rejects consumer
> requests
> > > it
> > > > > might
> > > > > > > > > still
> > > > > > > > > > > > > continue
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > > > > > > > > > > follower. Might it be
> > > helpful to
> > > > > > > clarify
> > > > > > > > > > > both
> > > > > > > > > > > > > > > decisions
> > > > > > > > > > > > > > > > > > > here?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Good question. I think a
> > > fenced
> > > > > broker
> > > > > > > > > should
> > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > > > > > > > > > > partitions it was already
> > > fetching
> > > > > > > before
> > > > > > > > > it
> > > > > > > > > > > was
> > > > > > > > > > > > > > > fenced,
> > > > > > > > > > > > > > > > > > > unless it
> > > > > > > > > > > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > > > > > > > > > > problem. At that point it
> > > won't be
> > > > > > > able to
> > > > > > > > > > > > > continue,
> > > > > > > > > > > > > > > > > since it
> > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > > > the new metadata. For
> > > example, it
> > > > > > > won't
> > > > > > > > > know
> > > > > > > > > > > about
> > > > > > > > > > > > > > > > > leadership
> > > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > the partitions it's
> fetching.
> > > The
> > > > > > > > > rationale
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > > continuing to
> > > > > > > > > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > > > > > > > try to avoid disruptions as
> > > much as
> > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I don't think fenced
> brokers
> > > should
> > > > > > > accept
> > > > > > > > > > > client
> > > > > > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > > > > The issue
> > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > that the fenced broker may
> or
> > > may
> > > > > not
> > > > > > > have
> > > > > > > > > any
> > > > > > > > > > > > > data it
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > > > > > > > > > > have. It may or may not
> have
> > > > > applied
> > > > > > > any
> > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > > it is supposed to have
> applied.
> > > > > So it
> > > > > > > > > could
> > > > > > > > > > > get
> > > > > > > > > > > > > pretty
> > > > > > > > > > > > > > > > > > > confusing, and
> > > > > > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > > potentially waste the
> client's
> > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > When fenced, how would the
> broker
> > > > > reply
> > > > > > > to a
> > > > > > > > > > > client
> > > > > > > > > > > > > > > which did
> > > > > > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The broker will respond with a
> > > > > retryable
> > > > > > > error
> > > > > > > > > in
> > > > > > > > > > > that
> > > > > > > > > > > > > > > case.
> > > > > > > > > > > > > > > > > Once
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > client has re-fetched its
> > > metadata, it
> > > > > > > will no
> > > > > > > > > > > longer
> > > > > > > > > > > > > see
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > > as part of the cluster. I
> added a
> > > > > note to
> > > > > > > the
> > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Thu, Dec 17, 2020, at 10:19, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the reply.
>
> 211. Hmm, I still don't see a clear benefit of registering a broker before
> recovery. It's possible for the recovery to take time. However, during the
> recovery mode, it seems the broker will still be in the fenced mode and
> won't be able to do work for the clients. So, registering and heartbeating
> early seems to only add unnecessary overhead. For your point on topic
> creation, I thought we now allow replicas to be created on
> unregistered brokers.
>
Hi Jun,
Thanks again for the reviews.
Topics cannot be created on unregistered brokers. They can be created on registered but fenced brokers. So for that reason I think it makes sense to register as early as possible.
> 230. Currently, we do have a ControllerId field in MetadataResponse. In the
> early discussion, I thought that we want to expose the controller for
> debugging purposes, but not used by the client library.
>
The current plan is that we will expose the controller node ID, but the controller will not be included in the list of nodes in the metadata response.
It's not really possible to include the controller in that list of nodes because the controller may not share the same set of listeners as the broker. So, for example, maybe the controller endpoint is using a different type of security than the broker. So while we could pass back a hostname and port, the client would have no way to connect since it doesn't know what security settings to use.
regards,
Colin
> Jun
>
> On Wed, Dec 16, 2020 at 9:13 PM Colin McCabe <cm...@apache.org> wrote:
>
> > On Wed, Dec 16, 2020, at 18:13, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply. Just a couple of more comments.
> > >
> > > 211. Currently, the broker only registers itself in ZK after log
> > recovery.
> > > Is there any benefit to change that? As you mentioned, the broker can't
> > do
> > > much before completing log recovery.
> > >
> >
> > Hi Jun,
> >
> > Previously, it wasn't possible to register in ZK without immediately
> > getting added to the MetadataResponse. So I think that's the main reason
> > why registration was delayed until after log recovery. Since that
> > constraint doesn't exist any more, there seems to be no reason to delay
> > registration.
> >
> > I think delaying registration would have some major downsides. If log
> > recovery takes a while, that's a longer window during which someone else
> > could register a broker with the same ID. Having parts of the cluster
> > missing for a while gives up some of the benefit of separating registration
> > from fencing. For example, if a broker somehow gets unregistered and we
> > want to re-register it, but we have to wait for a 10 minute log recovery to
> > do that, that could be a window during which topics can't be created that
> > need to be on that broker, and so forth.
> >
> > > 230. Regarding MetadataResponse, there is a slight awkwardness. We return
> > > rack for each node. However, if that node is for the controller, the rack
> > > field is not really relevant. Should we clean it up here or in another
> > KIP
> > > like KIP-700?
> >
> > Oh, controllers don't appear in the MetadataResponses returned to clients,
> > since clients can't access them. I should have been more clear about that
> > in the KIP-- I added a sentence to "Networking" describing this.
> >
> > best,
> > Colin
> >
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Dec 16, 2020 at 4:23 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Wed, Dec 16, 2020, at 13:40, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the reply. A few follow up comments.
> > > > >
> > > > > 211. When does the broker send the BrokerRegistration request to the
> > > > > controller? Is it after the recovery phase? If so, at that point, the
> > > > > broker has already caught up on the metadata (in order to clean up
> > > > deleted
> > > > > partitions). Then, it seems we don't need the ShouldFence field
> > > > > in BrokerHeartbeatRequest?
> > > >
> > > > Hi Jun,
> > > >
> > > > Thanks again for the reviews.
> > > >
> > > > The broker sends the registration request as soon as it starts up. It
> > > > cannot wait until the recovery phase is over since sometimes log
> > recovery
> > > > can take quite a long time.
> > > >
> > > > >
> > > > > 213. kafka-cluster.sh
> > > > > 213.1 For the decommision example, should the command take a broker
> > id?
> > > >
> > > > Yes, the example should have a broker id. Fixed.
> > > >
> > > > > 213.2 Existing tools use the "--" command line option (e.g.
> > kafka-topics
> > > > > --list --topic test). Should we follow the same convention
> > > > > for kafka-cluster.sh (and kafka-storage.sh)?
> > > >
> > > > Hmm. I don't think argparse4j supports using double dashes to identify
> > > > subcommands. I think it might be confusing as well, since the
> > subcommand
> > > > must come first, unlike a plain old argument which can be anywhere on
> > the
> > > > command line.
> > > >
> > > > > 213.3 Should we add an admin api for broker decommission so that
> > this can
> > > > > be done programmatically?
> > > > >
> > > >
> > > > Yes, there is an admin client API for decommissioning as well.
> > > >
> > > > > 220. DecommissionBroker: "NOT_CONTROLLER if the node that the
> > request was
> > > > > sent to is not the controller". I thought the clients never send a
> > > > request
> > > > > to the controller directly now and the broker will forward it to the
> > > > > controller?
> > > > >
> > > >
> > > > If the controller moved recently, it's possible that the broker could
> > send
> > > > to a controller that has just recently become inactive. In that case
> > > > NOT_CONTROLLER would be returned. (A standby controller returns
> > > > NOT_CONTROLLER for most APIs).
> > > >
> > > > > 221. Could we add the required ACL for the new requests?
> > > > >
> > > >
> > > > Good point. I added the required ACL for each new RPC.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Dec 16, 2020 at 11:38 AM Colin McCabe <cm...@apache.org>
> > > > wrote:
> > > > >
> > > > > > On Wed, Dec 16, 2020, at 09:59, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the reply. A few more comments below.
> > > > > > >
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Thanks for the comments.
> > > > > >
> > > > > > > 206. "RemoveTopic is the last step, that scrubs all metadata
> > about
> > > > the
> > > > > > > topic. In order to get to that last step, the topic data needs
> > to
> > > > > > removed
> > > > > > > from all brokers (after each broker notices that the topic is
> > being
> > > > > > > deleted)." Currently, this is done based on the response of
> > > > > > > StopReplicaRequest. Since the controller no longer sends this
> > > > request,
> > > > > > how
> > > > > > > does the controller know that the data for the deleted topic has
> > > > > > > been removed in the brokers?
> > > > > > >
> > > > > >
> > > > > > That's a good point... definitely an oversight.
> > > > > >
> > > > > > It seems complex to force the controller to track when log
> > directories
> > > > > > have been deleted. Let's just assume that KIP-516 has been
> > > > implemented,
> > > > > > and track them by UUID. Then we can just have a single topic
> > deletion
> > > > > > record.
> > > > > >
> > > > > > I added a section on "topic identifiers" describing this.
> > > > > >
> > > > > > > 210. Thanks for the explanation. Sounds good to me.
> > > > > > >
> > > > > > > 211. I still don't see an example when ShouldFence is set to
> > true in
> > > > > > > BrokerHeartbeatReques. Could we add one?
> > > > > > >
> > > > > >
> > > > > > It is sent to true when the broker is first starting up and
> > doesn't yet
> > > > > > want to be unfenced. I added a longer explanation of this in the
> > > > "Broker
> > > > > > Leases" section.
> > > > > >
> > > > > > > 213. The KIP now allows replicas to be assigned on brokers that
> > are
> > > > > > fenced,
> > > > > > > which is an improvement. How do we permanently remove a broker
> > (e.g.
> > > > > > > cluster shrinking) to prevent it from being used for future
> > replica
> > > > > > > assignments?
> > > > > > >
> > > > > >
> > > > > > This is a fair point. I will create a kafka-cluster.sh script
> > which
> > > > can
> > > > > > do this, plus a DecommissionBrokerRequest.
> > > > > >
> > > > > > As a bonus the kafka-cluster.sh script can help people find the
> > > > cluster ID
> > > > > > of brokers, something that people have wanted a tool for in the
> > past.
> > > > > >
> > > > > > > 214. "Currently, when a node is down, all of its ZK registration
> > > > > > > information is gone. But we need this information in order to
> > > > > > understand
> > > > > > > things like whether the replicas of a particular partition are
> > > > > > > well-balanced across racks." This is not quite true. Currently,
> > even
> > > > when
> > > > > > > ZK registration is gone, the existing replica assignment is still
> > > > > > available
> > > > > > > in the metadata response.
> > > > > > >
> > > > > >
> > > > > > I agree that the existing replica assignment is still available.
> > But
> > > > that
> > > > > > just tells you partition X is on nodes A, B, and C. If you don't
> > have
> > > > the
> > > > > > ZK registration for one or more of A, B, or C then you don't know
> > > > whether
> > > > > > we are following the policy of "two replicas on one rack, one
> > replica
> > > > on
> > > > > > another." Or any other more complex rack placement policy that you
> > > > might
> > > > > > have.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Wed, Dec 16, 2020 at 8:48 AM Colin McCabe <cmccabe@apache.org
> > >
> > > > wrote:
> > > > > > >
> > > > > > > > On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> > > > > > > > > Hi, Colin,
> > > > > > > > >
> > > > > > > > > Thanks for the reply. A few more follow up comments.
> > > > > > > > >
> > > > > > > > > 210. initial.broker.registration.timeout.ms: The default
> > value
> > > > is
> > > > > > 90sec,
> > > > > > > > > which seems long. If a broker fails the registration because
> > of
> > > > > > incorrect
> > > > > > > > > configs, we want to fail faster. In comparison, the defaults
> > for
> > > > > > > > > zookeeper.connection.timeout.ms is 18 secs.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > I agree that the initial connection timeout here is longer than
> > > > what we
> > > > > > > > had with ZK. The main reason I selected a slightly longer
> > timeout
> > > > > > here is
> > > > > > > > to handle the case where the controllers and the brokers are
> > > > > > co-located.
> > > > > > > > For example, if you have a 3 node cluster and all three nodes
> > are
> > > > > > > > controllers+brokers, when you first bring up the cluster, we
> > will
> > > > have
> > > > > > to
> > > > > > > > stand up the controller quorum and then handle broker
> > > > registrations.
> > > > > > > > Although we believe Raft will start up relatively quickly, it's
> > > > good to
> > > > > > > > leave some extra margin here.
> > > > > > > >
> > > > > > > > I don't think there's a big disadvantage to having a slightly
> > > > longer
> > > > > > > > timeout here. After all, starting up the brokers with no
> > > > controllers
> > > > > > is an
> > > > > > > > admin mistake, which we don't expect to see very often. Maybe
> > > > let's
> > > > > > set it
> > > > > > > > to 60 seconds for now and maybe see if we can tweak it in the
> > > > future if
> > > > > > > > that turns out to be too long or short?
> > > > > > > >
> > > > > > > > > 211. "Once they are all moved, the controller responds to the
> > > > > > heartbeat
> > > > > > > > > with a nextState of SHUTDOWN." It seems that nextState is no
> > > > longer
> > > > > > > > > relevant. Also, could you add a bit more explanation on when
> > > > > > ShouldFence
> > > > > > > > is
> > > > > > > > > set to true in BrokerHeartbeatRequest?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Good catch. I removed the obsolete section referring to
> > nextState
> > > > and
> > > > > > > > added a reference to the new boolean. I also added some more
> > > > > > information
> > > > > > > > about ShouldFence and about the rationale for separating
> > fencing
> > > > from
> > > > > > > > registration.
> > > > > > > >
> > > > > > > > > 212. Related to the above, what does the broker do if
> > IsFenced is
> > > > > > true in
> > > > > > > > > the BrokerHeartbeatResponse? Will the broker do the same
> > thing on
> > > > > > > > receiving
> > > > > > > > > a FenceBrokerRecord from the metadata log?
> > > > > > > > >
> > > > > > > >
> > > > > > > > The broker only checks this boolean during the startup process.
> > > > After
> > > > > > the
> > > > > > > > startup process is finished, it ignores this boolean.
> > > > > > > >
> > > > > > > > The broker uses fence / unfence records in the metadata log to
> > > > > > determine
> > > > > > > > which brokers should appear in its MetadataResponse.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <
> > cmccabe@apache.org
> > > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > > > > > > > > > Hi Colin,
> > > > > > > > > > >
> > > > > > > > > > > The KIP says that "brokers which are fenced will not
> > appear
> > > > in
> > > > > > > > > > > MetadataResponses. So clients that have up-to-date
> > metadata
> > > > will
> > > > > > > > not try
> > > > > > > > > > > to contact fenced brokers.", which is fine, but it
> > doesn't
> > > > seem
> > > > > > to
> > > > > > > > > > > elaborate on what happens for clients which try to
> > contact a
> > > > > > broker
> > > > > > > > > > (using
> > > > > > > > > > > stale metadata, for example) and find it in the
> > > > > > registered-but-fenced
> > > > > > > > > > > state.
> > > > > > > > > >
> > > > > > > > > > Hi Tom,
> > > > > > > > > >
> > > > > > > > > > I have dropped the broker-side fencing from this
> > proposal. So
> > > > now
> > > > > > the
> > > > > > > > > > fencing is basically the same as today's fencing: it's
> > > > > > controller-side
> > > > > > > > > > only. That means that clients using stale metadata can
> > contact
> > > > > > fenced
> > > > > > > > > > brokers and communicate with them. The only case where the
> > > > broker
> > > > > > > > will be
> > > > > > > > > > inaccessible is when it hasn't finished starting yet
> > (i.e., has
> > > > > > not yet
> > > > > > > > > > attained RUNNING state.)
> > > > > > > > > >
> > > > > > > > > > Just like today, we expect the safeguards built into the
> > > > > > replication
> > > > > > > > > > protocol to prevent the worst corner cases that could
> > result
> > > > from
> > > > > > > > this. I
> > > > > > > > > > do think we will probably take up this issue later and
> > find a
> > > > > > better
> > > > > > > > > > solution for client-side metadata, but this KIP is big
> > enough
> > > > > > as-is.
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > You said previously that the broker will respond with a
> > > > retriable
> > > > > > > > > > > error, which is again fine, but you didn't answer my
> > question
> > > > > > about
> > > > > > > > which
> > > > > > > > > > > error code you've chosen for this. I realise that this
> > > > doesn't
> > > > > > really
> > > > > > > > > > > affect the correctness of the behaviour, but I'm not
> > aware
> > > > of any
> > > > > > > > > > existing
> > > > > > > > > > > error code which is a good fit. So it would be good to
> > > > understand
> > > > > > > > about
> > > > > > > > > > how
> > > > > > > > > > > you're making this backward compatible for clients.
> > > > > > > > > > >
> > > > > > > > > > > Many thanks,
> > > > > > > > > > >
> > > > > > > > > > > Tom
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <
> > > > cmccabe@apache.org
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the reply. Just a couple of more comments
> > > > below.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 210. Since we are deprecating
> > > > > > zookeeper.connection.timeout.ms,
> > > > > > > > > > should we
> > > > > > > > > > > > > add a new config to bound the time for a broker to
> > > > connect
> > > > > > to the
> > > > > > > > > > > > > controller during starting up?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Good idea. I added
> > initial.broker.registration.timeout.ms
> > > > for
> > > > > > > > this.
> > > > > > > > > > > >
> > > > > > > > > > > > > 211. BrokerHeartbeat no longer has the state field
> > in the
> > > > > > > > > > > > request/response.
> > > > > > > > > > > > > However, (a) the controller shutdown section still
> > has
> > > > "In
> > > > > > its
> > > > > > > > > > periodic
> > > > > > > > > > > > > heartbeats, the broker asks the controller if it can
> > > > > > transition
> > > > > > > > into
> > > > > > > > > > the
> > > > > > > > > > > > > SHUTDOWN state. This motivates the controller to
> > move
> > > > all
> > > > > > of the
> > > > > > > > > > leaders
> > > > > > > > > > > > > off of that broker. Once they are all moved, the
> > > > controller
> > > > > > > > > > responds to
> > > > > > > > > > > > > the heartbeat with a nextState of SHUTDOWN."; (2) the
> > > > > > > > description of
> > > > > > > > > > > > > BrokerHeartbeat still references currentState and
> > > > > > targetState.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks. I've made these sections clearer and removed
> > the
> > > > > > obsolete
> > > > > > > > > > > > references to sending states.
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > > > Jun
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <
> > > > > > cmccabe@apache.org
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the update. A few more follow up
> > comments.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks again for the review.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 100. FailedReplicaRecord: Since this is reported
> > by
> > > > each
> > > > > > > > broker
> > > > > > > > > > > > > > > independently, perhaps we could use a more
> > concise
> > > > > > > > representation
> > > > > > > > > > > > that
> > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > a top level broker field, an array of topics,
> > which
> > > > has
> > > > > > an
> > > > > > > > array
> > > > > > > > > > of
> > > > > > > > > > > > > > > partitions.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The issue is that there is a size limit on the each
> > > > record.
> > > > > > > > > > Putting
> > > > > > > > > > > > all
> > > > > > > > > > > > > > of the partitions of a log directory into a single
> > > > record
> > > > > > would
> > > > > > > > > > > > probably
> > > > > > > > > > > > > > break that in many cases. Still, we can optimize a
> > > > bit by
> > > > > > > > having
> > > > > > > > > > an
> > > > > > > > > > > > array
> > > > > > > > > > > > > > of partition IDs, since nearly all the time, we
> > have
> > > > more
> > > > > > than
> > > > > > > > one
> > > > > > > > > > > > from the
> > > > > > > > > > > > > > same topic.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 200. Sounds good. If we remove the broker-side
> > > > fencing
> > > > > > > > logic, do
> > > > > > > > > > we
> > > > > > > > > > > > plan
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > still keep FENCED in broker state? Do we plan to
> > > > expose
> > > > > > the
> > > > > > > > new
> > > > > > > > > > > > states
> > > > > > > > > > > > > > > through the existing BrokerState metric and if
> > so,
> > > > what
> > > > > > are
> > > > > > > > the
> > > > > > > > > > > > values
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the new states?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > No, we don't need FENCED any more. I have removed
> > it
> > > > from
> > > > > > the
> > > > > > > > KIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The new states are very similar to the current
> > ones,
> > > > > > actually.
> > > > > > > > > > There
> > > > > > > > > > > > are
> > > > > > > > > > > > > > no new states or removed ones. The main change in
> > the
> > > > > > broker
> > > > > > > > state
> > > > > > > > > > > > machine
> > > > > > > > > > > > > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state
> > has
> > > > been
> > > > > > > > > > renamed to
> > > > > > > > > > > > > > RECOVERY. Also, unlike previously, the broker will
> > > > always
> > > > > > pass
> > > > > > > > > > through
> > > > > > > > > > > > > > RECOVERY (although it may only stay in this state
> > for a
> > > > > > very
> > > > > > > > short
> > > > > > > > > > > > amount
> > > > > > > > > > > > > > of time).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 201. This may be fine too. Could we document what
> > > > happens
> > > > > > > > when
> > > > > > > > > > the
> > > > > > > > > > > > > > > broker.id/controller.id in metadata.properties
> > don't
> > > > > > match
> > > > > > > > the
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > config when the broker starts up?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I added some documentation about this.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 204. There is still "The highest metadata offset
> > > > which
> > > > > > the
> > > > > > > > broker
> > > > > > > > > > > > has not
> > > > > > > > > > > > > > > reached" referenced under BrokerRegistration.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 206. Is that separate step needed given KIP-516?
> > With
> > > > > > > > KIP-516 (
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > > > > > > > > > ),
> > > > > > > > > > > > > > > we don't need to wait for the topic data to be
> > > > removed
> > > > > > from
> > > > > > > > all
> > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > before removing the topic metadata. The
> > combination
> > > > of
> > > > > > > > unmatching
> > > > > > > > > > > > > > > topicId
> > > > > > > > > > > > > > > or the missing topicId from the metadata is
> > enough
> > > > for
> > > > > > the
> > > > > > > > > > broker to
> > > > > > > > > > > > > > > clean
> > > > > > > > > > > > > > > up deleted topics asynchronously.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It won't be needed once KIP-516 is adopted, but
> > this
> > > > hasn't
> > > > > > > > been
> > > > > > > > > > > > > > implemented yet.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <
> > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for the updated KIP. A few more
> > comments
> > > > > > below.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks again for the reviews.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 80.2 For deprecated configs, we need to
> > include
> > > > > > > > zookeeper.*
> > > > > > > > > > and
> > > > > > > > > > > > > > > > > broker.id.generation.enable.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Added.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 83.1 If a broker is down, does the controller
> > > > keep
> > > > > > the
> > > > > > > > > > previously
> > > > > > > > > > > > > > > > > registered broker epoch forever? If not, how
> > long
> > > > > > does
> > > > > > > > the
> > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > > > it? What does the controller do when
> > receiving a
> > > > > > broker
> > > > > > > > > > heartbeat
> > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > with an unfound broker epoch?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Yes, the controller keeps the previous
> > registration
> > > > > > > > forever.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Broker heartbeat requests with an incorrect
> > broker
> > > > > > epoch
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > > > rejected
> > > > > > > > > > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 100. Have you figured out if we need to add
> > a new
> > > > > > record
> > > > > > > > > > type for
> > > > > > > > > > > > > > > > reporting
> > > > > > > > > > > > > > > > > partitions on failed disks?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I added FailedReplicaRecord to reflect the case
> > > > where a
> > > > > > > > JBOD
> > > > > > > > > > > > directory
> > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > failed, leading to failed replicas.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 102. For debugging purposes, sometimes it's
> > > > useful to
> > > > > > > > read
> > > > > > > > > > the
> > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > topic using tools like console-consumer.
> > Should
> > > > we
> > > > > > > > support
> > > > > > > > > > that
> > > > > > > > > > > > and
> > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > so,
> > > > > > > > > > > > > > > > > how?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > For now, we have the ability to read the
> > metadata
> > > > logs
> > > > > > > > with the
> > > > > > > > > > > > > > dump-logs
> > > > > > > > > > > > > > > > tool. I think we will come up with some other
> > > > tools
> > > > > > in the
> > > > > > > > > > future
> > > > > > > > > > > > as
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > get experience.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 200. "brokers which are fenced will not
> > appear in
> > > > > > > > > > > > MetadataResponses.
> > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > broker will not respond to these requests--
> > > > instead,
> > > > > > it
> > > > > > > > will
> > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > disconnect." If the controller is
> > partitioned off
> > > > > > from
> > > > > > > > the
> > > > > > > > > > > > brokers,
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > design will cause every broker to stop
> > accepting
> > > > new
> > > > > > > > client
> > > > > > > > > > > > > > requests. In
> > > > > > > > > > > > > > > > > contrast, if ZK is partitioned off, the
> > existing
> > > > > > > > behavior is
> > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > brokers can continue to work based on the
> > last
> > > > known
> > > > > > > > > > metadata.
> > > > > > > > > > > > So, I
> > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > sure if we should change the existing
> > behavior
> > > > > > because
> > > > > > > > of the
> > > > > > > > > > > > bigger
> > > > > > > > > > > > > > > > impact
> > > > > > > > > > > > > > > > > in the new one. Another option is to keep the
> > > > > > existing
> > > > > > > > > > behavior
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > expose
> > > > > > > > > > > > > > > > > a metric for fenced brokers so that the
> > operator
> > > > > > could be
> > > > > > > > > > > > alerted.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm skeptical about how well running without ZK
> > > > > > currently
> > > > > > > > > > works.
> > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > I will move the broker-side fencing into a
> > > > follow-up
> > > > > > KIP.
> > > > > > > > This
> > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > already pretty large and there is no hard
> > > > dependency on
> > > > > > > > this.
> > > > > > > > > > > > There
> > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > also be other ways of accomplishing the
> > positive
> > > > > > effects of
> > > > > > > > > > what
> > > > > > > > > > > > > > > > broker-side fencing, so more discussion is
> > needed.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 201. I read Ron's comment, but I am still not
> > > > sure
> > > > > > the
> > > > > > > > > > benefit of
> > > > > > > > > > > > > > keeping
> > > > > > > > > > > > > > > > > broker.id and controller.id in
> > meta.properties.
> > > > It
> > > > > > seems
> > > > > > > > > > that
> > > > > > > > > > > > we are
> > > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > duplicating the same info in two places and
> > have
> > > > the
> > > > > > > > > > additional
> > > > > > > > > > > > > > burden of
> > > > > > > > > > > > > > > > > making sure the values in the two places are
> > > > > > consistent.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think the reasoning is that having broker.id
> > > > > > protects us
> > > > > > > > > > against
> > > > > > > > > > > > > > > > accidentally bringing up a broker with a disk
> > from
> > > > a
> > > > > > > > different
> > > > > > > > > > > > > > broker. I
> > > > > > > > > > > > > > > > don't feel strongly about this but it seemed
> > > > simpler to
> > > > > > > > keep
> > > > > > > > > > it.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 202. controller.connect.security.protocol: Is
> > > > this
> > > > > > needed
> > > > > > > > > > since
> > > > > > > > > > > > > > > > > controller.listener.names and
> > > > > > > > listener.security.protocol.map
> > > > > > > > > > > > imply
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > security protocol already?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > You're right, this isn't needed. I'll remove
> > it.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 203. registration.heartbeat.interval.ms: It
> > > > > > defaults to
> > > > > > > > 2k.
> > > > > > > > > > ZK
> > > > > > > > > > > > uses
> > > > > > > > > > > > > > 1/3
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the session timeout for heartbeat. So, given
> > the
> > > > > > default
> > > > > > > > 18k
> > > > > > > > > > for
> > > > > > > > > > > > > > > > > registration.lease.timeout.ms, should we
> > default
> > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms to 6k?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 6 seconds seems like a pretty long time between
> > > > > > > > heartbeats. It
> > > > > > > > > > > > might
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > useful to know when a broker is missing
> > heartbeats,
> > > > > > with
> > > > > > > > less
> > > > > > > > > > time
> > > > > > > > > > > > than
> > > > > > > > > > > > > > > > that. I provisionally set it to 3 seconds (we
> > can
> > > > > > always
> > > > > > > > > > change
> > > > > > > > > > > > > > later...)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I also changed the name of these
> > configurations to
> > > > "
> > > > > > > > > > > > > > > > broker.heartbeat.interval.ms" and "
> > > > > > > > > > broker.registration.timeout.ms"
> > > > > > > > > > > > to
> > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > > to clarify them a bit.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 204. "The highest metadata offset which the
> > > > broker
> > > > > > has
> > > > > > > > not
> > > > > > > > > > > > reached."
> > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > seems this should be "has reached".
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I changed this to "one more than the highest
> > > > metadata
> > > > > > > > offset
> > > > > > > > > > which
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > broker has reached."
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 205. UnfenceBrokerRecord and
> > > > UnregisterBrokerRecord:
> > > > > > To
> > > > > > > > me,
> > > > > > > > > > they
> > > > > > > > > > > > > > seem to
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > the same. Do we need both?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Unregistration means that the broker has been
> > > > removed
> > > > > > from
> > > > > > > > the
> > > > > > > > > > > > cluster.
> > > > > > > > > > > > > > > > That is different than unfencing, which marks
> > the
> > > > > > broker as
> > > > > > > > > > active.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 206. TopicRecord: The Deleting field is used
> > to
> > > > > > indicate
> > > > > > > > > > that the
> > > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > being deleted. I am wondering if this is
> > really
> > > > > > needed
> > > > > > > > since
> > > > > > > > > > > > > > RemoveTopic
> > > > > > > > > > > > > > > > > already indicates the same thing.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > RemoveTopic is the last step, that scrubs all
> > > > metadata
> > > > > > > > about
> > > > > > > > > > the
> > > > > > > > > > > > topic.
> > > > > > > > > > > > > > > > In order to get to that last step, the topic
> > data
> > > > > > needs to
> > > > > > > > > > removed
> > > > > > > > > > > > > > from all
> > > > > > > > > > > > > > > > brokers (after each broker notices that the
> > topic
> > > > is
> > > > > > being
> > > > > > > > > > > > deleted).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <
> > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron
> > Dagostino
> > > > wrote:
> > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the updates. It's
> > now
> > > > > > clear
> > > > > > > > to me
> > > > > > > > > > that
> > > > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > > > > keep their broker epoch for the life of
> > their
> > > > > > JVM --
> > > > > > > > they
> > > > > > > > > > > > > > register
> > > > > > > > > > > > > > > > > > > once, get their broker epoch in the
> > > > response, and
> > > > > > > > then
> > > > > > > > > > never
> > > > > > > > > > > > > > > > > > > re-register again. Brokers may get
> > fenced,
> > > > but
> > > > > > they
> > > > > > > > > > keep the
> > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > broker epoch for the life of their JVM.
> > The
> > > > > > > > incarnation
> > > > > > > > > > ID
> > > > > > > > > > > > is
> > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > kept for the life of the JVM but is
> > > > generated by
> > > > > > the
> > > > > > > > > > broker
> > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > upon startup, and the combination of the
> > two
> > > > > > allows
> > > > > > > > the
> > > > > > > > > > > > > > Controller to
> > > > > > > > > > > > > > > > > > > act idempotently if any previously-sent
> > > > > > registration
> > > > > > > > > > response
> > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks, Ron. That's a good summary.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > One thing I wonder about is if it might
> > be
> > > > > > helpful
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > send the Cluster ID as determined from
> > its
> > > > > > > > > > meta.properties
> > > > > > > > > > > > file
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > > registration request. Does it even make
> > > > sense
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > > successfully register and enter the
> > Fenced
> > > > state
> > > > > > if
> > > > > > > > it
> > > > > > > > > > has
> > > > > > > > > > > > the
> > > > > > > > > > > > > > wrong
> > > > > > > > > > > > > > > > > > > Cluster ID?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Yeah, that's a good idea. Let's have the
> > > > broker
> > > > > > pass
> > > > > > > > its
> > > > > > > > > > > > cluster
> > > > > > > > > > > > > > ID in
> > > > > > > > > > > > > > > > > > the registration RPC, and then
> > registration can
> > > > > > fail
> > > > > > > > if the
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The nextMetadatOffset value that the
> > broker
> > > > > > > > communicates
> > > > > > > > > > > > > > > > > > > in its registration request only has
> > meaning
> > > > > > within
> > > > > > > > the
> > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > cluster, so it feels to me that the
> > > > Controller
> > > > > > should
> > > > > > > > > > have
> > > > > > > > > > > > some
> > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > perform this sanity check. There is
> > > > currently
> > > > > > > > (pre-KIP
> > > > > > > > > > 500)
> > > > > > > > > > > > a
> > > > > > > > > > > > > > check
> > > > > > > > > > > > > > > > > > > in the broker to make sure its configured
> > > > > > cluster ID
> > > > > > > > > > matches
> > > > > > > > > > > > the
> > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > > stored in ZooKeeper, and we will have to
> > > > perform
> > > > > > this
> > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > somewhere in the KIP-500 world. If the
> > > > > > Controller
> > > > > > > > > > doesn't
> > > > > > > > > > > > do it
> > > > > > > > > > > > > > > > > > > within the registration request then the
> > > > broker
> > > > > > will
> > > > > > > > > > have to
> > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > metadata request to the Controller,
> > retrieve
> > > > the
> > > > > > > > Cluster
> > > > > > > > > > ID,
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > perform the check itself. It feels to me
> > > > that it
> > > > > > > > might
> > > > > > > > > > be
> > > > > > > > > > > > > > better for
> > > > > > > > > > > > > > > > > > > the Controller to just do it, and then
> > the
> > > > broker
> > > > > > > > doesn't
> > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > worry about it anymore once it
> > successfully
> > > > > > > > registers.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I also have a question about the
> > broker.id
> > > > > > value and
> > > > > > > > > > > > > > > > meta.properties.
> > > > > > > > > > > > > > > > > > > The KIP now says "In version 0 of
> > > > > > meta.properties,
> > > > > > > > there
> > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > broker.id field. Version 1 does not
> > have
> > > > this
> > > > > > > > field.
> > > > > > > > > > It
> > > > > > > > > > > > is no
> > > > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > > > > needed because we no longer support
> > dynamic
> > > > > > broker id
> > > > > > > > > > > > > > assignment."
> > > > > > > > > > > > > > > > > > > But then there is an example version 1
> > > > > > > > meta.properties
> > > > > > > > > > file
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > shows
> > > > > > > > > > > > > > > > > > > the broker.id value. I actually wonder
> > if
> > > > > > maybe the
> > > > > > > > > > > > broker.id
> > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > would be good to keep in the version 1
> > > > > > > > meta.properties
> > > > > > > > > > file
> > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > currently (pre-KIP 500, version 0) acts
> > as a
> > > > > > sanity
> > > > > > > > > > check to
> > > > > > > > > > > > make
> > > > > > > > > > > > > > > > sure
> > > > > > > > > > > > > > > > > > > the broker is using the correct log
> > > > directory.
> > > > > > > > Similarly
> > > > > > > > > > > > with
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > controller.id value on controllers -- it
> > > > would
> > > > > > > > allow the
> > > > > > > > > > > > same
> > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > sanity check for quorum controllers.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > That's a good point. I will add broker.id
> > > > back,
> > > > > > and
> > > > > > > > also
> > > > > > > > > > add
> > > > > > > > > > > > > > > > > > controller.id as a possibility.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin
> > McCabe
> > > > <
> > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao
> > > > wrote:
> > > > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for the reply. A few more
> > > > comments.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks again for the reply. Sorry for
> > the
> > > > long
> > > > > > > > > > hiatus. I
> > > > > > > > > > > > was
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > vacation for a while.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 55. There is still text that favors
> > new
> > > > > > broker
> > > > > > > > > > > > registration.
> > > > > > > > > > > > > > > > "When a
> > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > first starts up, when it is in the
> > > > INITIAL
> > > > > > > > state, it
> > > > > > > > > > will
> > > > > > > > > > > > > > always
> > > > > > > > > > > > > > > > > > "win"
> > > > > > > > > > > > > > > > > > > > > broker ID conflicts. However, once
> > it is
> > > > > > > > granted a
> > > > > > > > > > > > lease, it
> > > > > > > > > > > > > > > > > > transitions
> > > > > > > > > > > > > > > > > > > > > out of the INITIAL state.
> > Thereafter,
> > > > it may
> > > > > > > > lose
> > > > > > > > > > > > subsequent
> > > > > > > > > > > > > > > > > > conflicts if
> > > > > > > > > > > > > > > > > > > > > its broker epoch is stale. (See
> > KIP-380
> > > > for
> > > > > > some
> > > > > > > > > > > > background
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > epoch.) The reason for favoring new
> > > > > > processes
> > > > > > > > is to
> > > > > > > > > > > > > > accommodate
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > > > > > case where a process is killed with
> > kill
> > > > -9
> > > > > > and
> > > > > > > > then
> > > > > > > > > > > > > > restarted.
> > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > want it
> > > > > > > > > > > > > > > > > > > > > to be able to reclaim its old ID
> > quickly
> > > > in
> > > > > > this
> > > > > > > > > > case."
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for the reminder. I have
> > clarified
> > > > the
> > > > > > > > language
> > > > > > > > > > > > here.
> > > > > > > > > > > > > > > > > > Hopefully now it is clear that we don't
> > allow
> > > > quick
> > > > > > > > re-use
> > > > > > > > > > of
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 80.1 Sounds good. Could you document
> > that
> > > > > > > > listeners
> > > > > > > > > > is a
> > > > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > > now? It would also be useful to
> > annotate
> > > > > > other
> > > > > > > > > > required
> > > > > > > > > > > > > > configs.
> > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > > example, controller.connect should be
> > > > > > required.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I added a note specifying that these
> > are
> > > > > > required.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 80.2 Could you list all deprecated
> > > > existing
> > > > > > > > configs?
> > > > > > > > > > > > Another
> > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > control.plane.listener.name since
> > the
> > > > > > > > controller no
> > > > > > > > > > > > longer
> > > > > > > > > > > > > > sends
> > > > > > > > > > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata and
> > > > StopReplica
> > > > > > > > > > requests.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I added a section specifying some
> > > > deprecated
> > > > > > > > configs.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 83.1 It seems that the broker can
> > > > transition
> > > > > > from
> > > > > > > > > > FENCED
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > RUNNING
> > > > > > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > > > > > registering for a new broker epoch.
> > I am
> > > > not
> > > > > > > > sure how
> > > > > > > > > > > > this
> > > > > > > > > > > > > > works.
> > > > > > > > > > > > > > > > > > Once the
> > > > > > > > > > > > > > > > > > > > > controller fences a broker, there is
> > no
> > > > need
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > > > > > > boker epoch around. So, if the fenced
> > > > > > broker's
> > > > > > > > > > heartbeat
> > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > > > existing broker epoch will be
> > rejected,
> > > > > > leading
> > > > > > > > the
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The broker epoch refers to the broker
> > > > > > registration.
> > > > > > > > > > So we
> > > > > > > > > > > > DO
> > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > broker epoch around even while the broker
> > is
> > > > > > fenced.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The broker epoch changes only when
> > there
> > > > is a
> > > > > > new
> > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > registration. Fencing or unfencing the
> > broker
> > > > > > doesn't
> > > > > > > > > > change
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 83.5 Good point on KIP-590. Then
> > should
> > > > we
> > > > > > > > expose the
> > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > debugging purposes? If not, we should
> > > > > > deprecate
> > > > > > > > the
> > > > > > > > > > > > > > controllerID
> > > > > > > > > > > > > > > > > > field in
> > > > > > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think it's OK to expose it for now,
> > with
> > > > the
> > > > > > > > proviso
> > > > > > > > > > > > that it
> > > > > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > > > > be reachable by clients.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 90. We rejected the shared ID with
> > just
> > > > one
> > > > > > > > reason
> > > > > > > > > > "This
> > > > > > > > > > > > is
> > > > > > > > > > > > > > not a
> > > > > > > > > > > > > > > > > > good idea
> > > > > > > > > > > > > > > > > > > > > because NetworkClient assumes a
> > single ID
> > > > > > > > space. So
> > > > > > > > > > if
> > > > > > > > > > > > > > there is
> > > > > > > > > > > > > > > > > > both a
> > > > > > > > > > > > > > > > > > > > > controller 1 and a broker 1, we don't
> > > > have a
> > > > > > way
> > > > > > > > of
> > > > > > > > > > > > picking
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > "right"
> > > > > > > > > > > > > > > > > > > > > one." This doesn't seem to be a
> > strong
> > > > > > reason.
> > > > > > > > For
> > > > > > > > > > > > example,
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > > > > address the NetworkClient issue with
> > the
> > > > node
> > > > > > > > type
> > > > > > > > > > as you
> > > > > > > > > > > > > > pointed
> > > > > > > > > > > > > > > > > > out or
> > > > > > > > > > > > > > > > > > > > > using the negative value of a broker
> > ID
> > > > as
> > > > > > the
> > > > > > > > > > > > controller ID.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > It would require a lot of code changes
> > to
> > > > > > support
> > > > > > > > > > multiple
> > > > > > > > > > > > > > types of
> > > > > > > > > > > > > > > > > > node IDs. It's not clear to me that the
> > end
> > > > result
> > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > > > > better --
> > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > tend to think it would be worse, since it
> > > > would be
> > > > > > more
> > > > > > > > > > > > complex.
> > > > > > > > > > > > > > In a
> > > > > > > > > > > > > > > > > > similar vein, using negative numbers seems
> > > > > > dangerous,
> > > > > > > > > > since we
> > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > negatives or -1 as "special values" in many
> > > > places.
> > > > > > > > For
> > > > > > > > > > > > example,
> > > > > > > > > > > > > > -1
> > > > > > > > > > > > > > > > often
> > > > > > > > > > > > > > > > > > represents "no such node."
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > One important thing to keep in mind is
> > > > that we
> > > > > > > > want to
> > > > > > > > > > be
> > > > > > > > > > > > able
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > transition from a broker and a controller
> > being
> > > > > > > > co-located
> > > > > > > > > > to
> > > > > > > > > > > > them
> > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > > longer being co-located. This is much
> > easier
> > > > to do
> > > > > > > > when
> > > > > > > > > > they
> > > > > > > > > > > > have
> > > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > > the broker reports all offline
> > replicas
> > > > due
> > > > > > to a
> > > > > > > > disk
> > > > > > > > > > > > > > failure to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > controller. It seems this information
> > > > needs
> > > > > > to be
> > > > > > > > > > > > persisted
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > > > > log. Do we have a corresponding
> > record
> > > > for
> > > > > > that?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hmm, I have to look into this a little
> > bit
> > > > > > more.
> > > > > > > > We
> > > > > > > > > > may
> > > > > > > > > > > > need
> > > > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > > > record type.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 101. Currently, StopReplica request
> > has 2
> > > > > > modes,
> > > > > > > > > > without
> > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > and with
> > > > > > > > > > > > > > > > > > > > > deletion. The former is used for
> > > > controlled
> > > > > > > > shutdown
> > > > > > > > > > and
> > > > > > > > > > > > > > handling
> > > > > > > > > > > > > > > > > > disk
> > > > > > > > > > > > > > > > > > > > > failure, and causes the follower to
> > > > stop. The
> > > > > > > > latter
> > > > > > > > > > is
> > > > > > > > > > > > for
> > > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > > > > and partition reassignment, and
> > causes
> > > > the
> > > > > > > > replica
> > > > > > > > > > to be
> > > > > > > > > > > > > > deleted.
> > > > > > > > > > > > > > > > > > Since we
> > > > > > > > > > > > > > > > > > > > > are deprecating StopReplica, could we
> > > > > > document
> > > > > > > > what
> > > > > > > > > > > > triggers
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > stopping
> > > > > > > > > > > > > > > > > > > > > of a follower and the deleting of a
> > > > replica
> > > > > > now?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > RemoveTopic triggers deletion. In
> > general
> > > > the
> > > > > > > > > > > > functionality of
> > > > > > > > > > > > > > > > > > StopReplica is subsumed by the metadata
> > > > records.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 102. Should we include the metadata
> > > > topic in
> > > > > > the
> > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > If so,
> > > > > > > > > > > > > > > > > > > > > when it will be included and what
> > will
> > > > the
> > > > > > > > metadata
> > > > > > > > > > > > response
> > > > > > > > > > > > > > look
> > > > > > > > > > > > > > > > > > like?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > No, it won't be included in the
> > metadata
> > > > > > response
> > > > > > > > sent
> > > > > > > > > > back
> > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > brokers.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 103. "The active controller assigns
> > the
> > > > > > broker a
> > > > > > > > new
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > epoch,
> > > > > > > > > > > > > > > > > > based on
> > > > > > > > > > > > > > > > > > > > > the latest committed offset in the
> > log."
> > > > This
> > > > > > > > seems
> > > > > > > > > > > > > > inaccurate
> > > > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > latest committed offset doesn't
> > always
> > > > > > advance on
> > > > > > > > > > every
> > > > > > > > > > > > log
> > > > > > > > > > > > > > > > append.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Given that the new broker epoch won't
> > be
> > > > > > visible
> > > > > > > > until
> > > > > > > > > > the
> > > > > > > > > > > > > > commit
> > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > happened, I have changed this to "the next
> > > > > > available
> > > > > > > > > > offset in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > log"
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 104. REGISTERING(1) : It says
> > > > "Otherwise, the
> > > > > > > > broker
> > > > > > > > > > > > moves
> > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > > > > > > > state.". It seems this should be
> > RUNNING?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 105. RUNNING: Should we require the
> > > > broker to
> > > > > > > > catch
> > > > > > > > > > up
> > > > > > > > > > > > to the
> > > > > > > > > > > > > > > > > > metadata log
> > > > > > > > > > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > For 104 and 105, these sections have
> > been
> > > > > > reworked.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin
> > > > McCabe
> > > > > > <
> > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom
> > > > Bentley
> > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at 08:59,
> > Ron
> > > > > > Dagostino
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the
> > hard
> > > > work
> > > > > > on
> > > > > > > > this
> > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I have some questions about
> > what
> > > > > > happens
> > > > > > > > to a
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > when it
> > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > > > > fenced (e.g. because it can't
> > > > send a
> > > > > > > > > > heartbeat
> > > > > > > > > > > > > > request to
> > > > > > > > > > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > > > > > > > > > lease). The KIP says "When a
> > > > broker
> > > > > > is
> > > > > > > > > > fenced,
> > > > > > > > > > > > it
> > > > > > > > > > > > > > cannot
> > > > > > > > > > > > > > > > > > process any
> > > > > > > > > > > > > > > > > > > > > > > > > client requests. This
> > prevents
> > > > > > brokers
> > > > > > > > which
> > > > > > > > > > > > are not
> > > > > > > > > > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > > > > > > > > > metadata updates or that are
> > not
> > > > > > > > receiving
> > > > > > > > > > and
> > > > > > > > > > > > > > processing
> > > > > > > > > > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > > > > > > > > > enough from causing issues to
> > > > > > clients."
> > > > > > > > And
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > > > > > > > > > FENCED(4) state it likewise
> > says
> > > > > > "While
> > > > > > > > in
> > > > > > > > > > this
> > > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > > > > > not respond to client
> > > > requests." It
> > > > > > > > makes
> > > > > > > > > > sense
> > > > > > > > > > > > > > that a
> > > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > > > > should not accept producer
> > > > requests
> > > > > > -- I
> > > > > > > > > > assume
> > > > > > > > > > > > any
> > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > > > > > > > > would result in
> > > > > > > > NotLeaderOrFollowerException.
> > > > > > > > > > > > But
> > > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > > > > > > > > > (fetch from follower)
> > consumer
> > > > > > > > requests? It
> > > > > > > > > > is
> > > > > > > > > > > > > > > > conceivable
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > > > > could continue. Related to
> > that,
> > > > > > would a
> > > > > > > > > > fenced
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > > > > > fetch data for partitions
> > where
> > > > it
> > > > > > > > thinks it
> > > > > > > > > > is a
> > > > > > > > > > > > > > > > follower?
> > > > > > > > > > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > > > > > > > > > it rejects consumer requests
> > it
> > > > might
> > > > > > > > still
> > > > > > > > > > > > continue
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > > > > > > > > > follower. Might it be
> > helpful to
> > > > > > clarify
> > > > > > > > > > both
> > > > > > > > > > > > > > decisions
> > > > > > > > > > > > > > > > > > here?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Good question. I think a
> > fenced
> > > > broker
> > > > > > > > should
> > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > > > > > > > > > partitions it was already
> > fetching
> > > > > > before
> > > > > > > > it
> > > > > > > > > > was
> > > > > > > > > > > > > > fenced,
> > > > > > > > > > > > > > > > > > unless it
> > > > > > > > > > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > > > > > > > > > problem. At that point it
> > won't be
> > > > > > able to
> > > > > > > > > > > > continue,
> > > > > > > > > > > > > > > > since it
> > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > > the new metadata. For
> > example, it
> > > > > > won't
> > > > > > > > know
> > > > > > > > > > about
> > > > > > > > > > > > > > > > leadership
> > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > the partitions it's fetching.
> > The
> > > > > > > > rationale
> > > > > > > > > > for
> > > > > > > > > > > > > > > > continuing to
> > > > > > > > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > > > > > > try to avoid disruptions as
> > much as
> > > > > > > > possible.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I don't think fenced brokers
> > should
> > > > > > accept
> > > > > > > > > > client
> > > > > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > > > The issue
> > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > that the fenced broker may or
> > may
> > > > not
> > > > > > have
> > > > > > > > any
> > > > > > > > > > > > data it
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > > > > > > > > > have. It may or may not have
> > > > applied
> > > > > > any
> > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > it is supposed to have applied.
> > > > So it
> > > > > > > > could
> > > > > > > > > > get
> > > > > > > > > > > > pretty
> > > > > > > > > > > > > > > > > > confusing, and
> > > > > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > potentially waste the client's
> > > > time.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > When fenced, how would the broker
> > > > reply
> > > > > > to a
> > > > > > > > > > client
> > > > > > > > > > > > > > which did
> > > > > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The broker will respond with a
> > > > retryable
> > > > > > error
> > > > > > > > in
> > > > > > > > > > that
> > > > > > > > > > > > > > case.
> > > > > > > > > > > > > > > > Once
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > client has re-fetched its
> > metadata, it
> > > > > > will no
> > > > > > > > > > longer
> > > > > > > > > > > > see
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > as part of the cluster. I added a
> > > > note to
> > > > > > the
> > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the reply.
211. Hmm, I still don't see a clear benefit of registering a broker before
recovery. It's possible for the recovery to take time. However, during the
recovery mode, it seems the broker will still be in the fenced mode and
won't be able to do work for the clients. So, registering and heartbeating
early seems to only add unnecessary overhead. For your point on topic
creation, I thought we now allow replicas to be created on
unregistered brokers.
230. Currently, we do have a ControllerId field in MetadataResponse. In the
early discussion, I thought that we want to expose the controller for
debugging purposes, but not used by the client library.
Jun
On Wed, Dec 16, 2020 at 9:13 PM Colin McCabe <cm...@apache.org> wrote:
> On Wed, Dec 16, 2020, at 18:13, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the reply. Just a couple of more comments.
> >
> > 211. Currently, the broker only registers itself in ZK after log
> recovery.
> > Is there any benefit to change that? As you mentioned, the broker can't
> do
> > much before completing log recovery.
> >
>
> Hi Jun,
>
> Previously, it wasn't possible to register in ZK without immediately
> getting added to the MetadataResponse. So I think that's the main reason
> why registration was delayed until after log recovery. Since that
> constraint doesn't exist any more, there seems to be no reason to delay
> registration.
>
> I think delaying registration would have some major downsides. If log
> recovery takes a while, that's a longer window during which someone else
> could register a broker with the same ID. Having parts of the cluster
> missing for a while gives up some of the benefit of separating registration
> from fencing. For example, if a broker somehow gets unregistered and we
> want to re-register it, but we have to wait for a 10 minute log recovery to
> do that, that could be a window during which topics can't be created that
> need to be on that broker, and so forth.
>
> > 230. Regarding MetadataResponse, there is a slight awkwardness. We return
> > rack for each node. However, if that node is for the controller, the rack
> > field is not really relevant. Should we clean it up here or in another
> KIP
> > like KIP-700?
>
> Oh, controllers don't appear in the MetadataResponses returned to clients,
> since clients can't access them. I should have been more clear about that
> in the KIP-- I added a sentence to "Networking" describing this.
>
> best,
> Colin
>
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Dec 16, 2020 at 4:23 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Wed, Dec 16, 2020, at 13:40, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply. A few follow up comments.
> > > >
> > > > 211. When does the broker send the BrokerRegistration request to the
> > > > controller? Is it after the recovery phase? If so, at that point, the
> > > > broker has already caught up on the metadata (in order to clean up
> > > deleted
> > > > partitions). Then, it seems we don't need the ShouldFence field
> > > > in BrokerHeartbeatRequest?
> > >
> > > Hi Jun,
> > >
> > > Thanks again for the reviews.
> > >
> > > The broker sends the registration request as soon as it starts up. It
> > > cannot wait until the recovery phase is over since sometimes log
> recovery
> > > can take quite a long time.
> > >
> > > >
> > > > 213. kafka-cluster.sh
> > > > 213.1 For the decommision example, should the command take a broker
> id?
> > >
> > > Yes, the example should have a broker id. Fixed.
> > >
> > > > 213.2 Existing tools use the "--" command line option (e.g.
> kafka-topics
> > > > --list --topic test). Should we follow the same convention
> > > > for kafka-cluster.sh (and kafka-storage.sh)?
> > >
> > > Hmm. I don't think argparse4j supports using double dashes to identify
> > > subcommands. I think it might be confusing as well, since the
> subcommand
> > > must come first, unlike a plain old argument which can be anywhere on
> the
> > > command line.
> > >
> > > > 213.3 Should we add an admin api for broker decommission so that
> this can
> > > > be done programmatically?
> > > >
> > >
> > > Yes, there is an admin client API for decommissioning as well.
> > >
> > > > 220. DecommissionBroker: "NOT_CONTROLLER if the node that the
> request was
> > > > sent to is not the controller". I thought the clients never send a
> > > request
> > > > to the controller directly now and the broker will forward it to the
> > > > controller?
> > > >
> > >
> > > If the controller moved recently, it's possible that the broker could
> send
> > > to a controller that has just recently become inactive. In that case
> > > NOT_CONTROLLER would be returned. (A standby controller returns
> > > NOT_CONTROLLER for most APIs).
> > >
> > > > 221. Could we add the required ACL for the new requests?
> > > >
> > >
> > > Good point. I added the required ACL for each new RPC.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > > Jun
> > > >
> > > > On Wed, Dec 16, 2020 at 11:38 AM Colin McCabe <cm...@apache.org>
> > > wrote:
> > > >
> > > > > On Wed, Dec 16, 2020, at 09:59, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the reply. A few more comments below.
> > > > > >
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > Thanks for the comments.
> > > > >
> > > > > > 206. "RemoveTopic is the last step, that scrubs all metadata
> about
> > > the
> > > > > > topic. In order to get to that last step, the topic data needs
> to
> > > > > removed
> > > > > > from all brokers (after each broker notices that the topic is
> being
> > > > > > deleted)." Currently, this is done based on the response of
> > > > > > StopReplicaRequest. Since the controller no longer sends this
> > > request,
> > > > > how
> > > > > > does the controller know that the data for the deleted topic has
> > > > > > been removed in the brokers?
> > > > > >
> > > > >
> > > > > That's a good point... definitely an oversight.
> > > > >
> > > > > It seems complex to force the controller to track when log
> directories
> > > > > have been deleted. Let's just assume that KIP-516 has been
> > > implemented,
> > > > > and track them by UUID. Then we can just have a single topic
> deletion
> > > > > record.
> > > > >
> > > > > I added a section on "topic identifiers" describing this.
> > > > >
> > > > > > 210. Thanks for the explanation. Sounds good to me.
> > > > > >
> > > > > > 211. I still don't see an example when ShouldFence is set to
> true in
> > > > > > BrokerHeartbeatReques. Could we add one?
> > > > > >
> > > > >
> > > > > It is sent to true when the broker is first starting up and
> doesn't yet
> > > > > want to be unfenced. I added a longer explanation of this in the
> > > "Broker
> > > > > Leases" section.
> > > > >
> > > > > > 213. The KIP now allows replicas to be assigned on brokers that
> are
> > > > > fenced,
> > > > > > which is an improvement. How do we permanently remove a broker
> (e.g.
> > > > > > cluster shrinking) to prevent it from being used for future
> replica
> > > > > > assignments?
> > > > > >
> > > > >
> > > > > This is a fair point. I will create a kafka-cluster.sh script
> which
> > > can
> > > > > do this, plus a DecommissionBrokerRequest.
> > > > >
> > > > > As a bonus the kafka-cluster.sh script can help people find the
> > > cluster ID
> > > > > of brokers, something that people have wanted a tool for in the
> past.
> > > > >
> > > > > > 214. "Currently, when a node is down, all of its ZK registration
> > > > > > information is gone. But we need this information in order to
> > > > > understand
> > > > > > things like whether the replicas of a particular partition are
> > > > > > well-balanced across racks." This is not quite true. Currently,
> even
> > > when
> > > > > > ZK registration is gone, the existing replica assignment is still
> > > > > available
> > > > > > in the metadata response.
> > > > > >
> > > > >
> > > > > I agree that the existing replica assignment is still available.
> But
> > > that
> > > > > just tells you partition X is on nodes A, B, and C. If you don't
> have
> > > the
> > > > > ZK registration for one or more of A, B, or C then you don't know
> > > whether
> > > > > we are following the policy of "two replicas on one rack, one
> replica
> > > on
> > > > > another." Or any other more complex rack placement policy that you
> > > might
> > > > > have.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, Dec 16, 2020 at 8:48 AM Colin McCabe <cmccabe@apache.org
> >
> > > wrote:
> > > > > >
> > > > > > > On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the reply. A few more follow up comments.
> > > > > > > >
> > > > > > > > 210. initial.broker.registration.timeout.ms: The default
> value
> > > is
> > > > > 90sec,
> > > > > > > > which seems long. If a broker fails the registration because
> of
> > > > > incorrect
> > > > > > > > configs, we want to fail faster. In comparison, the defaults
> for
> > > > > > > > zookeeper.connection.timeout.ms is 18 secs.
> > > > > > > >
> > > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > I agree that the initial connection timeout here is longer than
> > > what we
> > > > > > > had with ZK. The main reason I selected a slightly longer
> timeout
> > > > > here is
> > > > > > > to handle the case where the controllers and the brokers are
> > > > > co-located.
> > > > > > > For example, if you have a 3 node cluster and all three nodes
> are
> > > > > > > controllers+brokers, when you first bring up the cluster, we
> will
> > > have
> > > > > to
> > > > > > > stand up the controller quorum and then handle broker
> > > registrations.
> > > > > > > Although we believe Raft will start up relatively quickly, it's
> > > good to
> > > > > > > leave some extra margin here.
> > > > > > >
> > > > > > > I don't think there's a big disadvantage to having a slightly
> > > longer
> > > > > > > timeout here. After all, starting up the brokers with no
> > > controllers
> > > > > is an
> > > > > > > admin mistake, which we don't expect to see very often. Maybe
> > > let's
> > > > > set it
> > > > > > > to 60 seconds for now and maybe see if we can tweak it in the
> > > future if
> > > > > > > that turns out to be too long or short?
> > > > > > >
> > > > > > > > 211. "Once they are all moved, the controller responds to the
> > > > > heartbeat
> > > > > > > > with a nextState of SHUTDOWN." It seems that nextState is no
> > > longer
> > > > > > > > relevant. Also, could you add a bit more explanation on when
> > > > > ShouldFence
> > > > > > > is
> > > > > > > > set to true in BrokerHeartbeatRequest?
> > > > > > > >
> > > > > > >
> > > > > > > Good catch. I removed the obsolete section referring to
> nextState
> > > and
> > > > > > > added a reference to the new boolean. I also added some more
> > > > > information
> > > > > > > about ShouldFence and about the rationale for separating
> fencing
> > > from
> > > > > > > registration.
> > > > > > >
> > > > > > > > 212. Related to the above, what does the broker do if
> IsFenced is
> > > > > true in
> > > > > > > > the BrokerHeartbeatResponse? Will the broker do the same
> thing on
> > > > > > > receiving
> > > > > > > > a FenceBrokerRecord from the metadata log?
> > > > > > > >
> > > > > > >
> > > > > > > The broker only checks this boolean during the startup process.
> > > After
> > > > > the
> > > > > > > startup process is finished, it ignores this boolean.
> > > > > > >
> > > > > > > The broker uses fence / unfence records in the metadata log to
> > > > > determine
> > > > > > > which brokers should appear in its MetadataResponse.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <
> cmccabe@apache.org
> > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > > > > > > > > Hi Colin,
> > > > > > > > > >
> > > > > > > > > > The KIP says that "brokers which are fenced will not
> appear
> > > in
> > > > > > > > > > MetadataResponses. So clients that have up-to-date
> metadata
> > > will
> > > > > > > not try
> > > > > > > > > > to contact fenced brokers.", which is fine, but it
> doesn't
> > > seem
> > > > > to
> > > > > > > > > > elaborate on what happens for clients which try to
> contact a
> > > > > broker
> > > > > > > > > (using
> > > > > > > > > > stale metadata, for example) and find it in the
> > > > > registered-but-fenced
> > > > > > > > > > state.
> > > > > > > > >
> > > > > > > > > Hi Tom,
> > > > > > > > >
> > > > > > > > > I have dropped the broker-side fencing from this
> proposal. So
> > > now
> > > > > the
> > > > > > > > > fencing is basically the same as today's fencing: it's
> > > > > controller-side
> > > > > > > > > only. That means that clients using stale metadata can
> contact
> > > > > fenced
> > > > > > > > > brokers and communicate with them. The only case where the
> > > broker
> > > > > > > will be
> > > > > > > > > inaccessible is when it hasn't finished starting yet
> (i.e., has
> > > > > not yet
> > > > > > > > > attained RUNNING state.)
> > > > > > > > >
> > > > > > > > > Just like today, we expect the safeguards built into the
> > > > > replication
> > > > > > > > > protocol to prevent the worst corner cases that could
> result
> > > from
> > > > > > > this. I
> > > > > > > > > do think we will probably take up this issue later and
> find a
> > > > > better
> > > > > > > > > solution for client-side metadata, but this KIP is big
> enough
> > > > > as-is.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > You said previously that the broker will respond with a
> > > retriable
> > > > > > > > > > error, which is again fine, but you didn't answer my
> question
> > > > > about
> > > > > > > which
> > > > > > > > > > error code you've chosen for this. I realise that this
> > > doesn't
> > > > > really
> > > > > > > > > > affect the correctness of the behaviour, but I'm not
> aware
> > > of any
> > > > > > > > > existing
> > > > > > > > > > error code which is a good fit. So it would be good to
> > > understand
> > > > > > > about
> > > > > > > > > how
> > > > > > > > > > you're making this backward compatible for clients.
> > > > > > > > > >
> > > > > > > > > > Many thanks,
> > > > > > > > > >
> > > > > > > > > > Tom
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <
> > > cmccabe@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the reply. Just a couple of more comments
> > > below.
> > > > > > > > > > > >
> > > > > > > > > > > > 210. Since we are deprecating
> > > > > zookeeper.connection.timeout.ms,
> > > > > > > > > should we
> > > > > > > > > > > > add a new config to bound the time for a broker to
> > > connect
> > > > > to the
> > > > > > > > > > > > controller during starting up?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Good idea. I added
> initial.broker.registration.timeout.ms
> > > for
> > > > > > > this.
> > > > > > > > > > >
> > > > > > > > > > > > 211. BrokerHeartbeat no longer has the state field
> in the
> > > > > > > > > > > request/response.
> > > > > > > > > > > > However, (a) the controller shutdown section still
> has
> > > "In
> > > > > its
> > > > > > > > > periodic
> > > > > > > > > > > > heartbeats, the broker asks the controller if it can
> > > > > transition
> > > > > > > into
> > > > > > > > > the
> > > > > > > > > > > > SHUTDOWN state. This motivates the controller to
> move
> > > all
> > > > > of the
> > > > > > > > > leaders
> > > > > > > > > > > > off of that broker. Once they are all moved, the
> > > controller
> > > > > > > > > responds to
> > > > > > > > > > > > the heartbeat with a nextState of SHUTDOWN."; (2) the
> > > > > > > description of
> > > > > > > > > > > > BrokerHeartbeat still references currentState and
> > > > > targetState.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks. I've made these sections clearer and removed
> the
> > > > > obsolete
> > > > > > > > > > > references to sending states.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > > Jun
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <
> > > > > cmccabe@apache.org
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the update. A few more follow up
> comments.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks again for the review.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 100. FailedReplicaRecord: Since this is reported
> by
> > > each
> > > > > > > broker
> > > > > > > > > > > > > > independently, perhaps we could use a more
> concise
> > > > > > > representation
> > > > > > > > > > > that
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > a top level broker field, an array of topics,
> which
> > > has
> > > > > an
> > > > > > > array
> > > > > > > > > of
> > > > > > > > > > > > > > partitions.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > The issue is that there is a size limit on the each
> > > record.
> > > > > > > > > Putting
> > > > > > > > > > > all
> > > > > > > > > > > > > of the partitions of a log directory into a single
> > > record
> > > > > would
> > > > > > > > > > > probably
> > > > > > > > > > > > > break that in many cases. Still, we can optimize a
> > > bit by
> > > > > > > having
> > > > > > > > > an
> > > > > > > > > > > array
> > > > > > > > > > > > > of partition IDs, since nearly all the time, we
> have
> > > more
> > > > > than
> > > > > > > one
> > > > > > > > > > > from the
> > > > > > > > > > > > > same topic.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 200. Sounds good. If we remove the broker-side
> > > fencing
> > > > > > > logic, do
> > > > > > > > > we
> > > > > > > > > > > plan
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > still keep FENCED in broker state? Do we plan to
> > > expose
> > > > > the
> > > > > > > new
> > > > > > > > > > > states
> > > > > > > > > > > > > > through the existing BrokerState metric and if
> so,
> > > what
> > > > > are
> > > > > > > the
> > > > > > > > > > > values
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > the new states?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > No, we don't need FENCED any more. I have removed
> it
> > > from
> > > > > the
> > > > > > > KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The new states are very similar to the current
> ones,
> > > > > actually.
> > > > > > > > > There
> > > > > > > > > > > are
> > > > > > > > > > > > > no new states or removed ones. The main change in
> the
> > > > > broker
> > > > > > > state
> > > > > > > > > > > machine
> > > > > > > > > > > > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state
> has
> > > been
> > > > > > > > > renamed to
> > > > > > > > > > > > > RECOVERY. Also, unlike previously, the broker will
> > > always
> > > > > pass
> > > > > > > > > through
> > > > > > > > > > > > > RECOVERY (although it may only stay in this state
> for a
> > > > > very
> > > > > > > short
> > > > > > > > > > > amount
> > > > > > > > > > > > > of time).
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 201. This may be fine too. Could we document what
> > > happens
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > > > > > broker.id/controller.id in metadata.properties
> don't
> > > > > match
> > > > > > > the
> > > > > > > > > > > broker
> > > > > > > > > > > > > > config when the broker starts up?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I added some documentation about this.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 204. There is still "The highest metadata offset
> > > which
> > > > > the
> > > > > > > broker
> > > > > > > > > > > has not
> > > > > > > > > > > > > > reached" referenced under BrokerRegistration.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 206. Is that separate step needed given KIP-516?
> With
> > > > > > > KIP-516 (
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > > > > > > > > ),
> > > > > > > > > > > > > > we don't need to wait for the topic data to be
> > > removed
> > > > > from
> > > > > > > all
> > > > > > > > > > > brokers
> > > > > > > > > > > > > > before removing the topic metadata. The
> combination
> > > of
> > > > > > > unmatching
> > > > > > > > > > > > > > topicId
> > > > > > > > > > > > > > or the missing topicId from the metadata is
> enough
> > > for
> > > > > the
> > > > > > > > > broker to
> > > > > > > > > > > > > > clean
> > > > > > > > > > > > > > up deleted topics asynchronously.
> > > > > > > > > > > > >
> > > > > > > > > > > > > It won't be needed once KIP-516 is adopted, but
> this
> > > hasn't
> > > > > > > been
> > > > > > > > > > > > > implemented yet.
> > > > > > > > > > > > >
> > > > > > > > > > > > > best,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <
> > > > > > > cmccabe@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the updated KIP. A few more
> comments
> > > > > below.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks again for the reviews.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 80.2 For deprecated configs, we need to
> include
> > > > > > > zookeeper.*
> > > > > > > > > and
> > > > > > > > > > > > > > > > broker.id.generation.enable.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Added.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 83.1 If a broker is down, does the controller
> > > keep
> > > > > the
> > > > > > > > > previously
> > > > > > > > > > > > > > > > registered broker epoch forever? If not, how
> long
> > > > > does
> > > > > > > the
> > > > > > > > > > > controller
> > > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > > it? What does the controller do when
> receiving a
> > > > > broker
> > > > > > > > > heartbeat
> > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > with an unfound broker epoch?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Yes, the controller keeps the previous
> registration
> > > > > > > forever.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Broker heartbeat requests with an incorrect
> broker
> > > > > epoch
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > > rejected
> > > > > > > > > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 100. Have you figured out if we need to add
> a new
> > > > > record
> > > > > > > > > type for
> > > > > > > > > > > > > > > reporting
> > > > > > > > > > > > > > > > partitions on failed disks?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I added FailedReplicaRecord to reflect the case
> > > where a
> > > > > > > JBOD
> > > > > > > > > > > directory
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > failed, leading to failed replicas.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 102. For debugging purposes, sometimes it's
> > > useful to
> > > > > > > read
> > > > > > > > > the
> > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > topic using tools like console-consumer.
> Should
> > > we
> > > > > > > support
> > > > > > > > > that
> > > > > > > > > > > and
> > > > > > > > > > > > > if
> > > > > > > > > > > > > > > so,
> > > > > > > > > > > > > > > > how?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For now, we have the ability to read the
> metadata
> > > logs
> > > > > > > with the
> > > > > > > > > > > > > dump-logs
> > > > > > > > > > > > > > > tool. I think we will come up with some other
> > > tools
> > > > > in the
> > > > > > > > > future
> > > > > > > > > > > as
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > get experience.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 200. "brokers which are fenced will not
> appear in
> > > > > > > > > > > MetadataResponses.
> > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > broker will not respond to these requests--
> > > instead,
> > > > > it
> > > > > > > will
> > > > > > > > > > > simply
> > > > > > > > > > > > > > > > disconnect." If the controller is
> partitioned off
> > > > > from
> > > > > > > the
> > > > > > > > > > > brokers,
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > design will cause every broker to stop
> accepting
> > > new
> > > > > > > client
> > > > > > > > > > > > > requests. In
> > > > > > > > > > > > > > > > contrast, if ZK is partitioned off, the
> existing
> > > > > > > behavior is
> > > > > > > > > > > that the
> > > > > > > > > > > > > > > > brokers can continue to work based on the
> last
> > > known
> > > > > > > > > metadata.
> > > > > > > > > > > So, I
> > > > > > > > > > > > > am
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > sure if we should change the existing
> behavior
> > > > > because
> > > > > > > of the
> > > > > > > > > > > bigger
> > > > > > > > > > > > > > > impact
> > > > > > > > > > > > > > > > in the new one. Another option is to keep the
> > > > > existing
> > > > > > > > > behavior
> > > > > > > > > > > and
> > > > > > > > > > > > > > > expose
> > > > > > > > > > > > > > > > a metric for fenced brokers so that the
> operator
> > > > > could be
> > > > > > > > > > > alerted.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm skeptical about how well running without ZK
> > > > > currently
> > > > > > > > > works.
> > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > I will move the broker-side fencing into a
> > > follow-up
> > > > > KIP.
> > > > > > > This
> > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > already pretty large and there is no hard
> > > dependency on
> > > > > > > this.
> > > > > > > > > > > There
> > > > > > > > > > > > > may
> > > > > > > > > > > > > > > also be other ways of accomplishing the
> positive
> > > > > effects of
> > > > > > > > > what
> > > > > > > > > > > > > > > broker-side fencing, so more discussion is
> needed.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 201. I read Ron's comment, but I am still not
> > > sure
> > > > > the
> > > > > > > > > benefit of
> > > > > > > > > > > > > keeping
> > > > > > > > > > > > > > > > broker.id and controller.id in
> meta.properties.
> > > It
> > > > > seems
> > > > > > > > > that
> > > > > > > > > > > we are
> > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > duplicating the same info in two places and
> have
> > > the
> > > > > > > > > additional
> > > > > > > > > > > > > burden of
> > > > > > > > > > > > > > > > making sure the values in the two places are
> > > > > consistent.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think the reasoning is that having broker.id
> > > > > protects us
> > > > > > > > > against
> > > > > > > > > > > > > > > accidentally bringing up a broker with a disk
> from
> > > a
> > > > > > > different
> > > > > > > > > > > > > broker. I
> > > > > > > > > > > > > > > don't feel strongly about this but it seemed
> > > simpler to
> > > > > > > keep
> > > > > > > > > it.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 202. controller.connect.security.protocol: Is
> > > this
> > > > > needed
> > > > > > > > > since
> > > > > > > > > > > > > > > > controller.listener.names and
> > > > > > > listener.security.protocol.map
> > > > > > > > > > > imply
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > security protocol already?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You're right, this isn't needed. I'll remove
> it.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 203. registration.heartbeat.interval.ms: It
> > > > > defaults to
> > > > > > > 2k.
> > > > > > > > > ZK
> > > > > > > > > > > uses
> > > > > > > > > > > > > 1/3
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the session timeout for heartbeat. So, given
> the
> > > > > default
> > > > > > > 18k
> > > > > > > > > for
> > > > > > > > > > > > > > > > registration.lease.timeout.ms, should we
> default
> > > > > > > > > > > > > > > > registration.heartbeat.interval.ms to 6k?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 6 seconds seems like a pretty long time between
> > > > > > > heartbeats. It
> > > > > > > > > > > might
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > useful to know when a broker is missing
> heartbeats,
> > > > > with
> > > > > > > less
> > > > > > > > > time
> > > > > > > > > > > than
> > > > > > > > > > > > > > > that. I provisionally set it to 3 seconds (we
> can
> > > > > always
> > > > > > > > > change
> > > > > > > > > > > > > later...)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I also changed the name of these
> configurations to
> > > "
> > > > > > > > > > > > > > > broker.heartbeat.interval.ms" and "
> > > > > > > > > broker.registration.timeout.ms"
> > > > > > > > > > > to
> > > > > > > > > > > > > try
> > > > > > > > > > > > > > > to clarify them a bit.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 204. "The highest metadata offset which the
> > > broker
> > > > > has
> > > > > > > not
> > > > > > > > > > > reached."
> > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > seems this should be "has reached".
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I changed this to "one more than the highest
> > > metadata
> > > > > > > offset
> > > > > > > > > which
> > > > > > > > > > > the
> > > > > > > > > > > > > > > broker has reached."
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 205. UnfenceBrokerRecord and
> > > UnregisterBrokerRecord:
> > > > > To
> > > > > > > me,
> > > > > > > > > they
> > > > > > > > > > > > > seem to
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > the same. Do we need both?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Unregistration means that the broker has been
> > > removed
> > > > > from
> > > > > > > the
> > > > > > > > > > > cluster.
> > > > > > > > > > > > > > > That is different than unfencing, which marks
> the
> > > > > broker as
> > > > > > > > > active.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 206. TopicRecord: The Deleting field is used
> to
> > > > > indicate
> > > > > > > > > that the
> > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > being deleted. I am wondering if this is
> really
> > > > > needed
> > > > > > > since
> > > > > > > > > > > > > RemoveTopic
> > > > > > > > > > > > > > > > already indicates the same thing.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > RemoveTopic is the last step, that scrubs all
> > > metadata
> > > > > > > about
> > > > > > > > > the
> > > > > > > > > > > topic.
> > > > > > > > > > > > > > > In order to get to that last step, the topic
> data
> > > > > needs to
> > > > > > > > > removed
> > > > > > > > > > > > > from all
> > > > > > > > > > > > > > > brokers (after each broker notices that the
> topic
> > > is
> > > > > being
> > > > > > > > > > > deleted).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <
> > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron
> Dagostino
> > > wrote:
> > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the updates. It's
> now
> > > > > clear
> > > > > > > to me
> > > > > > > > > that
> > > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > > > keep their broker epoch for the life of
> their
> > > > > JVM --
> > > > > > > they
> > > > > > > > > > > > > register
> > > > > > > > > > > > > > > > > > once, get their broker epoch in the
> > > response, and
> > > > > > > then
> > > > > > > > > never
> > > > > > > > > > > > > > > > > > re-register again. Brokers may get
> fenced,
> > > but
> > > > > they
> > > > > > > > > keep the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > broker epoch for the life of their JVM.
> The
> > > > > > > incarnation
> > > > > > > > > ID
> > > > > > > > > > > is
> > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > kept for the life of the JVM but is
> > > generated by
> > > > > the
> > > > > > > > > broker
> > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > upon startup, and the combination of the
> two
> > > > > allows
> > > > > > > the
> > > > > > > > > > > > > Controller to
> > > > > > > > > > > > > > > > > > act idempotently if any previously-sent
> > > > > registration
> > > > > > > > > response
> > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks, Ron. That's a good summary.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > One thing I wonder about is if it might
> be
> > > > > helpful
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > send the Cluster ID as determined from
> its
> > > > > > > > > meta.properties
> > > > > > > > > > > file
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > registration request. Does it even make
> > > sense
> > > > > for
> > > > > > > the
> > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > > successfully register and enter the
> Fenced
> > > state
> > > > > if
> > > > > > > it
> > > > > > > > > has
> > > > > > > > > > > the
> > > > > > > > > > > > > wrong
> > > > > > > > > > > > > > > > > > Cluster ID?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Yeah, that's a good idea. Let's have the
> > > broker
> > > > > pass
> > > > > > > its
> > > > > > > > > > > cluster
> > > > > > > > > > > > > ID in
> > > > > > > > > > > > > > > > > the registration RPC, and then
> registration can
> > > > > fail
> > > > > > > if the
> > > > > > > > > > > broker
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The nextMetadatOffset value that the
> broker
> > > > > > > communicates
> > > > > > > > > > > > > > > > > > in its registration request only has
> meaning
> > > > > within
> > > > > > > the
> > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > cluster, so it feels to me that the
> > > Controller
> > > > > should
> > > > > > > > > have
> > > > > > > > > > > some
> > > > > > > > > > > > > way
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > perform this sanity check. There is
> > > currently
> > > > > > > (pre-KIP
> > > > > > > > > 500)
> > > > > > > > > > > a
> > > > > > > > > > > > > check
> > > > > > > > > > > > > > > > > > in the broker to make sure its configured
> > > > > cluster ID
> > > > > > > > > matches
> > > > > > > > > > > the
> > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > > stored in ZooKeeper, and we will have to
> > > perform
> > > > > this
> > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > somewhere in the KIP-500 world. If the
> > > > > Controller
> > > > > > > > > doesn't
> > > > > > > > > > > do it
> > > > > > > > > > > > > > > > > > within the registration request then the
> > > broker
> > > > > will
> > > > > > > > > have to
> > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > metadata request to the Controller,
> retrieve
> > > the
> > > > > > > Cluster
> > > > > > > > > ID,
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > perform the check itself. It feels to me
> > > that it
> > > > > > > might
> > > > > > > > > be
> > > > > > > > > > > > > better for
> > > > > > > > > > > > > > > > > > the Controller to just do it, and then
> the
> > > broker
> > > > > > > doesn't
> > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > worry about it anymore once it
> successfully
> > > > > > > registers.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I also have a question about the
> broker.id
> > > > > value and
> > > > > > > > > > > > > > > meta.properties.
> > > > > > > > > > > > > > > > > > The KIP now says "In version 0 of
> > > > > meta.properties,
> > > > > > > there
> > > > > > > > > is a
> > > > > > > > > > > > > > > > > > broker.id field. Version 1 does not
> have
> > > this
> > > > > > > field.
> > > > > > > > > It
> > > > > > > > > > > is no
> > > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > > > needed because we no longer support
> dynamic
> > > > > broker id
> > > > > > > > > > > > > assignment."
> > > > > > > > > > > > > > > > > > But then there is an example version 1
> > > > > > > meta.properties
> > > > > > > > > file
> > > > > > > > > > > that
> > > > > > > > > > > > > > > shows
> > > > > > > > > > > > > > > > > > the broker.id value. I actually wonder
> if
> > > > > maybe the
> > > > > > > > > > > broker.id
> > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > would be good to keep in the version 1
> > > > > > > meta.properties
> > > > > > > > > file
> > > > > > > > > > > > > because
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > currently (pre-KIP 500, version 0) acts
> as a
> > > > > sanity
> > > > > > > > > check to
> > > > > > > > > > > make
> > > > > > > > > > > > > > > sure
> > > > > > > > > > > > > > > > > > the broker is using the correct log
> > > directory.
> > > > > > > Similarly
> > > > > > > > > > > with
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > controller.id value on controllers -- it
> > > would
> > > > > > > allow the
> > > > > > > > > > > same
> > > > > > > > > > > > > type
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > sanity check for quorum controllers.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > That's a good point. I will add broker.id
> > > back,
> > > > > and
> > > > > > > also
> > > > > > > > > add
> > > > > > > > > > > > > > > > > controller.id as a possibility.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin
> McCabe
> > > <
> > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao
> > > wrote:
> > > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for the reply. A few more
> > > comments.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks again for the reply. Sorry for
> the
> > > long
> > > > > > > > > hiatus. I
> > > > > > > > > > > was
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > vacation for a while.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 55. There is still text that favors
> new
> > > > > broker
> > > > > > > > > > > registration.
> > > > > > > > > > > > > > > "When a
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > first starts up, when it is in the
> > > INITIAL
> > > > > > > state, it
> > > > > > > > > will
> > > > > > > > > > > > > always
> > > > > > > > > > > > > > > > > "win"
> > > > > > > > > > > > > > > > > > > > broker ID conflicts. However, once
> it is
> > > > > > > granted a
> > > > > > > > > > > lease, it
> > > > > > > > > > > > > > > > > transitions
> > > > > > > > > > > > > > > > > > > > out of the INITIAL state.
> Thereafter,
> > > it may
> > > > > > > lose
> > > > > > > > > > > subsequent
> > > > > > > > > > > > > > > > > conflicts if
> > > > > > > > > > > > > > > > > > > > its broker epoch is stale. (See
> KIP-380
> > > for
> > > > > some
> > > > > > > > > > > background
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > epoch.) The reason for favoring new
> > > > > processes
> > > > > > > is to
> > > > > > > > > > > > > accommodate
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > > > > case where a process is killed with
> kill
> > > -9
> > > > > and
> > > > > > > then
> > > > > > > > > > > > > restarted.
> > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > want it
> > > > > > > > > > > > > > > > > > > > to be able to reclaim its old ID
> quickly
> > > in
> > > > > this
> > > > > > > > > case."
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for the reminder. I have
> clarified
> > > the
> > > > > > > language
> > > > > > > > > > > here.
> > > > > > > > > > > > > > > > > Hopefully now it is clear that we don't
> allow
> > > quick
> > > > > > > re-use
> > > > > > > > > of
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 80.1 Sounds good. Could you document
> that
> > > > > > > listeners
> > > > > > > > > is a
> > > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > now? It would also be useful to
> annotate
> > > > > other
> > > > > > > > > required
> > > > > > > > > > > > > configs.
> > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > example, controller.connect should be
> > > > > required.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I added a note specifying that these
> are
> > > > > required.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 80.2 Could you list all deprecated
> > > existing
> > > > > > > configs?
> > > > > > > > > > > Another
> > > > > > > > > > > > > one
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > control.plane.listener.name since
> the
> > > > > > > controller no
> > > > > > > > > > > longer
> > > > > > > > > > > > > sends
> > > > > > > > > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata and
> > > StopReplica
> > > > > > > > > requests.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I added a section specifying some
> > > deprecated
> > > > > > > configs.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 83.1 It seems that the broker can
> > > transition
> > > > > from
> > > > > > > > > FENCED
> > > > > > > > > > > to
> > > > > > > > > > > > > > > RUNNING
> > > > > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > > > > registering for a new broker epoch.
> I am
> > > not
> > > > > > > sure how
> > > > > > > > > > > this
> > > > > > > > > > > > > works.
> > > > > > > > > > > > > > > > > Once the
> > > > > > > > > > > > > > > > > > > > controller fences a broker, there is
> no
> > > need
> > > > > for
> > > > > > > the
> > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > > > > > boker epoch around. So, if the fenced
> > > > > broker's
> > > > > > > > > heartbeat
> > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > > existing broker epoch will be
> rejected,
> > > > > leading
> > > > > > > the
> > > > > > > > > > > broker
> > > > > > > > > > > > > back
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The broker epoch refers to the broker
> > > > > registration.
> > > > > > > > > So we
> > > > > > > > > > > DO
> > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > broker epoch around even while the broker
> is
> > > > > fenced.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The broker epoch changes only when
> there
> > > is a
> > > > > new
> > > > > > > > > broker
> > > > > > > > > > > > > > > > > registration. Fencing or unfencing the
> broker
> > > > > doesn't
> > > > > > > > > change
> > > > > > > > > > > the
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 83.5 Good point on KIP-590. Then
> should
> > > we
> > > > > > > expose the
> > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > debugging purposes? If not, we should
> > > > > deprecate
> > > > > > > the
> > > > > > > > > > > > > controllerID
> > > > > > > > > > > > > > > > > field in
> > > > > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think it's OK to expose it for now,
> with
> > > the
> > > > > > > proviso
> > > > > > > > > > > that it
> > > > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > > > be reachable by clients.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 90. We rejected the shared ID with
> just
> > > one
> > > > > > > reason
> > > > > > > > > "This
> > > > > > > > > > > is
> > > > > > > > > > > > > not a
> > > > > > > > > > > > > > > > > good idea
> > > > > > > > > > > > > > > > > > > > because NetworkClient assumes a
> single ID
> > > > > > > space. So
> > > > > > > > > if
> > > > > > > > > > > > > there is
> > > > > > > > > > > > > > > > > both a
> > > > > > > > > > > > > > > > > > > > controller 1 and a broker 1, we don't
> > > have a
> > > > > way
> > > > > > > of
> > > > > > > > > > > picking
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > "right"
> > > > > > > > > > > > > > > > > > > > one." This doesn't seem to be a
> strong
> > > > > reason.
> > > > > > > For
> > > > > > > > > > > example,
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > > > address the NetworkClient issue with
> the
> > > node
> > > > > > > type
> > > > > > > > > as you
> > > > > > > > > > > > > pointed
> > > > > > > > > > > > > > > > > out or
> > > > > > > > > > > > > > > > > > > > using the negative value of a broker
> ID
> > > as
> > > > > the
> > > > > > > > > > > controller ID.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It would require a lot of code changes
> to
> > > > > support
> > > > > > > > > multiple
> > > > > > > > > > > > > types of
> > > > > > > > > > > > > > > > > node IDs. It's not clear to me that the
> end
> > > result
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > > > > better --
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > tend to think it would be worse, since it
> > > would be
> > > > > more
> > > > > > > > > > > complex.
> > > > > > > > > > > > > In a
> > > > > > > > > > > > > > > > > similar vein, using negative numbers seems
> > > > > dangerous,
> > > > > > > > > since we
> > > > > > > > > > > use
> > > > > > > > > > > > > > > > > negatives or -1 as "special values" in many
> > > places.
> > > > > > > For
> > > > > > > > > > > example,
> > > > > > > > > > > > > -1
> > > > > > > > > > > > > > > often
> > > > > > > > > > > > > > > > > represents "no such node."
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > One important thing to keep in mind is
> > > that we
> > > > > > > want to
> > > > > > > > > be
> > > > > > > > > > > able
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > transition from a broker and a controller
> being
> > > > > > > co-located
> > > > > > > > > to
> > > > > > > > > > > them
> > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > longer being co-located. This is much
> easier
> > > to do
> > > > > > > when
> > > > > > > > > they
> > > > > > > > > > > have
> > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > the broker reports all offline
> replicas
> > > due
> > > > > to a
> > > > > > > disk
> > > > > > > > > > > > > failure to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > controller. It seems this information
> > > needs
> > > > > to be
> > > > > > > > > > > persisted
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > > > log. Do we have a corresponding
> record
> > > for
> > > > > that?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hmm, I have to look into this a little
> bit
> > > > > more.
> > > > > > > We
> > > > > > > > > may
> > > > > > > > > > > need
> > > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > > record type.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 101. Currently, StopReplica request
> has 2
> > > > > modes,
> > > > > > > > > without
> > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > and with
> > > > > > > > > > > > > > > > > > > > deletion. The former is used for
> > > controlled
> > > > > > > shutdown
> > > > > > > > > and
> > > > > > > > > > > > > handling
> > > > > > > > > > > > > > > > > disk
> > > > > > > > > > > > > > > > > > > > failure, and causes the follower to
> > > stop. The
> > > > > > > latter
> > > > > > > > > is
> > > > > > > > > > > for
> > > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > > > and partition reassignment, and
> causes
> > > the
> > > > > > > replica
> > > > > > > > > to be
> > > > > > > > > > > > > deleted.
> > > > > > > > > > > > > > > > > Since we
> > > > > > > > > > > > > > > > > > > > are deprecating StopReplica, could we
> > > > > document
> > > > > > > what
> > > > > > > > > > > triggers
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > stopping
> > > > > > > > > > > > > > > > > > > > of a follower and the deleting of a
> > > replica
> > > > > now?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > RemoveTopic triggers deletion. In
> general
> > > the
> > > > > > > > > > > functionality of
> > > > > > > > > > > > > > > > > StopReplica is subsumed by the metadata
> > > records.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 102. Should we include the metadata
> > > topic in
> > > > > the
> > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > If so,
> > > > > > > > > > > > > > > > > > > > when it will be included and what
> will
> > > the
> > > > > > > metadata
> > > > > > > > > > > response
> > > > > > > > > > > > > look
> > > > > > > > > > > > > > > > > like?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > No, it won't be included in the
> metadata
> > > > > response
> > > > > > > sent
> > > > > > > > > back
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > brokers.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 103. "The active controller assigns
> the
> > > > > broker a
> > > > > > > new
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > epoch,
> > > > > > > > > > > > > > > > > based on
> > > > > > > > > > > > > > > > > > > > the latest committed offset in the
> log."
> > > This
> > > > > > > seems
> > > > > > > > > > > > > inaccurate
> > > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > latest committed offset doesn't
> always
> > > > > advance on
> > > > > > > > > every
> > > > > > > > > > > log
> > > > > > > > > > > > > > > append.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Given that the new broker epoch won't
> be
> > > > > visible
> > > > > > > until
> > > > > > > > > the
> > > > > > > > > > > > > commit
> > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > happened, I have changed this to "the next
> > > > > available
> > > > > > > > > offset in
> > > > > > > > > > > the
> > > > > > > > > > > > > log"
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 104. REGISTERING(1) : It says
> > > "Otherwise, the
> > > > > > > broker
> > > > > > > > > > > moves
> > > > > > > > > > > > > into
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > > > > > > state.". It seems this should be
> RUNNING?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 105. RUNNING: Should we require the
> > > broker to
> > > > > > > catch
> > > > > > > > > up
> > > > > > > > > > > to the
> > > > > > > > > > > > > > > > > metadata log
> > > > > > > > > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > For 104 and 105, these sections have
> been
> > > > > reworked.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin
> > > McCabe
> > > > > <
> > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom
> > > Bentley
> > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at 08:59,
> Ron
> > > > > Dagostino
> > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the
> hard
> > > work
> > > > > on
> > > > > > > this
> > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I have some questions about
> what
> > > > > happens
> > > > > > > to a
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > when it
> > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > > > fenced (e.g. because it can't
> > > send a
> > > > > > > > > heartbeat
> > > > > > > > > > > > > request to
> > > > > > > > > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > > > > > > > > lease). The KIP says "When a
> > > broker
> > > > > is
> > > > > > > > > fenced,
> > > > > > > > > > > it
> > > > > > > > > > > > > cannot
> > > > > > > > > > > > > > > > > process any
> > > > > > > > > > > > > > > > > > > > > > > > client requests. This
> prevents
> > > > > brokers
> > > > > > > which
> > > > > > > > > > > are not
> > > > > > > > > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > > > > > > > > metadata updates or that are
> not
> > > > > > > receiving
> > > > > > > > > and
> > > > > > > > > > > > > processing
> > > > > > > > > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > > > > > > > > enough from causing issues to
> > > > > clients."
> > > > > > > And
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > > > > > > > > FENCED(4) state it likewise
> says
> > > > > "While
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > > > > not respond to client
> > > requests." It
> > > > > > > makes
> > > > > > > > > sense
> > > > > > > > > > > > > that a
> > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > > > should not accept producer
> > > requests
> > > > > -- I
> > > > > > > > > assume
> > > > > > > > > > > any
> > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > > > > > > > would result in
> > > > > > > NotLeaderOrFollowerException.
> > > > > > > > > > > But
> > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > > > > > > > > (fetch from follower)
> consumer
> > > > > > > requests? It
> > > > > > > > > is
> > > > > > > > > > > > > > > conceivable
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > > > could continue. Related to
> that,
> > > > > would a
> > > > > > > > > fenced
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > > > > fetch data for partitions
> where
> > > it
> > > > > > > thinks it
> > > > > > > > > is a
> > > > > > > > > > > > > > > follower?
> > > > > > > > > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > > > > > > > > it rejects consumer requests
> it
> > > might
> > > > > > > still
> > > > > > > > > > > continue
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > > > > > > > > follower. Might it be
> helpful to
> > > > > clarify
> > > > > > > > > both
> > > > > > > > > > > > > decisions
> > > > > > > > > > > > > > > > > here?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Good question. I think a
> fenced
> > > broker
> > > > > > > should
> > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > > > > > > > > partitions it was already
> fetching
> > > > > before
> > > > > > > it
> > > > > > > > > was
> > > > > > > > > > > > > fenced,
> > > > > > > > > > > > > > > > > unless it
> > > > > > > > > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > > > > > > > > problem. At that point it
> won't be
> > > > > able to
> > > > > > > > > > > continue,
> > > > > > > > > > > > > > > since it
> > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > the new metadata. For
> example, it
> > > > > won't
> > > > > > > know
> > > > > > > > > about
> > > > > > > > > > > > > > > leadership
> > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > the partitions it's fetching.
> The
> > > > > > > rationale
> > > > > > > > > for
> > > > > > > > > > > > > > > continuing to
> > > > > > > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > > > > > try to avoid disruptions as
> much as
> > > > > > > possible.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I don't think fenced brokers
> should
> > > > > accept
> > > > > > > > > client
> > > > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > > The issue
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > that the fenced broker may or
> may
> > > not
> > > > > have
> > > > > > > any
> > > > > > > > > > > data it
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > > > > > > > > have. It may or may not have
> > > applied
> > > > > any
> > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > it is supposed to have applied.
> > > So it
> > > > > > > could
> > > > > > > > > get
> > > > > > > > > > > pretty
> > > > > > > > > > > > > > > > > confusing, and
> > > > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > potentially waste the client's
> > > time.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > When fenced, how would the broker
> > > reply
> > > > > to a
> > > > > > > > > client
> > > > > > > > > > > > > which did
> > > > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The broker will respond with a
> > > retryable
> > > > > error
> > > > > > > in
> > > > > > > > > that
> > > > > > > > > > > > > case.
> > > > > > > > > > > > > > > Once
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > client has re-fetched its
> metadata, it
> > > > > will no
> > > > > > > > > longer
> > > > > > > > > > > see
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > as part of the cluster. I added a
> > > note to
> > > > > the
> > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Wed, Dec 16, 2020, at 18:13, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the reply. Just a couple of more comments.
>
> 211. Currently, the broker only registers itself in ZK after log recovery.
> Is there any benefit to change that? As you mentioned, the broker can't do
> much before completing log recovery.
>
Hi Jun,
Previously, it wasn't possible to register in ZK without immediately getting added to the MetadataResponse. So I think that's the main reason why registration was delayed until after log recovery. Since that constraint doesn't exist any more, there seems to be no reason to delay registration.
I think delaying registration would have some major downsides. If log recovery takes a while, that's a longer window during which someone else could register a broker with the same ID. Having parts of the cluster missing for a while gives up some of the benefit of separating registration from fencing. For example, if a broker somehow gets unregistered and we want to re-register it, but we have to wait for a 10 minute log recovery to do that, that could be a window during which topics can't be created that need to be on that broker, and so forth.
> 230. Regarding MetadataResponse, there is a slight awkwardness. We return
> rack for each node. However, if that node is for the controller, the rack
> field is not really relevant. Should we clean it up here or in another KIP
> like KIP-700?
Oh, controllers don't appear in the MetadataResponses returned to clients, since clients can't access them. I should have been more clear about that in the KIP-- I added a sentence to "Networking" describing this.
best,
Colin
>
> Thanks,
>
> Jun
>
> On Wed, Dec 16, 2020 at 4:23 PM Colin McCabe <cm...@apache.org> wrote:
>
> > On Wed, Dec 16, 2020, at 13:40, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply. A few follow up comments.
> > >
> > > 211. When does the broker send the BrokerRegistration request to the
> > > controller? Is it after the recovery phase? If so, at that point, the
> > > broker has already caught up on the metadata (in order to clean up
> > deleted
> > > partitions). Then, it seems we don't need the ShouldFence field
> > > in BrokerHeartbeatRequest?
> >
> > Hi Jun,
> >
> > Thanks again for the reviews.
> >
> > The broker sends the registration request as soon as it starts up. It
> > cannot wait until the recovery phase is over since sometimes log recovery
> > can take quite a long time.
> >
> > >
> > > 213. kafka-cluster.sh
> > > 213.1 For the decommision example, should the command take a broker id?
> >
> > Yes, the example should have a broker id. Fixed.
> >
> > > 213.2 Existing tools use the "--" command line option (e.g. kafka-topics
> > > --list --topic test). Should we follow the same convention
> > > for kafka-cluster.sh (and kafka-storage.sh)?
> >
> > Hmm. I don't think argparse4j supports using double dashes to identify
> > subcommands. I think it might be confusing as well, since the subcommand
> > must come first, unlike a plain old argument which can be anywhere on the
> > command line.
> >
> > > 213.3 Should we add an admin api for broker decommission so that this can
> > > be done programmatically?
> > >
> >
> > Yes, there is an admin client API for decommissioning as well.
> >
> > > 220. DecommissionBroker: "NOT_CONTROLLER if the node that the request was
> > > sent to is not the controller". I thought the clients never send a
> > request
> > > to the controller directly now and the broker will forward it to the
> > > controller?
> > >
> >
> > If the controller moved recently, it's possible that the broker could send
> > to a controller that has just recently become inactive. In that case
> > NOT_CONTROLLER would be returned. (A standby controller returns
> > NOT_CONTROLLER for most APIs).
> >
> > > 221. Could we add the required ACL for the new requests?
> > >
> >
> > Good point. I added the required ACL for each new RPC.
> >
> > best,
> > Colin
> >
> >
> > > Jun
> > >
> > > On Wed, Dec 16, 2020 at 11:38 AM Colin McCabe <cm...@apache.org>
> > wrote:
> > >
> > > > On Wed, Dec 16, 2020, at 09:59, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the reply. A few more comments below.
> > > > >
> > > >
> > > > Hi Jun,
> > > >
> > > > Thanks for the comments.
> > > >
> > > > > 206. "RemoveTopic is the last step, that scrubs all metadata about
> > the
> > > > > topic. In order to get to that last step, the topic data needs to
> > > > removed
> > > > > from all brokers (after each broker notices that the topic is being
> > > > > deleted)." Currently, this is done based on the response of
> > > > > StopReplicaRequest. Since the controller no longer sends this
> > request,
> > > > how
> > > > > does the controller know that the data for the deleted topic has
> > > > > been removed in the brokers?
> > > > >
> > > >
> > > > That's a good point... definitely an oversight.
> > > >
> > > > It seems complex to force the controller to track when log directories
> > > > have been deleted. Let's just assume that KIP-516 has been
> > implemented,
> > > > and track them by UUID. Then we can just have a single topic deletion
> > > > record.
> > > >
> > > > I added a section on "topic identifiers" describing this.
> > > >
> > > > > 210. Thanks for the explanation. Sounds good to me.
> > > > >
> > > > > 211. I still don't see an example when ShouldFence is set to true in
> > > > > BrokerHeartbeatReques. Could we add one?
> > > > >
> > > >
> > > > It is sent to true when the broker is first starting up and doesn't yet
> > > > want to be unfenced. I added a longer explanation of this in the
> > "Broker
> > > > Leases" section.
> > > >
> > > > > 213. The KIP now allows replicas to be assigned on brokers that are
> > > > fenced,
> > > > > which is an improvement. How do we permanently remove a broker (e.g.
> > > > > cluster shrinking) to prevent it from being used for future replica
> > > > > assignments?
> > > > >
> > > >
> > > > This is a fair point. I will create a kafka-cluster.sh script which
> > can
> > > > do this, plus a DecommissionBrokerRequest.
> > > >
> > > > As a bonus the kafka-cluster.sh script can help people find the
> > cluster ID
> > > > of brokers, something that people have wanted a tool for in the past.
> > > >
> > > > > 214. "Currently, when a node is down, all of its ZK registration
> > > > > information is gone. But we need this information in order to
> > > > understand
> > > > > things like whether the replicas of a particular partition are
> > > > > well-balanced across racks." This is not quite true. Currently, even
> > when
> > > > > ZK registration is gone, the existing replica assignment is still
> > > > available
> > > > > in the metadata response.
> > > > >
> > > >
> > > > I agree that the existing replica assignment is still available. But
> > that
> > > > just tells you partition X is on nodes A, B, and C. If you don't have
> > the
> > > > ZK registration for one or more of A, B, or C then you don't know
> > whether
> > > > we are following the policy of "two replicas on one rack, one replica
> > on
> > > > another." Or any other more complex rack placement policy that you
> > might
> > > > have.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Dec 16, 2020 at 8:48 AM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the reply. A few more follow up comments.
> > > > > > >
> > > > > > > 210. initial.broker.registration.timeout.ms: The default value
> > is
> > > > 90sec,
> > > > > > > which seems long. If a broker fails the registration because of
> > > > incorrect
> > > > > > > configs, we want to fail faster. In comparison, the defaults for
> > > > > > > zookeeper.connection.timeout.ms is 18 secs.
> > > > > > >
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > I agree that the initial connection timeout here is longer than
> > what we
> > > > > > had with ZK. The main reason I selected a slightly longer timeout
> > > > here is
> > > > > > to handle the case where the controllers and the brokers are
> > > > co-located.
> > > > > > For example, if you have a 3 node cluster and all three nodes are
> > > > > > controllers+brokers, when you first bring up the cluster, we will
> > have
> > > > to
> > > > > > stand up the controller quorum and then handle broker
> > registrations.
> > > > > > Although we believe Raft will start up relatively quickly, it's
> > good to
> > > > > > leave some extra margin here.
> > > > > >
> > > > > > I don't think there's a big disadvantage to having a slightly
> > longer
> > > > > > timeout here. After all, starting up the brokers with no
> > controllers
> > > > is an
> > > > > > admin mistake, which we don't expect to see very often. Maybe
> > let's
> > > > set it
> > > > > > to 60 seconds for now and maybe see if we can tweak it in the
> > future if
> > > > > > that turns out to be too long or short?
> > > > > >
> > > > > > > 211. "Once they are all moved, the controller responds to the
> > > > heartbeat
> > > > > > > with a nextState of SHUTDOWN." It seems that nextState is no
> > longer
> > > > > > > relevant. Also, could you add a bit more explanation on when
> > > > ShouldFence
> > > > > > is
> > > > > > > set to true in BrokerHeartbeatRequest?
> > > > > > >
> > > > > >
> > > > > > Good catch. I removed the obsolete section referring to nextState
> > and
> > > > > > added a reference to the new boolean. I also added some more
> > > > information
> > > > > > about ShouldFence and about the rationale for separating fencing
> > from
> > > > > > registration.
> > > > > >
> > > > > > > 212. Related to the above, what does the broker do if IsFenced is
> > > > true in
> > > > > > > the BrokerHeartbeatResponse? Will the broker do the same thing on
> > > > > > receiving
> > > > > > > a FenceBrokerRecord from the metadata log?
> > > > > > >
> > > > > >
> > > > > > The broker only checks this boolean during the startup process.
> > After
> > > > the
> > > > > > startup process is finished, it ignores this boolean.
> > > > > >
> > > > > > The broker uses fence / unfence records in the metadata log to
> > > > determine
> > > > > > which brokers should appear in its MetadataResponse.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <cmccabe@apache.org
> > >
> > > > wrote:
> > > > > > >
> > > > > > > > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > > > > > > > Hi Colin,
> > > > > > > > >
> > > > > > > > > The KIP says that "brokers which are fenced will not appear
> > in
> > > > > > > > > MetadataResponses. So clients that have up-to-date metadata
> > will
> > > > > > not try
> > > > > > > > > to contact fenced brokers.", which is fine, but it doesn't
> > seem
> > > > to
> > > > > > > > > elaborate on what happens for clients which try to contact a
> > > > broker
> > > > > > > > (using
> > > > > > > > > stale metadata, for example) and find it in the
> > > > registered-but-fenced
> > > > > > > > > state.
> > > > > > > >
> > > > > > > > Hi Tom,
> > > > > > > >
> > > > > > > > I have dropped the broker-side fencing from this proposal. So
> > now
> > > > the
> > > > > > > > fencing is basically the same as today's fencing: it's
> > > > controller-side
> > > > > > > > only. That means that clients using stale metadata can contact
> > > > fenced
> > > > > > > > brokers and communicate with them. The only case where the
> > broker
> > > > > > will be
> > > > > > > > inaccessible is when it hasn't finished starting yet (i.e., has
> > > > not yet
> > > > > > > > attained RUNNING state.)
> > > > > > > >
> > > > > > > > Just like today, we expect the safeguards built into the
> > > > replication
> > > > > > > > protocol to prevent the worst corner cases that could result
> > from
> > > > > > this. I
> > > > > > > > do think we will probably take up this issue later and find a
> > > > better
> > > > > > > > solution for client-side metadata, but this KIP is big enough
> > > > as-is.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > >
> > > > > > > > > You said previously that the broker will respond with a
> > retriable
> > > > > > > > > error, which is again fine, but you didn't answer my question
> > > > about
> > > > > > which
> > > > > > > > > error code you've chosen for this. I realise that this
> > doesn't
> > > > really
> > > > > > > > > affect the correctness of the behaviour, but I'm not aware
> > of any
> > > > > > > > existing
> > > > > > > > > error code which is a good fit. So it would be good to
> > understand
> > > > > > about
> > > > > > > > how
> > > > > > > > > you're making this backward compatible for clients.
> > > > > > > > >
> > > > > > > > > Many thanks,
> > > > > > > > >
> > > > > > > > > Tom
> > > > > > > > >
> > > > > > > > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <
> > cmccabe@apache.org
> > > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > > > > > > > Hi, Colin,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the reply. Just a couple of more comments
> > below.
> > > > > > > > > > >
> > > > > > > > > > > 210. Since we are deprecating
> > > > zookeeper.connection.timeout.ms,
> > > > > > > > should we
> > > > > > > > > > > add a new config to bound the time for a broker to
> > connect
> > > > to the
> > > > > > > > > > > controller during starting up?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Good idea. I added initial.broker.registration.timeout.ms
> > for
> > > > > > this.
> > > > > > > > > >
> > > > > > > > > > > 211. BrokerHeartbeat no longer has the state field in the
> > > > > > > > > > request/response.
> > > > > > > > > > > However, (a) the controller shutdown section still has
> > "In
> > > > its
> > > > > > > > periodic
> > > > > > > > > > > heartbeats, the broker asks the controller if it can
> > > > transition
> > > > > > into
> > > > > > > > the
> > > > > > > > > > > SHUTDOWN state. This motivates the controller to move
> > all
> > > > of the
> > > > > > > > leaders
> > > > > > > > > > > off of that broker. Once they are all moved, the
> > controller
> > > > > > > > responds to
> > > > > > > > > > > the heartbeat with a nextState of SHUTDOWN."; (2) the
> > > > > > description of
> > > > > > > > > > > BrokerHeartbeat still references currentState and
> > > > targetState.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks. I've made these sections clearer and removed the
> > > > obsolete
> > > > > > > > > > references to sending states.
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <
> > > > cmccabe@apache.org
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the update. A few more follow up comments.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks again for the review.
> > > > > > > > > > > >
> > > > > > > > > > > > > 100. FailedReplicaRecord: Since this is reported by
> > each
> > > > > > broker
> > > > > > > > > > > > > independently, perhaps we could use a more concise
> > > > > > representation
> > > > > > > > > > that
> > > > > > > > > > > > has
> > > > > > > > > > > > > a top level broker field, an array of topics, which
> > has
> > > > an
> > > > > > array
> > > > > > > > of
> > > > > > > > > > > > > partitions.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > The issue is that there is a size limit on the each
> > record.
> > > > > > > > Putting
> > > > > > > > > > all
> > > > > > > > > > > > of the partitions of a log directory into a single
> > record
> > > > would
> > > > > > > > > > probably
> > > > > > > > > > > > break that in many cases. Still, we can optimize a
> > bit by
> > > > > > having
> > > > > > > > an
> > > > > > > > > > array
> > > > > > > > > > > > of partition IDs, since nearly all the time, we have
> > more
> > > > than
> > > > > > one
> > > > > > > > > > from the
> > > > > > > > > > > > same topic.
> > > > > > > > > > > >
> > > > > > > > > > > > > 200. Sounds good. If we remove the broker-side
> > fencing
> > > > > > logic, do
> > > > > > > > we
> > > > > > > > > > plan
> > > > > > > > > > > > to
> > > > > > > > > > > > > still keep FENCED in broker state? Do we plan to
> > expose
> > > > the
> > > > > > new
> > > > > > > > > > states
> > > > > > > > > > > > > through the existing BrokerState metric and if so,
> > what
> > > > are
> > > > > > the
> > > > > > > > > > values
> > > > > > > > > > > > for
> > > > > > > > > > > > > the new states?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > No, we don't need FENCED any more. I have removed it
> > from
> > > > the
> > > > > > KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > The new states are very similar to the current ones,
> > > > actually.
> > > > > > > > There
> > > > > > > > > > are
> > > > > > > > > > > > no new states or removed ones. The main change in the
> > > > broker
> > > > > > state
> > > > > > > > > > machine
> > > > > > > > > > > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state has
> > been
> > > > > > > > renamed to
> > > > > > > > > > > > RECOVERY. Also, unlike previously, the broker will
> > always
> > > > pass
> > > > > > > > through
> > > > > > > > > > > > RECOVERY (although it may only stay in this state for a
> > > > very
> > > > > > short
> > > > > > > > > > amount
> > > > > > > > > > > > of time).
> > > > > > > > > > > >
> > > > > > > > > > > > > 201. This may be fine too. Could we document what
> > happens
> > > > > > when
> > > > > > > > the
> > > > > > > > > > > > > broker.id/controller.id in metadata.properties don't
> > > > match
> > > > > > the
> > > > > > > > > > broker
> > > > > > > > > > > > > config when the broker starts up?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I added some documentation about this.
> > > > > > > > > > > >
> > > > > > > > > > > > > 204. There is still "The highest metadata offset
> > which
> > > > the
> > > > > > broker
> > > > > > > > > > has not
> > > > > > > > > > > > > reached" referenced under BrokerRegistration.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > > > > > > > >
> > > > > > > > > > > > > 206. Is that separate step needed given KIP-516? With
> > > > > > KIP-516 (
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > > > > > > > ),
> > > > > > > > > > > > > we don't need to wait for the topic data to be
> > removed
> > > > from
> > > > > > all
> > > > > > > > > > brokers
> > > > > > > > > > > > > before removing the topic metadata. The combination
> > of
> > > > > > unmatching
> > > > > > > > > > > > > topicId
> > > > > > > > > > > > > or the missing topicId from the metadata is enough
> > for
> > > > the
> > > > > > > > broker to
> > > > > > > > > > > > > clean
> > > > > > > > > > > > > up deleted topics asynchronously.
> > > > > > > > > > > >
> > > > > > > > > > > > It won't be needed once KIP-516 is adopted, but this
> > hasn't
> > > > > > been
> > > > > > > > > > > > implemented yet.
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jun
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <
> > > > > > cmccabe@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the updated KIP. A few more comments
> > > > below.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks again for the reviews.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 80.2 For deprecated configs, we need to include
> > > > > > zookeeper.*
> > > > > > > > and
> > > > > > > > > > > > > > > broker.id.generation.enable.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Added.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 83.1 If a broker is down, does the controller
> > keep
> > > > the
> > > > > > > > previously
> > > > > > > > > > > > > > > registered broker epoch forever? If not, how long
> > > > does
> > > > > > the
> > > > > > > > > > controller
> > > > > > > > > > > > > > keep
> > > > > > > > > > > > > > > it? What does the controller do when receiving a
> > > > broker
> > > > > > > > heartbeat
> > > > > > > > > > > > request
> > > > > > > > > > > > > > > with an unfound broker epoch?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes, the controller keeps the previous registration
> > > > > > forever.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Broker heartbeat requests with an incorrect broker
> > > > epoch
> > > > > > will
> > > > > > > > be
> > > > > > > > > > > > rejected
> > > > > > > > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 100. Have you figured out if we need to add a new
> > > > record
> > > > > > > > type for
> > > > > > > > > > > > > > reporting
> > > > > > > > > > > > > > > partitions on failed disks?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I added FailedReplicaRecord to reflect the case
> > where a
> > > > > > JBOD
> > > > > > > > > > directory
> > > > > > > > > > > > has
> > > > > > > > > > > > > > failed, leading to failed replicas.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 102. For debugging purposes, sometimes it's
> > useful to
> > > > > > read
> > > > > > > > the
> > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > topic using tools like console-consumer. Should
> > we
> > > > > > support
> > > > > > > > that
> > > > > > > > > > and
> > > > > > > > > > > > if
> > > > > > > > > > > > > > so,
> > > > > > > > > > > > > > > how?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For now, we have the ability to read the metadata
> > logs
> > > > > > with the
> > > > > > > > > > > > dump-logs
> > > > > > > > > > > > > > tool. I think we will come up with some other
> > tools
> > > > in the
> > > > > > > > future
> > > > > > > > > > as
> > > > > > > > > > > > we
> > > > > > > > > > > > > > get experience.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 200. "brokers which are fenced will not appear in
> > > > > > > > > > MetadataResponses.
> > > > > > > > > > > > The
> > > > > > > > > > > > > > > broker will not respond to these requests--
> > instead,
> > > > it
> > > > > > will
> > > > > > > > > > simply
> > > > > > > > > > > > > > > disconnect." If the controller is partitioned off
> > > > from
> > > > > > the
> > > > > > > > > > brokers,
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > design will cause every broker to stop accepting
> > new
> > > > > > client
> > > > > > > > > > > > requests. In
> > > > > > > > > > > > > > > contrast, if ZK is partitioned off, the existing
> > > > > > behavior is
> > > > > > > > > > that the
> > > > > > > > > > > > > > > brokers can continue to work based on the last
> > known
> > > > > > > > metadata.
> > > > > > > > > > So, I
> > > > > > > > > > > > am
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > sure if we should change the existing behavior
> > > > because
> > > > > > of the
> > > > > > > > > > bigger
> > > > > > > > > > > > > > impact
> > > > > > > > > > > > > > > in the new one. Another option is to keep the
> > > > existing
> > > > > > > > behavior
> > > > > > > > > > and
> > > > > > > > > > > > > > expose
> > > > > > > > > > > > > > > a metric for fenced brokers so that the operator
> > > > could be
> > > > > > > > > > alerted.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'm skeptical about how well running without ZK
> > > > currently
> > > > > > > > works.
> > > > > > > > > > > > However,
> > > > > > > > > > > > > > I will move the broker-side fencing into a
> > follow-up
> > > > KIP.
> > > > > > This
> > > > > > > > > > KIP is
> > > > > > > > > > > > > > already pretty large and there is no hard
> > dependency on
> > > > > > this.
> > > > > > > > > > There
> > > > > > > > > > > > may
> > > > > > > > > > > > > > also be other ways of accomplishing the positive
> > > > effects of
> > > > > > > > what
> > > > > > > > > > > > > > broker-side fencing, so more discussion is needed.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 201. I read Ron's comment, but I am still not
> > sure
> > > > the
> > > > > > > > benefit of
> > > > > > > > > > > > keeping
> > > > > > > > > > > > > > > broker.id and controller.id in meta.properties.
> > It
> > > > seems
> > > > > > > > that
> > > > > > > > > > we are
> > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > duplicating the same info in two places and have
> > the
> > > > > > > > additional
> > > > > > > > > > > > burden of
> > > > > > > > > > > > > > > making sure the values in the two places are
> > > > consistent.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think the reasoning is that having broker.id
> > > > protects us
> > > > > > > > against
> > > > > > > > > > > > > > accidentally bringing up a broker with a disk from
> > a
> > > > > > different
> > > > > > > > > > > > broker. I
> > > > > > > > > > > > > > don't feel strongly about this but it seemed
> > simpler to
> > > > > > keep
> > > > > > > > it.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 202. controller.connect.security.protocol: Is
> > this
> > > > needed
> > > > > > > > since
> > > > > > > > > > > > > > > controller.listener.names and
> > > > > > listener.security.protocol.map
> > > > > > > > > > imply
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > security protocol already?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You're right, this isn't needed. I'll remove it.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 203. registration.heartbeat.interval.ms: It
> > > > defaults to
> > > > > > 2k.
> > > > > > > > ZK
> > > > > > > > > > uses
> > > > > > > > > > > > 1/3
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the session timeout for heartbeat. So, given the
> > > > default
> > > > > > 18k
> > > > > > > > for
> > > > > > > > > > > > > > > registration.lease.timeout.ms, should we default
> > > > > > > > > > > > > > > registration.heartbeat.interval.ms to 6k?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 6 seconds seems like a pretty long time between
> > > > > > heartbeats. It
> > > > > > > > > > might
> > > > > > > > > > > > be
> > > > > > > > > > > > > > useful to know when a broker is missing heartbeats,
> > > > with
> > > > > > less
> > > > > > > > time
> > > > > > > > > > than
> > > > > > > > > > > > > > that. I provisionally set it to 3 seconds (we can
> > > > always
> > > > > > > > change
> > > > > > > > > > > > later...)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I also changed the name of these configurations to
> > "
> > > > > > > > > > > > > > broker.heartbeat.interval.ms" and "
> > > > > > > > broker.registration.timeout.ms"
> > > > > > > > > > to
> > > > > > > > > > > > try
> > > > > > > > > > > > > > to clarify them a bit.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 204. "The highest metadata offset which the
> > broker
> > > > has
> > > > > > not
> > > > > > > > > > reached."
> > > > > > > > > > > > It
> > > > > > > > > > > > > > > seems this should be "has reached".
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I changed this to "one more than the highest
> > metadata
> > > > > > offset
> > > > > > > > which
> > > > > > > > > > the
> > > > > > > > > > > > > > broker has reached."
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 205. UnfenceBrokerRecord and
> > UnregisterBrokerRecord:
> > > > To
> > > > > > me,
> > > > > > > > they
> > > > > > > > > > > > seem to
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > the same. Do we need both?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Unregistration means that the broker has been
> > removed
> > > > from
> > > > > > the
> > > > > > > > > > cluster.
> > > > > > > > > > > > > > That is different than unfencing, which marks the
> > > > broker as
> > > > > > > > active.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 206. TopicRecord: The Deleting field is used to
> > > > indicate
> > > > > > > > that the
> > > > > > > > > > > > topic
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > being deleted. I am wondering if this is really
> > > > needed
> > > > > > since
> > > > > > > > > > > > RemoveTopic
> > > > > > > > > > > > > > > already indicates the same thing.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > RemoveTopic is the last step, that scrubs all
> > metadata
> > > > > > about
> > > > > > > > the
> > > > > > > > > > topic.
> > > > > > > > > > > > > > In order to get to that last step, the topic data
> > > > needs to
> > > > > > > > removed
> > > > > > > > > > > > from all
> > > > > > > > > > > > > > brokers (after each broker notices that the topic
> > is
> > > > being
> > > > > > > > > > deleted).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <
> > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino
> > wrote:
> > > > > > > > > > > > > > > > > Hi Colin. Thanks for the updates. It's now
> > > > clear
> > > > > > to me
> > > > > > > > that
> > > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > > keep their broker epoch for the life of their
> > > > JVM --
> > > > > > they
> > > > > > > > > > > > register
> > > > > > > > > > > > > > > > > once, get their broker epoch in the
> > response, and
> > > > > > then
> > > > > > > > never
> > > > > > > > > > > > > > > > > re-register again. Brokers may get fenced,
> > but
> > > > they
> > > > > > > > keep the
> > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > broker epoch for the life of their JVM. The
> > > > > > incarnation
> > > > > > > > ID
> > > > > > > > > > is
> > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > kept for the life of the JVM but is
> > generated by
> > > > the
> > > > > > > > broker
> > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > upon startup, and the combination of the two
> > > > allows
> > > > > > the
> > > > > > > > > > > > Controller to
> > > > > > > > > > > > > > > > > act idempotently if any previously-sent
> > > > registration
> > > > > > > > response
> > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks, Ron. That's a good summary.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > One thing I wonder about is if it might be
> > > > helpful
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > send the Cluster ID as determined from its
> > > > > > > > meta.properties
> > > > > > > > > > file
> > > > > > > > > > > > in
> > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > registration request. Does it even make
> > sense
> > > > for
> > > > > > the
> > > > > > > > > > broker to
> > > > > > > > > > > > > > > > > successfully register and enter the Fenced
> > state
> > > > if
> > > > > > it
> > > > > > > > has
> > > > > > > > > > the
> > > > > > > > > > > > wrong
> > > > > > > > > > > > > > > > > Cluster ID?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Yeah, that's a good idea. Let's have the
> > broker
> > > > pass
> > > > > > its
> > > > > > > > > > cluster
> > > > > > > > > > > > ID in
> > > > > > > > > > > > > > > > the registration RPC, and then registration can
> > > > fail
> > > > > > if the
> > > > > > > > > > broker
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The nextMetadatOffset value that the broker
> > > > > > communicates
> > > > > > > > > > > > > > > > > in its registration request only has meaning
> > > > within
> > > > > > the
> > > > > > > > > > correct
> > > > > > > > > > > > > > > > > cluster, so it feels to me that the
> > Controller
> > > > should
> > > > > > > > have
> > > > > > > > > > some
> > > > > > > > > > > > way
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > perform this sanity check. There is
> > currently
> > > > > > (pre-KIP
> > > > > > > > 500)
> > > > > > > > > > a
> > > > > > > > > > > > check
> > > > > > > > > > > > > > > > > in the broker to make sure its configured
> > > > cluster ID
> > > > > > > > matches
> > > > > > > > > > the
> > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > stored in ZooKeeper, and we will have to
> > perform
> > > > this
> > > > > > > > > > validation
> > > > > > > > > > > > > > > > > somewhere in the KIP-500 world. If the
> > > > Controller
> > > > > > > > doesn't
> > > > > > > > > > do it
> > > > > > > > > > > > > > > > > within the registration request then the
> > broker
> > > > will
> > > > > > > > have to
> > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > metadata request to the Controller, retrieve
> > the
> > > > > > Cluster
> > > > > > > > ID,
> > > > > > > > > > and
> > > > > > > > > > > > > > > > > perform the check itself. It feels to me
> > that it
> > > > > > might
> > > > > > > > be
> > > > > > > > > > > > better for
> > > > > > > > > > > > > > > > > the Controller to just do it, and then the
> > broker
> > > > > > doesn't
> > > > > > > > > > have to
> > > > > > > > > > > > > > > > > worry about it anymore once it successfully
> > > > > > registers.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I also have a question about the broker.id
> > > > value and
> > > > > > > > > > > > > > meta.properties.
> > > > > > > > > > > > > > > > > The KIP now says "In version 0 of
> > > > meta.properties,
> > > > > > there
> > > > > > > > is a
> > > > > > > > > > > > > > > > > broker.id field. Version 1 does not have
> > this
> > > > > > field.
> > > > > > > > It
> > > > > > > > > > is no
> > > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > > needed because we no longer support dynamic
> > > > broker id
> > > > > > > > > > > > assignment."
> > > > > > > > > > > > > > > > > But then there is an example version 1
> > > > > > meta.properties
> > > > > > > > file
> > > > > > > > > > that
> > > > > > > > > > > > > > shows
> > > > > > > > > > > > > > > > > the broker.id value. I actually wonder if
> > > > maybe the
> > > > > > > > > > broker.id
> > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > would be good to keep in the version 1
> > > > > > meta.properties
> > > > > > > > file
> > > > > > > > > > > > because
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > currently (pre-KIP 500, version 0) acts as a
> > > > sanity
> > > > > > > > check to
> > > > > > > > > > make
> > > > > > > > > > > > > > sure
> > > > > > > > > > > > > > > > > the broker is using the correct log
> > directory.
> > > > > > Similarly
> > > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > controller.id value on controllers -- it
> > would
> > > > > > allow the
> > > > > > > > > > same
> > > > > > > > > > > > type
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > sanity check for quorum controllers.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > That's a good point. I will add broker.id
> > back,
> > > > and
> > > > > > also
> > > > > > > > add
> > > > > > > > > > > > > > > > controller.id as a possibility.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe
> > <
> > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao
> > wrote:
> > > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for the reply. A few more
> > comments.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks again for the reply. Sorry for the
> > long
> > > > > > > > hiatus. I
> > > > > > > > > > was
> > > > > > > > > > > > on
> > > > > > > > > > > > > > > > vacation for a while.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 55. There is still text that favors new
> > > > broker
> > > > > > > > > > registration.
> > > > > > > > > > > > > > "When a
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > first starts up, when it is in the
> > INITIAL
> > > > > > state, it
> > > > > > > > will
> > > > > > > > > > > > always
> > > > > > > > > > > > > > > > "win"
> > > > > > > > > > > > > > > > > > > broker ID conflicts. However, once it is
> > > > > > granted a
> > > > > > > > > > lease, it
> > > > > > > > > > > > > > > > transitions
> > > > > > > > > > > > > > > > > > > out of the INITIAL state. Thereafter,
> > it may
> > > > > > lose
> > > > > > > > > > subsequent
> > > > > > > > > > > > > > > > conflicts if
> > > > > > > > > > > > > > > > > > > its broker epoch is stale. (See KIP-380
> > for
> > > > some
> > > > > > > > > > background
> > > > > > > > > > > > on
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > epoch.) The reason for favoring new
> > > > processes
> > > > > > is to
> > > > > > > > > > > > accommodate
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > > > case where a process is killed with kill
> > -9
> > > > and
> > > > > > then
> > > > > > > > > > > > restarted.
> > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > want it
> > > > > > > > > > > > > > > > > > > to be able to reclaim its old ID quickly
> > in
> > > > this
> > > > > > > > case."
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for the reminder. I have clarified
> > the
> > > > > > language
> > > > > > > > > > here.
> > > > > > > > > > > > > > > > Hopefully now it is clear that we don't allow
> > quick
> > > > > > re-use
> > > > > > > > of
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 80.1 Sounds good. Could you document that
> > > > > > listeners
> > > > > > > > is a
> > > > > > > > > > > > required
> > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > now? It would also be useful to annotate
> > > > other
> > > > > > > > required
> > > > > > > > > > > > configs.
> > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > example, controller.connect should be
> > > > required.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I added a note specifying that these are
> > > > required.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 80.2 Could you list all deprecated
> > existing
> > > > > > configs?
> > > > > > > > > > Another
> > > > > > > > > > > > one
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > control.plane.listener.name since the
> > > > > > controller no
> > > > > > > > > > longer
> > > > > > > > > > > > sends
> > > > > > > > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata and
> > StopReplica
> > > > > > > > requests.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I added a section specifying some
> > deprecated
> > > > > > configs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 83.1 It seems that the broker can
> > transition
> > > > from
> > > > > > > > FENCED
> > > > > > > > > > to
> > > > > > > > > > > > > > RUNNING
> > > > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > > > registering for a new broker epoch. I am
> > not
> > > > > > sure how
> > > > > > > > > > this
> > > > > > > > > > > > works.
> > > > > > > > > > > > > > > > Once the
> > > > > > > > > > > > > > > > > > > controller fences a broker, there is no
> > need
> > > > for
> > > > > > the
> > > > > > > > > > > > controller
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > > > > boker epoch around. So, if the fenced
> > > > broker's
> > > > > > > > heartbeat
> > > > > > > > > > > > request
> > > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > existing broker epoch will be rejected,
> > > > leading
> > > > > > the
> > > > > > > > > > broker
> > > > > > > > > > > > back
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The broker epoch refers to the broker
> > > > registration.
> > > > > > > > So we
> > > > > > > > > > DO
> > > > > > > > > > > > keep
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > broker epoch around even while the broker is
> > > > fenced.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The broker epoch changes only when there
> > is a
> > > > new
> > > > > > > > broker
> > > > > > > > > > > > > > > > registration. Fencing or unfencing the broker
> > > > doesn't
> > > > > > > > change
> > > > > > > > > > the
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 83.5 Good point on KIP-590. Then should
> > we
> > > > > > expose the
> > > > > > > > > > > > controller
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > debugging purposes? If not, we should
> > > > deprecate
> > > > > > the
> > > > > > > > > > > > controllerID
> > > > > > > > > > > > > > > > field in
> > > > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think it's OK to expose it for now, with
> > the
> > > > > > proviso
> > > > > > > > > > that it
> > > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > > be reachable by clients.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 90. We rejected the shared ID with just
> > one
> > > > > > reason
> > > > > > > > "This
> > > > > > > > > > is
> > > > > > > > > > > > not a
> > > > > > > > > > > > > > > > good idea
> > > > > > > > > > > > > > > > > > > because NetworkClient assumes a single ID
> > > > > > space. So
> > > > > > > > if
> > > > > > > > > > > > there is
> > > > > > > > > > > > > > > > both a
> > > > > > > > > > > > > > > > > > > controller 1 and a broker 1, we don't
> > have a
> > > > way
> > > > > > of
> > > > > > > > > > picking
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > "right"
> > > > > > > > > > > > > > > > > > > one." This doesn't seem to be a strong
> > > > reason.
> > > > > > For
> > > > > > > > > > example,
> > > > > > > > > > > > we
> > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > > address the NetworkClient issue with the
> > node
> > > > > > type
> > > > > > > > as you
> > > > > > > > > > > > pointed
> > > > > > > > > > > > > > > > out or
> > > > > > > > > > > > > > > > > > > using the negative value of a broker ID
> > as
> > > > the
> > > > > > > > > > controller ID.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It would require a lot of code changes to
> > > > support
> > > > > > > > multiple
> > > > > > > > > > > > types of
> > > > > > > > > > > > > > > > node IDs. It's not clear to me that the end
> > result
> > > > > > would
> > > > > > > > be
> > > > > > > > > > > > better --
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > tend to think it would be worse, since it
> > would be
> > > > more
> > > > > > > > > > complex.
> > > > > > > > > > > > In a
> > > > > > > > > > > > > > > > similar vein, using negative numbers seems
> > > > dangerous,
> > > > > > > > since we
> > > > > > > > > > use
> > > > > > > > > > > > > > > > negatives or -1 as "special values" in many
> > places.
> > > > > > For
> > > > > > > > > > example,
> > > > > > > > > > > > -1
> > > > > > > > > > > > > > often
> > > > > > > > > > > > > > > > represents "no such node."
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > One important thing to keep in mind is
> > that we
> > > > > > want to
> > > > > > > > be
> > > > > > > > > > able
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > transition from a broker and a controller being
> > > > > > co-located
> > > > > > > > to
> > > > > > > > > > them
> > > > > > > > > > > > no
> > > > > > > > > > > > > > > > longer being co-located. This is much easier
> > to do
> > > > > > when
> > > > > > > > they
> > > > > > > > > > have
> > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > the broker reports all offline replicas
> > due
> > > > to a
> > > > > > disk
> > > > > > > > > > > > failure to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > controller. It seems this information
> > needs
> > > > to be
> > > > > > > > > > persisted
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > > log. Do we have a corresponding record
> > for
> > > > that?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hmm, I have to look into this a little bit
> > > > more.
> > > > > > We
> > > > > > > > may
> > > > > > > > > > need
> > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > record type.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 101. Currently, StopReplica request has 2
> > > > modes,
> > > > > > > > without
> > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > and with
> > > > > > > > > > > > > > > > > > > deletion. The former is used for
> > controlled
> > > > > > shutdown
> > > > > > > > and
> > > > > > > > > > > > handling
> > > > > > > > > > > > > > > > disk
> > > > > > > > > > > > > > > > > > > failure, and causes the follower to
> > stop. The
> > > > > > latter
> > > > > > > > is
> > > > > > > > > > for
> > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > > and partition reassignment, and causes
> > the
> > > > > > replica
> > > > > > > > to be
> > > > > > > > > > > > deleted.
> > > > > > > > > > > > > > > > Since we
> > > > > > > > > > > > > > > > > > > are deprecating StopReplica, could we
> > > > document
> > > > > > what
> > > > > > > > > > triggers
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > stopping
> > > > > > > > > > > > > > > > > > > of a follower and the deleting of a
> > replica
> > > > now?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > RemoveTopic triggers deletion. In general
> > the
> > > > > > > > > > functionality of
> > > > > > > > > > > > > > > > StopReplica is subsumed by the metadata
> > records.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 102. Should we include the metadata
> > topic in
> > > > the
> > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > If so,
> > > > > > > > > > > > > > > > > > > when it will be included and what will
> > the
> > > > > > metadata
> > > > > > > > > > response
> > > > > > > > > > > > look
> > > > > > > > > > > > > > > > like?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > No, it won't be included in the metadata
> > > > response
> > > > > > sent
> > > > > > > > back
> > > > > > > > > > > > from
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > brokers.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 103. "The active controller assigns the
> > > > broker a
> > > > > > new
> > > > > > > > > > broker
> > > > > > > > > > > > > > epoch,
> > > > > > > > > > > > > > > > based on
> > > > > > > > > > > > > > > > > > > the latest committed offset in the log."
> > This
> > > > > > seems
> > > > > > > > > > > > inaccurate
> > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > latest committed offset doesn't always
> > > > advance on
> > > > > > > > every
> > > > > > > > > > log
> > > > > > > > > > > > > > append.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Given that the new broker epoch won't be
> > > > visible
> > > > > > until
> > > > > > > > the
> > > > > > > > > > > > commit
> > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > happened, I have changed this to "the next
> > > > available
> > > > > > > > offset in
> > > > > > > > > > the
> > > > > > > > > > > > log"
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 104. REGISTERING(1) : It says
> > "Otherwise, the
> > > > > > broker
> > > > > > > > > > moves
> > > > > > > > > > > > into
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > > > > > state.". It seems this should be RUNNING?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 105. RUNNING: Should we require the
> > broker to
> > > > > > catch
> > > > > > > > up
> > > > > > > > > > to the
> > > > > > > > > > > > > > > > metadata log
> > > > > > > > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For 104 and 105, these sections have been
> > > > reworked.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin
> > McCabe
> > > > <
> > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom
> > Bentley
> > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron
> > > > Dagostino
> > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the hard
> > work
> > > > on
> > > > > > this
> > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I have some questions about what
> > > > happens
> > > > > > to a
> > > > > > > > > > broker
> > > > > > > > > > > > > > when it
> > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > > fenced (e.g. because it can't
> > send a
> > > > > > > > heartbeat
> > > > > > > > > > > > request to
> > > > > > > > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > > > > > > > lease). The KIP says "When a
> > broker
> > > > is
> > > > > > > > fenced,
> > > > > > > > > > it
> > > > > > > > > > > > cannot
> > > > > > > > > > > > > > > > process any
> > > > > > > > > > > > > > > > > > > > > > > client requests. This prevents
> > > > brokers
> > > > > > which
> > > > > > > > > > are not
> > > > > > > > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > > > > > > > metadata updates or that are not
> > > > > > receiving
> > > > > > > > and
> > > > > > > > > > > > processing
> > > > > > > > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > > > > > > > enough from causing issues to
> > > > clients."
> > > > > > And
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > > > > > > > FENCED(4) state it likewise says
> > > > "While
> > > > > > in
> > > > > > > > this
> > > > > > > > > > > > state,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > > > not respond to client
> > requests." It
> > > > > > makes
> > > > > > > > sense
> > > > > > > > > > > > that a
> > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > > should not accept producer
> > requests
> > > > -- I
> > > > > > > > assume
> > > > > > > > > > any
> > > > > > > > > > > > such
> > > > > > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > > > > > > would result in
> > > > > > NotLeaderOrFollowerException.
> > > > > > > > > > But
> > > > > > > > > > > > what
> > > > > > > > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > > > > > > > (fetch from follower) consumer
> > > > > > requests? It
> > > > > > > > is
> > > > > > > > > > > > > > conceivable
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > > could continue. Related to that,
> > > > would a
> > > > > > > > fenced
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > > > fetch data for partitions where
> > it
> > > > > > thinks it
> > > > > > > > is a
> > > > > > > > > > > > > > follower?
> > > > > > > > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > > > > > > > it rejects consumer requests it
> > might
> > > > > > still
> > > > > > > > > > continue
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > > > > > > > follower. Might it be helpful to
> > > > clarify
> > > > > > > > both
> > > > > > > > > > > > decisions
> > > > > > > > > > > > > > > > here?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Good question. I think a fenced
> > broker
> > > > > > should
> > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > > > > > > > partitions it was already fetching
> > > > before
> > > > > > it
> > > > > > > > was
> > > > > > > > > > > > fenced,
> > > > > > > > > > > > > > > > unless it
> > > > > > > > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > > > > > > > problem. At that point it won't be
> > > > able to
> > > > > > > > > > continue,
> > > > > > > > > > > > > > since it
> > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > the new metadata. For example, it
> > > > won't
> > > > > > know
> > > > > > > > about
> > > > > > > > > > > > > > leadership
> > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the partitions it's fetching. The
> > > > > > rationale
> > > > > > > > for
> > > > > > > > > > > > > > continuing to
> > > > > > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > > > > try to avoid disruptions as much as
> > > > > > possible.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I don't think fenced brokers should
> > > > accept
> > > > > > > > client
> > > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > The issue
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > that the fenced broker may or may
> > not
> > > > have
> > > > > > any
> > > > > > > > > > data it
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > > > > > > > have. It may or may not have
> > applied
> > > > any
> > > > > > > > > > configuration
> > > > > > > > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > it is supposed to have applied.
> > So it
> > > > > > could
> > > > > > > > get
> > > > > > > > > > pretty
> > > > > > > > > > > > > > > > confusing, and
> > > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > potentially waste the client's
> > time.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > When fenced, how would the broker
> > reply
> > > > to a
> > > > > > > > client
> > > > > > > > > > > > which did
> > > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The broker will respond with a
> > retryable
> > > > error
> > > > > > in
> > > > > > > > that
> > > > > > > > > > > > case.
> > > > > > > > > > > > > > Once
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > client has re-fetched its metadata, it
> > > > will no
> > > > > > > > longer
> > > > > > > > > > see
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > as part of the cluster. I added a
> > note to
> > > > the
> > > > > > KIP.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the reply. Just a couple of more comments.
211. Currently, the broker only registers itself in ZK after log recovery.
Is there any benefit to change that? As you mentioned, the broker can't do
much before completing log recovery.
230. Regarding MetadataResponse, there is a slight awkwardness. We return
rack for each node. However, if that node is for the controller, the rack
field is not really relevant. Should we clean it up here or in another KIP
like KIP-700?
Thanks,
Jun
On Wed, Dec 16, 2020 at 4:23 PM Colin McCabe <cm...@apache.org> wrote:
> On Wed, Dec 16, 2020, at 13:40, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the reply. A few follow up comments.
> >
> > 211. When does the broker send the BrokerRegistration request to the
> > controller? Is it after the recovery phase? If so, at that point, the
> > broker has already caught up on the metadata (in order to clean up
> deleted
> > partitions). Then, it seems we don't need the ShouldFence field
> > in BrokerHeartbeatRequest?
>
> Hi Jun,
>
> Thanks again for the reviews.
>
> The broker sends the registration request as soon as it starts up. It
> cannot wait until the recovery phase is over since sometimes log recovery
> can take quite a long time.
>
> >
> > 213. kafka-cluster.sh
> > 213.1 For the decommision example, should the command take a broker id?
>
> Yes, the example should have a broker id. Fixed.
>
> > 213.2 Existing tools use the "--" command line option (e.g. kafka-topics
> > --list --topic test). Should we follow the same convention
> > for kafka-cluster.sh (and kafka-storage.sh)?
>
> Hmm. I don't think argparse4j supports using double dashes to identify
> subcommands. I think it might be confusing as well, since the subcommand
> must come first, unlike a plain old argument which can be anywhere on the
> command line.
>
> > 213.3 Should we add an admin api for broker decommission so that this can
> > be done programmatically?
> >
>
> Yes, there is an admin client API for decommissioning as well.
>
> > 220. DecommissionBroker: "NOT_CONTROLLER if the node that the request was
> > sent to is not the controller". I thought the clients never send a
> request
> > to the controller directly now and the broker will forward it to the
> > controller?
> >
>
> If the controller moved recently, it's possible that the broker could send
> to a controller that has just recently become inactive. In that case
> NOT_CONTROLLER would be returned. (A standby controller returns
> NOT_CONTROLLER for most APIs).
>
> > 221. Could we add the required ACL for the new requests?
> >
>
> Good point. I added the required ACL for each new RPC.
>
> best,
> Colin
>
>
> > Jun
> >
> > On Wed, Dec 16, 2020 at 11:38 AM Colin McCabe <cm...@apache.org>
> wrote:
> >
> > > On Wed, Dec 16, 2020, at 09:59, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply. A few more comments below.
> > > >
> > >
> > > Hi Jun,
> > >
> > > Thanks for the comments.
> > >
> > > > 206. "RemoveTopic is the last step, that scrubs all metadata about
> the
> > > > topic. In order to get to that last step, the topic data needs to
> > > removed
> > > > from all brokers (after each broker notices that the topic is being
> > > > deleted)." Currently, this is done based on the response of
> > > > StopReplicaRequest. Since the controller no longer sends this
> request,
> > > how
> > > > does the controller know that the data for the deleted topic has
> > > > been removed in the brokers?
> > > >
> > >
> > > That's a good point... definitely an oversight.
> > >
> > > It seems complex to force the controller to track when log directories
> > > have been deleted. Let's just assume that KIP-516 has been
> implemented,
> > > and track them by UUID. Then we can just have a single topic deletion
> > > record.
> > >
> > > I added a section on "topic identifiers" describing this.
> > >
> > > > 210. Thanks for the explanation. Sounds good to me.
> > > >
> > > > 211. I still don't see an example when ShouldFence is set to true in
> > > > BrokerHeartbeatReques. Could we add one?
> > > >
> > >
> > > It is sent to true when the broker is first starting up and doesn't yet
> > > want to be unfenced. I added a longer explanation of this in the
> "Broker
> > > Leases" section.
> > >
> > > > 213. The KIP now allows replicas to be assigned on brokers that are
> > > fenced,
> > > > which is an improvement. How do we permanently remove a broker (e.g.
> > > > cluster shrinking) to prevent it from being used for future replica
> > > > assignments?
> > > >
> > >
> > > This is a fair point. I will create a kafka-cluster.sh script which
> can
> > > do this, plus a DecommissionBrokerRequest.
> > >
> > > As a bonus the kafka-cluster.sh script can help people find the
> cluster ID
> > > of brokers, something that people have wanted a tool for in the past.
> > >
> > > > 214. "Currently, when a node is down, all of its ZK registration
> > > > information is gone. But we need this information in order to
> > > understand
> > > > things like whether the replicas of a particular partition are
> > > > well-balanced across racks." This is not quite true. Currently, even
> when
> > > > ZK registration is gone, the existing replica assignment is still
> > > available
> > > > in the metadata response.
> > > >
> > >
> > > I agree that the existing replica assignment is still available. But
> that
> > > just tells you partition X is on nodes A, B, and C. If you don't have
> the
> > > ZK registration for one or more of A, B, or C then you don't know
> whether
> > > we are following the policy of "two replicas on one rack, one replica
> on
> > > another." Or any other more complex rack placement policy that you
> might
> > > have.
> > >
> > > best,
> > > Colin
> > >
> > > > Jun
> > > >
> > > > On Wed, Dec 16, 2020 at 8:48 AM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the reply. A few more follow up comments.
> > > > > >
> > > > > > 210. initial.broker.registration.timeout.ms: The default value
> is
> > > 90sec,
> > > > > > which seems long. If a broker fails the registration because of
> > > incorrect
> > > > > > configs, we want to fail faster. In comparison, the defaults for
> > > > > > zookeeper.connection.timeout.ms is 18 secs.
> > > > > >
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > I agree that the initial connection timeout here is longer than
> what we
> > > > > had with ZK. The main reason I selected a slightly longer timeout
> > > here is
> > > > > to handle the case where the controllers and the brokers are
> > > co-located.
> > > > > For example, if you have a 3 node cluster and all three nodes are
> > > > > controllers+brokers, when you first bring up the cluster, we will
> have
> > > to
> > > > > stand up the controller quorum and then handle broker
> registrations.
> > > > > Although we believe Raft will start up relatively quickly, it's
> good to
> > > > > leave some extra margin here.
> > > > >
> > > > > I don't think there's a big disadvantage to having a slightly
> longer
> > > > > timeout here. After all, starting up the brokers with no
> controllers
> > > is an
> > > > > admin mistake, which we don't expect to see very often. Maybe
> let's
> > > set it
> > > > > to 60 seconds for now and maybe see if we can tweak it in the
> future if
> > > > > that turns out to be too long or short?
> > > > >
> > > > > > 211. "Once they are all moved, the controller responds to the
> > > heartbeat
> > > > > > with a nextState of SHUTDOWN." It seems that nextState is no
> longer
> > > > > > relevant. Also, could you add a bit more explanation on when
> > > ShouldFence
> > > > > is
> > > > > > set to true in BrokerHeartbeatRequest?
> > > > > >
> > > > >
> > > > > Good catch. I removed the obsolete section referring to nextState
> and
> > > > > added a reference to the new boolean. I also added some more
> > > information
> > > > > about ShouldFence and about the rationale for separating fencing
> from
> > > > > registration.
> > > > >
> > > > > > 212. Related to the above, what does the broker do if IsFenced is
> > > true in
> > > > > > the BrokerHeartbeatResponse? Will the broker do the same thing on
> > > > > receiving
> > > > > > a FenceBrokerRecord from the metadata log?
> > > > > >
> > > > >
> > > > > The broker only checks this boolean during the startup process.
> After
> > > the
> > > > > startup process is finished, it ignores this boolean.
> > > > >
> > > > > The broker uses fence / unfence records in the metadata log to
> > > determine
> > > > > which brokers should appear in its MetadataResponse.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <cmccabe@apache.org
> >
> > > wrote:
> > > > > >
> > > > > > > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > > > > > > Hi Colin,
> > > > > > > >
> > > > > > > > The KIP says that "brokers which are fenced will not appear
> in
> > > > > > > > MetadataResponses. So clients that have up-to-date metadata
> will
> > > > > not try
> > > > > > > > to contact fenced brokers.", which is fine, but it doesn't
> seem
> > > to
> > > > > > > > elaborate on what happens for clients which try to contact a
> > > broker
> > > > > > > (using
> > > > > > > > stale metadata, for example) and find it in the
> > > registered-but-fenced
> > > > > > > > state.
> > > > > > >
> > > > > > > Hi Tom,
> > > > > > >
> > > > > > > I have dropped the broker-side fencing from this proposal. So
> now
> > > the
> > > > > > > fencing is basically the same as today's fencing: it's
> > > controller-side
> > > > > > > only. That means that clients using stale metadata can contact
> > > fenced
> > > > > > > brokers and communicate with them. The only case where the
> broker
> > > > > will be
> > > > > > > inaccessible is when it hasn't finished starting yet (i.e., has
> > > not yet
> > > > > > > attained RUNNING state.)
> > > > > > >
> > > > > > > Just like today, we expect the safeguards built into the
> > > replication
> > > > > > > protocol to prevent the worst corner cases that could result
> from
> > > > > this. I
> > > > > > > do think we will probably take up this issue later and find a
> > > better
> > > > > > > solution for client-side metadata, but this KIP is big enough
> > > as-is.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > >
> > > > > > > > You said previously that the broker will respond with a
> retriable
> > > > > > > > error, which is again fine, but you didn't answer my question
> > > about
> > > > > which
> > > > > > > > error code you've chosen for this. I realise that this
> doesn't
> > > really
> > > > > > > > affect the correctness of the behaviour, but I'm not aware
> of any
> > > > > > > existing
> > > > > > > > error code which is a good fit. So it would be good to
> understand
> > > > > about
> > > > > > > how
> > > > > > > > you're making this backward compatible for clients.
> > > > > > > >
> > > > > > > > Many thanks,
> > > > > > > >
> > > > > > > > Tom
> > > > > > > >
> > > > > > > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <
> cmccabe@apache.org
> > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > > > > > > Hi, Colin,
> > > > > > > > > >
> > > > > > > > > > Thanks for the reply. Just a couple of more comments
> below.
> > > > > > > > > >
> > > > > > > > > > 210. Since we are deprecating
> > > zookeeper.connection.timeout.ms,
> > > > > > > should we
> > > > > > > > > > add a new config to bound the time for a broker to
> connect
> > > to the
> > > > > > > > > > controller during starting up?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Good idea. I added initial.broker.registration.timeout.ms
> for
> > > > > this.
> > > > > > > > >
> > > > > > > > > > 211. BrokerHeartbeat no longer has the state field in the
> > > > > > > > > request/response.
> > > > > > > > > > However, (a) the controller shutdown section still has
> "In
> > > its
> > > > > > > periodic
> > > > > > > > > > heartbeats, the broker asks the controller if it can
> > > transition
> > > > > into
> > > > > > > the
> > > > > > > > > > SHUTDOWN state. This motivates the controller to move
> all
> > > of the
> > > > > > > leaders
> > > > > > > > > > off of that broker. Once they are all moved, the
> controller
> > > > > > > responds to
> > > > > > > > > > the heartbeat with a nextState of SHUTDOWN."; (2) the
> > > > > description of
> > > > > > > > > > BrokerHeartbeat still references currentState and
> > > targetState.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks. I've made these sections clearer and removed the
> > > obsolete
> > > > > > > > > references to sending states.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <
> > > cmccabe@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the update. A few more follow up comments.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Jun,
> > > > > > > > > > >
> > > > > > > > > > > Thanks again for the review.
> > > > > > > > > > >
> > > > > > > > > > > > 100. FailedReplicaRecord: Since this is reported by
> each
> > > > > broker
> > > > > > > > > > > > independently, perhaps we could use a more concise
> > > > > representation
> > > > > > > > > that
> > > > > > > > > > > has
> > > > > > > > > > > > a top level broker field, an array of topics, which
> has
> > > an
> > > > > array
> > > > > > > of
> > > > > > > > > > > > partitions.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The issue is that there is a size limit on the each
> record.
> > > > > > > Putting
> > > > > > > > > all
> > > > > > > > > > > of the partitions of a log directory into a single
> record
> > > would
> > > > > > > > > probably
> > > > > > > > > > > break that in many cases. Still, we can optimize a
> bit by
> > > > > having
> > > > > > > an
> > > > > > > > > array
> > > > > > > > > > > of partition IDs, since nearly all the time, we have
> more
> > > than
> > > > > one
> > > > > > > > > from the
> > > > > > > > > > > same topic.
> > > > > > > > > > >
> > > > > > > > > > > > 200. Sounds good. If we remove the broker-side
> fencing
> > > > > logic, do
> > > > > > > we
> > > > > > > > > plan
> > > > > > > > > > > to
> > > > > > > > > > > > still keep FENCED in broker state? Do we plan to
> expose
> > > the
> > > > > new
> > > > > > > > > states
> > > > > > > > > > > > through the existing BrokerState metric and if so,
> what
> > > are
> > > > > the
> > > > > > > > > values
> > > > > > > > > > > for
> > > > > > > > > > > > the new states?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > No, we don't need FENCED any more. I have removed it
> from
> > > the
> > > > > KIP.
> > > > > > > > > > >
> > > > > > > > > > > The new states are very similar to the current ones,
> > > actually.
> > > > > > > There
> > > > > > > > > are
> > > > > > > > > > > no new states or removed ones. The main change in the
> > > broker
> > > > > state
> > > > > > > > > machine
> > > > > > > > > > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state has
> been
> > > > > > > renamed to
> > > > > > > > > > > RECOVERY. Also, unlike previously, the broker will
> always
> > > pass
> > > > > > > through
> > > > > > > > > > > RECOVERY (although it may only stay in this state for a
> > > very
> > > > > short
> > > > > > > > > amount
> > > > > > > > > > > of time).
> > > > > > > > > > >
> > > > > > > > > > > > 201. This may be fine too. Could we document what
> happens
> > > > > when
> > > > > > > the
> > > > > > > > > > > > broker.id/controller.id in metadata.properties don't
> > > match
> > > > > the
> > > > > > > > > broker
> > > > > > > > > > > > config when the broker starts up?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I added some documentation about this.
> > > > > > > > > > >
> > > > > > > > > > > > 204. There is still "The highest metadata offset
> which
> > > the
> > > > > broker
> > > > > > > > > has not
> > > > > > > > > > > > reached" referenced under BrokerRegistration.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > > > > > > >
> > > > > > > > > > > > 206. Is that separate step needed given KIP-516? With
> > > > > KIP-516 (
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > > > > > > ),
> > > > > > > > > > > > we don't need to wait for the topic data to be
> removed
> > > from
> > > > > all
> > > > > > > > > brokers
> > > > > > > > > > > > before removing the topic metadata. The combination
> of
> > > > > unmatching
> > > > > > > > > > > > topicId
> > > > > > > > > > > > or the missing topicId from the metadata is enough
> for
> > > the
> > > > > > > broker to
> > > > > > > > > > > > clean
> > > > > > > > > > > > up deleted topics asynchronously.
> > > > > > > > > > >
> > > > > > > > > > > It won't be needed once KIP-516 is adopted, but this
> hasn't
> > > > > been
> > > > > > > > > > > implemented yet.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Jun
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <
> > > > > cmccabe@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the updated KIP. A few more comments
> > > below.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks again for the reviews.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 80.2 For deprecated configs, we need to include
> > > > > zookeeper.*
> > > > > > > and
> > > > > > > > > > > > > > broker.id.generation.enable.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Added.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 83.1 If a broker is down, does the controller
> keep
> > > the
> > > > > > > previously
> > > > > > > > > > > > > > registered broker epoch forever? If not, how long
> > > does
> > > > > the
> > > > > > > > > controller
> > > > > > > > > > > > > keep
> > > > > > > > > > > > > > it? What does the controller do when receiving a
> > > broker
> > > > > > > heartbeat
> > > > > > > > > > > request
> > > > > > > > > > > > > > with an unfound broker epoch?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, the controller keeps the previous registration
> > > > > forever.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Broker heartbeat requests with an incorrect broker
> > > epoch
> > > > > will
> > > > > > > be
> > > > > > > > > > > rejected
> > > > > > > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 100. Have you figured out if we need to add a new
> > > record
> > > > > > > type for
> > > > > > > > > > > > > reporting
> > > > > > > > > > > > > > partitions on failed disks?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I added FailedReplicaRecord to reflect the case
> where a
> > > > > JBOD
> > > > > > > > > directory
> > > > > > > > > > > has
> > > > > > > > > > > > > failed, leading to failed replicas.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 102. For debugging purposes, sometimes it's
> useful to
> > > > > read
> > > > > > > the
> > > > > > > > > > > metadata
> > > > > > > > > > > > > > topic using tools like console-consumer. Should
> we
> > > > > support
> > > > > > > that
> > > > > > > > > and
> > > > > > > > > > > if
> > > > > > > > > > > > > so,
> > > > > > > > > > > > > > how?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > For now, we have the ability to read the metadata
> logs
> > > > > with the
> > > > > > > > > > > dump-logs
> > > > > > > > > > > > > tool. I think we will come up with some other
> tools
> > > in the
> > > > > > > future
> > > > > > > > > as
> > > > > > > > > > > we
> > > > > > > > > > > > > get experience.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 200. "brokers which are fenced will not appear in
> > > > > > > > > MetadataResponses.
> > > > > > > > > > > The
> > > > > > > > > > > > > > broker will not respond to these requests--
> instead,
> > > it
> > > > > will
> > > > > > > > > simply
> > > > > > > > > > > > > > disconnect." If the controller is partitioned off
> > > from
> > > > > the
> > > > > > > > > brokers,
> > > > > > > > > > > this
> > > > > > > > > > > > > > design will cause every broker to stop accepting
> new
> > > > > client
> > > > > > > > > > > requests. In
> > > > > > > > > > > > > > contrast, if ZK is partitioned off, the existing
> > > > > behavior is
> > > > > > > > > that the
> > > > > > > > > > > > > > brokers can continue to work based on the last
> known
> > > > > > > metadata.
> > > > > > > > > So, I
> > > > > > > > > > > am
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > sure if we should change the existing behavior
> > > because
> > > > > of the
> > > > > > > > > bigger
> > > > > > > > > > > > > impact
> > > > > > > > > > > > > > in the new one. Another option is to keep the
> > > existing
> > > > > > > behavior
> > > > > > > > > and
> > > > > > > > > > > > > expose
> > > > > > > > > > > > > > a metric for fenced brokers so that the operator
> > > could be
> > > > > > > > > alerted.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm skeptical about how well running without ZK
> > > currently
> > > > > > > works.
> > > > > > > > > > > However,
> > > > > > > > > > > > > I will move the broker-side fencing into a
> follow-up
> > > KIP.
> > > > > This
> > > > > > > > > KIP is
> > > > > > > > > > > > > already pretty large and there is no hard
> dependency on
> > > > > this.
> > > > > > > > > There
> > > > > > > > > > > may
> > > > > > > > > > > > > also be other ways of accomplishing the positive
> > > effects of
> > > > > > > what
> > > > > > > > > > > > > broker-side fencing, so more discussion is needed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 201. I read Ron's comment, but I am still not
> sure
> > > the
> > > > > > > benefit of
> > > > > > > > > > > keeping
> > > > > > > > > > > > > > broker.id and controller.id in meta.properties.
> It
> > > seems
> > > > > > > that
> > > > > > > > > we are
> > > > > > > > > > > > > just
> > > > > > > > > > > > > > duplicating the same info in two places and have
> the
> > > > > > > additional
> > > > > > > > > > > burden of
> > > > > > > > > > > > > > making sure the values in the two places are
> > > consistent.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think the reasoning is that having broker.id
> > > protects us
> > > > > > > against
> > > > > > > > > > > > > accidentally bringing up a broker with a disk from
> a
> > > > > different
> > > > > > > > > > > broker. I
> > > > > > > > > > > > > don't feel strongly about this but it seemed
> simpler to
> > > > > keep
> > > > > > > it.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 202. controller.connect.security.protocol: Is
> this
> > > needed
> > > > > > > since
> > > > > > > > > > > > > > controller.listener.names and
> > > > > listener.security.protocol.map
> > > > > > > > > imply
> > > > > > > > > > > the
> > > > > > > > > > > > > > security protocol already?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > You're right, this isn't needed. I'll remove it.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 203. registration.heartbeat.interval.ms: It
> > > defaults to
> > > > > 2k.
> > > > > > > ZK
> > > > > > > > > uses
> > > > > > > > > > > 1/3
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > the session timeout for heartbeat. So, given the
> > > default
> > > > > 18k
> > > > > > > for
> > > > > > > > > > > > > > registration.lease.timeout.ms, should we default
> > > > > > > > > > > > > > registration.heartbeat.interval.ms to 6k?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 6 seconds seems like a pretty long time between
> > > > > heartbeats. It
> > > > > > > > > might
> > > > > > > > > > > be
> > > > > > > > > > > > > useful to know when a broker is missing heartbeats,
> > > with
> > > > > less
> > > > > > > time
> > > > > > > > > than
> > > > > > > > > > > > > that. I provisionally set it to 3 seconds (we can
> > > always
> > > > > > > change
> > > > > > > > > > > later...)
> > > > > > > > > > > > >
> > > > > > > > > > > > > I also changed the name of these configurations to
> "
> > > > > > > > > > > > > broker.heartbeat.interval.ms" and "
> > > > > > > broker.registration.timeout.ms"
> > > > > > > > > to
> > > > > > > > > > > try
> > > > > > > > > > > > > to clarify them a bit.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 204. "The highest metadata offset which the
> broker
> > > has
> > > > > not
> > > > > > > > > reached."
> > > > > > > > > > > It
> > > > > > > > > > > > > > seems this should be "has reached".
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I changed this to "one more than the highest
> metadata
> > > > > offset
> > > > > > > which
> > > > > > > > > the
> > > > > > > > > > > > > broker has reached."
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 205. UnfenceBrokerRecord and
> UnregisterBrokerRecord:
> > > To
> > > > > me,
> > > > > > > they
> > > > > > > > > > > seem to
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > the same. Do we need both?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Unregistration means that the broker has been
> removed
> > > from
> > > > > the
> > > > > > > > > cluster.
> > > > > > > > > > > > > That is different than unfencing, which marks the
> > > broker as
> > > > > > > active.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 206. TopicRecord: The Deleting field is used to
> > > indicate
> > > > > > > that the
> > > > > > > > > > > topic
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > being deleted. I am wondering if this is really
> > > needed
> > > > > since
> > > > > > > > > > > RemoveTopic
> > > > > > > > > > > > > > already indicates the same thing.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > RemoveTopic is the last step, that scrubs all
> metadata
> > > > > about
> > > > > > > the
> > > > > > > > > topic.
> > > > > > > > > > > > > In order to get to that last step, the topic data
> > > needs to
> > > > > > > removed
> > > > > > > > > > > from all
> > > > > > > > > > > > > brokers (after each broker notices that the topic
> is
> > > being
> > > > > > > > > deleted).
> > > > > > > > > > > > >
> > > > > > > > > > > > > best,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <
> > > > > > > cmccabe@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino
> wrote:
> > > > > > > > > > > > > > > > Hi Colin. Thanks for the updates. It's now
> > > clear
> > > > > to me
> > > > > > > that
> > > > > > > > > > > brokers
> > > > > > > > > > > > > > > > keep their broker epoch for the life of their
> > > JVM --
> > > > > they
> > > > > > > > > > > register
> > > > > > > > > > > > > > > > once, get their broker epoch in the
> response, and
> > > > > then
> > > > > > > never
> > > > > > > > > > > > > > > > re-register again. Brokers may get fenced,
> but
> > > they
> > > > > > > keep the
> > > > > > > > > > > same
> > > > > > > > > > > > > > > > broker epoch for the life of their JVM. The
> > > > > incarnation
> > > > > > > ID
> > > > > > > > > is
> > > > > > > > > > > also
> > > > > > > > > > > > > > > > kept for the life of the JVM but is
> generated by
> > > the
> > > > > > > broker
> > > > > > > > > > > itself
> > > > > > > > > > > > > > > > upon startup, and the combination of the two
> > > allows
> > > > > the
> > > > > > > > > > > Controller to
> > > > > > > > > > > > > > > > act idempotently if any previously-sent
> > > registration
> > > > > > > response
> > > > > > > > > > > gets
> > > > > > > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks, Ron. That's a good summary.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > One thing I wonder about is if it might be
> > > helpful
> > > > > for
> > > > > > > the
> > > > > > > > > > > broker to
> > > > > > > > > > > > > > > > send the Cluster ID as determined from its
> > > > > > > meta.properties
> > > > > > > > > file
> > > > > > > > > > > in
> > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > registration request. Does it even make
> sense
> > > for
> > > > > the
> > > > > > > > > broker to
> > > > > > > > > > > > > > > > successfully register and enter the Fenced
> state
> > > if
> > > > > it
> > > > > > > has
> > > > > > > > > the
> > > > > > > > > > > wrong
> > > > > > > > > > > > > > > > Cluster ID?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Yeah, that's a good idea. Let's have the
> broker
> > > pass
> > > > > its
> > > > > > > > > cluster
> > > > > > > > > > > ID in
> > > > > > > > > > > > > > > the registration RPC, and then registration can
> > > fail
> > > > > if the
> > > > > > > > > broker
> > > > > > > > > > > is
> > > > > > > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The nextMetadatOffset value that the broker
> > > > > communicates
> > > > > > > > > > > > > > > > in its registration request only has meaning
> > > within
> > > > > the
> > > > > > > > > correct
> > > > > > > > > > > > > > > > cluster, so it feels to me that the
> Controller
> > > should
> > > > > > > have
> > > > > > > > > some
> > > > > > > > > > > way
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > perform this sanity check. There is
> currently
> > > > > (pre-KIP
> > > > > > > 500)
> > > > > > > > > a
> > > > > > > > > > > check
> > > > > > > > > > > > > > > > in the broker to make sure its configured
> > > cluster ID
> > > > > > > matches
> > > > > > > > > the
> > > > > > > > > > > one
> > > > > > > > > > > > > > > > stored in ZooKeeper, and we will have to
> perform
> > > this
> > > > > > > > > validation
> > > > > > > > > > > > > > > > somewhere in the KIP-500 world. If the
> > > Controller
> > > > > > > doesn't
> > > > > > > > > do it
> > > > > > > > > > > > > > > > within the registration request then the
> broker
> > > will
> > > > > > > have to
> > > > > > > > > > > make a
> > > > > > > > > > > > > > > > metadata request to the Controller, retrieve
> the
> > > > > Cluster
> > > > > > > ID,
> > > > > > > > > and
> > > > > > > > > > > > > > > > perform the check itself. It feels to me
> that it
> > > > > might
> > > > > > > be
> > > > > > > > > > > better for
> > > > > > > > > > > > > > > > the Controller to just do it, and then the
> broker
> > > > > doesn't
> > > > > > > > > have to
> > > > > > > > > > > > > > > > worry about it anymore once it successfully
> > > > > registers.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I also have a question about the broker.id
> > > value and
> > > > > > > > > > > > > meta.properties.
> > > > > > > > > > > > > > > > The KIP now says "In version 0 of
> > > meta.properties,
> > > > > there
> > > > > > > is a
> > > > > > > > > > > > > > > > broker.id field. Version 1 does not have
> this
> > > > > field.
> > > > > > > It
> > > > > > > > > is no
> > > > > > > > > > > > > longer
> > > > > > > > > > > > > > > > needed because we no longer support dynamic
> > > broker id
> > > > > > > > > > > assignment."
> > > > > > > > > > > > > > > > But then there is an example version 1
> > > > > meta.properties
> > > > > > > file
> > > > > > > > > that
> > > > > > > > > > > > > shows
> > > > > > > > > > > > > > > > the broker.id value. I actually wonder if
> > > maybe the
> > > > > > > > > broker.id
> > > > > > > > > > > value
> > > > > > > > > > > > > > > > would be good to keep in the version 1
> > > > > meta.properties
> > > > > > > file
> > > > > > > > > > > because
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > currently (pre-KIP 500, version 0) acts as a
> > > sanity
> > > > > > > check to
> > > > > > > > > make
> > > > > > > > > > > > > sure
> > > > > > > > > > > > > > > > the broker is using the correct log
> directory.
> > > > > Similarly
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > controller.id value on controllers -- it
> would
> > > > > allow the
> > > > > > > > > same
> > > > > > > > > > > type
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > sanity check for quorum controllers.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > That's a good point. I will add broker.id
> back,
> > > and
> > > > > also
> > > > > > > add
> > > > > > > > > > > > > > > controller.id as a possibility.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe
> <
> > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao
> wrote:
> > > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for the reply. A few more
> comments.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks again for the reply. Sorry for the
> long
> > > > > > > hiatus. I
> > > > > > > > > was
> > > > > > > > > > > on
> > > > > > > > > > > > > > > vacation for a while.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 55. There is still text that favors new
> > > broker
> > > > > > > > > registration.
> > > > > > > > > > > > > "When a
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > first starts up, when it is in the
> INITIAL
> > > > > state, it
> > > > > > > will
> > > > > > > > > > > always
> > > > > > > > > > > > > > > "win"
> > > > > > > > > > > > > > > > > > broker ID conflicts. However, once it is
> > > > > granted a
> > > > > > > > > lease, it
> > > > > > > > > > > > > > > transitions
> > > > > > > > > > > > > > > > > > out of the INITIAL state. Thereafter,
> it may
> > > > > lose
> > > > > > > > > subsequent
> > > > > > > > > > > > > > > conflicts if
> > > > > > > > > > > > > > > > > > its broker epoch is stale. (See KIP-380
> for
> > > some
> > > > > > > > > background
> > > > > > > > > > > on
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > epoch.) The reason for favoring new
> > > processes
> > > > > is to
> > > > > > > > > > > accommodate
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > > case where a process is killed with kill
> -9
> > > and
> > > > > then
> > > > > > > > > > > restarted.
> > > > > > > > > > > > > We
> > > > > > > > > > > > > > > want it
> > > > > > > > > > > > > > > > > > to be able to reclaim its old ID quickly
> in
> > > this
> > > > > > > case."
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for the reminder. I have clarified
> the
> > > > > language
> > > > > > > > > here.
> > > > > > > > > > > > > > > Hopefully now it is clear that we don't allow
> quick
> > > > > re-use
> > > > > > > of
> > > > > > > > > > > broker
> > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 80.1 Sounds good. Could you document that
> > > > > listeners
> > > > > > > is a
> > > > > > > > > > > required
> > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > now? It would also be useful to annotate
> > > other
> > > > > > > required
> > > > > > > > > > > configs.
> > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > example, controller.connect should be
> > > required.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I added a note specifying that these are
> > > required.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 80.2 Could you list all deprecated
> existing
> > > > > configs?
> > > > > > > > > Another
> > > > > > > > > > > one
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > control.plane.listener.name since the
> > > > > controller no
> > > > > > > > > longer
> > > > > > > > > > > sends
> > > > > > > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata and
> StopReplica
> > > > > > > requests.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I added a section specifying some
> deprecated
> > > > > configs.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 83.1 It seems that the broker can
> transition
> > > from
> > > > > > > FENCED
> > > > > > > > > to
> > > > > > > > > > > > > RUNNING
> > > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > > registering for a new broker epoch. I am
> not
> > > > > sure how
> > > > > > > > > this
> > > > > > > > > > > works.
> > > > > > > > > > > > > > > Once the
> > > > > > > > > > > > > > > > > > controller fences a broker, there is no
> need
> > > for
> > > > > the
> > > > > > > > > > > controller
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > > > boker epoch around. So, if the fenced
> > > broker's
> > > > > > > heartbeat
> > > > > > > > > > > request
> > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > existing broker epoch will be rejected,
> > > leading
> > > > > the
> > > > > > > > > broker
> > > > > > > > > > > back
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The broker epoch refers to the broker
> > > registration.
> > > > > > > So we
> > > > > > > > > DO
> > > > > > > > > > > keep
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > broker epoch around even while the broker is
> > > fenced.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The broker epoch changes only when there
> is a
> > > new
> > > > > > > broker
> > > > > > > > > > > > > > > registration. Fencing or unfencing the broker
> > > doesn't
> > > > > > > change
> > > > > > > > > the
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 83.5 Good point on KIP-590. Then should
> we
> > > > > expose the
> > > > > > > > > > > controller
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > debugging purposes? If not, we should
> > > deprecate
> > > > > the
> > > > > > > > > > > controllerID
> > > > > > > > > > > > > > > field in
> > > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think it's OK to expose it for now, with
> the
> > > > > proviso
> > > > > > > > > that it
> > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > be reachable by clients.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 90. We rejected the shared ID with just
> one
> > > > > reason
> > > > > > > "This
> > > > > > > > > is
> > > > > > > > > > > not a
> > > > > > > > > > > > > > > good idea
> > > > > > > > > > > > > > > > > > because NetworkClient assumes a single ID
> > > > > space. So
> > > > > > > if
> > > > > > > > > > > there is
> > > > > > > > > > > > > > > both a
> > > > > > > > > > > > > > > > > > controller 1 and a broker 1, we don't
> have a
> > > way
> > > > > of
> > > > > > > > > picking
> > > > > > > > > > > the
> > > > > > > > > > > > > > > "right"
> > > > > > > > > > > > > > > > > > one." This doesn't seem to be a strong
> > > reason.
> > > > > For
> > > > > > > > > example,
> > > > > > > > > > > we
> > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > address the NetworkClient issue with the
> node
> > > > > type
> > > > > > > as you
> > > > > > > > > > > pointed
> > > > > > > > > > > > > > > out or
> > > > > > > > > > > > > > > > > > using the negative value of a broker ID
> as
> > > the
> > > > > > > > > controller ID.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It would require a lot of code changes to
> > > support
> > > > > > > multiple
> > > > > > > > > > > types of
> > > > > > > > > > > > > > > node IDs. It's not clear to me that the end
> result
> > > > > would
> > > > > > > be
> > > > > > > > > > > better --
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > > tend to think it would be worse, since it
> would be
> > > more
> > > > > > > > > complex.
> > > > > > > > > > > In a
> > > > > > > > > > > > > > > similar vein, using negative numbers seems
> > > dangerous,
> > > > > > > since we
> > > > > > > > > use
> > > > > > > > > > > > > > > negatives or -1 as "special values" in many
> places.
> > > > > For
> > > > > > > > > example,
> > > > > > > > > > > -1
> > > > > > > > > > > > > often
> > > > > > > > > > > > > > > represents "no such node."
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > One important thing to keep in mind is
> that we
> > > > > want to
> > > > > > > be
> > > > > > > > > able
> > > > > > > > > > > to
> > > > > > > > > > > > > > > transition from a broker and a controller being
> > > > > co-located
> > > > > > > to
> > > > > > > > > them
> > > > > > > > > > > no
> > > > > > > > > > > > > > > longer being co-located. This is much easier
> to do
> > > > > when
> > > > > > > they
> > > > > > > > > have
> > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > the broker reports all offline replicas
> due
> > > to a
> > > > > disk
> > > > > > > > > > > failure to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > controller. It seems this information
> needs
> > > to be
> > > > > > > > > persisted
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > > log. Do we have a corresponding record
> for
> > > that?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hmm, I have to look into this a little bit
> > > more.
> > > > > We
> > > > > > > may
> > > > > > > > > need
> > > > > > > > > > > a new
> > > > > > > > > > > > > > > record type.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 101. Currently, StopReplica request has 2
> > > modes,
> > > > > > > without
> > > > > > > > > > > deletion
> > > > > > > > > > > > > > > and with
> > > > > > > > > > > > > > > > > > deletion. The former is used for
> controlled
> > > > > shutdown
> > > > > > > and
> > > > > > > > > > > handling
> > > > > > > > > > > > > > > disk
> > > > > > > > > > > > > > > > > > failure, and causes the follower to
> stop. The
> > > > > latter
> > > > > > > is
> > > > > > > > > for
> > > > > > > > > > > topic
> > > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > > and partition reassignment, and causes
> the
> > > > > replica
> > > > > > > to be
> > > > > > > > > > > deleted.
> > > > > > > > > > > > > > > Since we
> > > > > > > > > > > > > > > > > > are deprecating StopReplica, could we
> > > document
> > > > > what
> > > > > > > > > triggers
> > > > > > > > > > > the
> > > > > > > > > > > > > > > stopping
> > > > > > > > > > > > > > > > > > of a follower and the deleting of a
> replica
> > > now?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > RemoveTopic triggers deletion. In general
> the
> > > > > > > > > functionality of
> > > > > > > > > > > > > > > StopReplica is subsumed by the metadata
> records.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 102. Should we include the metadata
> topic in
> > > the
> > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > If so,
> > > > > > > > > > > > > > > > > > when it will be included and what will
> the
> > > > > metadata
> > > > > > > > > response
> > > > > > > > > > > look
> > > > > > > > > > > > > > > like?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > No, it won't be included in the metadata
> > > response
> > > > > sent
> > > > > > > back
> > > > > > > > > > > from
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > brokers.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 103. "The active controller assigns the
> > > broker a
> > > > > new
> > > > > > > > > broker
> > > > > > > > > > > > > epoch,
> > > > > > > > > > > > > > > based on
> > > > > > > > > > > > > > > > > > the latest committed offset in the log."
> This
> > > > > seems
> > > > > > > > > > > inaccurate
> > > > > > > > > > > > > since
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > latest committed offset doesn't always
> > > advance on
> > > > > > > every
> > > > > > > > > log
> > > > > > > > > > > > > append.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Given that the new broker epoch won't be
> > > visible
> > > > > until
> > > > > > > the
> > > > > > > > > > > commit
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > happened, I have changed this to "the next
> > > available
> > > > > > > offset in
> > > > > > > > > the
> > > > > > > > > > > log"
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 104. REGISTERING(1) : It says
> "Otherwise, the
> > > > > broker
> > > > > > > > > moves
> > > > > > > > > > > into
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > > > > state.". It seems this should be RUNNING?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 105. RUNNING: Should we require the
> broker to
> > > > > catch
> > > > > > > up
> > > > > > > > > to the
> > > > > > > > > > > > > > > metadata log
> > > > > > > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For 104 and 105, these sections have been
> > > reworked.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin
> McCabe
> > > <
> > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom
> Bentley
> > > > > wrote:
> > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron
> > > Dagostino
> > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the hard
> work
> > > on
> > > > > this
> > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I have some questions about what
> > > happens
> > > > > to a
> > > > > > > > > broker
> > > > > > > > > > > > > when it
> > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > fenced (e.g. because it can't
> send a
> > > > > > > heartbeat
> > > > > > > > > > > request to
> > > > > > > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > > > > > > lease). The KIP says "When a
> broker
> > > is
> > > > > > > fenced,
> > > > > > > > > it
> > > > > > > > > > > cannot
> > > > > > > > > > > > > > > process any
> > > > > > > > > > > > > > > > > > > > > > client requests. This prevents
> > > brokers
> > > > > which
> > > > > > > > > are not
> > > > > > > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > > > > > > metadata updates or that are not
> > > > > receiving
> > > > > > > and
> > > > > > > > > > > processing
> > > > > > > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > > > > > > enough from causing issues to
> > > clients."
> > > > > And
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > > > > > > FENCED(4) state it likewise says
> > > "While
> > > > > in
> > > > > > > this
> > > > > > > > > > > state,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > > not respond to client
> requests." It
> > > > > makes
> > > > > > > sense
> > > > > > > > > > > that a
> > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > > should not accept producer
> requests
> > > -- I
> > > > > > > assume
> > > > > > > > > any
> > > > > > > > > > > such
> > > > > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > > > > > would result in
> > > > > NotLeaderOrFollowerException.
> > > > > > > > > But
> > > > > > > > > > > what
> > > > > > > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > > > > > > (fetch from follower) consumer
> > > > > requests? It
> > > > > > > is
> > > > > > > > > > > > > conceivable
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > could continue. Related to that,
> > > would a
> > > > > > > fenced
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > > fetch data for partitions where
> it
> > > > > thinks it
> > > > > > > is a
> > > > > > > > > > > > > follower?
> > > > > > > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > > > > > > it rejects consumer requests it
> might
> > > > > still
> > > > > > > > > continue
> > > > > > > > > > > to
> > > > > > > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > > > > > > follower. Might it be helpful to
> > > clarify
> > > > > > > both
> > > > > > > > > > > decisions
> > > > > > > > > > > > > > > here?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Good question. I think a fenced
> broker
> > > > > should
> > > > > > > > > > > continue to
> > > > > > > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > > > > > > partitions it was already fetching
> > > before
> > > > > it
> > > > > > > was
> > > > > > > > > > > fenced,
> > > > > > > > > > > > > > > unless it
> > > > > > > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > > > > > > problem. At that point it won't be
> > > able to
> > > > > > > > > continue,
> > > > > > > > > > > > > since it
> > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > the new metadata. For example, it
> > > won't
> > > > > know
> > > > > > > about
> > > > > > > > > > > > > leadership
> > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the partitions it's fetching. The
> > > > > rationale
> > > > > > > for
> > > > > > > > > > > > > continuing to
> > > > > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > > > try to avoid disruptions as much as
> > > > > possible.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I don't think fenced brokers should
> > > accept
> > > > > > > client
> > > > > > > > > > > requests.
> > > > > > > > > > > > > > > The issue
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > that the fenced broker may or may
> not
> > > have
> > > > > any
> > > > > > > > > data it
> > > > > > > > > > > is
> > > > > > > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > > > > > > have. It may or may not have
> applied
> > > any
> > > > > > > > > configuration
> > > > > > > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > it is supposed to have applied.
> So it
> > > > > could
> > > > > > > get
> > > > > > > > > pretty
> > > > > > > > > > > > > > > confusing, and
> > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > potentially waste the client's
> time.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > When fenced, how would the broker
> reply
> > > to a
> > > > > > > client
> > > > > > > > > > > which did
> > > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The broker will respond with a
> retryable
> > > error
> > > > > in
> > > > > > > that
> > > > > > > > > > > case.
> > > > > > > > > > > > > Once
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > client has re-fetched its metadata, it
> > > will no
> > > > > > > longer
> > > > > > > > > see
> > > > > > > > > > > the
> > > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > as part of the cluster. I added a
> note to
> > > the
> > > > > KIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Wed, Dec 16, 2020, at 13:40, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the reply. A few follow up comments.
>
> 211. When does the broker send the BrokerRegistration request to the
> controller? Is it after the recovery phase? If so, at that point, the
> broker has already caught up on the metadata (in order to clean up deleted
> partitions). Then, it seems we don't need the ShouldFence field
> in BrokerHeartbeatRequest?
Hi Jun,
Thanks again for the reviews.
The broker sends the registration request as soon as it starts up. It cannot wait until the recovery phase is over since sometimes log recovery can take quite a long time.
>
> 213. kafka-cluster.sh
> 213.1 For the decommision example, should the command take a broker id?
Yes, the example should have a broker id. Fixed.
> 213.2 Existing tools use the "--" command line option (e.g. kafka-topics
> --list --topic test). Should we follow the same convention
> for kafka-cluster.sh (and kafka-storage.sh)?
Hmm. I don't think argparse4j supports using double dashes to identify subcommands. I think it might be confusing as well, since the subcommand must come first, unlike a plain old argument which can be anywhere on the command line.
> 213.3 Should we add an admin api for broker decommission so that this can
> be done programmatically?
>
Yes, there is an admin client API for decommissioning as well.
> 220. DecommissionBroker: "NOT_CONTROLLER if the node that the request was
> sent to is not the controller". I thought the clients never send a request
> to the controller directly now and the broker will forward it to the
> controller?
>
If the controller moved recently, it's possible that the broker could send to a controller that has just recently become inactive. In that case NOT_CONTROLLER would be returned. (A standby controller returns NOT_CONTROLLER for most APIs).
> 221. Could we add the required ACL for the new requests?
>
Good point. I added the required ACL for each new RPC.
best,
Colin
> Jun
>
> On Wed, Dec 16, 2020 at 11:38 AM Colin McCabe <cm...@apache.org> wrote:
>
> > On Wed, Dec 16, 2020, at 09:59, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply. A few more comments below.
> > >
> >
> > Hi Jun,
> >
> > Thanks for the comments.
> >
> > > 206. "RemoveTopic is the last step, that scrubs all metadata about the
> > > topic. In order to get to that last step, the topic data needs to
> > removed
> > > from all brokers (after each broker notices that the topic is being
> > > deleted)." Currently, this is done based on the response of
> > > StopReplicaRequest. Since the controller no longer sends this request,
> > how
> > > does the controller know that the data for the deleted topic has
> > > been removed in the brokers?
> > >
> >
> > That's a good point... definitely an oversight.
> >
> > It seems complex to force the controller to track when log directories
> > have been deleted. Let's just assume that KIP-516 has been implemented,
> > and track them by UUID. Then we can just have a single topic deletion
> > record.
> >
> > I added a section on "topic identifiers" describing this.
> >
> > > 210. Thanks for the explanation. Sounds good to me.
> > >
> > > 211. I still don't see an example when ShouldFence is set to true in
> > > BrokerHeartbeatReques. Could we add one?
> > >
> >
> > It is sent to true when the broker is first starting up and doesn't yet
> > want to be unfenced. I added a longer explanation of this in the "Broker
> > Leases" section.
> >
> > > 213. The KIP now allows replicas to be assigned on brokers that are
> > fenced,
> > > which is an improvement. How do we permanently remove a broker (e.g.
> > > cluster shrinking) to prevent it from being used for future replica
> > > assignments?
> > >
> >
> > This is a fair point. I will create a kafka-cluster.sh script which can
> > do this, plus a DecommissionBrokerRequest.
> >
> > As a bonus the kafka-cluster.sh script can help people find the cluster ID
> > of brokers, something that people have wanted a tool for in the past.
> >
> > > 214. "Currently, when a node is down, all of its ZK registration
> > > information is gone. But we need this information in order to
> > understand
> > > things like whether the replicas of a particular partition are
> > > well-balanced across racks." This is not quite true. Currently, even when
> > > ZK registration is gone, the existing replica assignment is still
> > available
> > > in the metadata response.
> > >
> >
> > I agree that the existing replica assignment is still available. But that
> > just tells you partition X is on nodes A, B, and C. If you don't have the
> > ZK registration for one or more of A, B, or C then you don't know whether
> > we are following the policy of "two replicas on one rack, one replica on
> > another." Or any other more complex rack placement policy that you might
> > have.
> >
> > best,
> > Colin
> >
> > > Jun
> > >
> > > On Wed, Dec 16, 2020 at 8:48 AM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the reply. A few more follow up comments.
> > > > >
> > > > > 210. initial.broker.registration.timeout.ms: The default value is
> > 90sec,
> > > > > which seems long. If a broker fails the registration because of
> > incorrect
> > > > > configs, we want to fail faster. In comparison, the defaults for
> > > > > zookeeper.connection.timeout.ms is 18 secs.
> > > > >
> > > >
> > > > Hi Jun,
> > > >
> > > > I agree that the initial connection timeout here is longer than what we
> > > > had with ZK. The main reason I selected a slightly longer timeout
> > here is
> > > > to handle the case where the controllers and the brokers are
> > co-located.
> > > > For example, if you have a 3 node cluster and all three nodes are
> > > > controllers+brokers, when you first bring up the cluster, we will have
> > to
> > > > stand up the controller quorum and then handle broker registrations.
> > > > Although we believe Raft will start up relatively quickly, it's good to
> > > > leave some extra margin here.
> > > >
> > > > I don't think there's a big disadvantage to having a slightly longer
> > > > timeout here. After all, starting up the brokers with no controllers
> > is an
> > > > admin mistake, which we don't expect to see very often. Maybe let's
> > set it
> > > > to 60 seconds for now and maybe see if we can tweak it in the future if
> > > > that turns out to be too long or short?
> > > >
> > > > > 211. "Once they are all moved, the controller responds to the
> > heartbeat
> > > > > with a nextState of SHUTDOWN." It seems that nextState is no longer
> > > > > relevant. Also, could you add a bit more explanation on when
> > ShouldFence
> > > > is
> > > > > set to true in BrokerHeartbeatRequest?
> > > > >
> > > >
> > > > Good catch. I removed the obsolete section referring to nextState and
> > > > added a reference to the new boolean. I also added some more
> > information
> > > > about ShouldFence and about the rationale for separating fencing from
> > > > registration.
> > > >
> > > > > 212. Related to the above, what does the broker do if IsFenced is
> > true in
> > > > > the BrokerHeartbeatResponse? Will the broker do the same thing on
> > > > receiving
> > > > > a FenceBrokerRecord from the metadata log?
> > > > >
> > > >
> > > > The broker only checks this boolean during the startup process. After
> > the
> > > > startup process is finished, it ignores this boolean.
> > > >
> > > > The broker uses fence / unfence records in the metadata log to
> > determine
> > > > which brokers should appear in its MetadataResponse.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > > Jun
> > > > >
> > > > > On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > > > > > Hi Colin,
> > > > > > >
> > > > > > > The KIP says that "brokers which are fenced will not appear in
> > > > > > > MetadataResponses. So clients that have up-to-date metadata will
> > > > not try
> > > > > > > to contact fenced brokers.", which is fine, but it doesn't seem
> > to
> > > > > > > elaborate on what happens for clients which try to contact a
> > broker
> > > > > > (using
> > > > > > > stale metadata, for example) and find it in the
> > registered-but-fenced
> > > > > > > state.
> > > > > >
> > > > > > Hi Tom,
> > > > > >
> > > > > > I have dropped the broker-side fencing from this proposal. So now
> > the
> > > > > > fencing is basically the same as today's fencing: it's
> > controller-side
> > > > > > only. That means that clients using stale metadata can contact
> > fenced
> > > > > > brokers and communicate with them. The only case where the broker
> > > > will be
> > > > > > inaccessible is when it hasn't finished starting yet (i.e., has
> > not yet
> > > > > > attained RUNNING state.)
> > > > > >
> > > > > > Just like today, we expect the safeguards built into the
> > replication
> > > > > > protocol to prevent the worst corner cases that could result from
> > > > this. I
> > > > > > do think we will probably take up this issue later and find a
> > better
> > > > > > solution for client-side metadata, but this KIP is big enough
> > as-is.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > > You said previously that the broker will respond with a retriable
> > > > > > > error, which is again fine, but you didn't answer my question
> > about
> > > > which
> > > > > > > error code you've chosen for this. I realise that this doesn't
> > really
> > > > > > > affect the correctness of the behaviour, but I'm not aware of any
> > > > > > existing
> > > > > > > error code which is a good fit. So it would be good to understand
> > > > about
> > > > > > how
> > > > > > > you're making this backward compatible for clients.
> > > > > > >
> > > > > > > Many thanks,
> > > > > > >
> > > > > > > Tom
> > > > > > >
> > > > > > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <cmccabe@apache.org
> > >
> > > > wrote:
> > > > > > >
> > > > > > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > > > > > Hi, Colin,
> > > > > > > > >
> > > > > > > > > Thanks for the reply. Just a couple of more comments below.
> > > > > > > > >
> > > > > > > > > 210. Since we are deprecating
> > zookeeper.connection.timeout.ms,
> > > > > > should we
> > > > > > > > > add a new config to bound the time for a broker to connect
> > to the
> > > > > > > > > controller during starting up?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Good idea. I added initial.broker.registration.timeout.ms for
> > > > this.
> > > > > > > >
> > > > > > > > > 211. BrokerHeartbeat no longer has the state field in the
> > > > > > > > request/response.
> > > > > > > > > However, (a) the controller shutdown section still has "In
> > its
> > > > > > periodic
> > > > > > > > > heartbeats, the broker asks the controller if it can
> > transition
> > > > into
> > > > > > the
> > > > > > > > > SHUTDOWN state. This motivates the controller to move all
> > of the
> > > > > > leaders
> > > > > > > > > off of that broker. Once they are all moved, the controller
> > > > > > responds to
> > > > > > > > > the heartbeat with a nextState of SHUTDOWN."; (2) the
> > > > description of
> > > > > > > > > BrokerHeartbeat still references currentState and
> > targetState.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Thanks. I've made these sections clearer and removed the
> > obsolete
> > > > > > > > references to sending states.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <
> > cmccabe@apache.org
> > > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > > > > > > > > Hi, Colin,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the update. A few more follow up comments.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi Jun,
> > > > > > > > > >
> > > > > > > > > > Thanks again for the review.
> > > > > > > > > >
> > > > > > > > > > > 100. FailedReplicaRecord: Since this is reported by each
> > > > broker
> > > > > > > > > > > independently, perhaps we could use a more concise
> > > > representation
> > > > > > > > that
> > > > > > > > > > has
> > > > > > > > > > > a top level broker field, an array of topics, which has
> > an
> > > > array
> > > > > > of
> > > > > > > > > > > partitions.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The issue is that there is a size limit on the each record.
> > > > > > Putting
> > > > > > > > all
> > > > > > > > > > of the partitions of a log directory into a single record
> > would
> > > > > > > > probably
> > > > > > > > > > break that in many cases. Still, we can optimize a bit by
> > > > having
> > > > > > an
> > > > > > > > array
> > > > > > > > > > of partition IDs, since nearly all the time, we have more
> > than
> > > > one
> > > > > > > > from the
> > > > > > > > > > same topic.
> > > > > > > > > >
> > > > > > > > > > > 200. Sounds good. If we remove the broker-side fencing
> > > > logic, do
> > > > > > we
> > > > > > > > plan
> > > > > > > > > > to
> > > > > > > > > > > still keep FENCED in broker state? Do we plan to expose
> > the
> > > > new
> > > > > > > > states
> > > > > > > > > > > through the existing BrokerState metric and if so, what
> > are
> > > > the
> > > > > > > > values
> > > > > > > > > > for
> > > > > > > > > > > the new states?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > No, we don't need FENCED any more. I have removed it from
> > the
> > > > KIP.
> > > > > > > > > >
> > > > > > > > > > The new states are very similar to the current ones,
> > actually.
> > > > > > There
> > > > > > > > are
> > > > > > > > > > no new states or removed ones. The main change in the
> > broker
> > > > state
> > > > > > > > machine
> > > > > > > > > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state has been
> > > > > > renamed to
> > > > > > > > > > RECOVERY. Also, unlike previously, the broker will always
> > pass
> > > > > > through
> > > > > > > > > > RECOVERY (although it may only stay in this state for a
> > very
> > > > short
> > > > > > > > amount
> > > > > > > > > > of time).
> > > > > > > > > >
> > > > > > > > > > > 201. This may be fine too. Could we document what happens
> > > > when
> > > > > > the
> > > > > > > > > > > broker.id/controller.id in metadata.properties don't
> > match
> > > > the
> > > > > > > > broker
> > > > > > > > > > > config when the broker starts up?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I added some documentation about this.
> > > > > > > > > >
> > > > > > > > > > > 204. There is still "The highest metadata offset which
> > the
> > > > broker
> > > > > > > > has not
> > > > > > > > > > > reached" referenced under BrokerRegistration.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > > > > > >
> > > > > > > > > > > 206. Is that separate step needed given KIP-516? With
> > > > KIP-516 (
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > > > > > ),
> > > > > > > > > > > we don't need to wait for the topic data to be removed
> > from
> > > > all
> > > > > > > > brokers
> > > > > > > > > > > before removing the topic metadata. The combination of
> > > > unmatching
> > > > > > > > > > > topicId
> > > > > > > > > > > or the missing topicId from the metadata is enough for
> > the
> > > > > > broker to
> > > > > > > > > > > clean
> > > > > > > > > > > up deleted topics asynchronously.
> > > > > > > > > >
> > > > > > > > > > It won't be needed once KIP-516 is adopted, but this hasn't
> > > > been
> > > > > > > > > > implemented yet.
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <
> > > > cmccabe@apache.org>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the updated KIP. A few more comments
> > below.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks again for the reviews.
> > > > > > > > > > > >
> > > > > > > > > > > > > 80.2 For deprecated configs, we need to include
> > > > zookeeper.*
> > > > > > and
> > > > > > > > > > > > > broker.id.generation.enable.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Added.
> > > > > > > > > > > >
> > > > > > > > > > > > > 83.1 If a broker is down, does the controller keep
> > the
> > > > > > previously
> > > > > > > > > > > > > registered broker epoch forever? If not, how long
> > does
> > > > the
> > > > > > > > controller
> > > > > > > > > > > > keep
> > > > > > > > > > > > > it? What does the controller do when receiving a
> > broker
> > > > > > heartbeat
> > > > > > > > > > request
> > > > > > > > > > > > > with an unfound broker epoch?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, the controller keeps the previous registration
> > > > forever.
> > > > > > > > > > > >
> > > > > > > > > > > > Broker heartbeat requests with an incorrect broker
> > epoch
> > > > will
> > > > > > be
> > > > > > > > > > rejected
> > > > > > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > > > > > >
> > > > > > > > > > > > > 100. Have you figured out if we need to add a new
> > record
> > > > > > type for
> > > > > > > > > > > > reporting
> > > > > > > > > > > > > partitions on failed disks?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I added FailedReplicaRecord to reflect the case where a
> > > > JBOD
> > > > > > > > directory
> > > > > > > > > > has
> > > > > > > > > > > > failed, leading to failed replicas.
> > > > > > > > > > > >
> > > > > > > > > > > > > 102. For debugging purposes, sometimes it's useful to
> > > > read
> > > > > > the
> > > > > > > > > > metadata
> > > > > > > > > > > > > topic using tools like console-consumer. Should we
> > > > support
> > > > > > that
> > > > > > > > and
> > > > > > > > > > if
> > > > > > > > > > > > so,
> > > > > > > > > > > > > how?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > For now, we have the ability to read the metadata logs
> > > > with the
> > > > > > > > > > dump-logs
> > > > > > > > > > > > tool. I think we will come up with some other tools
> > in the
> > > > > > future
> > > > > > > > as
> > > > > > > > > > we
> > > > > > > > > > > > get experience.
> > > > > > > > > > > >
> > > > > > > > > > > > > 200. "brokers which are fenced will not appear in
> > > > > > > > MetadataResponses.
> > > > > > > > > > The
> > > > > > > > > > > > > broker will not respond to these requests-- instead,
> > it
> > > > will
> > > > > > > > simply
> > > > > > > > > > > > > disconnect." If the controller is partitioned off
> > from
> > > > the
> > > > > > > > brokers,
> > > > > > > > > > this
> > > > > > > > > > > > > design will cause every broker to stop accepting new
> > > > client
> > > > > > > > > > requests. In
> > > > > > > > > > > > > contrast, if ZK is partitioned off, the existing
> > > > behavior is
> > > > > > > > that the
> > > > > > > > > > > > > brokers can continue to work based on the last known
> > > > > > metadata.
> > > > > > > > So, I
> > > > > > > > > > am
> > > > > > > > > > > > not
> > > > > > > > > > > > > sure if we should change the existing behavior
> > because
> > > > of the
> > > > > > > > bigger
> > > > > > > > > > > > impact
> > > > > > > > > > > > > in the new one. Another option is to keep the
> > existing
> > > > > > behavior
> > > > > > > > and
> > > > > > > > > > > > expose
> > > > > > > > > > > > > a metric for fenced brokers so that the operator
> > could be
> > > > > > > > alerted.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I'm skeptical about how well running without ZK
> > currently
> > > > > > works.
> > > > > > > > > > However,
> > > > > > > > > > > > I will move the broker-side fencing into a follow-up
> > KIP.
> > > > This
> > > > > > > > KIP is
> > > > > > > > > > > > already pretty large and there is no hard dependency on
> > > > this.
> > > > > > > > There
> > > > > > > > > > may
> > > > > > > > > > > > also be other ways of accomplishing the positive
> > effects of
> > > > > > what
> > > > > > > > > > > > broker-side fencing, so more discussion is needed.
> > > > > > > > > > > >
> > > > > > > > > > > > > 201. I read Ron's comment, but I am still not sure
> > the
> > > > > > benefit of
> > > > > > > > > > keeping
> > > > > > > > > > > > > broker.id and controller.id in meta.properties. It
> > seems
> > > > > > that
> > > > > > > > we are
> > > > > > > > > > > > just
> > > > > > > > > > > > > duplicating the same info in two places and have the
> > > > > > additional
> > > > > > > > > > burden of
> > > > > > > > > > > > > making sure the values in the two places are
> > consistent.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I think the reasoning is that having broker.id
> > protects us
> > > > > > against
> > > > > > > > > > > > accidentally bringing up a broker with a disk from a
> > > > different
> > > > > > > > > > broker. I
> > > > > > > > > > > > don't feel strongly about this but it seemed simpler to
> > > > keep
> > > > > > it.
> > > > > > > > > > > >
> > > > > > > > > > > > > 202. controller.connect.security.protocol: Is this
> > needed
> > > > > > since
> > > > > > > > > > > > > controller.listener.names and
> > > > listener.security.protocol.map
> > > > > > > > imply
> > > > > > > > > > the
> > > > > > > > > > > > > security protocol already?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > You're right, this isn't needed. I'll remove it.
> > > > > > > > > > > >
> > > > > > > > > > > > > 203. registration.heartbeat.interval.ms: It
> > defaults to
> > > > 2k.
> > > > > > ZK
> > > > > > > > uses
> > > > > > > > > > 1/3
> > > > > > > > > > > > of
> > > > > > > > > > > > > the session timeout for heartbeat. So, given the
> > default
> > > > 18k
> > > > > > for
> > > > > > > > > > > > > registration.lease.timeout.ms, should we default
> > > > > > > > > > > > > registration.heartbeat.interval.ms to 6k?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 6 seconds seems like a pretty long time between
> > > > heartbeats. It
> > > > > > > > might
> > > > > > > > > > be
> > > > > > > > > > > > useful to know when a broker is missing heartbeats,
> > with
> > > > less
> > > > > > time
> > > > > > > > than
> > > > > > > > > > > > that. I provisionally set it to 3 seconds (we can
> > always
> > > > > > change
> > > > > > > > > > later...)
> > > > > > > > > > > >
> > > > > > > > > > > > I also changed the name of these configurations to "
> > > > > > > > > > > > broker.heartbeat.interval.ms" and "
> > > > > > broker.registration.timeout.ms"
> > > > > > > > to
> > > > > > > > > > try
> > > > > > > > > > > > to clarify them a bit.
> > > > > > > > > > > >
> > > > > > > > > > > > > 204. "The highest metadata offset which the broker
> > has
> > > > not
> > > > > > > > reached."
> > > > > > > > > > It
> > > > > > > > > > > > > seems this should be "has reached".
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I changed this to "one more than the highest metadata
> > > > offset
> > > > > > which
> > > > > > > > the
> > > > > > > > > > > > broker has reached."
> > > > > > > > > > > >
> > > > > > > > > > > > > 205. UnfenceBrokerRecord and UnregisterBrokerRecord:
> > To
> > > > me,
> > > > > > they
> > > > > > > > > > seem to
> > > > > > > > > > > > be
> > > > > > > > > > > > > the same. Do we need both?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Unregistration means that the broker has been removed
> > from
> > > > the
> > > > > > > > cluster.
> > > > > > > > > > > > That is different than unfencing, which marks the
> > broker as
> > > > > > active.
> > > > > > > > > > > >
> > > > > > > > > > > > > 206. TopicRecord: The Deleting field is used to
> > indicate
> > > > > > that the
> > > > > > > > > > topic
> > > > > > > > > > > > is
> > > > > > > > > > > > > being deleted. I am wondering if this is really
> > needed
> > > > since
> > > > > > > > > > RemoveTopic
> > > > > > > > > > > > > already indicates the same thing.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > RemoveTopic is the last step, that scrubs all metadata
> > > > about
> > > > > > the
> > > > > > > > topic.
> > > > > > > > > > > > In order to get to that last step, the topic data
> > needs to
> > > > > > removed
> > > > > > > > > > from all
> > > > > > > > > > > > brokers (after each broker notices that the topic is
> > being
> > > > > > > > deleted).
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > > > Jun
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <
> > > > > > cmccabe@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > > > > > > > > > > > > > > Hi Colin. Thanks for the updates. It's now
> > clear
> > > > to me
> > > > > > that
> > > > > > > > > > brokers
> > > > > > > > > > > > > > > keep their broker epoch for the life of their
> > JVM --
> > > > they
> > > > > > > > > > register
> > > > > > > > > > > > > > > once, get their broker epoch in the response, and
> > > > then
> > > > > > never
> > > > > > > > > > > > > > > re-register again. Brokers may get fenced, but
> > they
> > > > > > keep the
> > > > > > > > > > same
> > > > > > > > > > > > > > > broker epoch for the life of their JVM. The
> > > > incarnation
> > > > > > ID
> > > > > > > > is
> > > > > > > > > > also
> > > > > > > > > > > > > > > kept for the life of the JVM but is generated by
> > the
> > > > > > broker
> > > > > > > > > > itself
> > > > > > > > > > > > > > > upon startup, and the combination of the two
> > allows
> > > > the
> > > > > > > > > > Controller to
> > > > > > > > > > > > > > > act idempotently if any previously-sent
> > registration
> > > > > > response
> > > > > > > > > > gets
> > > > > > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks, Ron. That's a good summary.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > One thing I wonder about is if it might be
> > helpful
> > > > for
> > > > > > the
> > > > > > > > > > broker to
> > > > > > > > > > > > > > > send the Cluster ID as determined from its
> > > > > > meta.properties
> > > > > > > > file
> > > > > > > > > > in
> > > > > > > > > > > > its
> > > > > > > > > > > > > > > registration request. Does it even make sense
> > for
> > > > the
> > > > > > > > broker to
> > > > > > > > > > > > > > > successfully register and enter the Fenced state
> > if
> > > > it
> > > > > > has
> > > > > > > > the
> > > > > > > > > > wrong
> > > > > > > > > > > > > > > Cluster ID?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yeah, that's a good idea. Let's have the broker
> > pass
> > > > its
> > > > > > > > cluster
> > > > > > > > > > ID in
> > > > > > > > > > > > > > the registration RPC, and then registration can
> > fail
> > > > if the
> > > > > > > > broker
> > > > > > > > > > is
> > > > > > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The nextMetadatOffset value that the broker
> > > > communicates
> > > > > > > > > > > > > > > in its registration request only has meaning
> > within
> > > > the
> > > > > > > > correct
> > > > > > > > > > > > > > > cluster, so it feels to me that the Controller
> > should
> > > > > > have
> > > > > > > > some
> > > > > > > > > > way
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > perform this sanity check. There is currently
> > > > (pre-KIP
> > > > > > 500)
> > > > > > > > a
> > > > > > > > > > check
> > > > > > > > > > > > > > > in the broker to make sure its configured
> > cluster ID
> > > > > > matches
> > > > > > > > the
> > > > > > > > > > one
> > > > > > > > > > > > > > > stored in ZooKeeper, and we will have to perform
> > this
> > > > > > > > validation
> > > > > > > > > > > > > > > somewhere in the KIP-500 world. If the
> > Controller
> > > > > > doesn't
> > > > > > > > do it
> > > > > > > > > > > > > > > within the registration request then the broker
> > will
> > > > > > have to
> > > > > > > > > > make a
> > > > > > > > > > > > > > > metadata request to the Controller, retrieve the
> > > > Cluster
> > > > > > ID,
> > > > > > > > and
> > > > > > > > > > > > > > > perform the check itself. It feels to me that it
> > > > might
> > > > > > be
> > > > > > > > > > better for
> > > > > > > > > > > > > > > the Controller to just do it, and then the broker
> > > > doesn't
> > > > > > > > have to
> > > > > > > > > > > > > > > worry about it anymore once it successfully
> > > > registers.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I also have a question about the broker.id
> > value and
> > > > > > > > > > > > meta.properties.
> > > > > > > > > > > > > > > The KIP now says "In version 0 of
> > meta.properties,
> > > > there
> > > > > > is a
> > > > > > > > > > > > > > > broker.id field. Version 1 does not have this
> > > > field.
> > > > > > It
> > > > > > > > is no
> > > > > > > > > > > > longer
> > > > > > > > > > > > > > > needed because we no longer support dynamic
> > broker id
> > > > > > > > > > assignment."
> > > > > > > > > > > > > > > But then there is an example version 1
> > > > meta.properties
> > > > > > file
> > > > > > > > that
> > > > > > > > > > > > shows
> > > > > > > > > > > > > > > the broker.id value. I actually wonder if
> > maybe the
> > > > > > > > broker.id
> > > > > > > > > > value
> > > > > > > > > > > > > > > would be good to keep in the version 1
> > > > meta.properties
> > > > > > file
> > > > > > > > > > because
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > currently (pre-KIP 500, version 0) acts as a
> > sanity
> > > > > > check to
> > > > > > > > make
> > > > > > > > > > > > sure
> > > > > > > > > > > > > > > the broker is using the correct log directory.
> > > > Similarly
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > > > controller.id value on controllers -- it would
> > > > allow the
> > > > > > > > same
> > > > > > > > > > type
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > sanity check for quorum controllers.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > That's a good point. I will add broker.id back,
> > and
> > > > also
> > > > > > add
> > > > > > > > > > > > > > controller.id as a possibility.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <
> > > > > > > > cmccabe@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for the reply. A few more comments.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks again for the reply. Sorry for the long
> > > > > > hiatus. I
> > > > > > > > was
> > > > > > > > > > on
> > > > > > > > > > > > > > vacation for a while.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 55. There is still text that favors new
> > broker
> > > > > > > > registration.
> > > > > > > > > > > > "When a
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > first starts up, when it is in the INITIAL
> > > > state, it
> > > > > > will
> > > > > > > > > > always
> > > > > > > > > > > > > > "win"
> > > > > > > > > > > > > > > > > broker ID conflicts. However, once it is
> > > > granted a
> > > > > > > > lease, it
> > > > > > > > > > > > > > transitions
> > > > > > > > > > > > > > > > > out of the INITIAL state. Thereafter, it may
> > > > lose
> > > > > > > > subsequent
> > > > > > > > > > > > > > conflicts if
> > > > > > > > > > > > > > > > > its broker epoch is stale. (See KIP-380 for
> > some
> > > > > > > > background
> > > > > > > > > > on
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > epoch.) The reason for favoring new
> > processes
> > > > is to
> > > > > > > > > > accommodate
> > > > > > > > > > > > the
> > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > case where a process is killed with kill -9
> > and
> > > > then
> > > > > > > > > > restarted.
> > > > > > > > > > > > We
> > > > > > > > > > > > > > want it
> > > > > > > > > > > > > > > > > to be able to reclaim its old ID quickly in
> > this
> > > > > > case."
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the reminder. I have clarified the
> > > > language
> > > > > > > > here.
> > > > > > > > > > > > > > Hopefully now it is clear that we don't allow quick
> > > > re-use
> > > > > > of
> > > > > > > > > > broker
> > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 80.1 Sounds good. Could you document that
> > > > listeners
> > > > > > is a
> > > > > > > > > > required
> > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > now? It would also be useful to annotate
> > other
> > > > > > required
> > > > > > > > > > configs.
> > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > example, controller.connect should be
> > required.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I added a note specifying that these are
> > required.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 80.2 Could you list all deprecated existing
> > > > configs?
> > > > > > > > Another
> > > > > > > > > > one
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > control.plane.listener.name since the
> > > > controller no
> > > > > > > > longer
> > > > > > > > > > sends
> > > > > > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata and StopReplica
> > > > > > requests.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I added a section specifying some deprecated
> > > > configs.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 83.1 It seems that the broker can transition
> > from
> > > > > > FENCED
> > > > > > > > to
> > > > > > > > > > > > RUNNING
> > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > registering for a new broker epoch. I am not
> > > > sure how
> > > > > > > > this
> > > > > > > > > > works.
> > > > > > > > > > > > > > Once the
> > > > > > > > > > > > > > > > > controller fences a broker, there is no need
> > for
> > > > the
> > > > > > > > > > controller
> > > > > > > > > > > > to
> > > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > > boker epoch around. So, if the fenced
> > broker's
> > > > > > heartbeat
> > > > > > > > > > request
> > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > existing broker epoch will be rejected,
> > leading
> > > > the
> > > > > > > > broker
> > > > > > > > > > back
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The broker epoch refers to the broker
> > registration.
> > > > > > So we
> > > > > > > > DO
> > > > > > > > > > keep
> > > > > > > > > > > > the
> > > > > > > > > > > > > > broker epoch around even while the broker is
> > fenced.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The broker epoch changes only when there is a
> > new
> > > > > > broker
> > > > > > > > > > > > > > registration. Fencing or unfencing the broker
> > doesn't
> > > > > > change
> > > > > > > > the
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 83.5 Good point on KIP-590. Then should we
> > > > expose the
> > > > > > > > > > controller
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > debugging purposes? If not, we should
> > deprecate
> > > > the
> > > > > > > > > > controllerID
> > > > > > > > > > > > > > field in
> > > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think it's OK to expose it for now, with the
> > > > proviso
> > > > > > > > that it
> > > > > > > > > > > > won't
> > > > > > > > > > > > > > be reachable by clients.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 90. We rejected the shared ID with just one
> > > > reason
> > > > > > "This
> > > > > > > > is
> > > > > > > > > > not a
> > > > > > > > > > > > > > good idea
> > > > > > > > > > > > > > > > > because NetworkClient assumes a single ID
> > > > space. So
> > > > > > if
> > > > > > > > > > there is
> > > > > > > > > > > > > > both a
> > > > > > > > > > > > > > > > > controller 1 and a broker 1, we don't have a
> > way
> > > > of
> > > > > > > > picking
> > > > > > > > > > the
> > > > > > > > > > > > > > "right"
> > > > > > > > > > > > > > > > > one." This doesn't seem to be a strong
> > reason.
> > > > For
> > > > > > > > example,
> > > > > > > > > > we
> > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > address the NetworkClient issue with the node
> > > > type
> > > > > > as you
> > > > > > > > > > pointed
> > > > > > > > > > > > > > out or
> > > > > > > > > > > > > > > > > using the negative value of a broker ID as
> > the
> > > > > > > > controller ID.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It would require a lot of code changes to
> > support
> > > > > > multiple
> > > > > > > > > > types of
> > > > > > > > > > > > > > node IDs. It's not clear to me that the end result
> > > > would
> > > > > > be
> > > > > > > > > > better --
> > > > > > > > > > > > I
> > > > > > > > > > > > > > tend to think it would be worse, since it would be
> > more
> > > > > > > > complex.
> > > > > > > > > > In a
> > > > > > > > > > > > > > similar vein, using negative numbers seems
> > dangerous,
> > > > > > since we
> > > > > > > > use
> > > > > > > > > > > > > > negatives or -1 as "special values" in many places.
> > > > For
> > > > > > > > example,
> > > > > > > > > > -1
> > > > > > > > > > > > often
> > > > > > > > > > > > > > represents "no such node."
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > One important thing to keep in mind is that we
> > > > want to
> > > > > > be
> > > > > > > > able
> > > > > > > > > > to
> > > > > > > > > > > > > > transition from a broker and a controller being
> > > > co-located
> > > > > > to
> > > > > > > > them
> > > > > > > > > > no
> > > > > > > > > > > > > > longer being co-located. This is much easier to do
> > > > when
> > > > > > they
> > > > > > > > have
> > > > > > > > > > > > separate
> > > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > the broker reports all offline replicas due
> > to a
> > > > disk
> > > > > > > > > > failure to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > controller. It seems this information needs
> > to be
> > > > > > > > persisted
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > > log. Do we have a corresponding record for
> > that?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hmm, I have to look into this a little bit
> > more.
> > > > We
> > > > > > may
> > > > > > > > need
> > > > > > > > > > a new
> > > > > > > > > > > > > > record type.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 101. Currently, StopReplica request has 2
> > modes,
> > > > > > without
> > > > > > > > > > deletion
> > > > > > > > > > > > > > and with
> > > > > > > > > > > > > > > > > deletion. The former is used for controlled
> > > > shutdown
> > > > > > and
> > > > > > > > > > handling
> > > > > > > > > > > > > > disk
> > > > > > > > > > > > > > > > > failure, and causes the follower to stop. The
> > > > latter
> > > > > > is
> > > > > > > > for
> > > > > > > > > > topic
> > > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > > and partition reassignment, and causes the
> > > > replica
> > > > > > to be
> > > > > > > > > > deleted.
> > > > > > > > > > > > > > Since we
> > > > > > > > > > > > > > > > > are deprecating StopReplica, could we
> > document
> > > > what
> > > > > > > > triggers
> > > > > > > > > > the
> > > > > > > > > > > > > > stopping
> > > > > > > > > > > > > > > > > of a follower and the deleting of a replica
> > now?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > RemoveTopic triggers deletion. In general the
> > > > > > > > functionality of
> > > > > > > > > > > > > > StopReplica is subsumed by the metadata records.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 102. Should we include the metadata topic in
> > the
> > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > If so,
> > > > > > > > > > > > > > > > > when it will be included and what will the
> > > > metadata
> > > > > > > > response
> > > > > > > > > > look
> > > > > > > > > > > > > > like?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > No, it won't be included in the metadata
> > response
> > > > sent
> > > > > > back
> > > > > > > > > > from
> > > > > > > > > > > > the
> > > > > > > > > > > > > > brokers.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 103. "The active controller assigns the
> > broker a
> > > > new
> > > > > > > > broker
> > > > > > > > > > > > epoch,
> > > > > > > > > > > > > > based on
> > > > > > > > > > > > > > > > > the latest committed offset in the log." This
> > > > seems
> > > > > > > > > > inaccurate
> > > > > > > > > > > > since
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > latest committed offset doesn't always
> > advance on
> > > > > > every
> > > > > > > > log
> > > > > > > > > > > > append.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Given that the new broker epoch won't be
> > visible
> > > > until
> > > > > > the
> > > > > > > > > > commit
> > > > > > > > > > > > has
> > > > > > > > > > > > > > happened, I have changed this to "the next
> > available
> > > > > > offset in
> > > > > > > > the
> > > > > > > > > > log"
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 104. REGISTERING(1) : It says "Otherwise, the
> > > > broker
> > > > > > > > moves
> > > > > > > > > > into
> > > > > > > > > > > > the
> > > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > > > state.". It seems this should be RUNNING?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 105. RUNNING: Should we require the broker to
> > > > catch
> > > > > > up
> > > > > > > > to the
> > > > > > > > > > > > > > metadata log
> > > > > > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > For 104 and 105, these sections have been
> > reworked.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe
> > <
> > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley
> > > > wrote:
> > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron
> > Dagostino
> > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the hard work
> > on
> > > > this
> > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I have some questions about what
> > happens
> > > > to a
> > > > > > > > broker
> > > > > > > > > > > > when it
> > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > fenced (e.g. because it can't send a
> > > > > > heartbeat
> > > > > > > > > > request to
> > > > > > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > > > > > lease). The KIP says "When a broker
> > is
> > > > > > fenced,
> > > > > > > > it
> > > > > > > > > > cannot
> > > > > > > > > > > > > > process any
> > > > > > > > > > > > > > > > > > > > > client requests. This prevents
> > brokers
> > > > which
> > > > > > > > are not
> > > > > > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > > > > > metadata updates or that are not
> > > > receiving
> > > > > > and
> > > > > > > > > > processing
> > > > > > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > > > > > enough from causing issues to
> > clients."
> > > > And
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > > > > > FENCED(4) state it likewise says
> > "While
> > > > in
> > > > > > this
> > > > > > > > > > state,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > not respond to client requests." It
> > > > makes
> > > > > > sense
> > > > > > > > > > that a
> > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > > should not accept producer requests
> > -- I
> > > > > > assume
> > > > > > > > any
> > > > > > > > > > such
> > > > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > > > > would result in
> > > > NotLeaderOrFollowerException.
> > > > > > > > But
> > > > > > > > > > what
> > > > > > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > > > > > (fetch from follower) consumer
> > > > requests? It
> > > > > > is
> > > > > > > > > > > > conceivable
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > could continue. Related to that,
> > would a
> > > > > > fenced
> > > > > > > > > > broker
> > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > fetch data for partitions where it
> > > > thinks it
> > > > > > is a
> > > > > > > > > > > > follower?
> > > > > > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > > > > > it rejects consumer requests it might
> > > > still
> > > > > > > > continue
> > > > > > > > > > to
> > > > > > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > > > > > follower. Might it be helpful to
> > clarify
> > > > > > both
> > > > > > > > > > decisions
> > > > > > > > > > > > > > here?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Good question. I think a fenced broker
> > > > should
> > > > > > > > > > continue to
> > > > > > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > > > > > partitions it was already fetching
> > before
> > > > it
> > > > > > was
> > > > > > > > > > fenced,
> > > > > > > > > > > > > > unless it
> > > > > > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > > > > > problem. At that point it won't be
> > able to
> > > > > > > > continue,
> > > > > > > > > > > > since it
> > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > the new metadata. For example, it
> > won't
> > > > know
> > > > > > about
> > > > > > > > > > > > leadership
> > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the partitions it's fetching. The
> > > > rationale
> > > > > > for
> > > > > > > > > > > > continuing to
> > > > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > > try to avoid disruptions as much as
> > > > possible.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I don't think fenced brokers should
> > accept
> > > > > > client
> > > > > > > > > > requests.
> > > > > > > > > > > > > > The issue
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > that the fenced broker may or may not
> > have
> > > > any
> > > > > > > > data it
> > > > > > > > > > is
> > > > > > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > > > > > have. It may or may not have applied
> > any
> > > > > > > > configuration
> > > > > > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > it is supposed to have applied. So it
> > > > could
> > > > > > get
> > > > > > > > pretty
> > > > > > > > > > > > > > confusing, and
> > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > potentially waste the client's time.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > When fenced, how would the broker reply
> > to a
> > > > > > client
> > > > > > > > > > which did
> > > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The broker will respond with a retryable
> > error
> > > > in
> > > > > > that
> > > > > > > > > > case.
> > > > > > > > > > > > Once
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > client has re-fetched its metadata, it
> > will no
> > > > > > longer
> > > > > > > > see
> > > > > > > > > > the
> > > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > as part of the cluster. I added a note to
> > the
> > > > KIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the reply. A few follow up comments.
211. When does the broker send the BrokerRegistration request to the
controller? Is it after the recovery phase? If so, at that point, the
broker has already caught up on the metadata (in order to clean up deleted
partitions). Then, it seems we don't need the ShouldFence field
in BrokerHeartbeatRequest?
213. kafka-cluster.sh
213.1 For the decommision example, should the command take a broker id?
213.2 Existing tools use the "--" command line option (e.g. kafka-topics
--list --topic test). Should we follow the same convention
for kafka-cluster.sh (and kafka-storage.sh)?
213.3 Should we add an admin api for broker decommission so that this can
be done programmatically?
220. DecommissionBroker: "NOT_CONTROLLER if the node that the request was
sent to is not the controller". I thought the clients never send a request
to the controller directly now and the broker will forward it to the
controller?
221. Could we add the required ACL for the new requests?
Jun
On Wed, Dec 16, 2020 at 11:38 AM Colin McCabe <cm...@apache.org> wrote:
> On Wed, Dec 16, 2020, at 09:59, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the reply. A few more comments below.
> >
>
> Hi Jun,
>
> Thanks for the comments.
>
> > 206. "RemoveTopic is the last step, that scrubs all metadata about the
> > topic. In order to get to that last step, the topic data needs to
> removed
> > from all brokers (after each broker notices that the topic is being
> > deleted)." Currently, this is done based on the response of
> > StopReplicaRequest. Since the controller no longer sends this request,
> how
> > does the controller know that the data for the deleted topic has
> > been removed in the brokers?
> >
>
> That's a good point... definitely an oversight.
>
> It seems complex to force the controller to track when log directories
> have been deleted. Let's just assume that KIP-516 has been implemented,
> and track them by UUID. Then we can just have a single topic deletion
> record.
>
> I added a section on "topic identifiers" describing this.
>
> > 210. Thanks for the explanation. Sounds good to me.
> >
> > 211. I still don't see an example when ShouldFence is set to true in
> > BrokerHeartbeatReques. Could we add one?
> >
>
> It is sent to true when the broker is first starting up and doesn't yet
> want to be unfenced. I added a longer explanation of this in the "Broker
> Leases" section.
>
> > 213. The KIP now allows replicas to be assigned on brokers that are
> fenced,
> > which is an improvement. How do we permanently remove a broker (e.g.
> > cluster shrinking) to prevent it from being used for future replica
> > assignments?
> >
>
> This is a fair point. I will create a kafka-cluster.sh script which can
> do this, plus a DecommissionBrokerRequest.
>
> As a bonus the kafka-cluster.sh script can help people find the cluster ID
> of brokers, something that people have wanted a tool for in the past.
>
> > 214. "Currently, when a node is down, all of its ZK registration
> > information is gone. But we need this information in order to
> understand
> > things like whether the replicas of a particular partition are
> > well-balanced across racks." This is not quite true. Currently, even when
> > ZK registration is gone, the existing replica assignment is still
> available
> > in the metadata response.
> >
>
> I agree that the existing replica assignment is still available. But that
> just tells you partition X is on nodes A, B, and C. If you don't have the
> ZK registration for one or more of A, B, or C then you don't know whether
> we are following the policy of "two replicas on one rack, one replica on
> another." Or any other more complex rack placement policy that you might
> have.
>
> best,
> Colin
>
> > Jun
> >
> > On Wed, Dec 16, 2020 at 8:48 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply. A few more follow up comments.
> > > >
> > > > 210. initial.broker.registration.timeout.ms: The default value is
> 90sec,
> > > > which seems long. If a broker fails the registration because of
> incorrect
> > > > configs, we want to fail faster. In comparison, the defaults for
> > > > zookeeper.connection.timeout.ms is 18 secs.
> > > >
> > >
> > > Hi Jun,
> > >
> > > I agree that the initial connection timeout here is longer than what we
> > > had with ZK. The main reason I selected a slightly longer timeout
> here is
> > > to handle the case where the controllers and the brokers are
> co-located.
> > > For example, if you have a 3 node cluster and all three nodes are
> > > controllers+brokers, when you first bring up the cluster, we will have
> to
> > > stand up the controller quorum and then handle broker registrations.
> > > Although we believe Raft will start up relatively quickly, it's good to
> > > leave some extra margin here.
> > >
> > > I don't think there's a big disadvantage to having a slightly longer
> > > timeout here. After all, starting up the brokers with no controllers
> is an
> > > admin mistake, which we don't expect to see very often. Maybe let's
> set it
> > > to 60 seconds for now and maybe see if we can tweak it in the future if
> > > that turns out to be too long or short?
> > >
> > > > 211. "Once they are all moved, the controller responds to the
> heartbeat
> > > > with a nextState of SHUTDOWN." It seems that nextState is no longer
> > > > relevant. Also, could you add a bit more explanation on when
> ShouldFence
> > > is
> > > > set to true in BrokerHeartbeatRequest?
> > > >
> > >
> > > Good catch. I removed the obsolete section referring to nextState and
> > > added a reference to the new boolean. I also added some more
> information
> > > about ShouldFence and about the rationale for separating fencing from
> > > registration.
> > >
> > > > 212. Related to the above, what does the broker do if IsFenced is
> true in
> > > > the BrokerHeartbeatResponse? Will the broker do the same thing on
> > > receiving
> > > > a FenceBrokerRecord from the metadata log?
> > > >
> > >
> > > The broker only checks this boolean during the startup process. After
> the
> > > startup process is finished, it ignores this boolean.
> > >
> > > The broker uses fence / unfence records in the metadata log to
> determine
> > > which brokers should appear in its MetadataResponse.
> > >
> > > best,
> > > Colin
> > >
> > > > Jun
> > > >
> > > > On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > > > > Hi Colin,
> > > > > >
> > > > > > The KIP says that "brokers which are fenced will not appear in
> > > > > > MetadataResponses. So clients that have up-to-date metadata will
> > > not try
> > > > > > to contact fenced brokers.", which is fine, but it doesn't seem
> to
> > > > > > elaborate on what happens for clients which try to contact a
> broker
> > > > > (using
> > > > > > stale metadata, for example) and find it in the
> registered-but-fenced
> > > > > > state.
> > > > >
> > > > > Hi Tom,
> > > > >
> > > > > I have dropped the broker-side fencing from this proposal. So now
> the
> > > > > fencing is basically the same as today's fencing: it's
> controller-side
> > > > > only. That means that clients using stale metadata can contact
> fenced
> > > > > brokers and communicate with them. The only case where the broker
> > > will be
> > > > > inaccessible is when it hasn't finished starting yet (i.e., has
> not yet
> > > > > attained RUNNING state.)
> > > > >
> > > > > Just like today, we expect the safeguards built into the
> replication
> > > > > protocol to prevent the worst corner cases that could result from
> > > this. I
> > > > > do think we will probably take up this issue later and find a
> better
> > > > > solution for client-side metadata, but this KIP is big enough
> as-is.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > > You said previously that the broker will respond with a retriable
> > > > > > error, which is again fine, but you didn't answer my question
> about
> > > which
> > > > > > error code you've chosen for this. I realise that this doesn't
> really
> > > > > > affect the correctness of the behaviour, but I'm not aware of any
> > > > > existing
> > > > > > error code which is a good fit. So it would be good to understand
> > > about
> > > > > how
> > > > > > you're making this backward compatible for clients.
> > > > > >
> > > > > > Many thanks,
> > > > > >
> > > > > > Tom
> > > > > >
> > > > > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <cmccabe@apache.org
> >
> > > wrote:
> > > > > >
> > > > > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the reply. Just a couple of more comments below.
> > > > > > > >
> > > > > > > > 210. Since we are deprecating
> zookeeper.connection.timeout.ms,
> > > > > should we
> > > > > > > > add a new config to bound the time for a broker to connect
> to the
> > > > > > > > controller during starting up?
> > > > > > > >
> > > > > > >
> > > > > > > Good idea. I added initial.broker.registration.timeout.ms for
> > > this.
> > > > > > >
> > > > > > > > 211. BrokerHeartbeat no longer has the state field in the
> > > > > > > request/response.
> > > > > > > > However, (a) the controller shutdown section still has "In
> its
> > > > > periodic
> > > > > > > > heartbeats, the broker asks the controller if it can
> transition
> > > into
> > > > > the
> > > > > > > > SHUTDOWN state. This motivates the controller to move all
> of the
> > > > > leaders
> > > > > > > > off of that broker. Once they are all moved, the controller
> > > > > responds to
> > > > > > > > the heartbeat with a nextState of SHUTDOWN."; (2) the
> > > description of
> > > > > > > > BrokerHeartbeat still references currentState and
> targetState.
> > > > > > > >
> > > > > > >
> > > > > > > Thanks. I've made these sections clearer and removed the
> obsolete
> > > > > > > references to sending states.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <
> cmccabe@apache.org
> > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > > > > > > > Hi, Colin,
> > > > > > > > > >
> > > > > > > > > > Thanks for the update. A few more follow up comments.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > Thanks again for the review.
> > > > > > > > >
> > > > > > > > > > 100. FailedReplicaRecord: Since this is reported by each
> > > broker
> > > > > > > > > > independently, perhaps we could use a more concise
> > > representation
> > > > > > > that
> > > > > > > > > has
> > > > > > > > > > a top level broker field, an array of topics, which has
> an
> > > array
> > > > > of
> > > > > > > > > > partitions.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > The issue is that there is a size limit on the each record.
> > > > > Putting
> > > > > > > all
> > > > > > > > > of the partitions of a log directory into a single record
> would
> > > > > > > probably
> > > > > > > > > break that in many cases. Still, we can optimize a bit by
> > > having
> > > > > an
> > > > > > > array
> > > > > > > > > of partition IDs, since nearly all the time, we have more
> than
> > > one
> > > > > > > from the
> > > > > > > > > same topic.
> > > > > > > > >
> > > > > > > > > > 200. Sounds good. If we remove the broker-side fencing
> > > logic, do
> > > > > we
> > > > > > > plan
> > > > > > > > > to
> > > > > > > > > > still keep FENCED in broker state? Do we plan to expose
> the
> > > new
> > > > > > > states
> > > > > > > > > > through the existing BrokerState metric and if so, what
> are
> > > the
> > > > > > > values
> > > > > > > > > for
> > > > > > > > > > the new states?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > No, we don't need FENCED any more. I have removed it from
> the
> > > KIP.
> > > > > > > > >
> > > > > > > > > The new states are very similar to the current ones,
> actually.
> > > > > There
> > > > > > > are
> > > > > > > > > no new states or removed ones. The main change in the
> broker
> > > state
> > > > > > > machine
> > > > > > > > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state has been
> > > > > renamed to
> > > > > > > > > RECOVERY. Also, unlike previously, the broker will always
> pass
> > > > > through
> > > > > > > > > RECOVERY (although it may only stay in this state for a
> very
> > > short
> > > > > > > amount
> > > > > > > > > of time).
> > > > > > > > >
> > > > > > > > > > 201. This may be fine too. Could we document what happens
> > > when
> > > > > the
> > > > > > > > > > broker.id/controller.id in metadata.properties don't
> match
> > > the
> > > > > > > broker
> > > > > > > > > > config when the broker starts up?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I added some documentation about this.
> > > > > > > > >
> > > > > > > > > > 204. There is still "The highest metadata offset which
> the
> > > broker
> > > > > > > has not
> > > > > > > > > > reached" referenced under BrokerRegistration.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > > > > >
> > > > > > > > > > 206. Is that separate step needed given KIP-516? With
> > > KIP-516 (
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > > > > ),
> > > > > > > > > > we don't need to wait for the topic data to be removed
> from
> > > all
> > > > > > > brokers
> > > > > > > > > > before removing the topic metadata. The combination of
> > > unmatching
> > > > > > > > > > topicId
> > > > > > > > > > or the missing topicId from the metadata is enough for
> the
> > > > > broker to
> > > > > > > > > > clean
> > > > > > > > > > up deleted topics asynchronously.
> > > > > > > > >
> > > > > > > > > It won't be needed once KIP-516 is adopted, but this hasn't
> > > been
> > > > > > > > > implemented yet.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <
> > > cmccabe@apache.org>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the updated KIP. A few more comments
> below.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Jun,
> > > > > > > > > > >
> > > > > > > > > > > Thanks again for the reviews.
> > > > > > > > > > >
> > > > > > > > > > > > 80.2 For deprecated configs, we need to include
> > > zookeeper.*
> > > > > and
> > > > > > > > > > > > broker.id.generation.enable.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Added.
> > > > > > > > > > >
> > > > > > > > > > > > 83.1 If a broker is down, does the controller keep
> the
> > > > > previously
> > > > > > > > > > > > registered broker epoch forever? If not, how long
> does
> > > the
> > > > > > > controller
> > > > > > > > > > > keep
> > > > > > > > > > > > it? What does the controller do when receiving a
> broker
> > > > > heartbeat
> > > > > > > > > request
> > > > > > > > > > > > with an unfound broker epoch?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Yes, the controller keeps the previous registration
> > > forever.
> > > > > > > > > > >
> > > > > > > > > > > Broker heartbeat requests with an incorrect broker
> epoch
> > > will
> > > > > be
> > > > > > > > > rejected
> > > > > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > > > > >
> > > > > > > > > > > > 100. Have you figured out if we need to add a new
> record
> > > > > type for
> > > > > > > > > > > reporting
> > > > > > > > > > > > partitions on failed disks?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I added FailedReplicaRecord to reflect the case where a
> > > JBOD
> > > > > > > directory
> > > > > > > > > has
> > > > > > > > > > > failed, leading to failed replicas.
> > > > > > > > > > >
> > > > > > > > > > > > 102. For debugging purposes, sometimes it's useful to
> > > read
> > > > > the
> > > > > > > > > metadata
> > > > > > > > > > > > topic using tools like console-consumer. Should we
> > > support
> > > > > that
> > > > > > > and
> > > > > > > > > if
> > > > > > > > > > > so,
> > > > > > > > > > > > how?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > For now, we have the ability to read the metadata logs
> > > with the
> > > > > > > > > dump-logs
> > > > > > > > > > > tool. I think we will come up with some other tools
> in the
> > > > > future
> > > > > > > as
> > > > > > > > > we
> > > > > > > > > > > get experience.
> > > > > > > > > > >
> > > > > > > > > > > > 200. "brokers which are fenced will not appear in
> > > > > > > MetadataResponses.
> > > > > > > > > The
> > > > > > > > > > > > broker will not respond to these requests-- instead,
> it
> > > will
> > > > > > > simply
> > > > > > > > > > > > disconnect." If the controller is partitioned off
> from
> > > the
> > > > > > > brokers,
> > > > > > > > > this
> > > > > > > > > > > > design will cause every broker to stop accepting new
> > > client
> > > > > > > > > requests. In
> > > > > > > > > > > > contrast, if ZK is partitioned off, the existing
> > > behavior is
> > > > > > > that the
> > > > > > > > > > > > brokers can continue to work based on the last known
> > > > > metadata.
> > > > > > > So, I
> > > > > > > > > am
> > > > > > > > > > > not
> > > > > > > > > > > > sure if we should change the existing behavior
> because
> > > of the
> > > > > > > bigger
> > > > > > > > > > > impact
> > > > > > > > > > > > in the new one. Another option is to keep the
> existing
> > > > > behavior
> > > > > > > and
> > > > > > > > > > > expose
> > > > > > > > > > > > a metric for fenced brokers so that the operator
> could be
> > > > > > > alerted.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I'm skeptical about how well running without ZK
> currently
> > > > > works.
> > > > > > > > > However,
> > > > > > > > > > > I will move the broker-side fencing into a follow-up
> KIP.
> > > This
> > > > > > > KIP is
> > > > > > > > > > > already pretty large and there is no hard dependency on
> > > this.
> > > > > > > There
> > > > > > > > > may
> > > > > > > > > > > also be other ways of accomplishing the positive
> effects of
> > > > > what
> > > > > > > > > > > broker-side fencing, so more discussion is needed.
> > > > > > > > > > >
> > > > > > > > > > > > 201. I read Ron's comment, but I am still not sure
> the
> > > > > benefit of
> > > > > > > > > keeping
> > > > > > > > > > > > broker.id and controller.id in meta.properties. It
> seems
> > > > > that
> > > > > > > we are
> > > > > > > > > > > just
> > > > > > > > > > > > duplicating the same info in two places and have the
> > > > > additional
> > > > > > > > > burden of
> > > > > > > > > > > > making sure the values in the two places are
> consistent.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I think the reasoning is that having broker.id
> protects us
> > > > > against
> > > > > > > > > > > accidentally bringing up a broker with a disk from a
> > > different
> > > > > > > > > broker. I
> > > > > > > > > > > don't feel strongly about this but it seemed simpler to
> > > keep
> > > > > it.
> > > > > > > > > > >
> > > > > > > > > > > > 202. controller.connect.security.protocol: Is this
> needed
> > > > > since
> > > > > > > > > > > > controller.listener.names and
> > > listener.security.protocol.map
> > > > > > > imply
> > > > > > > > > the
> > > > > > > > > > > > security protocol already?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > You're right, this isn't needed. I'll remove it.
> > > > > > > > > > >
> > > > > > > > > > > > 203. registration.heartbeat.interval.ms: It
> defaults to
> > > 2k.
> > > > > ZK
> > > > > > > uses
> > > > > > > > > 1/3
> > > > > > > > > > > of
> > > > > > > > > > > > the session timeout for heartbeat. So, given the
> default
> > > 18k
> > > > > for
> > > > > > > > > > > > registration.lease.timeout.ms, should we default
> > > > > > > > > > > > registration.heartbeat.interval.ms to 6k?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 6 seconds seems like a pretty long time between
> > > heartbeats. It
> > > > > > > might
> > > > > > > > > be
> > > > > > > > > > > useful to know when a broker is missing heartbeats,
> with
> > > less
> > > > > time
> > > > > > > than
> > > > > > > > > > > that. I provisionally set it to 3 seconds (we can
> always
> > > > > change
> > > > > > > > > later...)
> > > > > > > > > > >
> > > > > > > > > > > I also changed the name of these configurations to "
> > > > > > > > > > > broker.heartbeat.interval.ms" and "
> > > > > broker.registration.timeout.ms"
> > > > > > > to
> > > > > > > > > try
> > > > > > > > > > > to clarify them a bit.
> > > > > > > > > > >
> > > > > > > > > > > > 204. "The highest metadata offset which the broker
> has
> > > not
> > > > > > > reached."
> > > > > > > > > It
> > > > > > > > > > > > seems this should be "has reached".
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I changed this to "one more than the highest metadata
> > > offset
> > > > > which
> > > > > > > the
> > > > > > > > > > > broker has reached."
> > > > > > > > > > >
> > > > > > > > > > > > 205. UnfenceBrokerRecord and UnregisterBrokerRecord:
> To
> > > me,
> > > > > they
> > > > > > > > > seem to
> > > > > > > > > > > be
> > > > > > > > > > > > the same. Do we need both?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Unregistration means that the broker has been removed
> from
> > > the
> > > > > > > cluster.
> > > > > > > > > > > That is different than unfencing, which marks the
> broker as
> > > > > active.
> > > > > > > > > > >
> > > > > > > > > > > > 206. TopicRecord: The Deleting field is used to
> indicate
> > > > > that the
> > > > > > > > > topic
> > > > > > > > > > > is
> > > > > > > > > > > > being deleted. I am wondering if this is really
> needed
> > > since
> > > > > > > > > RemoveTopic
> > > > > > > > > > > > already indicates the same thing.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > RemoveTopic is the last step, that scrubs all metadata
> > > about
> > > > > the
> > > > > > > topic.
> > > > > > > > > > > In order to get to that last step, the topic data
> needs to
> > > > > removed
> > > > > > > > > from all
> > > > > > > > > > > brokers (after each broker notices that the topic is
> being
> > > > > > > deleted).
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > > Jun
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <
> > > > > cmccabe@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > > > > > > > > > > > > > Hi Colin. Thanks for the updates. It's now
> clear
> > > to me
> > > > > that
> > > > > > > > > brokers
> > > > > > > > > > > > > > keep their broker epoch for the life of their
> JVM --
> > > they
> > > > > > > > > register
> > > > > > > > > > > > > > once, get their broker epoch in the response, and
> > > then
> > > > > never
> > > > > > > > > > > > > > re-register again. Brokers may get fenced, but
> they
> > > > > keep the
> > > > > > > > > same
> > > > > > > > > > > > > > broker epoch for the life of their JVM. The
> > > incarnation
> > > > > ID
> > > > > > > is
> > > > > > > > > also
> > > > > > > > > > > > > > kept for the life of the JVM but is generated by
> the
> > > > > broker
> > > > > > > > > itself
> > > > > > > > > > > > > > upon startup, and the combination of the two
> allows
> > > the
> > > > > > > > > Controller to
> > > > > > > > > > > > > > act idempotently if any previously-sent
> registration
> > > > > response
> > > > > > > > > gets
> > > > > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks, Ron. That's a good summary.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > One thing I wonder about is if it might be
> helpful
> > > for
> > > > > the
> > > > > > > > > broker to
> > > > > > > > > > > > > > send the Cluster ID as determined from its
> > > > > meta.properties
> > > > > > > file
> > > > > > > > > in
> > > > > > > > > > > its
> > > > > > > > > > > > > > registration request. Does it even make sense
> for
> > > the
> > > > > > > broker to
> > > > > > > > > > > > > > successfully register and enter the Fenced state
> if
> > > it
> > > > > has
> > > > > > > the
> > > > > > > > > wrong
> > > > > > > > > > > > > > Cluster ID?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yeah, that's a good idea. Let's have the broker
> pass
> > > its
> > > > > > > cluster
> > > > > > > > > ID in
> > > > > > > > > > > > > the registration RPC, and then registration can
> fail
> > > if the
> > > > > > > broker
> > > > > > > > > is
> > > > > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > The nextMetadatOffset value that the broker
> > > communicates
> > > > > > > > > > > > > > in its registration request only has meaning
> within
> > > the
> > > > > > > correct
> > > > > > > > > > > > > > cluster, so it feels to me that the Controller
> should
> > > > > have
> > > > > > > some
> > > > > > > > > way
> > > > > > > > > > > to
> > > > > > > > > > > > > > perform this sanity check. There is currently
> > > (pre-KIP
> > > > > 500)
> > > > > > > a
> > > > > > > > > check
> > > > > > > > > > > > > > in the broker to make sure its configured
> cluster ID
> > > > > matches
> > > > > > > the
> > > > > > > > > one
> > > > > > > > > > > > > > stored in ZooKeeper, and we will have to perform
> this
> > > > > > > validation
> > > > > > > > > > > > > > somewhere in the KIP-500 world. If the
> Controller
> > > > > doesn't
> > > > > > > do it
> > > > > > > > > > > > > > within the registration request then the broker
> will
> > > > > have to
> > > > > > > > > make a
> > > > > > > > > > > > > > metadata request to the Controller, retrieve the
> > > Cluster
> > > > > ID,
> > > > > > > and
> > > > > > > > > > > > > > perform the check itself. It feels to me that it
> > > might
> > > > > be
> > > > > > > > > better for
> > > > > > > > > > > > > > the Controller to just do it, and then the broker
> > > doesn't
> > > > > > > have to
> > > > > > > > > > > > > > worry about it anymore once it successfully
> > > registers.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I also have a question about the broker.id
> value and
> > > > > > > > > > > meta.properties.
> > > > > > > > > > > > > > The KIP now says "In version 0 of
> meta.properties,
> > > there
> > > > > is a
> > > > > > > > > > > > > > broker.id field. Version 1 does not have this
> > > field.
> > > > > It
> > > > > > > is no
> > > > > > > > > > > longer
> > > > > > > > > > > > > > needed because we no longer support dynamic
> broker id
> > > > > > > > > assignment."
> > > > > > > > > > > > > > But then there is an example version 1
> > > meta.properties
> > > > > file
> > > > > > > that
> > > > > > > > > > > shows
> > > > > > > > > > > > > > the broker.id value. I actually wonder if
> maybe the
> > > > > > > broker.id
> > > > > > > > > value
> > > > > > > > > > > > > > would be good to keep in the version 1
> > > meta.properties
> > > > > file
> > > > > > > > > because
> > > > > > > > > > > it
> > > > > > > > > > > > > > currently (pre-KIP 500, version 0) acts as a
> sanity
> > > > > check to
> > > > > > > make
> > > > > > > > > > > sure
> > > > > > > > > > > > > > the broker is using the correct log directory.
> > > Similarly
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > > controller.id value on controllers -- it would
> > > allow the
> > > > > > > same
> > > > > > > > > type
> > > > > > > > > > > of
> > > > > > > > > > > > > > sanity check for quorum controllers.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > That's a good point. I will add broker.id back,
> and
> > > also
> > > > > add
> > > > > > > > > > > > > controller.id as a possibility.
> > > > > > > > > > > > >
> > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <
> > > > > > > cmccabe@apache.org
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the reply. A few more comments.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks again for the reply. Sorry for the long
> > > > > hiatus. I
> > > > > > > was
> > > > > > > > > on
> > > > > > > > > > > > > vacation for a while.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 55. There is still text that favors new
> broker
> > > > > > > registration.
> > > > > > > > > > > "When a
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > first starts up, when it is in the INITIAL
> > > state, it
> > > > > will
> > > > > > > > > always
> > > > > > > > > > > > > "win"
> > > > > > > > > > > > > > > > broker ID conflicts. However, once it is
> > > granted a
> > > > > > > lease, it
> > > > > > > > > > > > > transitions
> > > > > > > > > > > > > > > > out of the INITIAL state. Thereafter, it may
> > > lose
> > > > > > > subsequent
> > > > > > > > > > > > > conflicts if
> > > > > > > > > > > > > > > > its broker epoch is stale. (See KIP-380 for
> some
> > > > > > > background
> > > > > > > > > on
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > epoch.) The reason for favoring new
> processes
> > > is to
> > > > > > > > > accommodate
> > > > > > > > > > > the
> > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > case where a process is killed with kill -9
> and
> > > then
> > > > > > > > > restarted.
> > > > > > > > > > > We
> > > > > > > > > > > > > want it
> > > > > > > > > > > > > > > > to be able to reclaim its old ID quickly in
> this
> > > > > case."
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the reminder. I have clarified the
> > > language
> > > > > > > here.
> > > > > > > > > > > > > Hopefully now it is clear that we don't allow quick
> > > re-use
> > > > > of
> > > > > > > > > broker
> > > > > > > > > > > IDs.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 80.1 Sounds good. Could you document that
> > > listeners
> > > > > is a
> > > > > > > > > required
> > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > now? It would also be useful to annotate
> other
> > > > > required
> > > > > > > > > configs.
> > > > > > > > > > > For
> > > > > > > > > > > > > > > > example, controller.connect should be
> required.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I added a note specifying that these are
> required.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 80.2 Could you list all deprecated existing
> > > configs?
> > > > > > > Another
> > > > > > > > > one
> > > > > > > > > > > is
> > > > > > > > > > > > > > > > control.plane.listener.name since the
> > > controller no
> > > > > > > longer
> > > > > > > > > sends
> > > > > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata and StopReplica
> > > > > requests.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I added a section specifying some deprecated
> > > configs.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 83.1 It seems that the broker can transition
> from
> > > > > FENCED
> > > > > > > to
> > > > > > > > > > > RUNNING
> > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > registering for a new broker epoch. I am not
> > > sure how
> > > > > > > this
> > > > > > > > > works.
> > > > > > > > > > > > > Once the
> > > > > > > > > > > > > > > > controller fences a broker, there is no need
> for
> > > the
> > > > > > > > > controller
> > > > > > > > > > > to
> > > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > > boker epoch around. So, if the fenced
> broker's
> > > > > heartbeat
> > > > > > > > > request
> > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > existing broker epoch will be rejected,
> leading
> > > the
> > > > > > > broker
> > > > > > > > > back
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The broker epoch refers to the broker
> registration.
> > > > > So we
> > > > > > > DO
> > > > > > > > > keep
> > > > > > > > > > > the
> > > > > > > > > > > > > broker epoch around even while the broker is
> fenced.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The broker epoch changes only when there is a
> new
> > > > > broker
> > > > > > > > > > > > > registration. Fencing or unfencing the broker
> doesn't
> > > > > change
> > > > > > > the
> > > > > > > > > > > broker
> > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 83.5 Good point on KIP-590. Then should we
> > > expose the
> > > > > > > > > controller
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > debugging purposes? If not, we should
> deprecate
> > > the
> > > > > > > > > controllerID
> > > > > > > > > > > > > field in
> > > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think it's OK to expose it for now, with the
> > > proviso
> > > > > > > that it
> > > > > > > > > > > won't
> > > > > > > > > > > > > be reachable by clients.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 90. We rejected the shared ID with just one
> > > reason
> > > > > "This
> > > > > > > is
> > > > > > > > > not a
> > > > > > > > > > > > > good idea
> > > > > > > > > > > > > > > > because NetworkClient assumes a single ID
> > > space. So
> > > > > if
> > > > > > > > > there is
> > > > > > > > > > > > > both a
> > > > > > > > > > > > > > > > controller 1 and a broker 1, we don't have a
> way
> > > of
> > > > > > > picking
> > > > > > > > > the
> > > > > > > > > > > > > "right"
> > > > > > > > > > > > > > > > one." This doesn't seem to be a strong
> reason.
> > > For
> > > > > > > example,
> > > > > > > > > we
> > > > > > > > > > > could
> > > > > > > > > > > > > > > > address the NetworkClient issue with the node
> > > type
> > > > > as you
> > > > > > > > > pointed
> > > > > > > > > > > > > out or
> > > > > > > > > > > > > > > > using the negative value of a broker ID as
> the
> > > > > > > controller ID.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It would require a lot of code changes to
> support
> > > > > multiple
> > > > > > > > > types of
> > > > > > > > > > > > > node IDs. It's not clear to me that the end result
> > > would
> > > > > be
> > > > > > > > > better --
> > > > > > > > > > > I
> > > > > > > > > > > > > tend to think it would be worse, since it would be
> more
> > > > > > > complex.
> > > > > > > > > In a
> > > > > > > > > > > > > similar vein, using negative numbers seems
> dangerous,
> > > > > since we
> > > > > > > use
> > > > > > > > > > > > > negatives or -1 as "special values" in many places.
> > > For
> > > > > > > example,
> > > > > > > > > -1
> > > > > > > > > > > often
> > > > > > > > > > > > > represents "no such node."
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > One important thing to keep in mind is that we
> > > want to
> > > > > be
> > > > > > > able
> > > > > > > > > to
> > > > > > > > > > > > > transition from a broker and a controller being
> > > co-located
> > > > > to
> > > > > > > them
> > > > > > > > > no
> > > > > > > > > > > > > longer being co-located. This is much easier to do
> > > when
> > > > > they
> > > > > > > have
> > > > > > > > > > > separate
> > > > > > > > > > > > > IDs.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > > > > <
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > the broker reports all offline replicas due
> to a
> > > disk
> > > > > > > > > failure to
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > controller. It seems this information needs
> to be
> > > > > > > persisted
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > > log. Do we have a corresponding record for
> that?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hmm, I have to look into this a little bit
> more.
> > > We
> > > > > may
> > > > > > > need
> > > > > > > > > a new
> > > > > > > > > > > > > record type.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 101. Currently, StopReplica request has 2
> modes,
> > > > > without
> > > > > > > > > deletion
> > > > > > > > > > > > > and with
> > > > > > > > > > > > > > > > deletion. The former is used for controlled
> > > shutdown
> > > > > and
> > > > > > > > > handling
> > > > > > > > > > > > > disk
> > > > > > > > > > > > > > > > failure, and causes the follower to stop. The
> > > latter
> > > > > is
> > > > > > > for
> > > > > > > > > topic
> > > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > > and partition reassignment, and causes the
> > > replica
> > > > > to be
> > > > > > > > > deleted.
> > > > > > > > > > > > > Since we
> > > > > > > > > > > > > > > > are deprecating StopReplica, could we
> document
> > > what
> > > > > > > triggers
> > > > > > > > > the
> > > > > > > > > > > > > stopping
> > > > > > > > > > > > > > > > of a follower and the deleting of a replica
> now?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > RemoveTopic triggers deletion. In general the
> > > > > > > functionality of
> > > > > > > > > > > > > StopReplica is subsumed by the metadata records.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 102. Should we include the metadata topic in
> the
> > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > If so,
> > > > > > > > > > > > > > > > when it will be included and what will the
> > > metadata
> > > > > > > response
> > > > > > > > > look
> > > > > > > > > > > > > like?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > No, it won't be included in the metadata
> response
> > > sent
> > > > > back
> > > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > > brokers.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 103. "The active controller assigns the
> broker a
> > > new
> > > > > > > broker
> > > > > > > > > > > epoch,
> > > > > > > > > > > > > based on
> > > > > > > > > > > > > > > > the latest committed offset in the log." This
> > > seems
> > > > > > > > > inaccurate
> > > > > > > > > > > since
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > latest committed offset doesn't always
> advance on
> > > > > every
> > > > > > > log
> > > > > > > > > > > append.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Given that the new broker epoch won't be
> visible
> > > until
> > > > > the
> > > > > > > > > commit
> > > > > > > > > > > has
> > > > > > > > > > > > > happened, I have changed this to "the next
> available
> > > > > offset in
> > > > > > > the
> > > > > > > > > log"
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 104. REGISTERING(1) : It says "Otherwise, the
> > > broker
> > > > > > > moves
> > > > > > > > > into
> > > > > > > > > > > the
> > > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > > state.". It seems this should be RUNNING?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 105. RUNNING: Should we require the broker to
> > > catch
> > > > > up
> > > > > > > to the
> > > > > > > > > > > > > metadata log
> > > > > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For 104 and 105, these sections have been
> reworked.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe
> <
> > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley
> > > wrote:
> > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron
> Dagostino
> > > > > wrote:
> > > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the hard work
> on
> > > this
> > > > > KIP.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I have some questions about what
> happens
> > > to a
> > > > > > > broker
> > > > > > > > > > > when it
> > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > fenced (e.g. because it can't send a
> > > > > heartbeat
> > > > > > > > > request to
> > > > > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > > > > lease). The KIP says "When a broker
> is
> > > > > fenced,
> > > > > > > it
> > > > > > > > > cannot
> > > > > > > > > > > > > process any
> > > > > > > > > > > > > > > > > > > > client requests. This prevents
> brokers
> > > which
> > > > > > > are not
> > > > > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > > > > metadata updates or that are not
> > > receiving
> > > > > and
> > > > > > > > > processing
> > > > > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > > > > enough from causing issues to
> clients."
> > > And
> > > > > in
> > > > > > > the
> > > > > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > > > > FENCED(4) state it likewise says
> "While
> > > in
> > > > > this
> > > > > > > > > state,
> > > > > > > > > > > the
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > not respond to client requests." It
> > > makes
> > > > > sense
> > > > > > > > > that a
> > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > > should not accept producer requests
> -- I
> > > > > assume
> > > > > > > any
> > > > > > > > > such
> > > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > > > would result in
> > > NotLeaderOrFollowerException.
> > > > > > > But
> > > > > > > > > what
> > > > > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > > > > (fetch from follower) consumer
> > > requests? It
> > > > > is
> > > > > > > > > > > conceivable
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > could continue. Related to that,
> would a
> > > > > fenced
> > > > > > > > > broker
> > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > fetch data for partitions where it
> > > thinks it
> > > > > is a
> > > > > > > > > > > follower?
> > > > > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > > > > it rejects consumer requests it might
> > > still
> > > > > > > continue
> > > > > > > > > to
> > > > > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > > > > follower. Might it be helpful to
> clarify
> > > > > both
> > > > > > > > > decisions
> > > > > > > > > > > > > here?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Good question. I think a fenced broker
> > > should
> > > > > > > > > continue to
> > > > > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > > > > partitions it was already fetching
> before
> > > it
> > > > > was
> > > > > > > > > fenced,
> > > > > > > > > > > > > unless it
> > > > > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > > > > problem. At that point it won't be
> able to
> > > > > > > continue,
> > > > > > > > > > > since it
> > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > the new metadata. For example, it
> won't
> > > know
> > > > > about
> > > > > > > > > > > leadership
> > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the partitions it's fetching. The
> > > rationale
> > > > > for
> > > > > > > > > > > continuing to
> > > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > try to avoid disruptions as much as
> > > possible.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I don't think fenced brokers should
> accept
> > > > > client
> > > > > > > > > requests.
> > > > > > > > > > > > > The issue
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > that the fenced broker may or may not
> have
> > > any
> > > > > > > data it
> > > > > > > > > is
> > > > > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > > > > have. It may or may not have applied
> any
> > > > > > > configuration
> > > > > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > it is supposed to have applied. So it
> > > could
> > > > > get
> > > > > > > pretty
> > > > > > > > > > > > > confusing, and
> > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > potentially waste the client's time.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > When fenced, how would the broker reply
> to a
> > > > > client
> > > > > > > > > which did
> > > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The broker will respond with a retryable
> error
> > > in
> > > > > that
> > > > > > > > > case.
> > > > > > > > > > > Once
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > client has re-fetched its metadata, it
> will no
> > > > > longer
> > > > > > > see
> > > > > > > > > the
> > > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > as part of the cluster. I added a note to
> the
> > > KIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Wed, Dec 16, 2020, at 09:59, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the reply. A few more comments below.
>
Hi Jun,
Thanks for the comments.
> 206. "RemoveTopic is the last step, that scrubs all metadata about the
> topic. In order to get to that last step, the topic data needs to removed
> from all brokers (after each broker notices that the topic is being
> deleted)." Currently, this is done based on the response of
> StopReplicaRequest. Since the controller no longer sends this request, how
> does the controller know that the data for the deleted topic has
> been removed in the brokers?
>
That's a good point... definitely an oversight.
It seems complex to force the controller to track when log directories have been deleted. Let's just assume that KIP-516 has been implemented, and track them by UUID. Then we can just have a single topic deletion record.
I added a section on "topic identifiers" describing this.
> 210. Thanks for the explanation. Sounds good to me.
>
> 211. I still don't see an example when ShouldFence is set to true in
> BrokerHeartbeatReques. Could we add one?
>
It is sent to true when the broker is first starting up and doesn't yet want to be unfenced. I added a longer explanation of this in the "Broker Leases" section.
> 213. The KIP now allows replicas to be assigned on brokers that are fenced,
> which is an improvement. How do we permanently remove a broker (e.g.
> cluster shrinking) to prevent it from being used for future replica
> assignments?
>
This is a fair point. I will create a kafka-cluster.sh script which can do this, plus a DecommissionBrokerRequest.
As a bonus the kafka-cluster.sh script can help people find the cluster ID of brokers, something that people have wanted a tool for in the past.
> 214. "Currently, when a node is down, all of its ZK registration
> information is gone. But we need this information in order to understand
> things like whether the replicas of a particular partition are
> well-balanced across racks." This is not quite true. Currently, even when
> ZK registration is gone, the existing replica assignment is still available
> in the metadata response.
>
I agree that the existing replica assignment is still available. But that just tells you partition X is on nodes A, B, and C. If you don't have the ZK registration for one or more of A, B, or C then you don't know whether we are following the policy of "two replicas on one rack, one replica on another." Or any other more complex rack placement policy that you might have.
best,
Colin
> Jun
>
> On Wed, Dec 16, 2020 at 8:48 AM Colin McCabe <cm...@apache.org> wrote:
>
> > On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply. A few more follow up comments.
> > >
> > > 210. initial.broker.registration.timeout.ms: The default value is 90sec,
> > > which seems long. If a broker fails the registration because of incorrect
> > > configs, we want to fail faster. In comparison, the defaults for
> > > zookeeper.connection.timeout.ms is 18 secs.
> > >
> >
> > Hi Jun,
> >
> > I agree that the initial connection timeout here is longer than what we
> > had with ZK. The main reason I selected a slightly longer timeout here is
> > to handle the case where the controllers and the brokers are co-located.
> > For example, if you have a 3 node cluster and all three nodes are
> > controllers+brokers, when you first bring up the cluster, we will have to
> > stand up the controller quorum and then handle broker registrations.
> > Although we believe Raft will start up relatively quickly, it's good to
> > leave some extra margin here.
> >
> > I don't think there's a big disadvantage to having a slightly longer
> > timeout here. After all, starting up the brokers with no controllers is an
> > admin mistake, which we don't expect to see very often. Maybe let's set it
> > to 60 seconds for now and maybe see if we can tweak it in the future if
> > that turns out to be too long or short?
> >
> > > 211. "Once they are all moved, the controller responds to the heartbeat
> > > with a nextState of SHUTDOWN." It seems that nextState is no longer
> > > relevant. Also, could you add a bit more explanation on when ShouldFence
> > is
> > > set to true in BrokerHeartbeatRequest?
> > >
> >
> > Good catch. I removed the obsolete section referring to nextState and
> > added a reference to the new boolean. I also added some more information
> > about ShouldFence and about the rationale for separating fencing from
> > registration.
> >
> > > 212. Related to the above, what does the broker do if IsFenced is true in
> > > the BrokerHeartbeatResponse? Will the broker do the same thing on
> > receiving
> > > a FenceBrokerRecord from the metadata log?
> > >
> >
> > The broker only checks this boolean during the startup process. After the
> > startup process is finished, it ignores this boolean.
> >
> > The broker uses fence / unfence records in the metadata log to determine
> > which brokers should appear in its MetadataResponse.
> >
> > best,
> > Colin
> >
> > > Jun
> > >
> > > On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > > > Hi Colin,
> > > > >
> > > > > The KIP says that "brokers which are fenced will not appear in
> > > > > MetadataResponses. So clients that have up-to-date metadata will
> > not try
> > > > > to contact fenced brokers.", which is fine, but it doesn't seem to
> > > > > elaborate on what happens for clients which try to contact a broker
> > > > (using
> > > > > stale metadata, for example) and find it in the registered-but-fenced
> > > > > state.
> > > >
> > > > Hi Tom,
> > > >
> > > > I have dropped the broker-side fencing from this proposal. So now the
> > > > fencing is basically the same as today's fencing: it's controller-side
> > > > only. That means that clients using stale metadata can contact fenced
> > > > brokers and communicate with them. The only case where the broker
> > will be
> > > > inaccessible is when it hasn't finished starting yet (i.e., has not yet
> > > > attained RUNNING state.)
> > > >
> > > > Just like today, we expect the safeguards built into the replication
> > > > protocol to prevent the worst corner cases that could result from
> > this. I
> > > > do think we will probably take up this issue later and find a better
> > > > solution for client-side metadata, but this KIP is big enough as-is.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > > You said previously that the broker will respond with a retriable
> > > > > error, which is again fine, but you didn't answer my question about
> > which
> > > > > error code you've chosen for this. I realise that this doesn't really
> > > > > affect the correctness of the behaviour, but I'm not aware of any
> > > > existing
> > > > > error code which is a good fit. So it would be good to understand
> > about
> > > > how
> > > > > you're making this backward compatible for clients.
> > > > >
> > > > > Many thanks,
> > > > >
> > > > > Tom
> > > > >
> > > > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the reply. Just a couple of more comments below.
> > > > > > >
> > > > > > > 210. Since we are deprecating zookeeper.connection.timeout.ms,
> > > > should we
> > > > > > > add a new config to bound the time for a broker to connect to the
> > > > > > > controller during starting up?
> > > > > > >
> > > > > >
> > > > > > Good idea. I added initial.broker.registration.timeout.ms for
> > this.
> > > > > >
> > > > > > > 211. BrokerHeartbeat no longer has the state field in the
> > > > > > request/response.
> > > > > > > However, (a) the controller shutdown section still has "In its
> > > > periodic
> > > > > > > heartbeats, the broker asks the controller if it can transition
> > into
> > > > the
> > > > > > > SHUTDOWN state. This motivates the controller to move all of the
> > > > leaders
> > > > > > > off of that broker. Once they are all moved, the controller
> > > > responds to
> > > > > > > the heartbeat with a nextState of SHUTDOWN."; (2) the
> > description of
> > > > > > > BrokerHeartbeat still references currentState and targetState.
> > > > > > >
> > > > > >
> > > > > > Thanks. I've made these sections clearer and removed the obsolete
> > > > > > references to sending states.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <cmccabe@apache.org
> > >
> > > > wrote:
> > > > > > >
> > > > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > > > > > > Hi, Colin,
> > > > > > > > >
> > > > > > > > > Thanks for the update. A few more follow up comments.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > Thanks again for the review.
> > > > > > > >
> > > > > > > > > 100. FailedReplicaRecord: Since this is reported by each
> > broker
> > > > > > > > > independently, perhaps we could use a more concise
> > representation
> > > > > > that
> > > > > > > > has
> > > > > > > > > a top level broker field, an array of topics, which has an
> > array
> > > > of
> > > > > > > > > partitions.
> > > > > > > > >
> > > > > > > >
> > > > > > > > The issue is that there is a size limit on the each record.
> > > > Putting
> > > > > > all
> > > > > > > > of the partitions of a log directory into a single record would
> > > > > > probably
> > > > > > > > break that in many cases. Still, we can optimize a bit by
> > having
> > > > an
> > > > > > array
> > > > > > > > of partition IDs, since nearly all the time, we have more than
> > one
> > > > > > from the
> > > > > > > > same topic.
> > > > > > > >
> > > > > > > > > 200. Sounds good. If we remove the broker-side fencing
> > logic, do
> > > > we
> > > > > > plan
> > > > > > > > to
> > > > > > > > > still keep FENCED in broker state? Do we plan to expose the
> > new
> > > > > > states
> > > > > > > > > through the existing BrokerState metric and if so, what are
> > the
> > > > > > values
> > > > > > > > for
> > > > > > > > > the new states?
> > > > > > > > >
> > > > > > > >
> > > > > > > > No, we don't need FENCED any more. I have removed it from the
> > KIP.
> > > > > > > >
> > > > > > > > The new states are very similar to the current ones, actually.
> > > > There
> > > > > > are
> > > > > > > > no new states or removed ones. The main change in the broker
> > state
> > > > > > machine
> > > > > > > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state has been
> > > > renamed to
> > > > > > > > RECOVERY. Also, unlike previously, the broker will always pass
> > > > through
> > > > > > > > RECOVERY (although it may only stay in this state for a very
> > short
> > > > > > amount
> > > > > > > > of time).
> > > > > > > >
> > > > > > > > > 201. This may be fine too. Could we document what happens
> > when
> > > > the
> > > > > > > > > broker.id/controller.id in metadata.properties don't match
> > the
> > > > > > broker
> > > > > > > > > config when the broker starts up?
> > > > > > > > >
> > > > > > > >
> > > > > > > > I added some documentation about this.
> > > > > > > >
> > > > > > > > > 204. There is still "The highest metadata offset which the
> > broker
> > > > > > has not
> > > > > > > > > reached" referenced under BrokerRegistration.
> > > > > > > > >
> > > > > > > >
> > > > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > > > >
> > > > > > > > > 206. Is that separate step needed given KIP-516? With
> > KIP-516 (
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > > > ),
> > > > > > > > > we don't need to wait for the topic data to be removed from
> > all
> > > > > > brokers
> > > > > > > > > before removing the topic metadata. The combination of
> > unmatching
> > > > > > > > > topicId
> > > > > > > > > or the missing topicId from the metadata is enough for the
> > > > broker to
> > > > > > > > > clean
> > > > > > > > > up deleted topics asynchronously.
> > > > > > > >
> > > > > > > > It won't be needed once KIP-516 is adopted, but this hasn't
> > been
> > > > > > > > implemented yet.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <
> > cmccabe@apache.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > > > > > > > > Hi, Colin,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the updated KIP. A few more comments below.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi Jun,
> > > > > > > > > >
> > > > > > > > > > Thanks again for the reviews.
> > > > > > > > > >
> > > > > > > > > > > 80.2 For deprecated configs, we need to include
> > zookeeper.*
> > > > and
> > > > > > > > > > > broker.id.generation.enable.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Added.
> > > > > > > > > >
> > > > > > > > > > > 83.1 If a broker is down, does the controller keep the
> > > > previously
> > > > > > > > > > > registered broker epoch forever? If not, how long does
> > the
> > > > > > controller
> > > > > > > > > > keep
> > > > > > > > > > > it? What does the controller do when receiving a broker
> > > > heartbeat
> > > > > > > > request
> > > > > > > > > > > with an unfound broker epoch?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Yes, the controller keeps the previous registration
> > forever.
> > > > > > > > > >
> > > > > > > > > > Broker heartbeat requests with an incorrect broker epoch
> > will
> > > > be
> > > > > > > > rejected
> > > > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > > > >
> > > > > > > > > > > 100. Have you figured out if we need to add a new record
> > > > type for
> > > > > > > > > > reporting
> > > > > > > > > > > partitions on failed disks?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I added FailedReplicaRecord to reflect the case where a
> > JBOD
> > > > > > directory
> > > > > > > > has
> > > > > > > > > > failed, leading to failed replicas.
> > > > > > > > > >
> > > > > > > > > > > 102. For debugging purposes, sometimes it's useful to
> > read
> > > > the
> > > > > > > > metadata
> > > > > > > > > > > topic using tools like console-consumer. Should we
> > support
> > > > that
> > > > > > and
> > > > > > > > if
> > > > > > > > > > so,
> > > > > > > > > > > how?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > For now, we have the ability to read the metadata logs
> > with the
> > > > > > > > dump-logs
> > > > > > > > > > tool. I think we will come up with some other tools in the
> > > > future
> > > > > > as
> > > > > > > > we
> > > > > > > > > > get experience.
> > > > > > > > > >
> > > > > > > > > > > 200. "brokers which are fenced will not appear in
> > > > > > MetadataResponses.
> > > > > > > > The
> > > > > > > > > > > broker will not respond to these requests-- instead, it
> > will
> > > > > > simply
> > > > > > > > > > > disconnect." If the controller is partitioned off from
> > the
> > > > > > brokers,
> > > > > > > > this
> > > > > > > > > > > design will cause every broker to stop accepting new
> > client
> > > > > > > > requests. In
> > > > > > > > > > > contrast, if ZK is partitioned off, the existing
> > behavior is
> > > > > > that the
> > > > > > > > > > > brokers can continue to work based on the last known
> > > > metadata.
> > > > > > So, I
> > > > > > > > am
> > > > > > > > > > not
> > > > > > > > > > > sure if we should change the existing behavior because
> > of the
> > > > > > bigger
> > > > > > > > > > impact
> > > > > > > > > > > in the new one. Another option is to keep the existing
> > > > behavior
> > > > > > and
> > > > > > > > > > expose
> > > > > > > > > > > a metric for fenced brokers so that the operator could be
> > > > > > alerted.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I'm skeptical about how well running without ZK currently
> > > > works.
> > > > > > > > However,
> > > > > > > > > > I will move the broker-side fencing into a follow-up KIP.
> > This
> > > > > > KIP is
> > > > > > > > > > already pretty large and there is no hard dependency on
> > this.
> > > > > > There
> > > > > > > > may
> > > > > > > > > > also be other ways of accomplishing the positive effects of
> > > > what
> > > > > > > > > > broker-side fencing, so more discussion is needed.
> > > > > > > > > >
> > > > > > > > > > > 201. I read Ron's comment, but I am still not sure the
> > > > benefit of
> > > > > > > > keeping
> > > > > > > > > > > broker.id and controller.id in meta.properties. It seems
> > > > that
> > > > > > we are
> > > > > > > > > > just
> > > > > > > > > > > duplicating the same info in two places and have the
> > > > additional
> > > > > > > > burden of
> > > > > > > > > > > making sure the values in the two places are consistent.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I think the reasoning is that having broker.id protects us
> > > > against
> > > > > > > > > > accidentally bringing up a broker with a disk from a
> > different
> > > > > > > > broker. I
> > > > > > > > > > don't feel strongly about this but it seemed simpler to
> > keep
> > > > it.
> > > > > > > > > >
> > > > > > > > > > > 202. controller.connect.security.protocol: Is this needed
> > > > since
> > > > > > > > > > > controller.listener.names and
> > listener.security.protocol.map
> > > > > > imply
> > > > > > > > the
> > > > > > > > > > > security protocol already?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > You're right, this isn't needed. I'll remove it.
> > > > > > > > > >
> > > > > > > > > > > 203. registration.heartbeat.interval.ms: It defaults to
> > 2k.
> > > > ZK
> > > > > > uses
> > > > > > > > 1/3
> > > > > > > > > > of
> > > > > > > > > > > the session timeout for heartbeat. So, given the default
> > 18k
> > > > for
> > > > > > > > > > > registration.lease.timeout.ms, should we default
> > > > > > > > > > > registration.heartbeat.interval.ms to 6k?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 6 seconds seems like a pretty long time between
> > heartbeats. It
> > > > > > might
> > > > > > > > be
> > > > > > > > > > useful to know when a broker is missing heartbeats, with
> > less
> > > > time
> > > > > > than
> > > > > > > > > > that. I provisionally set it to 3 seconds (we can always
> > > > change
> > > > > > > > later...)
> > > > > > > > > >
> > > > > > > > > > I also changed the name of these configurations to "
> > > > > > > > > > broker.heartbeat.interval.ms" and "
> > > > broker.registration.timeout.ms"
> > > > > > to
> > > > > > > > try
> > > > > > > > > > to clarify them a bit.
> > > > > > > > > >
> > > > > > > > > > > 204. "The highest metadata offset which the broker has
> > not
> > > > > > reached."
> > > > > > > > It
> > > > > > > > > > > seems this should be "has reached".
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I changed this to "one more than the highest metadata
> > offset
> > > > which
> > > > > > the
> > > > > > > > > > broker has reached."
> > > > > > > > > >
> > > > > > > > > > > 205. UnfenceBrokerRecord and UnregisterBrokerRecord: To
> > me,
> > > > they
> > > > > > > > seem to
> > > > > > > > > > be
> > > > > > > > > > > the same. Do we need both?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Unregistration means that the broker has been removed from
> > the
> > > > > > cluster.
> > > > > > > > > > That is different than unfencing, which marks the broker as
> > > > active.
> > > > > > > > > >
> > > > > > > > > > > 206. TopicRecord: The Deleting field is used to indicate
> > > > that the
> > > > > > > > topic
> > > > > > > > > > is
> > > > > > > > > > > being deleted. I am wondering if this is really needed
> > since
> > > > > > > > RemoveTopic
> > > > > > > > > > > already indicates the same thing.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > RemoveTopic is the last step, that scrubs all metadata
> > about
> > > > the
> > > > > > topic.
> > > > > > > > > > In order to get to that last step, the topic data needs to
> > > > removed
> > > > > > > > from all
> > > > > > > > > > brokers (after each broker notices that the topic is being
> > > > > > deleted).
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <
> > > > cmccabe@apache.org>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > > > > > > > > > > > > Hi Colin. Thanks for the updates. It's now clear
> > to me
> > > > that
> > > > > > > > brokers
> > > > > > > > > > > > > keep their broker epoch for the life of their JVM --
> > they
> > > > > > > > register
> > > > > > > > > > > > > once, get their broker epoch in the response, and
> > then
> > > > never
> > > > > > > > > > > > > re-register again. Brokers may get fenced, but they
> > > > keep the
> > > > > > > > same
> > > > > > > > > > > > > broker epoch for the life of their JVM. The
> > incarnation
> > > > ID
> > > > > > is
> > > > > > > > also
> > > > > > > > > > > > > kept for the life of the JVM but is generated by the
> > > > broker
> > > > > > > > itself
> > > > > > > > > > > > > upon startup, and the combination of the two allows
> > the
> > > > > > > > Controller to
> > > > > > > > > > > > > act idempotently if any previously-sent registration
> > > > response
> > > > > > > > gets
> > > > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks, Ron. That's a good summary.
> > > > > > > > > > > >
> > > > > > > > > > > > > One thing I wonder about is if it might be helpful
> > for
> > > > the
> > > > > > > > broker to
> > > > > > > > > > > > > send the Cluster ID as determined from its
> > > > meta.properties
> > > > > > file
> > > > > > > > in
> > > > > > > > > > its
> > > > > > > > > > > > > registration request. Does it even make sense for
> > the
> > > > > > broker to
> > > > > > > > > > > > > successfully register and enter the Fenced state if
> > it
> > > > has
> > > > > > the
> > > > > > > > wrong
> > > > > > > > > > > > > Cluster ID?
> > > > > > > > > > > >
> > > > > > > > > > > > Yeah, that's a good idea. Let's have the broker pass
> > its
> > > > > > cluster
> > > > > > > > ID in
> > > > > > > > > > > > the registration RPC, and then registration can fail
> > if the
> > > > > > broker
> > > > > > > > is
> > > > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > > > >
> > > > > > > > > > > > > The nextMetadatOffset value that the broker
> > communicates
> > > > > > > > > > > > > in its registration request only has meaning within
> > the
> > > > > > correct
> > > > > > > > > > > > > cluster, so it feels to me that the Controller should
> > > > have
> > > > > > some
> > > > > > > > way
> > > > > > > > > > to
> > > > > > > > > > > > > perform this sanity check. There is currently
> > (pre-KIP
> > > > 500)
> > > > > > a
> > > > > > > > check
> > > > > > > > > > > > > in the broker to make sure its configured cluster ID
> > > > matches
> > > > > > the
> > > > > > > > one
> > > > > > > > > > > > > stored in ZooKeeper, and we will have to perform this
> > > > > > validation
> > > > > > > > > > > > > somewhere in the KIP-500 world. If the Controller
> > > > doesn't
> > > > > > do it
> > > > > > > > > > > > > within the registration request then the broker will
> > > > have to
> > > > > > > > make a
> > > > > > > > > > > > > metadata request to the Controller, retrieve the
> > Cluster
> > > > ID,
> > > > > > and
> > > > > > > > > > > > > perform the check itself. It feels to me that it
> > might
> > > > be
> > > > > > > > better for
> > > > > > > > > > > > > the Controller to just do it, and then the broker
> > doesn't
> > > > > > have to
> > > > > > > > > > > > > worry about it anymore once it successfully
> > registers.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I also have a question about the broker.id value and
> > > > > > > > > > meta.properties.
> > > > > > > > > > > > > The KIP now says "In version 0 of meta.properties,
> > there
> > > > is a
> > > > > > > > > > > > > broker.id field. Version 1 does not have this
> > field.
> > > > It
> > > > > > is no
> > > > > > > > > > longer
> > > > > > > > > > > > > needed because we no longer support dynamic broker id
> > > > > > > > assignment."
> > > > > > > > > > > > > But then there is an example version 1
> > meta.properties
> > > > file
> > > > > > that
> > > > > > > > > > shows
> > > > > > > > > > > > > the broker.id value. I actually wonder if maybe the
> > > > > > broker.id
> > > > > > > > value
> > > > > > > > > > > > > would be good to keep in the version 1
> > meta.properties
> > > > file
> > > > > > > > because
> > > > > > > > > > it
> > > > > > > > > > > > > currently (pre-KIP 500, version 0) acts as a sanity
> > > > check to
> > > > > > make
> > > > > > > > > > sure
> > > > > > > > > > > > > the broker is using the correct log directory.
> > Similarly
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > > > controller.id value on controllers -- it would
> > allow the
> > > > > > same
> > > > > > > > type
> > > > > > > > > > of
> > > > > > > > > > > > > sanity check for quorum controllers.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > That's a good point. I will add broker.id back, and
> > also
> > > > add
> > > > > > > > > > > > controller.id as a possibility.
> > > > > > > > > > > >
> > > > > > > > > > > > cheers,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <
> > > > > > cmccabe@apache.org
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the reply. A few more comments.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks again for the reply. Sorry for the long
> > > > hiatus. I
> > > > > > was
> > > > > > > > on
> > > > > > > > > > > > vacation for a while.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 55. There is still text that favors new broker
> > > > > > registration.
> > > > > > > > > > "When a
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > first starts up, when it is in the INITIAL
> > state, it
> > > > will
> > > > > > > > always
> > > > > > > > > > > > "win"
> > > > > > > > > > > > > > > broker ID conflicts. However, once it is
> > granted a
> > > > > > lease, it
> > > > > > > > > > > > transitions
> > > > > > > > > > > > > > > out of the INITIAL state. Thereafter, it may
> > lose
> > > > > > subsequent
> > > > > > > > > > > > conflicts if
> > > > > > > > > > > > > > > its broker epoch is stale. (See KIP-380 for some
> > > > > > background
> > > > > > > > on
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > epoch.) The reason for favoring new processes
> > is to
> > > > > > > > accommodate
> > > > > > > > > > the
> > > > > > > > > > > > common
> > > > > > > > > > > > > > > case where a process is killed with kill -9 and
> > then
> > > > > > > > restarted.
> > > > > > > > > > We
> > > > > > > > > > > > want it
> > > > > > > > > > > > > > > to be able to reclaim its old ID quickly in this
> > > > case."
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the reminder. I have clarified the
> > language
> > > > > > here.
> > > > > > > > > > > > Hopefully now it is clear that we don't allow quick
> > re-use
> > > > of
> > > > > > > > broker
> > > > > > > > > > IDs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 80.1 Sounds good. Could you document that
> > listeners
> > > > is a
> > > > > > > > required
> > > > > > > > > > > > config
> > > > > > > > > > > > > > > now? It would also be useful to annotate other
> > > > required
> > > > > > > > configs.
> > > > > > > > > > For
> > > > > > > > > > > > > > > example, controller.connect should be required.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I added a note specifying that these are required.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 80.2 Could you list all deprecated existing
> > configs?
> > > > > > Another
> > > > > > > > one
> > > > > > > > > > is
> > > > > > > > > > > > > > > control.plane.listener.name since the
> > controller no
> > > > > > longer
> > > > > > > > sends
> > > > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata and StopReplica
> > > > requests.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I added a section specifying some deprecated
> > configs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 83.1 It seems that the broker can transition from
> > > > FENCED
> > > > > > to
> > > > > > > > > > RUNNING
> > > > > > > > > > > > without
> > > > > > > > > > > > > > > registering for a new broker epoch. I am not
> > sure how
> > > > > > this
> > > > > > > > works.
> > > > > > > > > > > > Once the
> > > > > > > > > > > > > > > controller fences a broker, there is no need for
> > the
> > > > > > > > controller
> > > > > > > > > > to
> > > > > > > > > > > > keep the
> > > > > > > > > > > > > > > boker epoch around. So, if the fenced broker's
> > > > heartbeat
> > > > > > > > request
> > > > > > > > > > > > with the
> > > > > > > > > > > > > > > existing broker epoch will be rejected, leading
> > the
> > > > > > broker
> > > > > > > > back
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The broker epoch refers to the broker registration.
> > > > So we
> > > > > > DO
> > > > > > > > keep
> > > > > > > > > > the
> > > > > > > > > > > > broker epoch around even while the broker is fenced.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The broker epoch changes only when there is a new
> > > > broker
> > > > > > > > > > > > registration. Fencing or unfencing the broker doesn't
> > > > change
> > > > > > the
> > > > > > > > > > broker
> > > > > > > > > > > > epoch.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 83.5 Good point on KIP-590. Then should we
> > expose the
> > > > > > > > controller
> > > > > > > > > > for
> > > > > > > > > > > > > > > debugging purposes? If not, we should deprecate
> > the
> > > > > > > > controllerID
> > > > > > > > > > > > field in
> > > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it's OK to expose it for now, with the
> > proviso
> > > > > > that it
> > > > > > > > > > won't
> > > > > > > > > > > > be reachable by clients.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 90. We rejected the shared ID with just one
> > reason
> > > > "This
> > > > > > is
> > > > > > > > not a
> > > > > > > > > > > > good idea
> > > > > > > > > > > > > > > because NetworkClient assumes a single ID
> > space. So
> > > > if
> > > > > > > > there is
> > > > > > > > > > > > both a
> > > > > > > > > > > > > > > controller 1 and a broker 1, we don't have a way
> > of
> > > > > > picking
> > > > > > > > the
> > > > > > > > > > > > "right"
> > > > > > > > > > > > > > > one." This doesn't seem to be a strong reason.
> > For
> > > > > > example,
> > > > > > > > we
> > > > > > > > > > could
> > > > > > > > > > > > > > > address the NetworkClient issue with the node
> > type
> > > > as you
> > > > > > > > pointed
> > > > > > > > > > > > out or
> > > > > > > > > > > > > > > using the negative value of a broker ID as the
> > > > > > controller ID.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It would require a lot of code changes to support
> > > > multiple
> > > > > > > > types of
> > > > > > > > > > > > node IDs. It's not clear to me that the end result
> > would
> > > > be
> > > > > > > > better --
> > > > > > > > > > I
> > > > > > > > > > > > tend to think it would be worse, since it would be more
> > > > > > complex.
> > > > > > > > In a
> > > > > > > > > > > > similar vein, using negative numbers seems dangerous,
> > > > since we
> > > > > > use
> > > > > > > > > > > > negatives or -1 as "special values" in many places.
> > For
> > > > > > example,
> > > > > > > > -1
> > > > > > > > > > often
> > > > > > > > > > > > represents "no such node."
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > One important thing to keep in mind is that we
> > want to
> > > > be
> > > > > > able
> > > > > > > > to
> > > > > > > > > > > > transition from a broker and a controller being
> > co-located
> > > > to
> > > > > > them
> > > > > > > > no
> > > > > > > > > > > > longer being co-located. This is much easier to do
> > when
> > > > they
> > > > > > have
> > > > > > > > > > separate
> > > > > > > > > > > > IDs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > > > <
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > > > >,
> > > > > > > > > > > > > > > the broker reports all offline replicas due to a
> > disk
> > > > > > > > failure to
> > > > > > > > > > the
> > > > > > > > > > > > > > > controller. It seems this information needs to be
> > > > > > persisted
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > > log. Do we have a corresponding record for that?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hmm, I have to look into this a little bit more.
> > We
> > > > may
> > > > > > need
> > > > > > > > a new
> > > > > > > > > > > > record type.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 101. Currently, StopReplica request has 2 modes,
> > > > without
> > > > > > > > deletion
> > > > > > > > > > > > and with
> > > > > > > > > > > > > > > deletion. The former is used for controlled
> > shutdown
> > > > and
> > > > > > > > handling
> > > > > > > > > > > > disk
> > > > > > > > > > > > > > > failure, and causes the follower to stop. The
> > latter
> > > > is
> > > > > > for
> > > > > > > > topic
> > > > > > > > > > > > deletion
> > > > > > > > > > > > > > > and partition reassignment, and causes the
> > replica
> > > > to be
> > > > > > > > deleted.
> > > > > > > > > > > > Since we
> > > > > > > > > > > > > > > are deprecating StopReplica, could we document
> > what
> > > > > > triggers
> > > > > > > > the
> > > > > > > > > > > > stopping
> > > > > > > > > > > > > > > of a follower and the deleting of a replica now?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > RemoveTopic triggers deletion. In general the
> > > > > > functionality of
> > > > > > > > > > > > StopReplica is subsumed by the metadata records.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 102. Should we include the metadata topic in the
> > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > If so,
> > > > > > > > > > > > > > > when it will be included and what will the
> > metadata
> > > > > > response
> > > > > > > > look
> > > > > > > > > > > > like?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > No, it won't be included in the metadata response
> > sent
> > > > back
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > brokers.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 103. "The active controller assigns the broker a
> > new
> > > > > > broker
> > > > > > > > > > epoch,
> > > > > > > > > > > > based on
> > > > > > > > > > > > > > > the latest committed offset in the log." This
> > seems
> > > > > > > > inaccurate
> > > > > > > > > > since
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > latest committed offset doesn't always advance on
> > > > every
> > > > > > log
> > > > > > > > > > append.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Given that the new broker epoch won't be visible
> > until
> > > > the
> > > > > > > > commit
> > > > > > > > > > has
> > > > > > > > > > > > happened, I have changed this to "the next available
> > > > offset in
> > > > > > the
> > > > > > > > log"
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 104. REGISTERING(1) : It says "Otherwise, the
> > broker
> > > > > > moves
> > > > > > > > into
> > > > > > > > > > the
> > > > > > > > > > > > FENCED
> > > > > > > > > > > > > > > state.". It seems this should be RUNNING?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 105. RUNNING: Should we require the broker to
> > catch
> > > > up
> > > > > > to the
> > > > > > > > > > > > metadata log
> > > > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For 104 and 105, these sections have been reworked.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <
> > > > > > > > cmccabe@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley
> > wrote:
> > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino
> > > > wrote:
> > > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the hard work on
> > this
> > > > KIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I have some questions about what happens
> > to a
> > > > > > broker
> > > > > > > > > > when it
> > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > fenced (e.g. because it can't send a
> > > > heartbeat
> > > > > > > > request to
> > > > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > > > lease). The KIP says "When a broker is
> > > > fenced,
> > > > > > it
> > > > > > > > cannot
> > > > > > > > > > > > process any
> > > > > > > > > > > > > > > > > > > client requests. This prevents brokers
> > which
> > > > > > are not
> > > > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > > > metadata updates or that are not
> > receiving
> > > > and
> > > > > > > > processing
> > > > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > > > enough from causing issues to clients."
> > And
> > > > in
> > > > > > the
> > > > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > > > FENCED(4) state it likewise says "While
> > in
> > > > this
> > > > > > > > state,
> > > > > > > > > > the
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > not respond to client requests." It
> > makes
> > > > sense
> > > > > > > > that a
> > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > > should not accept producer requests -- I
> > > > assume
> > > > > > any
> > > > > > > > such
> > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > > would result in
> > NotLeaderOrFollowerException.
> > > > > > But
> > > > > > > > what
> > > > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > > > (fetch from follower) consumer
> > requests? It
> > > > is
> > > > > > > > > > conceivable
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > could continue. Related to that, would a
> > > > fenced
> > > > > > > > broker
> > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > fetch data for partitions where it
> > thinks it
> > > > is a
> > > > > > > > > > follower?
> > > > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > > > it rejects consumer requests it might
> > still
> > > > > > continue
> > > > > > > > to
> > > > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > > > follower. Might it be helpful to clarify
> > > > both
> > > > > > > > decisions
> > > > > > > > > > > > here?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Good question. I think a fenced broker
> > should
> > > > > > > > continue to
> > > > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > > > partitions it was already fetching before
> > it
> > > > was
> > > > > > > > fenced,
> > > > > > > > > > > > unless it
> > > > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > > > problem. At that point it won't be able to
> > > > > > continue,
> > > > > > > > > > since it
> > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > the new metadata. For example, it won't
> > know
> > > > about
> > > > > > > > > > leadership
> > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the partitions it's fetching. The
> > rationale
> > > > for
> > > > > > > > > > continuing to
> > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > try to avoid disruptions as much as
> > possible.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I don't think fenced brokers should accept
> > > > client
> > > > > > > > requests.
> > > > > > > > > > > > The issue
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > that the fenced broker may or may not have
> > any
> > > > > > data it
> > > > > > > > is
> > > > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > > > have. It may or may not have applied any
> > > > > > configuration
> > > > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > it is supposed to have applied. So it
> > could
> > > > get
> > > > > > pretty
> > > > > > > > > > > > confusing, and
> > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > potentially waste the client's time.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > When fenced, how would the broker reply to a
> > > > client
> > > > > > > > which did
> > > > > > > > > > > > make a
> > > > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The broker will respond with a retryable error
> > in
> > > > that
> > > > > > > > case.
> > > > > > > > > > Once
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > client has re-fetched its metadata, it will no
> > > > longer
> > > > > > see
> > > > > > > > the
> > > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > as part of the cluster. I added a note to the
> > KIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the reply. A few more comments below.
206. "RemoveTopic is the last step, that scrubs all metadata about the
topic. In order to get to that last step, the topic data needs to removed
from all brokers (after each broker notices that the topic is being
deleted)." Currently, this is done based on the response of
StopReplicaRequest. Since the controller no longer sends this request, how
does the controller know that the data for the deleted topic has
been removed in the brokers?
210. Thanks for the explanation. Sounds good to me.
211. I still don't see an example when ShouldFence is set to true in
BrokerHeartbeatReques. Could we add one?
213. The KIP now allows replicas to be assigned on brokers that are fenced,
which is an improvement. How do we permanently remove a broker (e.g.
cluster shrinking) to prevent it from being used for future replica
assignments?
214. "Currently, when a node is down, all of its ZK registration
information is gone. But we need this information in order to understand
things like whether the replicas of a particular partition are
well-balanced across racks." This is not quite true. Currently, even when
ZK registration is gone, the existing replica assignment is still available
in the metadata response.
Jun
On Wed, Dec 16, 2020 at 8:48 AM Colin McCabe <cm...@apache.org> wrote:
> On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the reply. A few more follow up comments.
> >
> > 210. initial.broker.registration.timeout.ms: The default value is 90sec,
> > which seems long. If a broker fails the registration because of incorrect
> > configs, we want to fail faster. In comparison, the defaults for
> > zookeeper.connection.timeout.ms is 18 secs.
> >
>
> Hi Jun,
>
> I agree that the initial connection timeout here is longer than what we
> had with ZK. The main reason I selected a slightly longer timeout here is
> to handle the case where the controllers and the brokers are co-located.
> For example, if you have a 3 node cluster and all three nodes are
> controllers+brokers, when you first bring up the cluster, we will have to
> stand up the controller quorum and then handle broker registrations.
> Although we believe Raft will start up relatively quickly, it's good to
> leave some extra margin here.
>
> I don't think there's a big disadvantage to having a slightly longer
> timeout here. After all, starting up the brokers with no controllers is an
> admin mistake, which we don't expect to see very often. Maybe let's set it
> to 60 seconds for now and maybe see if we can tweak it in the future if
> that turns out to be too long or short?
>
> > 211. "Once they are all moved, the controller responds to the heartbeat
> > with a nextState of SHUTDOWN." It seems that nextState is no longer
> > relevant. Also, could you add a bit more explanation on when ShouldFence
> is
> > set to true in BrokerHeartbeatRequest?
> >
>
> Good catch. I removed the obsolete section referring to nextState and
> added a reference to the new boolean. I also added some more information
> about ShouldFence and about the rationale for separating fencing from
> registration.
>
> > 212. Related to the above, what does the broker do if IsFenced is true in
> > the BrokerHeartbeatResponse? Will the broker do the same thing on
> receiving
> > a FenceBrokerRecord from the metadata log?
> >
>
> The broker only checks this boolean during the startup process. After the
> startup process is finished, it ignores this boolean.
>
> The broker uses fence / unfence records in the metadata log to determine
> which brokers should appear in its MetadataResponse.
>
> best,
> Colin
>
> > Jun
> >
> > On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > > Hi Colin,
> > > >
> > > > The KIP says that "brokers which are fenced will not appear in
> > > > MetadataResponses. So clients that have up-to-date metadata will
> not try
> > > > to contact fenced brokers.", which is fine, but it doesn't seem to
> > > > elaborate on what happens for clients which try to contact a broker
> > > (using
> > > > stale metadata, for example) and find it in the registered-but-fenced
> > > > state.
> > >
> > > Hi Tom,
> > >
> > > I have dropped the broker-side fencing from this proposal. So now the
> > > fencing is basically the same as today's fencing: it's controller-side
> > > only. That means that clients using stale metadata can contact fenced
> > > brokers and communicate with them. The only case where the broker
> will be
> > > inaccessible is when it hasn't finished starting yet (i.e., has not yet
> > > attained RUNNING state.)
> > >
> > > Just like today, we expect the safeguards built into the replication
> > > protocol to prevent the worst corner cases that could result from
> this. I
> > > do think we will probably take up this issue later and find a better
> > > solution for client-side metadata, but this KIP is big enough as-is.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > > You said previously that the broker will respond with a retriable
> > > > error, which is again fine, but you didn't answer my question about
> which
> > > > error code you've chosen for this. I realise that this doesn't really
> > > > affect the correctness of the behaviour, but I'm not aware of any
> > > existing
> > > > error code which is a good fit. So it would be good to understand
> about
> > > how
> > > > you're making this backward compatible for clients.
> > > >
> > > > Many thanks,
> > > >
> > > > Tom
> > > >
> > > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the reply. Just a couple of more comments below.
> > > > > >
> > > > > > 210. Since we are deprecating zookeeper.connection.timeout.ms,
> > > should we
> > > > > > add a new config to bound the time for a broker to connect to the
> > > > > > controller during starting up?
> > > > > >
> > > > >
> > > > > Good idea. I added initial.broker.registration.timeout.ms for
> this.
> > > > >
> > > > > > 211. BrokerHeartbeat no longer has the state field in the
> > > > > request/response.
> > > > > > However, (a) the controller shutdown section still has "In its
> > > periodic
> > > > > > heartbeats, the broker asks the controller if it can transition
> into
> > > the
> > > > > > SHUTDOWN state. This motivates the controller to move all of the
> > > leaders
> > > > > > off of that broker. Once they are all moved, the controller
> > > responds to
> > > > > > the heartbeat with a nextState of SHUTDOWN."; (2) the
> description of
> > > > > > BrokerHeartbeat still references currentState and targetState.
> > > > > >
> > > > >
> > > > > Thanks. I've made these sections clearer and removed the obsolete
> > > > > references to sending states.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <cmccabe@apache.org
> >
> > > wrote:
> > > > > >
> > > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the update. A few more follow up comments.
> > > > > > > >
> > > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > Thanks again for the review.
> > > > > > >
> > > > > > > > 100. FailedReplicaRecord: Since this is reported by each
> broker
> > > > > > > > independently, perhaps we could use a more concise
> representation
> > > > > that
> > > > > > > has
> > > > > > > > a top level broker field, an array of topics, which has an
> array
> > > of
> > > > > > > > partitions.
> > > > > > > >
> > > > > > >
> > > > > > > The issue is that there is a size limit on the each record.
> > > Putting
> > > > > all
> > > > > > > of the partitions of a log directory into a single record would
> > > > > probably
> > > > > > > break that in many cases. Still, we can optimize a bit by
> having
> > > an
> > > > > array
> > > > > > > of partition IDs, since nearly all the time, we have more than
> one
> > > > > from the
> > > > > > > same topic.
> > > > > > >
> > > > > > > > 200. Sounds good. If we remove the broker-side fencing
> logic, do
> > > we
> > > > > plan
> > > > > > > to
> > > > > > > > still keep FENCED in broker state? Do we plan to expose the
> new
> > > > > states
> > > > > > > > through the existing BrokerState metric and if so, what are
> the
> > > > > values
> > > > > > > for
> > > > > > > > the new states?
> > > > > > > >
> > > > > > >
> > > > > > > No, we don't need FENCED any more. I have removed it from the
> KIP.
> > > > > > >
> > > > > > > The new states are very similar to the current ones, actually.
> > > There
> > > > > are
> > > > > > > no new states or removed ones. The main change in the broker
> state
> > > > > machine
> > > > > > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state has been
> > > renamed to
> > > > > > > RECOVERY. Also, unlike previously, the broker will always pass
> > > through
> > > > > > > RECOVERY (although it may only stay in this state for a very
> short
> > > > > amount
> > > > > > > of time).
> > > > > > >
> > > > > > > > 201. This may be fine too. Could we document what happens
> when
> > > the
> > > > > > > > broker.id/controller.id in metadata.properties don't match
> the
> > > > > broker
> > > > > > > > config when the broker starts up?
> > > > > > > >
> > > > > > >
> > > > > > > I added some documentation about this.
> > > > > > >
> > > > > > > > 204. There is still "The highest metadata offset which the
> broker
> > > > > has not
> > > > > > > > reached" referenced under BrokerRegistration.
> > > > > > > >
> > > > > > >
> > > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > > >
> > > > > > > > 206. Is that separate step needed given KIP-516? With
> KIP-516 (
> > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > > ),
> > > > > > > > we don't need to wait for the topic data to be removed from
> all
> > > > > brokers
> > > > > > > > before removing the topic metadata. The combination of
> unmatching
> > > > > > > > topicId
> > > > > > > > or the missing topicId from the metadata is enough for the
> > > broker to
> > > > > > > > clean
> > > > > > > > up deleted topics asynchronously.
> > > > > > >
> > > > > > > It won't be needed once KIP-516 is adopted, but this hasn't
> been
> > > > > > > implemented yet.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <
> cmccabe@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > > > > > > > Hi, Colin,
> > > > > > > > > >
> > > > > > > > > > Thanks for the updated KIP. A few more comments below.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > Thanks again for the reviews.
> > > > > > > > >
> > > > > > > > > > 80.2 For deprecated configs, we need to include
> zookeeper.*
> > > and
> > > > > > > > > > broker.id.generation.enable.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Added.
> > > > > > > > >
> > > > > > > > > > 83.1 If a broker is down, does the controller keep the
> > > previously
> > > > > > > > > > registered broker epoch forever? If not, how long does
> the
> > > > > controller
> > > > > > > > > keep
> > > > > > > > > > it? What does the controller do when receiving a broker
> > > heartbeat
> > > > > > > request
> > > > > > > > > > with an unfound broker epoch?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes, the controller keeps the previous registration
> forever.
> > > > > > > > >
> > > > > > > > > Broker heartbeat requests with an incorrect broker epoch
> will
> > > be
> > > > > > > rejected
> > > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > > >
> > > > > > > > > > 100. Have you figured out if we need to add a new record
> > > type for
> > > > > > > > > reporting
> > > > > > > > > > partitions on failed disks?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I added FailedReplicaRecord to reflect the case where a
> JBOD
> > > > > directory
> > > > > > > has
> > > > > > > > > failed, leading to failed replicas.
> > > > > > > > >
> > > > > > > > > > 102. For debugging purposes, sometimes it's useful to
> read
> > > the
> > > > > > > metadata
> > > > > > > > > > topic using tools like console-consumer. Should we
> support
> > > that
> > > > > and
> > > > > > > if
> > > > > > > > > so,
> > > > > > > > > > how?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > For now, we have the ability to read the metadata logs
> with the
> > > > > > > dump-logs
> > > > > > > > > tool. I think we will come up with some other tools in the
> > > future
> > > > > as
> > > > > > > we
> > > > > > > > > get experience.
> > > > > > > > >
> > > > > > > > > > 200. "brokers which are fenced will not appear in
> > > > > MetadataResponses.
> > > > > > > The
> > > > > > > > > > broker will not respond to these requests-- instead, it
> will
> > > > > simply
> > > > > > > > > > disconnect." If the controller is partitioned off from
> the
> > > > > brokers,
> > > > > > > this
> > > > > > > > > > design will cause every broker to stop accepting new
> client
> > > > > > > requests. In
> > > > > > > > > > contrast, if ZK is partitioned off, the existing
> behavior is
> > > > > that the
> > > > > > > > > > brokers can continue to work based on the last known
> > > metadata.
> > > > > So, I
> > > > > > > am
> > > > > > > > > not
> > > > > > > > > > sure if we should change the existing behavior because
> of the
> > > > > bigger
> > > > > > > > > impact
> > > > > > > > > > in the new one. Another option is to keep the existing
> > > behavior
> > > > > and
> > > > > > > > > expose
> > > > > > > > > > a metric for fenced brokers so that the operator could be
> > > > > alerted.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I'm skeptical about how well running without ZK currently
> > > works.
> > > > > > > However,
> > > > > > > > > I will move the broker-side fencing into a follow-up KIP.
> This
> > > > > KIP is
> > > > > > > > > already pretty large and there is no hard dependency on
> this.
> > > > > There
> > > > > > > may
> > > > > > > > > also be other ways of accomplishing the positive effects of
> > > what
> > > > > > > > > broker-side fencing, so more discussion is needed.
> > > > > > > > >
> > > > > > > > > > 201. I read Ron's comment, but I am still not sure the
> > > benefit of
> > > > > > > keeping
> > > > > > > > > > broker.id and controller.id in meta.properties. It seems
> > > that
> > > > > we are
> > > > > > > > > just
> > > > > > > > > > duplicating the same info in two places and have the
> > > additional
> > > > > > > burden of
> > > > > > > > > > making sure the values in the two places are consistent.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I think the reasoning is that having broker.id protects us
> > > against
> > > > > > > > > accidentally bringing up a broker with a disk from a
> different
> > > > > > > broker. I
> > > > > > > > > don't feel strongly about this but it seemed simpler to
> keep
> > > it.
> > > > > > > > >
> > > > > > > > > > 202. controller.connect.security.protocol: Is this needed
> > > since
> > > > > > > > > > controller.listener.names and
> listener.security.protocol.map
> > > > > imply
> > > > > > > the
> > > > > > > > > > security protocol already?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > You're right, this isn't needed. I'll remove it.
> > > > > > > > >
> > > > > > > > > > 203. registration.heartbeat.interval.ms: It defaults to
> 2k.
> > > ZK
> > > > > uses
> > > > > > > 1/3
> > > > > > > > > of
> > > > > > > > > > the session timeout for heartbeat. So, given the default
> 18k
> > > for
> > > > > > > > > > registration.lease.timeout.ms, should we default
> > > > > > > > > > registration.heartbeat.interval.ms to 6k?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > 6 seconds seems like a pretty long time between
> heartbeats. It
> > > > > might
> > > > > > > be
> > > > > > > > > useful to know when a broker is missing heartbeats, with
> less
> > > time
> > > > > than
> > > > > > > > > that. I provisionally set it to 3 seconds (we can always
> > > change
> > > > > > > later...)
> > > > > > > > >
> > > > > > > > > I also changed the name of these configurations to "
> > > > > > > > > broker.heartbeat.interval.ms" and "
> > > broker.registration.timeout.ms"
> > > > > to
> > > > > > > try
> > > > > > > > > to clarify them a bit.
> > > > > > > > >
> > > > > > > > > > 204. "The highest metadata offset which the broker has
> not
> > > > > reached."
> > > > > > > It
> > > > > > > > > > seems this should be "has reached".
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I changed this to "one more than the highest metadata
> offset
> > > which
> > > > > the
> > > > > > > > > broker has reached."
> > > > > > > > >
> > > > > > > > > > 205. UnfenceBrokerRecord and UnregisterBrokerRecord: To
> me,
> > > they
> > > > > > > seem to
> > > > > > > > > be
> > > > > > > > > > the same. Do we need both?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Unregistration means that the broker has been removed from
> the
> > > > > cluster.
> > > > > > > > > That is different than unfencing, which marks the broker as
> > > active.
> > > > > > > > >
> > > > > > > > > > 206. TopicRecord: The Deleting field is used to indicate
> > > that the
> > > > > > > topic
> > > > > > > > > is
> > > > > > > > > > being deleted. I am wondering if this is really needed
> since
> > > > > > > RemoveTopic
> > > > > > > > > > already indicates the same thing.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > RemoveTopic is the last step, that scrubs all metadata
> about
> > > the
> > > > > topic.
> > > > > > > > > In order to get to that last step, the topic data needs to
> > > removed
> > > > > > > from all
> > > > > > > > > brokers (after each broker notices that the topic is being
> > > > > deleted).
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <
> > > cmccabe@apache.org>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > > > > > > > > > > > Hi Colin. Thanks for the updates. It's now clear
> to me
> > > that
> > > > > > > brokers
> > > > > > > > > > > > keep their broker epoch for the life of their JVM --
> they
> > > > > > > register
> > > > > > > > > > > > once, get their broker epoch in the response, and
> then
> > > never
> > > > > > > > > > > > re-register again. Brokers may get fenced, but they
> > > keep the
> > > > > > > same
> > > > > > > > > > > > broker epoch for the life of their JVM. The
> incarnation
> > > ID
> > > > > is
> > > > > > > also
> > > > > > > > > > > > kept for the life of the JVM but is generated by the
> > > broker
> > > > > > > itself
> > > > > > > > > > > > upon startup, and the combination of the two allows
> the
> > > > > > > Controller to
> > > > > > > > > > > > act idempotently if any previously-sent registration
> > > response
> > > > > > > gets
> > > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks, Ron. That's a good summary.
> > > > > > > > > > >
> > > > > > > > > > > > One thing I wonder about is if it might be helpful
> for
> > > the
> > > > > > > broker to
> > > > > > > > > > > > send the Cluster ID as determined from its
> > > meta.properties
> > > > > file
> > > > > > > in
> > > > > > > > > its
> > > > > > > > > > > > registration request. Does it even make sense for
> the
> > > > > broker to
> > > > > > > > > > > > successfully register and enter the Fenced state if
> it
> > > has
> > > > > the
> > > > > > > wrong
> > > > > > > > > > > > Cluster ID?
> > > > > > > > > > >
> > > > > > > > > > > Yeah, that's a good idea. Let's have the broker pass
> its
> > > > > cluster
> > > > > > > ID in
> > > > > > > > > > > the registration RPC, and then registration can fail
> if the
> > > > > broker
> > > > > > > is
> > > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > > >
> > > > > > > > > > > > The nextMetadatOffset value that the broker
> communicates
> > > > > > > > > > > > in its registration request only has meaning within
> the
> > > > > correct
> > > > > > > > > > > > cluster, so it feels to me that the Controller should
> > > have
> > > > > some
> > > > > > > way
> > > > > > > > > to
> > > > > > > > > > > > perform this sanity check. There is currently
> (pre-KIP
> > > 500)
> > > > > a
> > > > > > > check
> > > > > > > > > > > > in the broker to make sure its configured cluster ID
> > > matches
> > > > > the
> > > > > > > one
> > > > > > > > > > > > stored in ZooKeeper, and we will have to perform this
> > > > > validation
> > > > > > > > > > > > somewhere in the KIP-500 world. If the Controller
> > > doesn't
> > > > > do it
> > > > > > > > > > > > within the registration request then the broker will
> > > have to
> > > > > > > make a
> > > > > > > > > > > > metadata request to the Controller, retrieve the
> Cluster
> > > ID,
> > > > > and
> > > > > > > > > > > > perform the check itself. It feels to me that it
> might
> > > be
> > > > > > > better for
> > > > > > > > > > > > the Controller to just do it, and then the broker
> doesn't
> > > > > have to
> > > > > > > > > > > > worry about it anymore once it successfully
> registers.
> > > > > > > > > > > >
> > > > > > > > > > > > I also have a question about the broker.id value and
> > > > > > > > > meta.properties.
> > > > > > > > > > > > The KIP now says "In version 0 of meta.properties,
> there
> > > is a
> > > > > > > > > > > > broker.id field. Version 1 does not have this
> field.
> > > It
> > > > > is no
> > > > > > > > > longer
> > > > > > > > > > > > needed because we no longer support dynamic broker id
> > > > > > > assignment."
> > > > > > > > > > > > But then there is an example version 1
> meta.properties
> > > file
> > > > > that
> > > > > > > > > shows
> > > > > > > > > > > > the broker.id value. I actually wonder if maybe the
> > > > > broker.id
> > > > > > > value
> > > > > > > > > > > > would be good to keep in the version 1
> meta.properties
> > > file
> > > > > > > because
> > > > > > > > > it
> > > > > > > > > > > > currently (pre-KIP 500, version 0) acts as a sanity
> > > check to
> > > > > make
> > > > > > > > > sure
> > > > > > > > > > > > the broker is using the correct log directory.
> Similarly
> > > > > with
> > > > > > > the
> > > > > > > > > > > > controller.id value on controllers -- it would
> allow the
> > > > > same
> > > > > > > type
> > > > > > > > > of
> > > > > > > > > > > > sanity check for quorum controllers.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > That's a good point. I will add broker.id back, and
> also
> > > add
> > > > > > > > > > > controller.id as a possibility.
> > > > > > > > > > >
> > > > > > > > > > > cheers,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <
> > > > > cmccabe@apache.org
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the reply. A few more comments.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks again for the reply. Sorry for the long
> > > hiatus. I
> > > > > was
> > > > > > > on
> > > > > > > > > > > vacation for a while.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 55. There is still text that favors new broker
> > > > > registration.
> > > > > > > > > "When a
> > > > > > > > > > > broker
> > > > > > > > > > > > > > first starts up, when it is in the INITIAL
> state, it
> > > will
> > > > > > > always
> > > > > > > > > > > "win"
> > > > > > > > > > > > > > broker ID conflicts. However, once it is
> granted a
> > > > > lease, it
> > > > > > > > > > > transitions
> > > > > > > > > > > > > > out of the INITIAL state. Thereafter, it may
> lose
> > > > > subsequent
> > > > > > > > > > > conflicts if
> > > > > > > > > > > > > > its broker epoch is stale. (See KIP-380 for some
> > > > > background
> > > > > > > on
> > > > > > > > > > > broker
> > > > > > > > > > > > > > epoch.) The reason for favoring new processes
> is to
> > > > > > > accommodate
> > > > > > > > > the
> > > > > > > > > > > common
> > > > > > > > > > > > > > case where a process is killed with kill -9 and
> then
> > > > > > > restarted.
> > > > > > > > > We
> > > > > > > > > > > want it
> > > > > > > > > > > > > > to be able to reclaim its old ID quickly in this
> > > case."
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the reminder. I have clarified the
> language
> > > > > here.
> > > > > > > > > > > Hopefully now it is clear that we don't allow quick
> re-use
> > > of
> > > > > > > broker
> > > > > > > > > IDs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 80.1 Sounds good. Could you document that
> listeners
> > > is a
> > > > > > > required
> > > > > > > > > > > config
> > > > > > > > > > > > > > now? It would also be useful to annotate other
> > > required
> > > > > > > configs.
> > > > > > > > > For
> > > > > > > > > > > > > > example, controller.connect should be required.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I added a note specifying that these are required.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 80.2 Could you list all deprecated existing
> configs?
> > > > > Another
> > > > > > > one
> > > > > > > > > is
> > > > > > > > > > > > > > control.plane.listener.name since the
> controller no
> > > > > longer
> > > > > > > sends
> > > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata and StopReplica
> > > requests.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I added a section specifying some deprecated
> configs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 83.1 It seems that the broker can transition from
> > > FENCED
> > > > > to
> > > > > > > > > RUNNING
> > > > > > > > > > > without
> > > > > > > > > > > > > > registering for a new broker epoch. I am not
> sure how
> > > > > this
> > > > > > > works.
> > > > > > > > > > > Once the
> > > > > > > > > > > > > > controller fences a broker, there is no need for
> the
> > > > > > > controller
> > > > > > > > > to
> > > > > > > > > > > keep the
> > > > > > > > > > > > > > boker epoch around. So, if the fenced broker's
> > > heartbeat
> > > > > > > request
> > > > > > > > > > > with the
> > > > > > > > > > > > > > existing broker epoch will be rejected, leading
> the
> > > > > broker
> > > > > > > back
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > The broker epoch refers to the broker registration.
> > > So we
> > > > > DO
> > > > > > > keep
> > > > > > > > > the
> > > > > > > > > > > broker epoch around even while the broker is fenced.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The broker epoch changes only when there is a new
> > > broker
> > > > > > > > > > > registration. Fencing or unfencing the broker doesn't
> > > change
> > > > > the
> > > > > > > > > broker
> > > > > > > > > > > epoch.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 83.5 Good point on KIP-590. Then should we
> expose the
> > > > > > > controller
> > > > > > > > > for
> > > > > > > > > > > > > > debugging purposes? If not, we should deprecate
> the
> > > > > > > controllerID
> > > > > > > > > > > field in
> > > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think it's OK to expose it for now, with the
> proviso
> > > > > that it
> > > > > > > > > won't
> > > > > > > > > > > be reachable by clients.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 90. We rejected the shared ID with just one
> reason
> > > "This
> > > > > is
> > > > > > > not a
> > > > > > > > > > > good idea
> > > > > > > > > > > > > > because NetworkClient assumes a single ID
> space. So
> > > if
> > > > > > > there is
> > > > > > > > > > > both a
> > > > > > > > > > > > > > controller 1 and a broker 1, we don't have a way
> of
> > > > > picking
> > > > > > > the
> > > > > > > > > > > "right"
> > > > > > > > > > > > > > one." This doesn't seem to be a strong reason.
> For
> > > > > example,
> > > > > > > we
> > > > > > > > > could
> > > > > > > > > > > > > > address the NetworkClient issue with the node
> type
> > > as you
> > > > > > > pointed
> > > > > > > > > > > out or
> > > > > > > > > > > > > > using the negative value of a broker ID as the
> > > > > controller ID.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > It would require a lot of code changes to support
> > > multiple
> > > > > > > types of
> > > > > > > > > > > node IDs. It's not clear to me that the end result
> would
> > > be
> > > > > > > better --
> > > > > > > > > I
> > > > > > > > > > > tend to think it would be worse, since it would be more
> > > > > complex.
> > > > > > > In a
> > > > > > > > > > > similar vein, using negative numbers seems dangerous,
> > > since we
> > > > > use
> > > > > > > > > > > negatives or -1 as "special values" in many places.
> For
> > > > > example,
> > > > > > > -1
> > > > > > > > > often
> > > > > > > > > > > represents "no such node."
> > > > > > > > > > > > >
> > > > > > > > > > > > > One important thing to keep in mind is that we
> want to
> > > be
> > > > > able
> > > > > > > to
> > > > > > > > > > > transition from a broker and a controller being
> co-located
> > > to
> > > > > them
> > > > > > > no
> > > > > > > > > > > longer being co-located. This is much easier to do
> when
> > > they
> > > > > have
> > > > > > > > > separate
> > > > > > > > > > > IDs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > > <
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > > >,
> > > > > > > > > > > > > > the broker reports all offline replicas due to a
> disk
> > > > > > > failure to
> > > > > > > > > the
> > > > > > > > > > > > > > controller. It seems this information needs to be
> > > > > persisted
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > > metadata
> > > > > > > > > > > > > > log. Do we have a corresponding record for that?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hmm, I have to look into this a little bit more.
> We
> > > may
> > > > > need
> > > > > > > a new
> > > > > > > > > > > record type.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 101. Currently, StopReplica request has 2 modes,
> > > without
> > > > > > > deletion
> > > > > > > > > > > and with
> > > > > > > > > > > > > > deletion. The former is used for controlled
> shutdown
> > > and
> > > > > > > handling
> > > > > > > > > > > disk
> > > > > > > > > > > > > > failure, and causes the follower to stop. The
> latter
> > > is
> > > > > for
> > > > > > > topic
> > > > > > > > > > > deletion
> > > > > > > > > > > > > > and partition reassignment, and causes the
> replica
> > > to be
> > > > > > > deleted.
> > > > > > > > > > > Since we
> > > > > > > > > > > > > > are deprecating StopReplica, could we document
> what
> > > > > triggers
> > > > > > > the
> > > > > > > > > > > stopping
> > > > > > > > > > > > > > of a follower and the deleting of a replica now?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > RemoveTopic triggers deletion. In general the
> > > > > functionality of
> > > > > > > > > > > StopReplica is subsumed by the metadata records.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 102. Should we include the metadata topic in the
> > > > > > > > > MetadataResponse?
> > > > > > > > > > > If so,
> > > > > > > > > > > > > > when it will be included and what will the
> metadata
> > > > > response
> > > > > > > look
> > > > > > > > > > > like?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > No, it won't be included in the metadata response
> sent
> > > back
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > brokers.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 103. "The active controller assigns the broker a
> new
> > > > > broker
> > > > > > > > > epoch,
> > > > > > > > > > > based on
> > > > > > > > > > > > > > the latest committed offset in the log." This
> seems
> > > > > > > inaccurate
> > > > > > > > > since
> > > > > > > > > > > the
> > > > > > > > > > > > > > latest committed offset doesn't always advance on
> > > every
> > > > > log
> > > > > > > > > append.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Given that the new broker epoch won't be visible
> until
> > > the
> > > > > > > commit
> > > > > > > > > has
> > > > > > > > > > > happened, I have changed this to "the next available
> > > offset in
> > > > > the
> > > > > > > log"
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 104. REGISTERING(1) : It says "Otherwise, the
> broker
> > > > > moves
> > > > > > > into
> > > > > > > > > the
> > > > > > > > > > > FENCED
> > > > > > > > > > > > > > state.". It seems this should be RUNNING?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 105. RUNNING: Should we require the broker to
> catch
> > > up
> > > > > to the
> > > > > > > > > > > metadata log
> > > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > > >
> > > > > > > > > > > > > For 104 and 105, these sections have been reworked.
> > > > > > > > > > > > >
> > > > > > > > > > > > > best,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <
> > > > > > > cmccabe@apache.org
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley
> wrote:
> > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino
> > > wrote:
> > > > > > > > > > > > > > > > > > Hi Colin. Thanks for the hard work on
> this
> > > KIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I have some questions about what happens
> to a
> > > > > broker
> > > > > > > > > when it
> > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > fenced (e.g. because it can't send a
> > > heartbeat
> > > > > > > request to
> > > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > > lease). The KIP says "When a broker is
> > > fenced,
> > > > > it
> > > > > > > cannot
> > > > > > > > > > > process any
> > > > > > > > > > > > > > > > > > client requests. This prevents brokers
> which
> > > > > are not
> > > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > > metadata updates or that are not
> receiving
> > > and
> > > > > > > processing
> > > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > > enough from causing issues to clients."
> And
> > > in
> > > > > the
> > > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > > FENCED(4) state it likewise says "While
> in
> > > this
> > > > > > > state,
> > > > > > > > > the
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > not respond to client requests." It
> makes
> > > sense
> > > > > > > that a
> > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > > should not accept producer requests -- I
> > > assume
> > > > > any
> > > > > > > such
> > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > would result in
> NotLeaderOrFollowerException.
> > > > > But
> > > > > > > what
> > > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > > (fetch from follower) consumer
> requests? It
> > > is
> > > > > > > > > conceivable
> > > > > > > > > > > that
> > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > could continue. Related to that, would a
> > > fenced
> > > > > > > broker
> > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > fetch data for partitions where it
> thinks it
> > > is a
> > > > > > > > > follower?
> > > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > > it rejects consumer requests it might
> still
> > > > > continue
> > > > > > > to
> > > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > > follower. Might it be helpful to clarify
> > > both
> > > > > > > decisions
> > > > > > > > > > > here?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Good question. I think a fenced broker
> should
> > > > > > > continue to
> > > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > > partitions it was already fetching before
> it
> > > was
> > > > > > > fenced,
> > > > > > > > > > > unless it
> > > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > > problem. At that point it won't be able to
> > > > > continue,
> > > > > > > > > since it
> > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > the new metadata. For example, it won't
> know
> > > about
> > > > > > > > > leadership
> > > > > > > > > > > changes
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the partitions it's fetching. The
> rationale
> > > for
> > > > > > > > > continuing to
> > > > > > > > > > > fetch
> > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > try to avoid disruptions as much as
> possible.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I don't think fenced brokers should accept
> > > client
> > > > > > > requests.
> > > > > > > > > > > The issue
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > that the fenced broker may or may not have
> any
> > > > > data it
> > > > > > > is
> > > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > > have. It may or may not have applied any
> > > > > configuration
> > > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > it is supposed to have applied. So it
> could
> > > get
> > > > > pretty
> > > > > > > > > > > confusing, and
> > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > potentially waste the client's time.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > When fenced, how would the broker reply to a
> > > client
> > > > > > > which did
> > > > > > > > > > > make a
> > > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The broker will respond with a retryable error
> in
> > > that
> > > > > > > case.
> > > > > > > > > Once
> > > > > > > > > > > the
> > > > > > > > > > > > > > > client has re-fetched its metadata, it will no
> > > longer
> > > > > see
> > > > > > > the
> > > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > as part of the cluster. I added a note to the
> KIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Tue, Dec 15, 2020, at 13:08, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the reply. A few more follow up comments.
>
> 210. initial.broker.registration.timeout.ms: The default value is 90sec,
> which seems long. If a broker fails the registration because of incorrect
> configs, we want to fail faster. In comparison, the defaults for
> zookeeper.connection.timeout.ms is 18 secs.
>
Hi Jun,
I agree that the initial connection timeout here is longer than what we had with ZK. The main reason I selected a slightly longer timeout here is to handle the case where the controllers and the brokers are co-located. For example, if you have a 3 node cluster and all three nodes are controllers+brokers, when you first bring up the cluster, we will have to stand up the controller quorum and then handle broker registrations. Although we believe Raft will start up relatively quickly, it's good to leave some extra margin here.
I don't think there's a big disadvantage to having a slightly longer timeout here. After all, starting up the brokers with no controllers is an admin mistake, which we don't expect to see very often. Maybe let's set it to 60 seconds for now and maybe see if we can tweak it in the future if that turns out to be too long or short?
> 211. "Once they are all moved, the controller responds to the heartbeat
> with a nextState of SHUTDOWN." It seems that nextState is no longer
> relevant. Also, could you add a bit more explanation on when ShouldFence is
> set to true in BrokerHeartbeatRequest?
>
Good catch. I removed the obsolete section referring to nextState and added a reference to the new boolean. I also added some more information about ShouldFence and about the rationale for separating fencing from registration.
> 212. Related to the above, what does the broker do if IsFenced is true in
> the BrokerHeartbeatResponse? Will the broker do the same thing on receiving
> a FenceBrokerRecord from the metadata log?
>
The broker only checks this boolean during the startup process. After the startup process is finished, it ignores this boolean.
The broker uses fence / unfence records in the metadata log to determine which brokers should appear in its MetadataResponse.
best,
Colin
> Jun
>
> On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <cm...@apache.org> wrote:
>
> > On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > > Hi Colin,
> > >
> > > The KIP says that "brokers which are fenced will not appear in
> > > MetadataResponses. So clients that have up-to-date metadata will not try
> > > to contact fenced brokers.", which is fine, but it doesn't seem to
> > > elaborate on what happens for clients which try to contact a broker
> > (using
> > > stale metadata, for example) and find it in the registered-but-fenced
> > > state.
> >
> > Hi Tom,
> >
> > I have dropped the broker-side fencing from this proposal. So now the
> > fencing is basically the same as today's fencing: it's controller-side
> > only. That means that clients using stale metadata can contact fenced
> > brokers and communicate with them. The only case where the broker will be
> > inaccessible is when it hasn't finished starting yet (i.e., has not yet
> > attained RUNNING state.)
> >
> > Just like today, we expect the safeguards built into the replication
> > protocol to prevent the worst corner cases that could result from this. I
> > do think we will probably take up this issue later and find a better
> > solution for client-side metadata, but this KIP is big enough as-is.
> >
> > best,
> > Colin
> >
> >
> > > You said previously that the broker will respond with a retriable
> > > error, which is again fine, but you didn't answer my question about which
> > > error code you've chosen for this. I realise that this doesn't really
> > > affect the correctness of the behaviour, but I'm not aware of any
> > existing
> > > error code which is a good fit. So it would be good to understand about
> > how
> > > you're making this backward compatible for clients.
> > >
> > > Many thanks,
> > >
> > > Tom
> > >
> > > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the reply. Just a couple of more comments below.
> > > > >
> > > > > 210. Since we are deprecating zookeeper.connection.timeout.ms,
> > should we
> > > > > add a new config to bound the time for a broker to connect to the
> > > > > controller during starting up?
> > > > >
> > > >
> > > > Good idea. I added initial.broker.registration.timeout.ms for this.
> > > >
> > > > > 211. BrokerHeartbeat no longer has the state field in the
> > > > request/response.
> > > > > However, (a) the controller shutdown section still has "In its
> > periodic
> > > > > heartbeats, the broker asks the controller if it can transition into
> > the
> > > > > SHUTDOWN state. This motivates the controller to move all of the
> > leaders
> > > > > off of that broker. Once they are all moved, the controller
> > responds to
> > > > > the heartbeat with a nextState of SHUTDOWN."; (2) the description of
> > > > > BrokerHeartbeat still references currentState and targetState.
> > > > >
> > > >
> > > > Thanks. I've made these sections clearer and removed the obsolete
> > > > references to sending states.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > > Jun
> > > > >
> > > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the update. A few more follow up comments.
> > > > > > >
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Thanks again for the review.
> > > > > >
> > > > > > > 100. FailedReplicaRecord: Since this is reported by each broker
> > > > > > > independently, perhaps we could use a more concise representation
> > > > that
> > > > > > has
> > > > > > > a top level broker field, an array of topics, which has an array
> > of
> > > > > > > partitions.
> > > > > > >
> > > > > >
> > > > > > The issue is that there is a size limit on the each record.
> > Putting
> > > > all
> > > > > > of the partitions of a log directory into a single record would
> > > > probably
> > > > > > break that in many cases. Still, we can optimize a bit by having
> > an
> > > > array
> > > > > > of partition IDs, since nearly all the time, we have more than one
> > > > from the
> > > > > > same topic.
> > > > > >
> > > > > > > 200. Sounds good. If we remove the broker-side fencing logic, do
> > we
> > > > plan
> > > > > > to
> > > > > > > still keep FENCED in broker state? Do we plan to expose the new
> > > > states
> > > > > > > through the existing BrokerState metric and if so, what are the
> > > > values
> > > > > > for
> > > > > > > the new states?
> > > > > > >
> > > > > >
> > > > > > No, we don't need FENCED any more. I have removed it from the KIP.
> > > > > >
> > > > > > The new states are very similar to the current ones, actually.
> > There
> > > > are
> > > > > > no new states or removed ones. The main change in the broker state
> > > > machine
> > > > > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state has been
> > renamed to
> > > > > > RECOVERY. Also, unlike previously, the broker will always pass
> > through
> > > > > > RECOVERY (although it may only stay in this state for a very short
> > > > amount
> > > > > > of time).
> > > > > >
> > > > > > > 201. This may be fine too. Could we document what happens when
> > the
> > > > > > > broker.id/controller.id in metadata.properties don't match the
> > > > broker
> > > > > > > config when the broker starts up?
> > > > > > >
> > > > > >
> > > > > > I added some documentation about this.
> > > > > >
> > > > > > > 204. There is still "The highest metadata offset which the broker
> > > > has not
> > > > > > > reached" referenced under BrokerRegistration.
> > > > > > >
> > > > > >
> > > > > > It should be CurrentMetadataOffset. Fixed.
> > > > > >
> > > > > > > 206. Is that separate step needed given KIP-516? With KIP-516 (
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > > ),
> > > > > > > we don't need to wait for the topic data to be removed from all
> > > > brokers
> > > > > > > before removing the topic metadata. The combination of unmatching
> > > > > > > topicId
> > > > > > > or the missing topicId from the metadata is enough for the
> > broker to
> > > > > > > clean
> > > > > > > up deleted topics asynchronously.
> > > > > >
> > > > > > It won't be needed once KIP-516 is adopted, but this hasn't been
> > > > > > implemented yet.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <cm...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > > > > > > Hi, Colin,
> > > > > > > > >
> > > > > > > > > Thanks for the updated KIP. A few more comments below.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > Thanks again for the reviews.
> > > > > > > >
> > > > > > > > > 80.2 For deprecated configs, we need to include zookeeper.*
> > and
> > > > > > > > > broker.id.generation.enable.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Added.
> > > > > > > >
> > > > > > > > > 83.1 If a broker is down, does the controller keep the
> > previously
> > > > > > > > > registered broker epoch forever? If not, how long does the
> > > > controller
> > > > > > > > keep
> > > > > > > > > it? What does the controller do when receiving a broker
> > heartbeat
> > > > > > request
> > > > > > > > > with an unfound broker epoch?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Yes, the controller keeps the previous registration forever.
> > > > > > > >
> > > > > > > > Broker heartbeat requests with an incorrect broker epoch will
> > be
> > > > > > rejected
> > > > > > > > with STALE_BROKER_EPOCH.
> > > > > > > >
> > > > > > > > > 100. Have you figured out if we need to add a new record
> > type for
> > > > > > > > reporting
> > > > > > > > > partitions on failed disks?
> > > > > > > > >
> > > > > > > >
> > > > > > > > I added FailedReplicaRecord to reflect the case where a JBOD
> > > > directory
> > > > > > has
> > > > > > > > failed, leading to failed replicas.
> > > > > > > >
> > > > > > > > > 102. For debugging purposes, sometimes it's useful to read
> > the
> > > > > > metadata
> > > > > > > > > topic using tools like console-consumer. Should we support
> > that
> > > > and
> > > > > > if
> > > > > > > > so,
> > > > > > > > > how?
> > > > > > > > >
> > > > > > > >
> > > > > > > > For now, we have the ability to read the metadata logs with the
> > > > > > dump-logs
> > > > > > > > tool. I think we will come up with some other tools in the
> > future
> > > > as
> > > > > > we
> > > > > > > > get experience.
> > > > > > > >
> > > > > > > > > 200. "brokers which are fenced will not appear in
> > > > MetadataResponses.
> > > > > > The
> > > > > > > > > broker will not respond to these requests-- instead, it will
> > > > simply
> > > > > > > > > disconnect." If the controller is partitioned off from the
> > > > brokers,
> > > > > > this
> > > > > > > > > design will cause every broker to stop accepting new client
> > > > > > requests. In
> > > > > > > > > contrast, if ZK is partitioned off, the existing behavior is
> > > > that the
> > > > > > > > > brokers can continue to work based on the last known
> > metadata.
> > > > So, I
> > > > > > am
> > > > > > > > not
> > > > > > > > > sure if we should change the existing behavior because of the
> > > > bigger
> > > > > > > > impact
> > > > > > > > > in the new one. Another option is to keep the existing
> > behavior
> > > > and
> > > > > > > > expose
> > > > > > > > > a metric for fenced brokers so that the operator could be
> > > > alerted.
> > > > > > > > >
> > > > > > > >
> > > > > > > > I'm skeptical about how well running without ZK currently
> > works.
> > > > > > However,
> > > > > > > > I will move the broker-side fencing into a follow-up KIP. This
> > > > KIP is
> > > > > > > > already pretty large and there is no hard dependency on this.
> > > > There
> > > > > > may
> > > > > > > > also be other ways of accomplishing the positive effects of
> > what
> > > > > > > > broker-side fencing, so more discussion is needed.
> > > > > > > >
> > > > > > > > > 201. I read Ron's comment, but I am still not sure the
> > benefit of
> > > > > > keeping
> > > > > > > > > broker.id and controller.id in meta.properties. It seems
> > that
> > > > we are
> > > > > > > > just
> > > > > > > > > duplicating the same info in two places and have the
> > additional
> > > > > > burden of
> > > > > > > > > making sure the values in the two places are consistent.
> > > > > > > > >
> > > > > > > >
> > > > > > > > I think the reasoning is that having broker.id protects us
> > against
> > > > > > > > accidentally bringing up a broker with a disk from a different
> > > > > > broker. I
> > > > > > > > don't feel strongly about this but it seemed simpler to keep
> > it.
> > > > > > > >
> > > > > > > > > 202. controller.connect.security.protocol: Is this needed
> > since
> > > > > > > > > controller.listener.names and listener.security.protocol.map
> > > > imply
> > > > > > the
> > > > > > > > > security protocol already?
> > > > > > > > >
> > > > > > > >
> > > > > > > > You're right, this isn't needed. I'll remove it.
> > > > > > > >
> > > > > > > > > 203. registration.heartbeat.interval.ms: It defaults to 2k.
> > ZK
> > > > uses
> > > > > > 1/3
> > > > > > > > of
> > > > > > > > > the session timeout for heartbeat. So, given the default 18k
> > for
> > > > > > > > > registration.lease.timeout.ms, should we default
> > > > > > > > > registration.heartbeat.interval.ms to 6k?
> > > > > > > > >
> > > > > > > >
> > > > > > > > 6 seconds seems like a pretty long time between heartbeats. It
> > > > might
> > > > > > be
> > > > > > > > useful to know when a broker is missing heartbeats, with less
> > time
> > > > than
> > > > > > > > that. I provisionally set it to 3 seconds (we can always
> > change
> > > > > > later...)
> > > > > > > >
> > > > > > > > I also changed the name of these configurations to "
> > > > > > > > broker.heartbeat.interval.ms" and "
> > broker.registration.timeout.ms"
> > > > to
> > > > > > try
> > > > > > > > to clarify them a bit.
> > > > > > > >
> > > > > > > > > 204. "The highest metadata offset which the broker has not
> > > > reached."
> > > > > > It
> > > > > > > > > seems this should be "has reached".
> > > > > > > > >
> > > > > > > >
> > > > > > > > I changed this to "one more than the highest metadata offset
> > which
> > > > the
> > > > > > > > broker has reached."
> > > > > > > >
> > > > > > > > > 205. UnfenceBrokerRecord and UnregisterBrokerRecord: To me,
> > they
> > > > > > seem to
> > > > > > > > be
> > > > > > > > > the same. Do we need both?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Unregistration means that the broker has been removed from the
> > > > cluster.
> > > > > > > > That is different than unfencing, which marks the broker as
> > active.
> > > > > > > >
> > > > > > > > > 206. TopicRecord: The Deleting field is used to indicate
> > that the
> > > > > > topic
> > > > > > > > is
> > > > > > > > > being deleted. I am wondering if this is really needed since
> > > > > > RemoveTopic
> > > > > > > > > already indicates the same thing.
> > > > > > > > >
> > > > > > > >
> > > > > > > > RemoveTopic is the last step, that scrubs all metadata about
> > the
> > > > topic.
> > > > > > > > In order to get to that last step, the topic data needs to
> > removed
> > > > > > from all
> > > > > > > > brokers (after each broker notices that the topic is being
> > > > deleted).
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <
> > cmccabe@apache.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > > > > > > > > > > Hi Colin. Thanks for the updates. It's now clear to me
> > that
> > > > > > brokers
> > > > > > > > > > > keep their broker epoch for the life of their JVM -- they
> > > > > > register
> > > > > > > > > > > once, get their broker epoch in the response, and then
> > never
> > > > > > > > > > > re-register again. Brokers may get fenced, but they
> > keep the
> > > > > > same
> > > > > > > > > > > broker epoch for the life of their JVM. The incarnation
> > ID
> > > > is
> > > > > > also
> > > > > > > > > > > kept for the life of the JVM but is generated by the
> > broker
> > > > > > itself
> > > > > > > > > > > upon startup, and the combination of the two allows the
> > > > > > Controller to
> > > > > > > > > > > act idempotently if any previously-sent registration
> > response
> > > > > > gets
> > > > > > > > > > > lost. Makes sense.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks, Ron. That's a good summary.
> > > > > > > > > >
> > > > > > > > > > > One thing I wonder about is if it might be helpful for
> > the
> > > > > > broker to
> > > > > > > > > > > send the Cluster ID as determined from its
> > meta.properties
> > > > file
> > > > > > in
> > > > > > > > its
> > > > > > > > > > > registration request. Does it even make sense for the
> > > > broker to
> > > > > > > > > > > successfully register and enter the Fenced state if it
> > has
> > > > the
> > > > > > wrong
> > > > > > > > > > > Cluster ID?
> > > > > > > > > >
> > > > > > > > > > Yeah, that's a good idea. Let's have the broker pass its
> > > > cluster
> > > > > > ID in
> > > > > > > > > > the registration RPC, and then registration can fail if the
> > > > broker
> > > > > > is
> > > > > > > > > > configured for the wrong cluster.
> > > > > > > > > >
> > > > > > > > > > > The nextMetadatOffset value that the broker communicates
> > > > > > > > > > > in its registration request only has meaning within the
> > > > correct
> > > > > > > > > > > cluster, so it feels to me that the Controller should
> > have
> > > > some
> > > > > > way
> > > > > > > > to
> > > > > > > > > > > perform this sanity check. There is currently (pre-KIP
> > 500)
> > > > a
> > > > > > check
> > > > > > > > > > > in the broker to make sure its configured cluster ID
> > matches
> > > > the
> > > > > > one
> > > > > > > > > > > stored in ZooKeeper, and we will have to perform this
> > > > validation
> > > > > > > > > > > somewhere in the KIP-500 world. If the Controller
> > doesn't
> > > > do it
> > > > > > > > > > > within the registration request then the broker will
> > have to
> > > > > > make a
> > > > > > > > > > > metadata request to the Controller, retrieve the Cluster
> > ID,
> > > > and
> > > > > > > > > > > perform the check itself. It feels to me that it might
> > be
> > > > > > better for
> > > > > > > > > > > the Controller to just do it, and then the broker doesn't
> > > > have to
> > > > > > > > > > > worry about it anymore once it successfully registers.
> > > > > > > > > > >
> > > > > > > > > > > I also have a question about the broker.id value and
> > > > > > > > meta.properties.
> > > > > > > > > > > The KIP now says "In version 0 of meta.properties, there
> > is a
> > > > > > > > > > > broker.id field. Version 1 does not have this field.
> > It
> > > > is no
> > > > > > > > longer
> > > > > > > > > > > needed because we no longer support dynamic broker id
> > > > > > assignment."
> > > > > > > > > > > But then there is an example version 1 meta.properties
> > file
> > > > that
> > > > > > > > shows
> > > > > > > > > > > the broker.id value. I actually wonder if maybe the
> > > > broker.id
> > > > > > value
> > > > > > > > > > > would be good to keep in the version 1 meta.properties
> > file
> > > > > > because
> > > > > > > > it
> > > > > > > > > > > currently (pre-KIP 500, version 0) acts as a sanity
> > check to
> > > > make
> > > > > > > > sure
> > > > > > > > > > > the broker is using the correct log directory. Similarly
> > > > with
> > > > > > the
> > > > > > > > > > > controller.id value on controllers -- it would allow the
> > > > same
> > > > > > type
> > > > > > > > of
> > > > > > > > > > > sanity check for quorum controllers.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > That's a good point. I will add broker.id back, and also
> > add
> > > > > > > > > > controller.id as a possibility.
> > > > > > > > > >
> > > > > > > > > > cheers,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <
> > > > cmccabe@apache.org
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the reply. A few more comments.
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks again for the reply. Sorry for the long
> > hiatus. I
> > > > was
> > > > > > on
> > > > > > > > > > vacation for a while.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 55. There is still text that favors new broker
> > > > registration.
> > > > > > > > "When a
> > > > > > > > > > broker
> > > > > > > > > > > > > first starts up, when it is in the INITIAL state, it
> > will
> > > > > > always
> > > > > > > > > > "win"
> > > > > > > > > > > > > broker ID conflicts. However, once it is granted a
> > > > lease, it
> > > > > > > > > > transitions
> > > > > > > > > > > > > out of the INITIAL state. Thereafter, it may lose
> > > > subsequent
> > > > > > > > > > conflicts if
> > > > > > > > > > > > > its broker epoch is stale. (See KIP-380 for some
> > > > background
> > > > > > on
> > > > > > > > > > broker
> > > > > > > > > > > > > epoch.) The reason for favoring new processes is to
> > > > > > accommodate
> > > > > > > > the
> > > > > > > > > > common
> > > > > > > > > > > > > case where a process is killed with kill -9 and then
> > > > > > restarted.
> > > > > > > > We
> > > > > > > > > > want it
> > > > > > > > > > > > > to be able to reclaim its old ID quickly in this
> > case."
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the reminder. I have clarified the language
> > > > here.
> > > > > > > > > > Hopefully now it is clear that we don't allow quick re-use
> > of
> > > > > > broker
> > > > > > > > IDs.
> > > > > > > > > > > >
> > > > > > > > > > > > > 80.1 Sounds good. Could you document that listeners
> > is a
> > > > > > required
> > > > > > > > > > config
> > > > > > > > > > > > > now? It would also be useful to annotate other
> > required
> > > > > > configs.
> > > > > > > > For
> > > > > > > > > > > > > example, controller.connect should be required.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I added a note specifying that these are required.
> > > > > > > > > > > >
> > > > > > > > > > > > > 80.2 Could you list all deprecated existing configs?
> > > > Another
> > > > > > one
> > > > > > > > is
> > > > > > > > > > > > > control.plane.listener.name since the controller no
> > > > longer
> > > > > > sends
> > > > > > > > > > > > > LeaderAndIsr, UpdateMetadata and StopReplica
> > requests.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I added a section specifying some deprecated configs.
> > > > > > > > > > > >
> > > > > > > > > > > > > 83.1 It seems that the broker can transition from
> > FENCED
> > > > to
> > > > > > > > RUNNING
> > > > > > > > > > without
> > > > > > > > > > > > > registering for a new broker epoch. I am not sure how
> > > > this
> > > > > > works.
> > > > > > > > > > Once the
> > > > > > > > > > > > > controller fences a broker, there is no need for the
> > > > > > controller
> > > > > > > > to
> > > > > > > > > > keep the
> > > > > > > > > > > > > boker epoch around. So, if the fenced broker's
> > heartbeat
> > > > > > request
> > > > > > > > > > with the
> > > > > > > > > > > > > existing broker epoch will be rejected, leading the
> > > > broker
> > > > > > back
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > The broker epoch refers to the broker registration.
> > So we
> > > > DO
> > > > > > keep
> > > > > > > > the
> > > > > > > > > > broker epoch around even while the broker is fenced.
> > > > > > > > > > > >
> > > > > > > > > > > > The broker epoch changes only when there is a new
> > broker
> > > > > > > > > > registration. Fencing or unfencing the broker doesn't
> > change
> > > > the
> > > > > > > > broker
> > > > > > > > > > epoch.
> > > > > > > > > > > >
> > > > > > > > > > > > > 83.5 Good point on KIP-590. Then should we expose the
> > > > > > controller
> > > > > > > > for
> > > > > > > > > > > > > debugging purposes? If not, we should deprecate the
> > > > > > controllerID
> > > > > > > > > > field in
> > > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I think it's OK to expose it for now, with the proviso
> > > > that it
> > > > > > > > won't
> > > > > > > > > > be reachable by clients.
> > > > > > > > > > > >
> > > > > > > > > > > > > 90. We rejected the shared ID with just one reason
> > "This
> > > > is
> > > > > > not a
> > > > > > > > > > good idea
> > > > > > > > > > > > > because NetworkClient assumes a single ID space. So
> > if
> > > > > > there is
> > > > > > > > > > both a
> > > > > > > > > > > > > controller 1 and a broker 1, we don't have a way of
> > > > picking
> > > > > > the
> > > > > > > > > > "right"
> > > > > > > > > > > > > one." This doesn't seem to be a strong reason. For
> > > > example,
> > > > > > we
> > > > > > > > could
> > > > > > > > > > > > > address the NetworkClient issue with the node type
> > as you
> > > > > > pointed
> > > > > > > > > > out or
> > > > > > > > > > > > > using the negative value of a broker ID as the
> > > > controller ID.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > It would require a lot of code changes to support
> > multiple
> > > > > > types of
> > > > > > > > > > node IDs. It's not clear to me that the end result would
> > be
> > > > > > better --
> > > > > > > > I
> > > > > > > > > > tend to think it would be worse, since it would be more
> > > > complex.
> > > > > > In a
> > > > > > > > > > similar vein, using negative numbers seems dangerous,
> > since we
> > > > use
> > > > > > > > > > negatives or -1 as "special values" in many places. For
> > > > example,
> > > > > > -1
> > > > > > > > often
> > > > > > > > > > represents "no such node."
> > > > > > > > > > > >
> > > > > > > > > > > > One important thing to keep in mind is that we want to
> > be
> > > > able
> > > > > > to
> > > > > > > > > > transition from a broker and a controller being co-located
> > to
> > > > them
> > > > > > no
> > > > > > > > > > longer being co-located. This is much easier to do when
> > they
> > > > have
> > > > > > > > separate
> > > > > > > > > > IDs.
> > > > > > > > > > > >
> > > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > > <
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > > >,
> > > > > > > > > > > > > the broker reports all offline replicas due to a disk
> > > > > > failure to
> > > > > > > > the
> > > > > > > > > > > > > controller. It seems this information needs to be
> > > > persisted
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > > metadata
> > > > > > > > > > > > > log. Do we have a corresponding record for that?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hmm, I have to look into this a little bit more. We
> > may
> > > > need
> > > > > > a new
> > > > > > > > > > record type.
> > > > > > > > > > > >
> > > > > > > > > > > > > 101. Currently, StopReplica request has 2 modes,
> > without
> > > > > > deletion
> > > > > > > > > > and with
> > > > > > > > > > > > > deletion. The former is used for controlled shutdown
> > and
> > > > > > handling
> > > > > > > > > > disk
> > > > > > > > > > > > > failure, and causes the follower to stop. The latter
> > is
> > > > for
> > > > > > topic
> > > > > > > > > > deletion
> > > > > > > > > > > > > and partition reassignment, and causes the replica
> > to be
> > > > > > deleted.
> > > > > > > > > > Since we
> > > > > > > > > > > > > are deprecating StopReplica, could we document what
> > > > triggers
> > > > > > the
> > > > > > > > > > stopping
> > > > > > > > > > > > > of a follower and the deleting of a replica now?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > RemoveTopic triggers deletion. In general the
> > > > functionality of
> > > > > > > > > > StopReplica is subsumed by the metadata records.
> > > > > > > > > > > >
> > > > > > > > > > > > > 102. Should we include the metadata topic in the
> > > > > > > > MetadataResponse?
> > > > > > > > > > If so,
> > > > > > > > > > > > > when it will be included and what will the metadata
> > > > response
> > > > > > look
> > > > > > > > > > like?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > No, it won't be included in the metadata response sent
> > back
> > > > > > from
> > > > > > > > the
> > > > > > > > > > brokers.
> > > > > > > > > > > >
> > > > > > > > > > > > > 103. "The active controller assigns the broker a new
> > > > broker
> > > > > > > > epoch,
> > > > > > > > > > based on
> > > > > > > > > > > > > the latest committed offset in the log." This seems
> > > > > > inaccurate
> > > > > > > > since
> > > > > > > > > > the
> > > > > > > > > > > > > latest committed offset doesn't always advance on
> > every
> > > > log
> > > > > > > > append.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Given that the new broker epoch won't be visible until
> > the
> > > > > > commit
> > > > > > > > has
> > > > > > > > > > happened, I have changed this to "the next available
> > offset in
> > > > the
> > > > > > log"
> > > > > > > > > > > >
> > > > > > > > > > > > > 104. REGISTERING(1) : It says "Otherwise, the broker
> > > > moves
> > > > > > into
> > > > > > > > the
> > > > > > > > > > FENCED
> > > > > > > > > > > > > state.". It seems this should be RUNNING?
> > > > > > > > > > > > >
> > > > > > > > > > > > > 105. RUNNING: Should we require the broker to catch
> > up
> > > > to the
> > > > > > > > > > metadata log
> > > > > > > > > > > > > to get into this state?
> > > > > > > > > > > >
> > > > > > > > > > > > For 104 and 105, these sections have been reworked.
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jun
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <
> > > > > > cmccabe@apache.org
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino
> > wrote:
> > > > > > > > > > > > > > > > > Hi Colin. Thanks for the hard work on this
> > KIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I have some questions about what happens to a
> > > > broker
> > > > > > > > when it
> > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > fenced (e.g. because it can't send a
> > heartbeat
> > > > > > request to
> > > > > > > > > > keep its
> > > > > > > > > > > > > > > > > lease). The KIP says "When a broker is
> > fenced,
> > > > it
> > > > > > cannot
> > > > > > > > > > process any
> > > > > > > > > > > > > > > > > client requests. This prevents brokers which
> > > > are not
> > > > > > > > > > receiving
> > > > > > > > > > > > > > > > > metadata updates or that are not receiving
> > and
> > > > > > processing
> > > > > > > > > > them fast
> > > > > > > > > > > > > > > > > enough from causing issues to clients." And
> > in
> > > > the
> > > > > > > > > > description of the
> > > > > > > > > > > > > > > > > FENCED(4) state it likewise says "While in
> > this
> > > > > > state,
> > > > > > > > the
> > > > > > > > > > broker
> > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > not respond to client requests." It makes
> > sense
> > > > > > that a
> > > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > > should not accept producer requests -- I
> > assume
> > > > any
> > > > > > such
> > > > > > > > > > requests
> > > > > > > > > > > > > > > > > would result in NotLeaderOrFollowerException.
> > > > But
> > > > > > what
> > > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > > (fetch from follower) consumer requests? It
> > is
> > > > > > > > conceivable
> > > > > > > > > > that
> > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > could continue. Related to that, would a
> > fenced
> > > > > > broker
> > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > fetch data for partitions where it thinks it
> > is a
> > > > > > > > follower?
> > > > > > > > > > Even if
> > > > > > > > > > > > > > > > > it rejects consumer requests it might still
> > > > continue
> > > > > > to
> > > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > > follower. Might it be helpful to clarify
> > both
> > > > > > decisions
> > > > > > > > > > here?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Good question. I think a fenced broker should
> > > > > > continue to
> > > > > > > > > > fetch on
> > > > > > > > > > > > > > > > partitions it was already fetching before it
> > was
> > > > > > fenced,
> > > > > > > > > > unless it
> > > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > > problem. At that point it won't be able to
> > > > continue,
> > > > > > > > since it
> > > > > > > > > > doesn't
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > the new metadata. For example, it won't know
> > about
> > > > > > > > leadership
> > > > > > > > > > changes
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the partitions it's fetching. The rationale
> > for
> > > > > > > > continuing to
> > > > > > > > > > fetch
> > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > try to avoid disruptions as much as possible.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I don't think fenced brokers should accept
> > client
> > > > > > requests.
> > > > > > > > > > The issue
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > that the fenced broker may or may not have any
> > > > data it
> > > > > > is
> > > > > > > > > > supposed to
> > > > > > > > > > > > > > > > have. It may or may not have applied any
> > > > configuration
> > > > > > > > > > changes, etc.
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > it is supposed to have applied. So it could
> > get
> > > > pretty
> > > > > > > > > > confusing, and
> > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > potentially waste the client's time.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > When fenced, how would the broker reply to a
> > client
> > > > > > which did
> > > > > > > > > > make a
> > > > > > > > > > > > > > > request?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The broker will respond with a retryable error in
> > that
> > > > > > case.
> > > > > > > > Once
> > > > > > > > > > the
> > > > > > > > > > > > > > client has re-fetched its metadata, it will no
> > longer
> > > > see
> > > > > > the
> > > > > > > > > > fenced broker
> > > > > > > > > > > > > > as part of the cluster. I added a note to the KIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the reply. A few more follow up comments.
210. initial.broker.registration.timeout.ms: The default value is 90sec,
which seems long. If a broker fails the registration because of incorrect
configs, we want to fail faster. In comparison, the defaults for
zookeeper.connection.timeout.ms is 18 secs.
211. "Once they are all moved, the controller responds to the heartbeat
with a nextState of SHUTDOWN." It seems that nextState is no longer
relevant. Also, could you add a bit more explanation on when ShouldFence is
set to true in BrokerHeartbeatRequest?
212. Related to the above, what does the broker do if IsFenced is true in
the BrokerHeartbeatResponse? Will the broker do the same thing on receiving
a FenceBrokerRecord from the metadata log?
Jun
On Tue, Dec 15, 2020 at 8:51 AM Colin McCabe <cm...@apache.org> wrote:
> On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> > Hi Colin,
> >
> > The KIP says that "brokers which are fenced will not appear in
> > MetadataResponses. So clients that have up-to-date metadata will not try
> > to contact fenced brokers.", which is fine, but it doesn't seem to
> > elaborate on what happens for clients which try to contact a broker
> (using
> > stale metadata, for example) and find it in the registered-but-fenced
> > state.
>
> Hi Tom,
>
> I have dropped the broker-side fencing from this proposal. So now the
> fencing is basically the same as today's fencing: it's controller-side
> only. That means that clients using stale metadata can contact fenced
> brokers and communicate with them. The only case where the broker will be
> inaccessible is when it hasn't finished starting yet (i.e., has not yet
> attained RUNNING state.)
>
> Just like today, we expect the safeguards built into the replication
> protocol to prevent the worst corner cases that could result from this. I
> do think we will probably take up this issue later and find a better
> solution for client-side metadata, but this KIP is big enough as-is.
>
> best,
> Colin
>
>
> > You said previously that the broker will respond with a retriable
> > error, which is again fine, but you didn't answer my question about which
> > error code you've chosen for this. I realise that this doesn't really
> > affect the correctness of the behaviour, but I'm not aware of any
> existing
> > error code which is a good fit. So it would be good to understand about
> how
> > you're making this backward compatible for clients.
> >
> > Many thanks,
> >
> > Tom
> >
> > On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply. Just a couple of more comments below.
> > > >
> > > > 210. Since we are deprecating zookeeper.connection.timeout.ms,
> should we
> > > > add a new config to bound the time for a broker to connect to the
> > > > controller during starting up?
> > > >
> > >
> > > Good idea. I added initial.broker.registration.timeout.ms for this.
> > >
> > > > 211. BrokerHeartbeat no longer has the state field in the
> > > request/response.
> > > > However, (a) the controller shutdown section still has "In its
> periodic
> > > > heartbeats, the broker asks the controller if it can transition into
> the
> > > > SHUTDOWN state. This motivates the controller to move all of the
> leaders
> > > > off of that broker. Once they are all moved, the controller
> responds to
> > > > the heartbeat with a nextState of SHUTDOWN."; (2) the description of
> > > > BrokerHeartbeat still references currentState and targetState.
> > > >
> > >
> > > Thanks. I've made these sections clearer and removed the obsolete
> > > references to sending states.
> > >
> > > best,
> > > Colin
> > >
> > > > Jun
> > > >
> > > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the update. A few more follow up comments.
> > > > > >
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > Thanks again for the review.
> > > > >
> > > > > > 100. FailedReplicaRecord: Since this is reported by each broker
> > > > > > independently, perhaps we could use a more concise representation
> > > that
> > > > > has
> > > > > > a top level broker field, an array of topics, which has an array
> of
> > > > > > partitions.
> > > > > >
> > > > >
> > > > > The issue is that there is a size limit on the each record.
> Putting
> > > all
> > > > > of the partitions of a log directory into a single record would
> > > probably
> > > > > break that in many cases. Still, we can optimize a bit by having
> an
> > > array
> > > > > of partition IDs, since nearly all the time, we have more than one
> > > from the
> > > > > same topic.
> > > > >
> > > > > > 200. Sounds good. If we remove the broker-side fencing logic, do
> we
> > > plan
> > > > > to
> > > > > > still keep FENCED in broker state? Do we plan to expose the new
> > > states
> > > > > > through the existing BrokerState metric and if so, what are the
> > > values
> > > > > for
> > > > > > the new states?
> > > > > >
> > > > >
> > > > > No, we don't need FENCED any more. I have removed it from the KIP.
> > > > >
> > > > > The new states are very similar to the current ones, actually.
> There
> > > are
> > > > > no new states or removed ones. The main change in the broker state
> > > machine
> > > > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state has been
> renamed to
> > > > > RECOVERY. Also, unlike previously, the broker will always pass
> through
> > > > > RECOVERY (although it may only stay in this state for a very short
> > > amount
> > > > > of time).
> > > > >
> > > > > > 201. This may be fine too. Could we document what happens when
> the
> > > > > > broker.id/controller.id in metadata.properties don't match the
> > > broker
> > > > > > config when the broker starts up?
> > > > > >
> > > > >
> > > > > I added some documentation about this.
> > > > >
> > > > > > 204. There is still "The highest metadata offset which the broker
> > > has not
> > > > > > reached" referenced under BrokerRegistration.
> > > > > >
> > > > >
> > > > > It should be CurrentMetadataOffset. Fixed.
> > > > >
> > > > > > 206. Is that separate step needed given KIP-516? With KIP-516 (
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > > ),
> > > > > > we don't need to wait for the topic data to be removed from all
> > > brokers
> > > > > > before removing the topic metadata. The combination of unmatching
> > > > > > topicId
> > > > > > or the missing topicId from the metadata is enough for the
> broker to
> > > > > > clean
> > > > > > up deleted topics asynchronously.
> > > > >
> > > > > It won't be needed once KIP-516 is adopted, but this hasn't been
> > > > > implemented yet.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <cm...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the updated KIP. A few more comments below.
> > > > > > > >
> > > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > Thanks again for the reviews.
> > > > > > >
> > > > > > > > 80.2 For deprecated configs, we need to include zookeeper.*
> and
> > > > > > > > broker.id.generation.enable.
> > > > > > > >
> > > > > > >
> > > > > > > Added.
> > > > > > >
> > > > > > > > 83.1 If a broker is down, does the controller keep the
> previously
> > > > > > > > registered broker epoch forever? If not, how long does the
> > > controller
> > > > > > > keep
> > > > > > > > it? What does the controller do when receiving a broker
> heartbeat
> > > > > request
> > > > > > > > with an unfound broker epoch?
> > > > > > > >
> > > > > > >
> > > > > > > Yes, the controller keeps the previous registration forever.
> > > > > > >
> > > > > > > Broker heartbeat requests with an incorrect broker epoch will
> be
> > > > > rejected
> > > > > > > with STALE_BROKER_EPOCH.
> > > > > > >
> > > > > > > > 100. Have you figured out if we need to add a new record
> type for
> > > > > > > reporting
> > > > > > > > partitions on failed disks?
> > > > > > > >
> > > > > > >
> > > > > > > I added FailedReplicaRecord to reflect the case where a JBOD
> > > directory
> > > > > has
> > > > > > > failed, leading to failed replicas.
> > > > > > >
> > > > > > > > 102. For debugging purposes, sometimes it's useful to read
> the
> > > > > metadata
> > > > > > > > topic using tools like console-consumer. Should we support
> that
> > > and
> > > > > if
> > > > > > > so,
> > > > > > > > how?
> > > > > > > >
> > > > > > >
> > > > > > > For now, we have the ability to read the metadata logs with the
> > > > > dump-logs
> > > > > > > tool. I think we will come up with some other tools in the
> future
> > > as
> > > > > we
> > > > > > > get experience.
> > > > > > >
> > > > > > > > 200. "brokers which are fenced will not appear in
> > > MetadataResponses.
> > > > > The
> > > > > > > > broker will not respond to these requests-- instead, it will
> > > simply
> > > > > > > > disconnect." If the controller is partitioned off from the
> > > brokers,
> > > > > this
> > > > > > > > design will cause every broker to stop accepting new client
> > > > > requests. In
> > > > > > > > contrast, if ZK is partitioned off, the existing behavior is
> > > that the
> > > > > > > > brokers can continue to work based on the last known
> metadata.
> > > So, I
> > > > > am
> > > > > > > not
> > > > > > > > sure if we should change the existing behavior because of the
> > > bigger
> > > > > > > impact
> > > > > > > > in the new one. Another option is to keep the existing
> behavior
> > > and
> > > > > > > expose
> > > > > > > > a metric for fenced brokers so that the operator could be
> > > alerted.
> > > > > > > >
> > > > > > >
> > > > > > > I'm skeptical about how well running without ZK currently
> works.
> > > > > However,
> > > > > > > I will move the broker-side fencing into a follow-up KIP. This
> > > KIP is
> > > > > > > already pretty large and there is no hard dependency on this.
> > > There
> > > > > may
> > > > > > > also be other ways of accomplishing the positive effects of
> what
> > > > > > > broker-side fencing, so more discussion is needed.
> > > > > > >
> > > > > > > > 201. I read Ron's comment, but I am still not sure the
> benefit of
> > > > > keeping
> > > > > > > > broker.id and controller.id in meta.properties. It seems
> that
> > > we are
> > > > > > > just
> > > > > > > > duplicating the same info in two places and have the
> additional
> > > > > burden of
> > > > > > > > making sure the values in the two places are consistent.
> > > > > > > >
> > > > > > >
> > > > > > > I think the reasoning is that having broker.id protects us
> against
> > > > > > > accidentally bringing up a broker with a disk from a different
> > > > > broker. I
> > > > > > > don't feel strongly about this but it seemed simpler to keep
> it.
> > > > > > >
> > > > > > > > 202. controller.connect.security.protocol: Is this needed
> since
> > > > > > > > controller.listener.names and listener.security.protocol.map
> > > imply
> > > > > the
> > > > > > > > security protocol already?
> > > > > > > >
> > > > > > >
> > > > > > > You're right, this isn't needed. I'll remove it.
> > > > > > >
> > > > > > > > 203. registration.heartbeat.interval.ms: It defaults to 2k.
> ZK
> > > uses
> > > > > 1/3
> > > > > > > of
> > > > > > > > the session timeout for heartbeat. So, given the default 18k
> for
> > > > > > > > registration.lease.timeout.ms, should we default
> > > > > > > > registration.heartbeat.interval.ms to 6k?
> > > > > > > >
> > > > > > >
> > > > > > > 6 seconds seems like a pretty long time between heartbeats. It
> > > might
> > > > > be
> > > > > > > useful to know when a broker is missing heartbeats, with less
> time
> > > than
> > > > > > > that. I provisionally set it to 3 seconds (we can always
> change
> > > > > later...)
> > > > > > >
> > > > > > > I also changed the name of these configurations to "
> > > > > > > broker.heartbeat.interval.ms" and "
> broker.registration.timeout.ms"
> > > to
> > > > > try
> > > > > > > to clarify them a bit.
> > > > > > >
> > > > > > > > 204. "The highest metadata offset which the broker has not
> > > reached."
> > > > > It
> > > > > > > > seems this should be "has reached".
> > > > > > > >
> > > > > > >
> > > > > > > I changed this to "one more than the highest metadata offset
> which
> > > the
> > > > > > > broker has reached."
> > > > > > >
> > > > > > > > 205. UnfenceBrokerRecord and UnregisterBrokerRecord: To me,
> they
> > > > > seem to
> > > > > > > be
> > > > > > > > the same. Do we need both?
> > > > > > > >
> > > > > > >
> > > > > > > Unregistration means that the broker has been removed from the
> > > cluster.
> > > > > > > That is different than unfencing, which marks the broker as
> active.
> > > > > > >
> > > > > > > > 206. TopicRecord: The Deleting field is used to indicate
> that the
> > > > > topic
> > > > > > > is
> > > > > > > > being deleted. I am wondering if this is really needed since
> > > > > RemoveTopic
> > > > > > > > already indicates the same thing.
> > > > > > > >
> > > > > > >
> > > > > > > RemoveTopic is the last step, that scrubs all metadata about
> the
> > > topic.
> > > > > > > In order to get to that last step, the topic data needs to
> removed
> > > > > from all
> > > > > > > brokers (after each broker notices that the topic is being
> > > deleted).
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <
> cmccabe@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > > > > > > > > > Hi Colin. Thanks for the updates. It's now clear to me
> that
> > > > > brokers
> > > > > > > > > > keep their broker epoch for the life of their JVM -- they
> > > > > register
> > > > > > > > > > once, get their broker epoch in the response, and then
> never
> > > > > > > > > > re-register again. Brokers may get fenced, but they
> keep the
> > > > > same
> > > > > > > > > > broker epoch for the life of their JVM. The incarnation
> ID
> > > is
> > > > > also
> > > > > > > > > > kept for the life of the JVM but is generated by the
> broker
> > > > > itself
> > > > > > > > > > upon startup, and the combination of the two allows the
> > > > > Controller to
> > > > > > > > > > act idempotently if any previously-sent registration
> response
> > > > > gets
> > > > > > > > > > lost. Makes sense.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks, Ron. That's a good summary.
> > > > > > > > >
> > > > > > > > > > One thing I wonder about is if it might be helpful for
> the
> > > > > broker to
> > > > > > > > > > send the Cluster ID as determined from its
> meta.properties
> > > file
> > > > > in
> > > > > > > its
> > > > > > > > > > registration request. Does it even make sense for the
> > > broker to
> > > > > > > > > > successfully register and enter the Fenced state if it
> has
> > > the
> > > > > wrong
> > > > > > > > > > Cluster ID?
> > > > > > > > >
> > > > > > > > > Yeah, that's a good idea. Let's have the broker pass its
> > > cluster
> > > > > ID in
> > > > > > > > > the registration RPC, and then registration can fail if the
> > > broker
> > > > > is
> > > > > > > > > configured for the wrong cluster.
> > > > > > > > >
> > > > > > > > > > The nextMetadatOffset value that the broker communicates
> > > > > > > > > > in its registration request only has meaning within the
> > > correct
> > > > > > > > > > cluster, so it feels to me that the Controller should
> have
> > > some
> > > > > way
> > > > > > > to
> > > > > > > > > > perform this sanity check. There is currently (pre-KIP
> 500)
> > > a
> > > > > check
> > > > > > > > > > in the broker to make sure its configured cluster ID
> matches
> > > the
> > > > > one
> > > > > > > > > > stored in ZooKeeper, and we will have to perform this
> > > validation
> > > > > > > > > > somewhere in the KIP-500 world. If the Controller
> doesn't
> > > do it
> > > > > > > > > > within the registration request then the broker will
> have to
> > > > > make a
> > > > > > > > > > metadata request to the Controller, retrieve the Cluster
> ID,
> > > and
> > > > > > > > > > perform the check itself. It feels to me that it might
> be
> > > > > better for
> > > > > > > > > > the Controller to just do it, and then the broker doesn't
> > > have to
> > > > > > > > > > worry about it anymore once it successfully registers.
> > > > > > > > > >
> > > > > > > > > > I also have a question about the broker.id value and
> > > > > > > meta.properties.
> > > > > > > > > > The KIP now says "In version 0 of meta.properties, there
> is a
> > > > > > > > > > broker.id field. Version 1 does not have this field.
> It
> > > is no
> > > > > > > longer
> > > > > > > > > > needed because we no longer support dynamic broker id
> > > > > assignment."
> > > > > > > > > > But then there is an example version 1 meta.properties
> file
> > > that
> > > > > > > shows
> > > > > > > > > > the broker.id value. I actually wonder if maybe the
> > > broker.id
> > > > > value
> > > > > > > > > > would be good to keep in the version 1 meta.properties
> file
> > > > > because
> > > > > > > it
> > > > > > > > > > currently (pre-KIP 500, version 0) acts as a sanity
> check to
> > > make
> > > > > > > sure
> > > > > > > > > > the broker is using the correct log directory. Similarly
> > > with
> > > > > the
> > > > > > > > > > controller.id value on controllers -- it would allow the
> > > same
> > > > > type
> > > > > > > of
> > > > > > > > > > sanity check for quorum controllers.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > That's a good point. I will add broker.id back, and also
> add
> > > > > > > > > controller.id as a possibility.
> > > > > > > > >
> > > > > > > > > cheers,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <
> > > cmccabe@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the reply. A few more comments.
> > > > > > > > > > >
> > > > > > > > > > > Hi Jun,
> > > > > > > > > > >
> > > > > > > > > > > Thanks again for the reply. Sorry for the long
> hiatus. I
> > > was
> > > > > on
> > > > > > > > > vacation for a while.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 55. There is still text that favors new broker
> > > registration.
> > > > > > > "When a
> > > > > > > > > broker
> > > > > > > > > > > > first starts up, when it is in the INITIAL state, it
> will
> > > > > always
> > > > > > > > > "win"
> > > > > > > > > > > > broker ID conflicts. However, once it is granted a
> > > lease, it
> > > > > > > > > transitions
> > > > > > > > > > > > out of the INITIAL state. Thereafter, it may lose
> > > subsequent
> > > > > > > > > conflicts if
> > > > > > > > > > > > its broker epoch is stale. (See KIP-380 for some
> > > background
> > > > > on
> > > > > > > > > broker
> > > > > > > > > > > > epoch.) The reason for favoring new processes is to
> > > > > accommodate
> > > > > > > the
> > > > > > > > > common
> > > > > > > > > > > > case where a process is killed with kill -9 and then
> > > > > restarted.
> > > > > > > We
> > > > > > > > > want it
> > > > > > > > > > > > to be able to reclaim its old ID quickly in this
> case."
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the reminder. I have clarified the language
> > > here.
> > > > > > > > > Hopefully now it is clear that we don't allow quick re-use
> of
> > > > > broker
> > > > > > > IDs.
> > > > > > > > > > >
> > > > > > > > > > > > 80.1 Sounds good. Could you document that listeners
> is a
> > > > > required
> > > > > > > > > config
> > > > > > > > > > > > now? It would also be useful to annotate other
> required
> > > > > configs.
> > > > > > > For
> > > > > > > > > > > > example, controller.connect should be required.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I added a note specifying that these are required.
> > > > > > > > > > >
> > > > > > > > > > > > 80.2 Could you list all deprecated existing configs?
> > > Another
> > > > > one
> > > > > > > is
> > > > > > > > > > > > control.plane.listener.name since the controller no
> > > longer
> > > > > sends
> > > > > > > > > > > > LeaderAndIsr, UpdateMetadata and StopReplica
> requests.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I added a section specifying some deprecated configs.
> > > > > > > > > > >
> > > > > > > > > > > > 83.1 It seems that the broker can transition from
> FENCED
> > > to
> > > > > > > RUNNING
> > > > > > > > > without
> > > > > > > > > > > > registering for a new broker epoch. I am not sure how
> > > this
> > > > > works.
> > > > > > > > > Once the
> > > > > > > > > > > > controller fences a broker, there is no need for the
> > > > > controller
> > > > > > > to
> > > > > > > > > keep the
> > > > > > > > > > > > boker epoch around. So, if the fenced broker's
> heartbeat
> > > > > request
> > > > > > > > > with the
> > > > > > > > > > > > existing broker epoch will be rejected, leading the
> > > broker
> > > > > back
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > FENCED state again.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The broker epoch refers to the broker registration.
> So we
> > > DO
> > > > > keep
> > > > > > > the
> > > > > > > > > broker epoch around even while the broker is fenced.
> > > > > > > > > > >
> > > > > > > > > > > The broker epoch changes only when there is a new
> broker
> > > > > > > > > registration. Fencing or unfencing the broker doesn't
> change
> > > the
> > > > > > > broker
> > > > > > > > > epoch.
> > > > > > > > > > >
> > > > > > > > > > > > 83.5 Good point on KIP-590. Then should we expose the
> > > > > controller
> > > > > > > for
> > > > > > > > > > > > debugging purposes? If not, we should deprecate the
> > > > > controllerID
> > > > > > > > > field in
> > > > > > > > > > > > MetadataResponse?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I think it's OK to expose it for now, with the proviso
> > > that it
> > > > > > > won't
> > > > > > > > > be reachable by clients.
> > > > > > > > > > >
> > > > > > > > > > > > 90. We rejected the shared ID with just one reason
> "This
> > > is
> > > > > not a
> > > > > > > > > good idea
> > > > > > > > > > > > because NetworkClient assumes a single ID space. So
> if
> > > > > there is
> > > > > > > > > both a
> > > > > > > > > > > > controller 1 and a broker 1, we don't have a way of
> > > picking
> > > > > the
> > > > > > > > > "right"
> > > > > > > > > > > > one." This doesn't seem to be a strong reason. For
> > > example,
> > > > > we
> > > > > > > could
> > > > > > > > > > > > address the NetworkClient issue with the node type
> as you
> > > > > pointed
> > > > > > > > > out or
> > > > > > > > > > > > using the negative value of a broker ID as the
> > > controller ID.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > It would require a lot of code changes to support
> multiple
> > > > > types of
> > > > > > > > > node IDs. It's not clear to me that the end result would
> be
> > > > > better --
> > > > > > > I
> > > > > > > > > tend to think it would be worse, since it would be more
> > > complex.
> > > > > In a
> > > > > > > > > similar vein, using negative numbers seems dangerous,
> since we
> > > use
> > > > > > > > > negatives or -1 as "special values" in many places. For
> > > example,
> > > > > -1
> > > > > > > often
> > > > > > > > > represents "no such node."
> > > > > > > > > > >
> > > > > > > > > > > One important thing to keep in mind is that we want to
> be
> > > able
> > > > > to
> > > > > > > > > transition from a broker and a controller being co-located
> to
> > > them
> > > > > no
> > > > > > > > > longer being co-located. This is much easier to do when
> they
> > > have
> > > > > > > separate
> > > > > > > > > IDs.
> > > > > > > > > > >
> > > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > > <
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > > >,
> > > > > > > > > > > > the broker reports all offline replicas due to a disk
> > > > > failure to
> > > > > > > the
> > > > > > > > > > > > controller. It seems this information needs to be
> > > persisted
> > > > > to
> > > > > > > the
> > > > > > > > > > > > metadata
> > > > > > > > > > > > log. Do we have a corresponding record for that?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hmm, I have to look into this a little bit more. We
> may
> > > need
> > > > > a new
> > > > > > > > > record type.
> > > > > > > > > > >
> > > > > > > > > > > > 101. Currently, StopReplica request has 2 modes,
> without
> > > > > deletion
> > > > > > > > > and with
> > > > > > > > > > > > deletion. The former is used for controlled shutdown
> and
> > > > > handling
> > > > > > > > > disk
> > > > > > > > > > > > failure, and causes the follower to stop. The latter
> is
> > > for
> > > > > topic
> > > > > > > > > deletion
> > > > > > > > > > > > and partition reassignment, and causes the replica
> to be
> > > > > deleted.
> > > > > > > > > Since we
> > > > > > > > > > > > are deprecating StopReplica, could we document what
> > > triggers
> > > > > the
> > > > > > > > > stopping
> > > > > > > > > > > > of a follower and the deleting of a replica now?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > RemoveTopic triggers deletion. In general the
> > > functionality of
> > > > > > > > > StopReplica is subsumed by the metadata records.
> > > > > > > > > > >
> > > > > > > > > > > > 102. Should we include the metadata topic in the
> > > > > > > MetadataResponse?
> > > > > > > > > If so,
> > > > > > > > > > > > when it will be included and what will the metadata
> > > response
> > > > > look
> > > > > > > > > like?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > No, it won't be included in the metadata response sent
> back
> > > > > from
> > > > > > > the
> > > > > > > > > brokers.
> > > > > > > > > > >
> > > > > > > > > > > > 103. "The active controller assigns the broker a new
> > > broker
> > > > > > > epoch,
> > > > > > > > > based on
> > > > > > > > > > > > the latest committed offset in the log." This seems
> > > > > inaccurate
> > > > > > > since
> > > > > > > > > the
> > > > > > > > > > > > latest committed offset doesn't always advance on
> every
> > > log
> > > > > > > append.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Given that the new broker epoch won't be visible until
> the
> > > > > commit
> > > > > > > has
> > > > > > > > > happened, I have changed this to "the next available
> offset in
> > > the
> > > > > log"
> > > > > > > > > > >
> > > > > > > > > > > > 104. REGISTERING(1) : It says "Otherwise, the broker
> > > moves
> > > > > into
> > > > > > > the
> > > > > > > > > FENCED
> > > > > > > > > > > > state.". It seems this should be RUNNING?
> > > > > > > > > > > >
> > > > > > > > > > > > 105. RUNNING: Should we require the broker to catch
> up
> > > to the
> > > > > > > > > metadata log
> > > > > > > > > > > > to get into this state?
> > > > > > > > > > >
> > > > > > > > > > > For 104 and 105, these sections have been reworked.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jun
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <
> > > > > cmccabe@apache.org
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino
> wrote:
> > > > > > > > > > > > > > > > Hi Colin. Thanks for the hard work on this
> KIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have some questions about what happens to a
> > > broker
> > > > > > > when it
> > > > > > > > > becomes
> > > > > > > > > > > > > > > > fenced (e.g. because it can't send a
> heartbeat
> > > > > request to
> > > > > > > > > keep its
> > > > > > > > > > > > > > > > lease). The KIP says "When a broker is
> fenced,
> > > it
> > > > > cannot
> > > > > > > > > process any
> > > > > > > > > > > > > > > > client requests. This prevents brokers which
> > > are not
> > > > > > > > > receiving
> > > > > > > > > > > > > > > > metadata updates or that are not receiving
> and
> > > > > processing
> > > > > > > > > them fast
> > > > > > > > > > > > > > > > enough from causing issues to clients." And
> in
> > > the
> > > > > > > > > description of the
> > > > > > > > > > > > > > > > FENCED(4) state it likewise says "While in
> this
> > > > > state,
> > > > > > > the
> > > > > > > > > broker
> > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > not respond to client requests." It makes
> sense
> > > > > that a
> > > > > > > > > fenced broker
> > > > > > > > > > > > > > > > should not accept producer requests -- I
> assume
> > > any
> > > > > such
> > > > > > > > > requests
> > > > > > > > > > > > > > > > would result in NotLeaderOrFollowerException.
> > > But
> > > > > what
> > > > > > > > > about KIP-392
> > > > > > > > > > > > > > > > (fetch from follower) consumer requests? It
> is
> > > > > > > conceivable
> > > > > > > > > that
> > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > could continue. Related to that, would a
> fenced
> > > > > broker
> > > > > > > > > continue to
> > > > > > > > > > > > > > > > fetch data for partitions where it thinks it
> is a
> > > > > > > follower?
> > > > > > > > > Even if
> > > > > > > > > > > > > > > > it rejects consumer requests it might still
> > > continue
> > > > > to
> > > > > > > > > fetch as a
> > > > > > > > > > > > > > > > follower. Might it be helpful to clarify
> both
> > > > > decisions
> > > > > > > > > here?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Good question. I think a fenced broker should
> > > > > continue to
> > > > > > > > > fetch on
> > > > > > > > > > > > > > > partitions it was already fetching before it
> was
> > > > > fenced,
> > > > > > > > > unless it
> > > > > > > > > > > > > hits a
> > > > > > > > > > > > > > > problem. At that point it won't be able to
> > > continue,
> > > > > > > since it
> > > > > > > > > doesn't
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > the new metadata. For example, it won't know
> about
> > > > > > > leadership
> > > > > > > > > changes
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the partitions it's fetching. The rationale
> for
> > > > > > > continuing to
> > > > > > > > > fetch
> > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > try to avoid disruptions as much as possible.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I don't think fenced brokers should accept
> client
> > > > > requests.
> > > > > > > > > The issue
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > that the fenced broker may or may not have any
> > > data it
> > > > > is
> > > > > > > > > supposed to
> > > > > > > > > > > > > > > have. It may or may not have applied any
> > > configuration
> > > > > > > > > changes, etc.
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > it is supposed to have applied. So it could
> get
> > > pretty
> > > > > > > > > confusing, and
> > > > > > > > > > > > > also
> > > > > > > > > > > > > > > potentially waste the client's time.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > When fenced, how would the broker reply to a
> client
> > > > > which did
> > > > > > > > > make a
> > > > > > > > > > > > > > request?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > >
> > > > > > > > > > > > > The broker will respond with a retryable error in
> that
> > > > > case.
> > > > > > > Once
> > > > > > > > > the
> > > > > > > > > > > > > client has re-fetched its metadata, it will no
> longer
> > > see
> > > > > the
> > > > > > > > > fenced broker
> > > > > > > > > > > > > as part of the cluster. I added a note to the KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > best,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Tom
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Tue, Dec 15, 2020, at 04:13, Tom Bentley wrote:
> Hi Colin,
>
> The KIP says that "brokers which are fenced will not appear in
> MetadataResponses. So clients that have up-to-date metadata will not try
> to contact fenced brokers.", which is fine, but it doesn't seem to
> elaborate on what happens for clients which try to contact a broker (using
> stale metadata, for example) and find it in the registered-but-fenced
> state.
Hi Tom,
I have dropped the broker-side fencing from this proposal. So now the fencing is basically the same as today's fencing: it's controller-side only. That means that clients using stale metadata can contact fenced brokers and communicate with them. The only case where the broker will be inaccessible is when it hasn't finished starting yet (i.e., has not yet attained RUNNING state.)
Just like today, we expect the safeguards built into the replication protocol to prevent the worst corner cases that could result from this. I do think we will probably take up this issue later and find a better solution for client-side metadata, but this KIP is big enough as-is.
best,
Colin
> You said previously that the broker will respond with a retriable
> error, which is again fine, but you didn't answer my question about which
> error code you've chosen for this. I realise that this doesn't really
> affect the correctness of the behaviour, but I'm not aware of any existing
> error code which is a good fit. So it would be good to understand about how
> you're making this backward compatible for clients.
>
> Many thanks,
>
> Tom
>
> On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <cm...@apache.org> wrote:
>
> > On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply. Just a couple of more comments below.
> > >
> > > 210. Since we are deprecating zookeeper.connection.timeout.ms, should we
> > > add a new config to bound the time for a broker to connect to the
> > > controller during starting up?
> > >
> >
> > Good idea. I added initial.broker.registration.timeout.ms for this.
> >
> > > 211. BrokerHeartbeat no longer has the state field in the
> > request/response.
> > > However, (a) the controller shutdown section still has "In its periodic
> > > heartbeats, the broker asks the controller if it can transition into the
> > > SHUTDOWN state. This motivates the controller to move all of the leaders
> > > off of that broker. Once they are all moved, the controller responds to
> > > the heartbeat with a nextState of SHUTDOWN."; (2) the description of
> > > BrokerHeartbeat still references currentState and targetState.
> > >
> >
> > Thanks. I've made these sections clearer and removed the obsolete
> > references to sending states.
> >
> > best,
> > Colin
> >
> > > Jun
> > >
> > > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the update. A few more follow up comments.
> > > > >
> > > >
> > > > Hi Jun,
> > > >
> > > > Thanks again for the review.
> > > >
> > > > > 100. FailedReplicaRecord: Since this is reported by each broker
> > > > > independently, perhaps we could use a more concise representation
> > that
> > > > has
> > > > > a top level broker field, an array of topics, which has an array of
> > > > > partitions.
> > > > >
> > > >
> > > > The issue is that there is a size limit on the each record. Putting
> > all
> > > > of the partitions of a log directory into a single record would
> > probably
> > > > break that in many cases. Still, we can optimize a bit by having an
> > array
> > > > of partition IDs, since nearly all the time, we have more than one
> > from the
> > > > same topic.
> > > >
> > > > > 200. Sounds good. If we remove the broker-side fencing logic, do we
> > plan
> > > > to
> > > > > still keep FENCED in broker state? Do we plan to expose the new
> > states
> > > > > through the existing BrokerState metric and if so, what are the
> > values
> > > > for
> > > > > the new states?
> > > > >
> > > >
> > > > No, we don't need FENCED any more. I have removed it from the KIP.
> > > >
> > > > The new states are very similar to the current ones, actually. There
> > are
> > > > no new states or removed ones. The main change in the broker state
> > machine
> > > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state has been renamed to
> > > > RECOVERY. Also, unlike previously, the broker will always pass through
> > > > RECOVERY (although it may only stay in this state for a very short
> > amount
> > > > of time).
> > > >
> > > > > 201. This may be fine too. Could we document what happens when the
> > > > > broker.id/controller.id in metadata.properties don't match the
> > broker
> > > > > config when the broker starts up?
> > > > >
> > > >
> > > > I added some documentation about this.
> > > >
> > > > > 204. There is still "The highest metadata offset which the broker
> > has not
> > > > > reached" referenced under BrokerRegistration.
> > > > >
> > > >
> > > > It should be CurrentMetadataOffset. Fixed.
> > > >
> > > > > 206. Is that separate step needed given KIP-516? With KIP-516 (
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > > ),
> > > > > we don't need to wait for the topic data to be removed from all
> > brokers
> > > > > before removing the topic metadata. The combination of unmatching
> > > > > topicId
> > > > > or the missing topicId from the metadata is enough for the broker to
> > > > > clean
> > > > > up deleted topics asynchronously.
> > > >
> > > > It won't be needed once KIP-516 is adopted, but this hasn't been
> > > > implemented yet.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the updated KIP. A few more comments below.
> > > > > > >
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Thanks again for the reviews.
> > > > > >
> > > > > > > 80.2 For deprecated configs, we need to include zookeeper.* and
> > > > > > > broker.id.generation.enable.
> > > > > > >
> > > > > >
> > > > > > Added.
> > > > > >
> > > > > > > 83.1 If a broker is down, does the controller keep the previously
> > > > > > > registered broker epoch forever? If not, how long does the
> > controller
> > > > > > keep
> > > > > > > it? What does the controller do when receiving a broker heartbeat
> > > > request
> > > > > > > with an unfound broker epoch?
> > > > > > >
> > > > > >
> > > > > > Yes, the controller keeps the previous registration forever.
> > > > > >
> > > > > > Broker heartbeat requests with an incorrect broker epoch will be
> > > > rejected
> > > > > > with STALE_BROKER_EPOCH.
> > > > > >
> > > > > > > 100. Have you figured out if we need to add a new record type for
> > > > > > reporting
> > > > > > > partitions on failed disks?
> > > > > > >
> > > > > >
> > > > > > I added FailedReplicaRecord to reflect the case where a JBOD
> > directory
> > > > has
> > > > > > failed, leading to failed replicas.
> > > > > >
> > > > > > > 102. For debugging purposes, sometimes it's useful to read the
> > > > metadata
> > > > > > > topic using tools like console-consumer. Should we support that
> > and
> > > > if
> > > > > > so,
> > > > > > > how?
> > > > > > >
> > > > > >
> > > > > > For now, we have the ability to read the metadata logs with the
> > > > dump-logs
> > > > > > tool. I think we will come up with some other tools in the future
> > as
> > > > we
> > > > > > get experience.
> > > > > >
> > > > > > > 200. "brokers which are fenced will not appear in
> > MetadataResponses.
> > > > The
> > > > > > > broker will not respond to these requests-- instead, it will
> > simply
> > > > > > > disconnect." If the controller is partitioned off from the
> > brokers,
> > > > this
> > > > > > > design will cause every broker to stop accepting new client
> > > > requests. In
> > > > > > > contrast, if ZK is partitioned off, the existing behavior is
> > that the
> > > > > > > brokers can continue to work based on the last known metadata.
> > So, I
> > > > am
> > > > > > not
> > > > > > > sure if we should change the existing behavior because of the
> > bigger
> > > > > > impact
> > > > > > > in the new one. Another option is to keep the existing behavior
> > and
> > > > > > expose
> > > > > > > a metric for fenced brokers so that the operator could be
> > alerted.
> > > > > > >
> > > > > >
> > > > > > I'm skeptical about how well running without ZK currently works.
> > > > However,
> > > > > > I will move the broker-side fencing into a follow-up KIP. This
> > KIP is
> > > > > > already pretty large and there is no hard dependency on this.
> > There
> > > > may
> > > > > > also be other ways of accomplishing the positive effects of what
> > > > > > broker-side fencing, so more discussion is needed.
> > > > > >
> > > > > > > 201. I read Ron's comment, but I am still not sure the benefit of
> > > > keeping
> > > > > > > broker.id and controller.id in meta.properties. It seems that
> > we are
> > > > > > just
> > > > > > > duplicating the same info in two places and have the additional
> > > > burden of
> > > > > > > making sure the values in the two places are consistent.
> > > > > > >
> > > > > >
> > > > > > I think the reasoning is that having broker.id protects us against
> > > > > > accidentally bringing up a broker with a disk from a different
> > > > broker. I
> > > > > > don't feel strongly about this but it seemed simpler to keep it.
> > > > > >
> > > > > > > 202. controller.connect.security.protocol: Is this needed since
> > > > > > > controller.listener.names and listener.security.protocol.map
> > imply
> > > > the
> > > > > > > security protocol already?
> > > > > > >
> > > > > >
> > > > > > You're right, this isn't needed. I'll remove it.
> > > > > >
> > > > > > > 203. registration.heartbeat.interval.ms: It defaults to 2k. ZK
> > uses
> > > > 1/3
> > > > > > of
> > > > > > > the session timeout for heartbeat. So, given the default 18k for
> > > > > > > registration.lease.timeout.ms, should we default
> > > > > > > registration.heartbeat.interval.ms to 6k?
> > > > > > >
> > > > > >
> > > > > > 6 seconds seems like a pretty long time between heartbeats. It
> > might
> > > > be
> > > > > > useful to know when a broker is missing heartbeats, with less time
> > than
> > > > > > that. I provisionally set it to 3 seconds (we can always change
> > > > later...)
> > > > > >
> > > > > > I also changed the name of these configurations to "
> > > > > > broker.heartbeat.interval.ms" and "broker.registration.timeout.ms"
> > to
> > > > try
> > > > > > to clarify them a bit.
> > > > > >
> > > > > > > 204. "The highest metadata offset which the broker has not
> > reached."
> > > > It
> > > > > > > seems this should be "has reached".
> > > > > > >
> > > > > >
> > > > > > I changed this to "one more than the highest metadata offset which
> > the
> > > > > > broker has reached."
> > > > > >
> > > > > > > 205. UnfenceBrokerRecord and UnregisterBrokerRecord: To me, they
> > > > seem to
> > > > > > be
> > > > > > > the same. Do we need both?
> > > > > > >
> > > > > >
> > > > > > Unregistration means that the broker has been removed from the
> > cluster.
> > > > > > That is different than unfencing, which marks the broker as active.
> > > > > >
> > > > > > > 206. TopicRecord: The Deleting field is used to indicate that the
> > > > topic
> > > > > > is
> > > > > > > being deleted. I am wondering if this is really needed since
> > > > RemoveTopic
> > > > > > > already indicates the same thing.
> > > > > > >
> > > > > >
> > > > > > RemoveTopic is the last step, that scrubs all metadata about the
> > topic.
> > > > > > In order to get to that last step, the topic data needs to removed
> > > > from all
> > > > > > brokers (after each broker notices that the topic is being
> > deleted).
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <cm...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > > > > > > > > Hi Colin. Thanks for the updates. It's now clear to me that
> > > > brokers
> > > > > > > > > keep their broker epoch for the life of their JVM -- they
> > > > register
> > > > > > > > > once, get their broker epoch in the response, and then never
> > > > > > > > > re-register again. Brokers may get fenced, but they keep the
> > > > same
> > > > > > > > > broker epoch for the life of their JVM. The incarnation ID
> > is
> > > > also
> > > > > > > > > kept for the life of the JVM but is generated by the broker
> > > > itself
> > > > > > > > > upon startup, and the combination of the two allows the
> > > > Controller to
> > > > > > > > > act idempotently if any previously-sent registration response
> > > > gets
> > > > > > > > > lost. Makes sense.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Thanks, Ron. That's a good summary.
> > > > > > > >
> > > > > > > > > One thing I wonder about is if it might be helpful for the
> > > > broker to
> > > > > > > > > send the Cluster ID as determined from its meta.properties
> > file
> > > > in
> > > > > > its
> > > > > > > > > registration request. Does it even make sense for the
> > broker to
> > > > > > > > > successfully register and enter the Fenced state if it has
> > the
> > > > wrong
> > > > > > > > > Cluster ID?
> > > > > > > >
> > > > > > > > Yeah, that's a good idea. Let's have the broker pass its
> > cluster
> > > > ID in
> > > > > > > > the registration RPC, and then registration can fail if the
> > broker
> > > > is
> > > > > > > > configured for the wrong cluster.
> > > > > > > >
> > > > > > > > > The nextMetadatOffset value that the broker communicates
> > > > > > > > > in its registration request only has meaning within the
> > correct
> > > > > > > > > cluster, so it feels to me that the Controller should have
> > some
> > > > way
> > > > > > to
> > > > > > > > > perform this sanity check. There is currently (pre-KIP 500)
> > a
> > > > check
> > > > > > > > > in the broker to make sure its configured cluster ID matches
> > the
> > > > one
> > > > > > > > > stored in ZooKeeper, and we will have to perform this
> > validation
> > > > > > > > > somewhere in the KIP-500 world. If the Controller doesn't
> > do it
> > > > > > > > > within the registration request then the broker will have to
> > > > make a
> > > > > > > > > metadata request to the Controller, retrieve the Cluster ID,
> > and
> > > > > > > > > perform the check itself. It feels to me that it might be
> > > > better for
> > > > > > > > > the Controller to just do it, and then the broker doesn't
> > have to
> > > > > > > > > worry about it anymore once it successfully registers.
> > > > > > > > >
> > > > > > > > > I also have a question about the broker.id value and
> > > > > > meta.properties.
> > > > > > > > > The KIP now says "In version 0 of meta.properties, there is a
> > > > > > > > > broker.id field. Version 1 does not have this field. It
> > is no
> > > > > > longer
> > > > > > > > > needed because we no longer support dynamic broker id
> > > > assignment."
> > > > > > > > > But then there is an example version 1 meta.properties file
> > that
> > > > > > shows
> > > > > > > > > the broker.id value. I actually wonder if maybe the
> > broker.id
> > > > value
> > > > > > > > > would be good to keep in the version 1 meta.properties file
> > > > because
> > > > > > it
> > > > > > > > > currently (pre-KIP 500, version 0) acts as a sanity check to
> > make
> > > > > > sure
> > > > > > > > > the broker is using the correct log directory. Similarly
> > with
> > > > the
> > > > > > > > > controller.id value on controllers -- it would allow the
> > same
> > > > type
> > > > > > of
> > > > > > > > > sanity check for quorum controllers.
> > > > > > > > >
> > > > > > > >
> > > > > > > > That's a good point. I will add broker.id back, and also add
> > > > > > > > controller.id as a possibility.
> > > > > > > >
> > > > > > > > cheers,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <
> > cmccabe@apache.org
> > > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > > > > > > > > Hi, Colin,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the reply. A few more comments.
> > > > > > > > > >
> > > > > > > > > > Hi Jun,
> > > > > > > > > >
> > > > > > > > > > Thanks again for the reply. Sorry for the long hiatus. I
> > was
> > > > on
> > > > > > > > vacation for a while.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 55. There is still text that favors new broker
> > registration.
> > > > > > "When a
> > > > > > > > broker
> > > > > > > > > > > first starts up, when it is in the INITIAL state, it will
> > > > always
> > > > > > > > "win"
> > > > > > > > > > > broker ID conflicts. However, once it is granted a
> > lease, it
> > > > > > > > transitions
> > > > > > > > > > > out of the INITIAL state. Thereafter, it may lose
> > subsequent
> > > > > > > > conflicts if
> > > > > > > > > > > its broker epoch is stale. (See KIP-380 for some
> > background
> > > > on
> > > > > > > > broker
> > > > > > > > > > > epoch.) The reason for favoring new processes is to
> > > > accommodate
> > > > > > the
> > > > > > > > common
> > > > > > > > > > > case where a process is killed with kill -9 and then
> > > > restarted.
> > > > > > We
> > > > > > > > want it
> > > > > > > > > > > to be able to reclaim its old ID quickly in this case."
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks for the reminder. I have clarified the language
> > here.
> > > > > > > > Hopefully now it is clear that we don't allow quick re-use of
> > > > broker
> > > > > > IDs.
> > > > > > > > > >
> > > > > > > > > > > 80.1 Sounds good. Could you document that listeners is a
> > > > required
> > > > > > > > config
> > > > > > > > > > > now? It would also be useful to annotate other required
> > > > configs.
> > > > > > For
> > > > > > > > > > > example, controller.connect should be required.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I added a note specifying that these are required.
> > > > > > > > > >
> > > > > > > > > > > 80.2 Could you list all deprecated existing configs?
> > Another
> > > > one
> > > > > > is
> > > > > > > > > > > control.plane.listener.name since the controller no
> > longer
> > > > sends
> > > > > > > > > > > LeaderAndIsr, UpdateMetadata and StopReplica requests.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I added a section specifying some deprecated configs.
> > > > > > > > > >
> > > > > > > > > > > 83.1 It seems that the broker can transition from FENCED
> > to
> > > > > > RUNNING
> > > > > > > > without
> > > > > > > > > > > registering for a new broker epoch. I am not sure how
> > this
> > > > works.
> > > > > > > > Once the
> > > > > > > > > > > controller fences a broker, there is no need for the
> > > > controller
> > > > > > to
> > > > > > > > keep the
> > > > > > > > > > > boker epoch around. So, if the fenced broker's heartbeat
> > > > request
> > > > > > > > with the
> > > > > > > > > > > existing broker epoch will be rejected, leading the
> > broker
> > > > back
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > FENCED state again.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The broker epoch refers to the broker registration. So we
> > DO
> > > > keep
> > > > > > the
> > > > > > > > broker epoch around even while the broker is fenced.
> > > > > > > > > >
> > > > > > > > > > The broker epoch changes only when there is a new broker
> > > > > > > > registration. Fencing or unfencing the broker doesn't change
> > the
> > > > > > broker
> > > > > > > > epoch.
> > > > > > > > > >
> > > > > > > > > > > 83.5 Good point on KIP-590. Then should we expose the
> > > > controller
> > > > > > for
> > > > > > > > > > > debugging purposes? If not, we should deprecate the
> > > > controllerID
> > > > > > > > field in
> > > > > > > > > > > MetadataResponse?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I think it's OK to expose it for now, with the proviso
> > that it
> > > > > > won't
> > > > > > > > be reachable by clients.
> > > > > > > > > >
> > > > > > > > > > > 90. We rejected the shared ID with just one reason "This
> > is
> > > > not a
> > > > > > > > good idea
> > > > > > > > > > > because NetworkClient assumes a single ID space. So if
> > > > there is
> > > > > > > > both a
> > > > > > > > > > > controller 1 and a broker 1, we don't have a way of
> > picking
> > > > the
> > > > > > > > "right"
> > > > > > > > > > > one." This doesn't seem to be a strong reason. For
> > example,
> > > > we
> > > > > > could
> > > > > > > > > > > address the NetworkClient issue with the node type as you
> > > > pointed
> > > > > > > > out or
> > > > > > > > > > > using the negative value of a broker ID as the
> > controller ID.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > It would require a lot of code changes to support multiple
> > > > types of
> > > > > > > > node IDs. It's not clear to me that the end result would be
> > > > better --
> > > > > > I
> > > > > > > > tend to think it would be worse, since it would be more
> > complex.
> > > > In a
> > > > > > > > similar vein, using negative numbers seems dangerous, since we
> > use
> > > > > > > > negatives or -1 as "special values" in many places. For
> > example,
> > > > -1
> > > > > > often
> > > > > > > > represents "no such node."
> > > > > > > > > >
> > > > > > > > > > One important thing to keep in mind is that we want to be
> > able
> > > > to
> > > > > > > > transition from a broker and a controller being co-located to
> > them
> > > > no
> > > > > > > > longer being co-located. This is much easier to do when they
> > have
> > > > > > separate
> > > > > > > > IDs.
> > > > > > > > > >
> > > > > > > > > > > 100. In KIP-589
> > > > > > > > > > > <
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > > >,
> > > > > > > > > > > the broker reports all offline replicas due to a disk
> > > > failure to
> > > > > > the
> > > > > > > > > > > controller. It seems this information needs to be
> > persisted
> > > > to
> > > > > > the
> > > > > > > > > > > metadata
> > > > > > > > > > > log. Do we have a corresponding record for that?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hmm, I have to look into this a little bit more. We may
> > need
> > > > a new
> > > > > > > > record type.
> > > > > > > > > >
> > > > > > > > > > > 101. Currently, StopReplica request has 2 modes, without
> > > > deletion
> > > > > > > > and with
> > > > > > > > > > > deletion. The former is used for controlled shutdown and
> > > > handling
> > > > > > > > disk
> > > > > > > > > > > failure, and causes the follower to stop. The latter is
> > for
> > > > topic
> > > > > > > > deletion
> > > > > > > > > > > and partition reassignment, and causes the replica to be
> > > > deleted.
> > > > > > > > Since we
> > > > > > > > > > > are deprecating StopReplica, could we document what
> > triggers
> > > > the
> > > > > > > > stopping
> > > > > > > > > > > of a follower and the deleting of a replica now?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > RemoveTopic triggers deletion. In general the
> > functionality of
> > > > > > > > StopReplica is subsumed by the metadata records.
> > > > > > > > > >
> > > > > > > > > > > 102. Should we include the metadata topic in the
> > > > > > MetadataResponse?
> > > > > > > > If so,
> > > > > > > > > > > when it will be included and what will the metadata
> > response
> > > > look
> > > > > > > > like?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > No, it won't be included in the metadata response sent back
> > > > from
> > > > > > the
> > > > > > > > brokers.
> > > > > > > > > >
> > > > > > > > > > > 103. "The active controller assigns the broker a new
> > broker
> > > > > > epoch,
> > > > > > > > based on
> > > > > > > > > > > the latest committed offset in the log." This seems
> > > > inaccurate
> > > > > > since
> > > > > > > > the
> > > > > > > > > > > latest committed offset doesn't always advance on every
> > log
> > > > > > append.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Given that the new broker epoch won't be visible until the
> > > > commit
> > > > > > has
> > > > > > > > happened, I have changed this to "the next available offset in
> > the
> > > > log"
> > > > > > > > > >
> > > > > > > > > > > 104. REGISTERING(1) : It says "Otherwise, the broker
> > moves
> > > > into
> > > > > > the
> > > > > > > > FENCED
> > > > > > > > > > > state.". It seems this should be RUNNING?
> > > > > > > > > > >
> > > > > > > > > > > 105. RUNNING: Should we require the broker to catch up
> > to the
> > > > > > > > metadata log
> > > > > > > > > > > to get into this state?
> > > > > > > > > >
> > > > > > > > > > For 104 and 105, these sections have been reworked.
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <
> > > > cmccabe@apache.org
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > > > > > > > > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have some questions about what happens to a
> > broker
> > > > > > when it
> > > > > > > > becomes
> > > > > > > > > > > > > > > fenced (e.g. because it can't send a heartbeat
> > > > request to
> > > > > > > > keep its
> > > > > > > > > > > > > > > lease). The KIP says "When a broker is fenced,
> > it
> > > > cannot
> > > > > > > > process any
> > > > > > > > > > > > > > > client requests. This prevents brokers which
> > are not
> > > > > > > > receiving
> > > > > > > > > > > > > > > metadata updates or that are not receiving and
> > > > processing
> > > > > > > > them fast
> > > > > > > > > > > > > > > enough from causing issues to clients." And in
> > the
> > > > > > > > description of the
> > > > > > > > > > > > > > > FENCED(4) state it likewise says "While in this
> > > > state,
> > > > > > the
> > > > > > > > broker
> > > > > > > > > > > > does
> > > > > > > > > > > > > > > not respond to client requests." It makes sense
> > > > that a
> > > > > > > > fenced broker
> > > > > > > > > > > > > > > should not accept producer requests -- I assume
> > any
> > > > such
> > > > > > > > requests
> > > > > > > > > > > > > > > would result in NotLeaderOrFollowerException.
> > But
> > > > what
> > > > > > > > about KIP-392
> > > > > > > > > > > > > > > (fetch from follower) consumer requests? It is
> > > > > > conceivable
> > > > > > > > that
> > > > > > > > > > > > these
> > > > > > > > > > > > > > > could continue. Related to that, would a fenced
> > > > broker
> > > > > > > > continue to
> > > > > > > > > > > > > > > fetch data for partitions where it thinks it is a
> > > > > > follower?
> > > > > > > > Even if
> > > > > > > > > > > > > > > it rejects consumer requests it might still
> > continue
> > > > to
> > > > > > > > fetch as a
> > > > > > > > > > > > > > > follower. Might it be helpful to clarify both
> > > > decisions
> > > > > > > > here?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Good question. I think a fenced broker should
> > > > continue to
> > > > > > > > fetch on
> > > > > > > > > > > > > > partitions it was already fetching before it was
> > > > fenced,
> > > > > > > > unless it
> > > > > > > > > > > > hits a
> > > > > > > > > > > > > > problem. At that point it won't be able to
> > continue,
> > > > > > since it
> > > > > > > > doesn't
> > > > > > > > > > > > have
> > > > > > > > > > > > > > the new metadata. For example, it won't know about
> > > > > > leadership
> > > > > > > > changes
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the partitions it's fetching. The rationale for
> > > > > > continuing to
> > > > > > > > fetch
> > > > > > > > > > > > is to
> > > > > > > > > > > > > > try to avoid disruptions as much as possible.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I don't think fenced brokers should accept client
> > > > requests.
> > > > > > > > The issue
> > > > > > > > > > > > is
> > > > > > > > > > > > > > that the fenced broker may or may not have any
> > data it
> > > > is
> > > > > > > > supposed to
> > > > > > > > > > > > > > have. It may or may not have applied any
> > configuration
> > > > > > > > changes, etc.
> > > > > > > > > > > > that
> > > > > > > > > > > > > > it is supposed to have applied. So it could get
> > pretty
> > > > > > > > confusing, and
> > > > > > > > > > > > also
> > > > > > > > > > > > > > potentially waste the client's time.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > When fenced, how would the broker reply to a client
> > > > which did
> > > > > > > > make a
> > > > > > > > > > > > > request?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > >
> > > > > > > > > > > > The broker will respond with a retryable error in that
> > > > case.
> > > > > > Once
> > > > > > > > the
> > > > > > > > > > > > client has re-fetched its metadata, it will no longer
> > see
> > > > the
> > > > > > > > fenced broker
> > > > > > > > > > > > as part of the cluster. I added a note to the KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Tom
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Tom Bentley <tb...@redhat.com>.
Hi Colin,
The KIP says that "brokers which are fenced will not appear in
MetadataResponses. So clients that have up-to-date metadata will not try
to contact fenced brokers.", which is fine, but it doesn't seem to
elaborate on what happens for clients which try to contact a broker (using
stale metadata, for example) and find it in the registered-but-fenced
state. You said previously that the broker will respond with a retriable
error, which is again fine, but you didn't answer my question about which
error code you've chosen for this. I realise that this doesn't really
affect the correctness of the behaviour, but I'm not aware of any existing
error code which is a good fit. So it would be good to understand about how
you're making this backward compatible for clients.
Many thanks,
Tom
On Tue, Dec 15, 2020 at 1:42 AM Colin McCabe <cm...@apache.org> wrote:
> On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the reply. Just a couple of more comments below.
> >
> > 210. Since we are deprecating zookeeper.connection.timeout.ms, should we
> > add a new config to bound the time for a broker to connect to the
> > controller during starting up?
> >
>
> Good idea. I added initial.broker.registration.timeout.ms for this.
>
> > 211. BrokerHeartbeat no longer has the state field in the
> request/response.
> > However, (a) the controller shutdown section still has "In its periodic
> > heartbeats, the broker asks the controller if it can transition into the
> > SHUTDOWN state. This motivates the controller to move all of the leaders
> > off of that broker. Once they are all moved, the controller responds to
> > the heartbeat with a nextState of SHUTDOWN."; (2) the description of
> > BrokerHeartbeat still references currentState and targetState.
> >
>
> Thanks. I've made these sections clearer and removed the obsolete
> references to sending states.
>
> best,
> Colin
>
> > Jun
> >
> > On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the update. A few more follow up comments.
> > > >
> > >
> > > Hi Jun,
> > >
> > > Thanks again for the review.
> > >
> > > > 100. FailedReplicaRecord: Since this is reported by each broker
> > > > independently, perhaps we could use a more concise representation
> that
> > > has
> > > > a top level broker field, an array of topics, which has an array of
> > > > partitions.
> > > >
> > >
> > > The issue is that there is a size limit on the each record. Putting
> all
> > > of the partitions of a log directory into a single record would
> probably
> > > break that in many cases. Still, we can optimize a bit by having an
> array
> > > of partition IDs, since nearly all the time, we have more than one
> from the
> > > same topic.
> > >
> > > > 200. Sounds good. If we remove the broker-side fencing logic, do we
> plan
> > > to
> > > > still keep FENCED in broker state? Do we plan to expose the new
> states
> > > > through the existing BrokerState metric and if so, what are the
> values
> > > for
> > > > the new states?
> > > >
> > >
> > > No, we don't need FENCED any more. I have removed it from the KIP.
> > >
> > > The new states are very similar to the current ones, actually. There
> are
> > > no new states or removed ones. The main change in the broker state
> machine
> > > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state has been renamed to
> > > RECOVERY. Also, unlike previously, the broker will always pass through
> > > RECOVERY (although it may only stay in this state for a very short
> amount
> > > of time).
> > >
> > > > 201. This may be fine too. Could we document what happens when the
> > > > broker.id/controller.id in metadata.properties don't match the
> broker
> > > > config when the broker starts up?
> > > >
> > >
> > > I added some documentation about this.
> > >
> > > > 204. There is still "The highest metadata offset which the broker
> has not
> > > > reached" referenced under BrokerRegistration.
> > > >
> > >
> > > It should be CurrentMetadataOffset. Fixed.
> > >
> > > > 206. Is that separate step needed given KIP-516? With KIP-516 (
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > > ),
> > > > we don't need to wait for the topic data to be removed from all
> brokers
> > > > before removing the topic metadata. The combination of unmatching
> > > > topicId
> > > > or the missing topicId from the metadata is enough for the broker to
> > > > clean
> > > > up deleted topics asynchronously.
> > >
> > > It won't be needed once KIP-516 is adopted, but this hasn't been
> > > implemented yet.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > Jun
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the updated KIP. A few more comments below.
> > > > > >
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > Thanks again for the reviews.
> > > > >
> > > > > > 80.2 For deprecated configs, we need to include zookeeper.* and
> > > > > > broker.id.generation.enable.
> > > > > >
> > > > >
> > > > > Added.
> > > > >
> > > > > > 83.1 If a broker is down, does the controller keep the previously
> > > > > > registered broker epoch forever? If not, how long does the
> controller
> > > > > keep
> > > > > > it? What does the controller do when receiving a broker heartbeat
> > > request
> > > > > > with an unfound broker epoch?
> > > > > >
> > > > >
> > > > > Yes, the controller keeps the previous registration forever.
> > > > >
> > > > > Broker heartbeat requests with an incorrect broker epoch will be
> > > rejected
> > > > > with STALE_BROKER_EPOCH.
> > > > >
> > > > > > 100. Have you figured out if we need to add a new record type for
> > > > > reporting
> > > > > > partitions on failed disks?
> > > > > >
> > > > >
> > > > > I added FailedReplicaRecord to reflect the case where a JBOD
> directory
> > > has
> > > > > failed, leading to failed replicas.
> > > > >
> > > > > > 102. For debugging purposes, sometimes it's useful to read the
> > > metadata
> > > > > > topic using tools like console-consumer. Should we support that
> and
> > > if
> > > > > so,
> > > > > > how?
> > > > > >
> > > > >
> > > > > For now, we have the ability to read the metadata logs with the
> > > dump-logs
> > > > > tool. I think we will come up with some other tools in the future
> as
> > > we
> > > > > get experience.
> > > > >
> > > > > > 200. "brokers which are fenced will not appear in
> MetadataResponses.
> > > The
> > > > > > broker will not respond to these requests-- instead, it will
> simply
> > > > > > disconnect." If the controller is partitioned off from the
> brokers,
> > > this
> > > > > > design will cause every broker to stop accepting new client
> > > requests. In
> > > > > > contrast, if ZK is partitioned off, the existing behavior is
> that the
> > > > > > brokers can continue to work based on the last known metadata.
> So, I
> > > am
> > > > > not
> > > > > > sure if we should change the existing behavior because of the
> bigger
> > > > > impact
> > > > > > in the new one. Another option is to keep the existing behavior
> and
> > > > > expose
> > > > > > a metric for fenced brokers so that the operator could be
> alerted.
> > > > > >
> > > > >
> > > > > I'm skeptical about how well running without ZK currently works.
> > > However,
> > > > > I will move the broker-side fencing into a follow-up KIP. This
> KIP is
> > > > > already pretty large and there is no hard dependency on this.
> There
> > > may
> > > > > also be other ways of accomplishing the positive effects of what
> > > > > broker-side fencing, so more discussion is needed.
> > > > >
> > > > > > 201. I read Ron's comment, but I am still not sure the benefit of
> > > keeping
> > > > > > broker.id and controller.id in meta.properties. It seems that
> we are
> > > > > just
> > > > > > duplicating the same info in two places and have the additional
> > > burden of
> > > > > > making sure the values in the two places are consistent.
> > > > > >
> > > > >
> > > > > I think the reasoning is that having broker.id protects us against
> > > > > accidentally bringing up a broker with a disk from a different
> > > broker. I
> > > > > don't feel strongly about this but it seemed simpler to keep it.
> > > > >
> > > > > > 202. controller.connect.security.protocol: Is this needed since
> > > > > > controller.listener.names and listener.security.protocol.map
> imply
> > > the
> > > > > > security protocol already?
> > > > > >
> > > > >
> > > > > You're right, this isn't needed. I'll remove it.
> > > > >
> > > > > > 203. registration.heartbeat.interval.ms: It defaults to 2k. ZK
> uses
> > > 1/3
> > > > > of
> > > > > > the session timeout for heartbeat. So, given the default 18k for
> > > > > > registration.lease.timeout.ms, should we default
> > > > > > registration.heartbeat.interval.ms to 6k?
> > > > > >
> > > > >
> > > > > 6 seconds seems like a pretty long time between heartbeats. It
> might
> > > be
> > > > > useful to know when a broker is missing heartbeats, with less time
> than
> > > > > that. I provisionally set it to 3 seconds (we can always change
> > > later...)
> > > > >
> > > > > I also changed the name of these configurations to "
> > > > > broker.heartbeat.interval.ms" and "broker.registration.timeout.ms"
> to
> > > try
> > > > > to clarify them a bit.
> > > > >
> > > > > > 204. "The highest metadata offset which the broker has not
> reached."
> > > It
> > > > > > seems this should be "has reached".
> > > > > >
> > > > >
> > > > > I changed this to "one more than the highest metadata offset which
> the
> > > > > broker has reached."
> > > > >
> > > > > > 205. UnfenceBrokerRecord and UnregisterBrokerRecord: To me, they
> > > seem to
> > > > > be
> > > > > > the same. Do we need both?
> > > > > >
> > > > >
> > > > > Unregistration means that the broker has been removed from the
> cluster.
> > > > > That is different than unfencing, which marks the broker as active.
> > > > >
> > > > > > 206. TopicRecord: The Deleting field is used to indicate that the
> > > topic
> > > > > is
> > > > > > being deleted. I am wondering if this is really needed since
> > > RemoveTopic
> > > > > > already indicates the same thing.
> > > > > >
> > > > >
> > > > > RemoveTopic is the last step, that scrubs all metadata about the
> topic.
> > > > > In order to get to that last step, the topic data needs to removed
> > > from all
> > > > > brokers (after each broker notices that the topic is being
> deleted).
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <cm...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > > > > > > > Hi Colin. Thanks for the updates. It's now clear to me that
> > > brokers
> > > > > > > > keep their broker epoch for the life of their JVM -- they
> > > register
> > > > > > > > once, get their broker epoch in the response, and then never
> > > > > > > > re-register again. Brokers may get fenced, but they keep the
> > > same
> > > > > > > > broker epoch for the life of their JVM. The incarnation ID
> is
> > > also
> > > > > > > > kept for the life of the JVM but is generated by the broker
> > > itself
> > > > > > > > upon startup, and the combination of the two allows the
> > > Controller to
> > > > > > > > act idempotently if any previously-sent registration response
> > > gets
> > > > > > > > lost. Makes sense.
> > > > > > > >
> > > > > > >
> > > > > > > Thanks, Ron. That's a good summary.
> > > > > > >
> > > > > > > > One thing I wonder about is if it might be helpful for the
> > > broker to
> > > > > > > > send the Cluster ID as determined from its meta.properties
> file
> > > in
> > > > > its
> > > > > > > > registration request. Does it even make sense for the
> broker to
> > > > > > > > successfully register and enter the Fenced state if it has
> the
> > > wrong
> > > > > > > > Cluster ID?
> > > > > > >
> > > > > > > Yeah, that's a good idea. Let's have the broker pass its
> cluster
> > > ID in
> > > > > > > the registration RPC, and then registration can fail if the
> broker
> > > is
> > > > > > > configured for the wrong cluster.
> > > > > > >
> > > > > > > > The nextMetadatOffset value that the broker communicates
> > > > > > > > in its registration request only has meaning within the
> correct
> > > > > > > > cluster, so it feels to me that the Controller should have
> some
> > > way
> > > > > to
> > > > > > > > perform this sanity check. There is currently (pre-KIP 500)
> a
> > > check
> > > > > > > > in the broker to make sure its configured cluster ID matches
> the
> > > one
> > > > > > > > stored in ZooKeeper, and we will have to perform this
> validation
> > > > > > > > somewhere in the KIP-500 world. If the Controller doesn't
> do it
> > > > > > > > within the registration request then the broker will have to
> > > make a
> > > > > > > > metadata request to the Controller, retrieve the Cluster ID,
> and
> > > > > > > > perform the check itself. It feels to me that it might be
> > > better for
> > > > > > > > the Controller to just do it, and then the broker doesn't
> have to
> > > > > > > > worry about it anymore once it successfully registers.
> > > > > > > >
> > > > > > > > I also have a question about the broker.id value and
> > > > > meta.properties.
> > > > > > > > The KIP now says "In version 0 of meta.properties, there is a
> > > > > > > > broker.id field. Version 1 does not have this field. It
> is no
> > > > > longer
> > > > > > > > needed because we no longer support dynamic broker id
> > > assignment."
> > > > > > > > But then there is an example version 1 meta.properties file
> that
> > > > > shows
> > > > > > > > the broker.id value. I actually wonder if maybe the
> broker.id
> > > value
> > > > > > > > would be good to keep in the version 1 meta.properties file
> > > because
> > > > > it
> > > > > > > > currently (pre-KIP 500, version 0) acts as a sanity check to
> make
> > > > > sure
> > > > > > > > the broker is using the correct log directory. Similarly
> with
> > > the
> > > > > > > > controller.id value on controllers -- it would allow the
> same
> > > type
> > > > > of
> > > > > > > > sanity check for quorum controllers.
> > > > > > > >
> > > > > > >
> > > > > > > That's a good point. I will add broker.id back, and also add
> > > > > > > controller.id as a possibility.
> > > > > > >
> > > > > > > cheers,
> > > > > > > Colin
> > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <
> cmccabe@apache.org
> > > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > > > > > > > Hi, Colin,
> > > > > > > > > >
> > > > > > > > > > Thanks for the reply. A few more comments.
> > > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > Thanks again for the reply. Sorry for the long hiatus. I
> was
> > > on
> > > > > > > vacation for a while.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 55. There is still text that favors new broker
> registration.
> > > > > "When a
> > > > > > > broker
> > > > > > > > > > first starts up, when it is in the INITIAL state, it will
> > > always
> > > > > > > "win"
> > > > > > > > > > broker ID conflicts. However, once it is granted a
> lease, it
> > > > > > > transitions
> > > > > > > > > > out of the INITIAL state. Thereafter, it may lose
> subsequent
> > > > > > > conflicts if
> > > > > > > > > > its broker epoch is stale. (See KIP-380 for some
> background
> > > on
> > > > > > > broker
> > > > > > > > > > epoch.) The reason for favoring new processes is to
> > > accommodate
> > > > > the
> > > > > > > common
> > > > > > > > > > case where a process is killed with kill -9 and then
> > > restarted.
> > > > > We
> > > > > > > want it
> > > > > > > > > > to be able to reclaim its old ID quickly in this case."
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks for the reminder. I have clarified the language
> here.
> > > > > > > Hopefully now it is clear that we don't allow quick re-use of
> > > broker
> > > > > IDs.
> > > > > > > > >
> > > > > > > > > > 80.1 Sounds good. Could you document that listeners is a
> > > required
> > > > > > > config
> > > > > > > > > > now? It would also be useful to annotate other required
> > > configs.
> > > > > For
> > > > > > > > > > example, controller.connect should be required.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I added a note specifying that these are required.
> > > > > > > > >
> > > > > > > > > > 80.2 Could you list all deprecated existing configs?
> Another
> > > one
> > > > > is
> > > > > > > > > > control.plane.listener.name since the controller no
> longer
> > > sends
> > > > > > > > > > LeaderAndIsr, UpdateMetadata and StopReplica requests.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I added a section specifying some deprecated configs.
> > > > > > > > >
> > > > > > > > > > 83.1 It seems that the broker can transition from FENCED
> to
> > > > > RUNNING
> > > > > > > without
> > > > > > > > > > registering for a new broker epoch. I am not sure how
> this
> > > works.
> > > > > > > Once the
> > > > > > > > > > controller fences a broker, there is no need for the
> > > controller
> > > > > to
> > > > > > > keep the
> > > > > > > > > > boker epoch around. So, if the fenced broker's heartbeat
> > > request
> > > > > > > with the
> > > > > > > > > > existing broker epoch will be rejected, leading the
> broker
> > > back
> > > > > to
> > > > > > > the
> > > > > > > > > > FENCED state again.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > The broker epoch refers to the broker registration. So we
> DO
> > > keep
> > > > > the
> > > > > > > broker epoch around even while the broker is fenced.
> > > > > > > > >
> > > > > > > > > The broker epoch changes only when there is a new broker
> > > > > > > registration. Fencing or unfencing the broker doesn't change
> the
> > > > > broker
> > > > > > > epoch.
> > > > > > > > >
> > > > > > > > > > 83.5 Good point on KIP-590. Then should we expose the
> > > controller
> > > > > for
> > > > > > > > > > debugging purposes? If not, we should deprecate the
> > > controllerID
> > > > > > > field in
> > > > > > > > > > MetadataResponse?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I think it's OK to expose it for now, with the proviso
> that it
> > > > > won't
> > > > > > > be reachable by clients.
> > > > > > > > >
> > > > > > > > > > 90. We rejected the shared ID with just one reason "This
> is
> > > not a
> > > > > > > good idea
> > > > > > > > > > because NetworkClient assumes a single ID space. So if
> > > there is
> > > > > > > both a
> > > > > > > > > > controller 1 and a broker 1, we don't have a way of
> picking
> > > the
> > > > > > > "right"
> > > > > > > > > > one." This doesn't seem to be a strong reason. For
> example,
> > > we
> > > > > could
> > > > > > > > > > address the NetworkClient issue with the node type as you
> > > pointed
> > > > > > > out or
> > > > > > > > > > using the negative value of a broker ID as the
> controller ID.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > It would require a lot of code changes to support multiple
> > > types of
> > > > > > > node IDs. It's not clear to me that the end result would be
> > > better --
> > > > > I
> > > > > > > tend to think it would be worse, since it would be more
> complex.
> > > In a
> > > > > > > similar vein, using negative numbers seems dangerous, since we
> use
> > > > > > > negatives or -1 as "special values" in many places. For
> example,
> > > -1
> > > > > often
> > > > > > > represents "no such node."
> > > > > > > > >
> > > > > > > > > One important thing to keep in mind is that we want to be
> able
> > > to
> > > > > > > transition from a broker and a controller being co-located to
> them
> > > no
> > > > > > > longer being co-located. This is much easier to do when they
> have
> > > > > separate
> > > > > > > IDs.
> > > > > > > > >
> > > > > > > > > > 100. In KIP-589
> > > > > > > > > > <
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > > >,
> > > > > > > > > > the broker reports all offline replicas due to a disk
> > > failure to
> > > > > the
> > > > > > > > > > controller. It seems this information needs to be
> persisted
> > > to
> > > > > the
> > > > > > > > > > metadata
> > > > > > > > > > log. Do we have a corresponding record for that?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hmm, I have to look into this a little bit more. We may
> need
> > > a new
> > > > > > > record type.
> > > > > > > > >
> > > > > > > > > > 101. Currently, StopReplica request has 2 modes, without
> > > deletion
> > > > > > > and with
> > > > > > > > > > deletion. The former is used for controlled shutdown and
> > > handling
> > > > > > > disk
> > > > > > > > > > failure, and causes the follower to stop. The latter is
> for
> > > topic
> > > > > > > deletion
> > > > > > > > > > and partition reassignment, and causes the replica to be
> > > deleted.
> > > > > > > Since we
> > > > > > > > > > are deprecating StopReplica, could we document what
> triggers
> > > the
> > > > > > > stopping
> > > > > > > > > > of a follower and the deleting of a replica now?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > RemoveTopic triggers deletion. In general the
> functionality of
> > > > > > > StopReplica is subsumed by the metadata records.
> > > > > > > > >
> > > > > > > > > > 102. Should we include the metadata topic in the
> > > > > MetadataResponse?
> > > > > > > If so,
> > > > > > > > > > when it will be included and what will the metadata
> response
> > > look
> > > > > > > like?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > No, it won't be included in the metadata response sent back
> > > from
> > > > > the
> > > > > > > brokers.
> > > > > > > > >
> > > > > > > > > > 103. "The active controller assigns the broker a new
> broker
> > > > > epoch,
> > > > > > > based on
> > > > > > > > > > the latest committed offset in the log." This seems
> > > inaccurate
> > > > > since
> > > > > > > the
> > > > > > > > > > latest committed offset doesn't always advance on every
> log
> > > > > append.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Given that the new broker epoch won't be visible until the
> > > commit
> > > > > has
> > > > > > > happened, I have changed this to "the next available offset in
> the
> > > log"
> > > > > > > > >
> > > > > > > > > > 104. REGISTERING(1) : It says "Otherwise, the broker
> moves
> > > into
> > > > > the
> > > > > > > FENCED
> > > > > > > > > > state.". It seems this should be RUNNING?
> > > > > > > > > >
> > > > > > > > > > 105. RUNNING: Should we require the broker to catch up
> to the
> > > > > > > metadata log
> > > > > > > > > > to get into this state?
> > > > > > > > >
> > > > > > > > > For 104 and 105, these sections have been reworked.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <
> > > cmccabe@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > > > > > > > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have some questions about what happens to a
> broker
> > > > > when it
> > > > > > > becomes
> > > > > > > > > > > > > > fenced (e.g. because it can't send a heartbeat
> > > request to
> > > > > > > keep its
> > > > > > > > > > > > > > lease). The KIP says "When a broker is fenced,
> it
> > > cannot
> > > > > > > process any
> > > > > > > > > > > > > > client requests. This prevents brokers which
> are not
> > > > > > > receiving
> > > > > > > > > > > > > > metadata updates or that are not receiving and
> > > processing
> > > > > > > them fast
> > > > > > > > > > > > > > enough from causing issues to clients." And in
> the
> > > > > > > description of the
> > > > > > > > > > > > > > FENCED(4) state it likewise says "While in this
> > > state,
> > > > > the
> > > > > > > broker
> > > > > > > > > > > does
> > > > > > > > > > > > > > not respond to client requests." It makes sense
> > > that a
> > > > > > > fenced broker
> > > > > > > > > > > > > > should not accept producer requests -- I assume
> any
> > > such
> > > > > > > requests
> > > > > > > > > > > > > > would result in NotLeaderOrFollowerException.
> But
> > > what
> > > > > > > about KIP-392
> > > > > > > > > > > > > > (fetch from follower) consumer requests? It is
> > > > > conceivable
> > > > > > > that
> > > > > > > > > > > these
> > > > > > > > > > > > > > could continue. Related to that, would a fenced
> > > broker
> > > > > > > continue to
> > > > > > > > > > > > > > fetch data for partitions where it thinks it is a
> > > > > follower?
> > > > > > > Even if
> > > > > > > > > > > > > > it rejects consumer requests it might still
> continue
> > > to
> > > > > > > fetch as a
> > > > > > > > > > > > > > follower. Might it be helpful to clarify both
> > > decisions
> > > > > > > here?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Good question. I think a fenced broker should
> > > continue to
> > > > > > > fetch on
> > > > > > > > > > > > > partitions it was already fetching before it was
> > > fenced,
> > > > > > > unless it
> > > > > > > > > > > hits a
> > > > > > > > > > > > > problem. At that point it won't be able to
> continue,
> > > > > since it
> > > > > > > doesn't
> > > > > > > > > > > have
> > > > > > > > > > > > > the new metadata. For example, it won't know about
> > > > > leadership
> > > > > > > changes
> > > > > > > > > > > in
> > > > > > > > > > > > > the partitions it's fetching. The rationale for
> > > > > continuing to
> > > > > > > fetch
> > > > > > > > > > > is to
> > > > > > > > > > > > > try to avoid disruptions as much as possible.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't think fenced brokers should accept client
> > > requests.
> > > > > > > The issue
> > > > > > > > > > > is
> > > > > > > > > > > > > that the fenced broker may or may not have any
> data it
> > > is
> > > > > > > supposed to
> > > > > > > > > > > > > have. It may or may not have applied any
> configuration
> > > > > > > changes, etc.
> > > > > > > > > > > that
> > > > > > > > > > > > > it is supposed to have applied. So it could get
> pretty
> > > > > > > confusing, and
> > > > > > > > > > > also
> > > > > > > > > > > > > potentially waste the client's time.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > When fenced, how would the broker reply to a client
> > > which did
> > > > > > > make a
> > > > > > > > > > > > request?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Tom,
> > > > > > > > > > >
> > > > > > > > > > > The broker will respond with a retryable error in that
> > > case.
> > > > > Once
> > > > > > > the
> > > > > > > > > > > client has re-fetched its metadata, it will no longer
> see
> > > the
> > > > > > > fenced broker
> > > > > > > > > > > as part of the cluster. I added a note to the KIP.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Tom
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Fri, Dec 11, 2020, at 17:07, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the reply. Just a couple of more comments below.
>
> 210. Since we are deprecating zookeeper.connection.timeout.ms, should we
> add a new config to bound the time for a broker to connect to the
> controller during starting up?
>
Good idea. I added initial.broker.registration.timeout.ms for this.
> 211. BrokerHeartbeat no longer has the state field in the request/response.
> However, (a) the controller shutdown section still has "In its periodic
> heartbeats, the broker asks the controller if it can transition into the
> SHUTDOWN state. This motivates the controller to move all of the leaders
> off of that broker. Once they are all moved, the controller responds to
> the heartbeat with a nextState of SHUTDOWN."; (2) the description of
> BrokerHeartbeat still references currentState and targetState.
>
Thanks. I've made these sections clearer and removed the obsolete references to sending states.
best,
Colin
> Jun
>
> On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <cm...@apache.org> wrote:
>
> > On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the update. A few more follow up comments.
> > >
> >
> > Hi Jun,
> >
> > Thanks again for the review.
> >
> > > 100. FailedReplicaRecord: Since this is reported by each broker
> > > independently, perhaps we could use a more concise representation that
> > has
> > > a top level broker field, an array of topics, which has an array of
> > > partitions.
> > >
> >
> > The issue is that there is a size limit on the each record. Putting all
> > of the partitions of a log directory into a single record would probably
> > break that in many cases. Still, we can optimize a bit by having an array
> > of partition IDs, since nearly all the time, we have more than one from the
> > same topic.
> >
> > > 200. Sounds good. If we remove the broker-side fencing logic, do we plan
> > to
> > > still keep FENCED in broker state? Do we plan to expose the new states
> > > through the existing BrokerState metric and if so, what are the values
> > for
> > > the new states?
> > >
> >
> > No, we don't need FENCED any more. I have removed it from the KIP.
> >
> > The new states are very similar to the current ones, actually. There are
> > no new states or removed ones. The main change in the broker state machine
> > is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state has been renamed to
> > RECOVERY. Also, unlike previously, the broker will always pass through
> > RECOVERY (although it may only stay in this state for a very short amount
> > of time).
> >
> > > 201. This may be fine too. Could we document what happens when the
> > > broker.id/controller.id in metadata.properties don't match the broker
> > > config when the broker starts up?
> > >
> >
> > I added some documentation about this.
> >
> > > 204. There is still "The highest metadata offset which the broker has not
> > > reached" referenced under BrokerRegistration.
> > >
> >
> > It should be CurrentMetadataOffset. Fixed.
> >
> > > 206. Is that separate step needed given KIP-516? With KIP-516 (
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> > ),
> > > we don't need to wait for the topic data to be removed from all brokers
> > > before removing the topic metadata. The combination of unmatching
> > > topicId
> > > or the missing topicId from the metadata is enough for the broker to
> > > clean
> > > up deleted topics asynchronously.
> >
> > It won't be needed once KIP-516 is adopted, but this hasn't been
> > implemented yet.
> >
> > best,
> > Colin
> >
> > >
> > > Jun
> > >
> > >
> > >
> > >
> > > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the updated KIP. A few more comments below.
> > > > >
> > > >
> > > > Hi Jun,
> > > >
> > > > Thanks again for the reviews.
> > > >
> > > > > 80.2 For deprecated configs, we need to include zookeeper.* and
> > > > > broker.id.generation.enable.
> > > > >
> > > >
> > > > Added.
> > > >
> > > > > 83.1 If a broker is down, does the controller keep the previously
> > > > > registered broker epoch forever? If not, how long does the controller
> > > > keep
> > > > > it? What does the controller do when receiving a broker heartbeat
> > request
> > > > > with an unfound broker epoch?
> > > > >
> > > >
> > > > Yes, the controller keeps the previous registration forever.
> > > >
> > > > Broker heartbeat requests with an incorrect broker epoch will be
> > rejected
> > > > with STALE_BROKER_EPOCH.
> > > >
> > > > > 100. Have you figured out if we need to add a new record type for
> > > > reporting
> > > > > partitions on failed disks?
> > > > >
> > > >
> > > > I added FailedReplicaRecord to reflect the case where a JBOD directory
> > has
> > > > failed, leading to failed replicas.
> > > >
> > > > > 102. For debugging purposes, sometimes it's useful to read the
> > metadata
> > > > > topic using tools like console-consumer. Should we support that and
> > if
> > > > so,
> > > > > how?
> > > > >
> > > >
> > > > For now, we have the ability to read the metadata logs with the
> > dump-logs
> > > > tool. I think we will come up with some other tools in the future as
> > we
> > > > get experience.
> > > >
> > > > > 200. "brokers which are fenced will not appear in MetadataResponses.
> > The
> > > > > broker will not respond to these requests-- instead, it will simply
> > > > > disconnect." If the controller is partitioned off from the brokers,
> > this
> > > > > design will cause every broker to stop accepting new client
> > requests. In
> > > > > contrast, if ZK is partitioned off, the existing behavior is that the
> > > > > brokers can continue to work based on the last known metadata. So, I
> > am
> > > > not
> > > > > sure if we should change the existing behavior because of the bigger
> > > > impact
> > > > > in the new one. Another option is to keep the existing behavior and
> > > > expose
> > > > > a metric for fenced brokers so that the operator could be alerted.
> > > > >
> > > >
> > > > I'm skeptical about how well running without ZK currently works.
> > However,
> > > > I will move the broker-side fencing into a follow-up KIP. This KIP is
> > > > already pretty large and there is no hard dependency on this. There
> > may
> > > > also be other ways of accomplishing the positive effects of what
> > > > broker-side fencing, so more discussion is needed.
> > > >
> > > > > 201. I read Ron's comment, but I am still not sure the benefit of
> > keeping
> > > > > broker.id and controller.id in meta.properties. It seems that we are
> > > > just
> > > > > duplicating the same info in two places and have the additional
> > burden of
> > > > > making sure the values in the two places are consistent.
> > > > >
> > > >
> > > > I think the reasoning is that having broker.id protects us against
> > > > accidentally bringing up a broker with a disk from a different
> > broker. I
> > > > don't feel strongly about this but it seemed simpler to keep it.
> > > >
> > > > > 202. controller.connect.security.protocol: Is this needed since
> > > > > controller.listener.names and listener.security.protocol.map imply
> > the
> > > > > security protocol already?
> > > > >
> > > >
> > > > You're right, this isn't needed. I'll remove it.
> > > >
> > > > > 203. registration.heartbeat.interval.ms: It defaults to 2k. ZK uses
> > 1/3
> > > > of
> > > > > the session timeout for heartbeat. So, given the default 18k for
> > > > > registration.lease.timeout.ms, should we default
> > > > > registration.heartbeat.interval.ms to 6k?
> > > > >
> > > >
> > > > 6 seconds seems like a pretty long time between heartbeats. It might
> > be
> > > > useful to know when a broker is missing heartbeats, with less time than
> > > > that. I provisionally set it to 3 seconds (we can always change
> > later...)
> > > >
> > > > I also changed the name of these configurations to "
> > > > broker.heartbeat.interval.ms" and "broker.registration.timeout.ms" to
> > try
> > > > to clarify them a bit.
> > > >
> > > > > 204. "The highest metadata offset which the broker has not reached."
> > It
> > > > > seems this should be "has reached".
> > > > >
> > > >
> > > > I changed this to "one more than the highest metadata offset which the
> > > > broker has reached."
> > > >
> > > > > 205. UnfenceBrokerRecord and UnregisterBrokerRecord: To me, they
> > seem to
> > > > be
> > > > > the same. Do we need both?
> > > > >
> > > >
> > > > Unregistration means that the broker has been removed from the cluster.
> > > > That is different than unfencing, which marks the broker as active.
> > > >
> > > > > 206. TopicRecord: The Deleting field is used to indicate that the
> > topic
> > > > is
> > > > > being deleted. I am wondering if this is really needed since
> > RemoveTopic
> > > > > already indicates the same thing.
> > > > >
> > > >
> > > > RemoveTopic is the last step, that scrubs all metadata about the topic.
> > > > In order to get to that last step, the topic data needs to removed
> > from all
> > > > brokers (after each broker notices that the topic is being deleted).
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > > > > > > Hi Colin. Thanks for the updates. It's now clear to me that
> > brokers
> > > > > > > keep their broker epoch for the life of their JVM -- they
> > register
> > > > > > > once, get their broker epoch in the response, and then never
> > > > > > > re-register again. Brokers may get fenced, but they keep the
> > same
> > > > > > > broker epoch for the life of their JVM. The incarnation ID is
> > also
> > > > > > > kept for the life of the JVM but is generated by the broker
> > itself
> > > > > > > upon startup, and the combination of the two allows the
> > Controller to
> > > > > > > act idempotently if any previously-sent registration response
> > gets
> > > > > > > lost. Makes sense.
> > > > > > >
> > > > > >
> > > > > > Thanks, Ron. That's a good summary.
> > > > > >
> > > > > > > One thing I wonder about is if it might be helpful for the
> > broker to
> > > > > > > send the Cluster ID as determined from its meta.properties file
> > in
> > > > its
> > > > > > > registration request. Does it even make sense for the broker to
> > > > > > > successfully register and enter the Fenced state if it has the
> > wrong
> > > > > > > Cluster ID?
> > > > > >
> > > > > > Yeah, that's a good idea. Let's have the broker pass its cluster
> > ID in
> > > > > > the registration RPC, and then registration can fail if the broker
> > is
> > > > > > configured for the wrong cluster.
> > > > > >
> > > > > > > The nextMetadatOffset value that the broker communicates
> > > > > > > in its registration request only has meaning within the correct
> > > > > > > cluster, so it feels to me that the Controller should have some
> > way
> > > > to
> > > > > > > perform this sanity check. There is currently (pre-KIP 500) a
> > check
> > > > > > > in the broker to make sure its configured cluster ID matches the
> > one
> > > > > > > stored in ZooKeeper, and we will have to perform this validation
> > > > > > > somewhere in the KIP-500 world. If the Controller doesn't do it
> > > > > > > within the registration request then the broker will have to
> > make a
> > > > > > > metadata request to the Controller, retrieve the Cluster ID, and
> > > > > > > perform the check itself. It feels to me that it might be
> > better for
> > > > > > > the Controller to just do it, and then the broker doesn't have to
> > > > > > > worry about it anymore once it successfully registers.
> > > > > > >
> > > > > > > I also have a question about the broker.id value and
> > > > meta.properties.
> > > > > > > The KIP now says "In version 0 of meta.properties, there is a
> > > > > > > broker.id field. Version 1 does not have this field. It is no
> > > > longer
> > > > > > > needed because we no longer support dynamic broker id
> > assignment."
> > > > > > > But then there is an example version 1 meta.properties file that
> > > > shows
> > > > > > > the broker.id value. I actually wonder if maybe the broker.id
> > value
> > > > > > > would be good to keep in the version 1 meta.properties file
> > because
> > > > it
> > > > > > > currently (pre-KIP 500, version 0) acts as a sanity check to make
> > > > sure
> > > > > > > the broker is using the correct log directory. Similarly with
> > the
> > > > > > > controller.id value on controllers -- it would allow the same
> > type
> > > > of
> > > > > > > sanity check for quorum controllers.
> > > > > > >
> > > > > >
> > > > > > That's a good point. I will add broker.id back, and also add
> > > > > > controller.id as a possibility.
> > > > > >
> > > > > > cheers,
> > > > > > Colin
> > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <cmccabe@apache.org
> > >
> > > > wrote:
> > > > > > > >
> > > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > > > > > > Hi, Colin,
> > > > > > > > >
> > > > > > > > > Thanks for the reply. A few more comments.
> > > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > Thanks again for the reply. Sorry for the long hiatus. I was
> > on
> > > > > > vacation for a while.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 55. There is still text that favors new broker registration.
> > > > "When a
> > > > > > broker
> > > > > > > > > first starts up, when it is in the INITIAL state, it will
> > always
> > > > > > "win"
> > > > > > > > > broker ID conflicts. However, once it is granted a lease, it
> > > > > > transitions
> > > > > > > > > out of the INITIAL state. Thereafter, it may lose subsequent
> > > > > > conflicts if
> > > > > > > > > its broker epoch is stale. (See KIP-380 for some background
> > on
> > > > > > broker
> > > > > > > > > epoch.) The reason for favoring new processes is to
> > accommodate
> > > > the
> > > > > > common
> > > > > > > > > case where a process is killed with kill -9 and then
> > restarted.
> > > > We
> > > > > > want it
> > > > > > > > > to be able to reclaim its old ID quickly in this case."
> > > > > > > > >
> > > > > > > >
> > > > > > > > Thanks for the reminder. I have clarified the language here.
> > > > > > Hopefully now it is clear that we don't allow quick re-use of
> > broker
> > > > IDs.
> > > > > > > >
> > > > > > > > > 80.1 Sounds good. Could you document that listeners is a
> > required
> > > > > > config
> > > > > > > > > now? It would also be useful to annotate other required
> > configs.
> > > > For
> > > > > > > > > example, controller.connect should be required.
> > > > > > > > >
> > > > > > > >
> > > > > > > > I added a note specifying that these are required.
> > > > > > > >
> > > > > > > > > 80.2 Could you list all deprecated existing configs? Another
> > one
> > > > is
> > > > > > > > > control.plane.listener.name since the controller no longer
> > sends
> > > > > > > > > LeaderAndIsr, UpdateMetadata and StopReplica requests.
> > > > > > > > >
> > > > > > > >
> > > > > > > > I added a section specifying some deprecated configs.
> > > > > > > >
> > > > > > > > > 83.1 It seems that the broker can transition from FENCED to
> > > > RUNNING
> > > > > > without
> > > > > > > > > registering for a new broker epoch. I am not sure how this
> > works.
> > > > > > Once the
> > > > > > > > > controller fences a broker, there is no need for the
> > controller
> > > > to
> > > > > > keep the
> > > > > > > > > boker epoch around. So, if the fenced broker's heartbeat
> > request
> > > > > > with the
> > > > > > > > > existing broker epoch will be rejected, leading the broker
> > back
> > > > to
> > > > > > the
> > > > > > > > > FENCED state again.
> > > > > > > > >
> > > > > > > >
> > > > > > > > The broker epoch refers to the broker registration. So we DO
> > keep
> > > > the
> > > > > > broker epoch around even while the broker is fenced.
> > > > > > > >
> > > > > > > > The broker epoch changes only when there is a new broker
> > > > > > registration. Fencing or unfencing the broker doesn't change the
> > > > broker
> > > > > > epoch.
> > > > > > > >
> > > > > > > > > 83.5 Good point on KIP-590. Then should we expose the
> > controller
> > > > for
> > > > > > > > > debugging purposes? If not, we should deprecate the
> > controllerID
> > > > > > field in
> > > > > > > > > MetadataResponse?
> > > > > > > > >
> > > > > > > >
> > > > > > > > I think it's OK to expose it for now, with the proviso that it
> > > > won't
> > > > > > be reachable by clients.
> > > > > > > >
> > > > > > > > > 90. We rejected the shared ID with just one reason "This is
> > not a
> > > > > > good idea
> > > > > > > > > because NetworkClient assumes a single ID space. So if
> > there is
> > > > > > both a
> > > > > > > > > controller 1 and a broker 1, we don't have a way of picking
> > the
> > > > > > "right"
> > > > > > > > > one." This doesn't seem to be a strong reason. For example,
> > we
> > > > could
> > > > > > > > > address the NetworkClient issue with the node type as you
> > pointed
> > > > > > out or
> > > > > > > > > using the negative value of a broker ID as the controller ID.
> > > > > > > > >
> > > > > > > >
> > > > > > > > It would require a lot of code changes to support multiple
> > types of
> > > > > > node IDs. It's not clear to me that the end result would be
> > better --
> > > > I
> > > > > > tend to think it would be worse, since it would be more complex.
> > In a
> > > > > > similar vein, using negative numbers seems dangerous, since we use
> > > > > > negatives or -1 as "special values" in many places. For example,
> > -1
> > > > often
> > > > > > represents "no such node."
> > > > > > > >
> > > > > > > > One important thing to keep in mind is that we want to be able
> > to
> > > > > > transition from a broker and a controller being co-located to them
> > no
> > > > > > longer being co-located. This is much easier to do when they have
> > > > separate
> > > > > > IDs.
> > > > > > > >
> > > > > > > > > 100. In KIP-589
> > > > > > > > > <
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > > >,
> > > > > > > > > the broker reports all offline replicas due to a disk
> > failure to
> > > > the
> > > > > > > > > controller. It seems this information needs to be persisted
> > to
> > > > the
> > > > > > > > > metadata
> > > > > > > > > log. Do we have a corresponding record for that?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hmm, I have to look into this a little bit more. We may need
> > a new
> > > > > > record type.
> > > > > > > >
> > > > > > > > > 101. Currently, StopReplica request has 2 modes, without
> > deletion
> > > > > > and with
> > > > > > > > > deletion. The former is used for controlled shutdown and
> > handling
> > > > > > disk
> > > > > > > > > failure, and causes the follower to stop. The latter is for
> > topic
> > > > > > deletion
> > > > > > > > > and partition reassignment, and causes the replica to be
> > deleted.
> > > > > > Since we
> > > > > > > > > are deprecating StopReplica, could we document what triggers
> > the
> > > > > > stopping
> > > > > > > > > of a follower and the deleting of a replica now?
> > > > > > > > >
> > > > > > > >
> > > > > > > > RemoveTopic triggers deletion. In general the functionality of
> > > > > > StopReplica is subsumed by the metadata records.
> > > > > > > >
> > > > > > > > > 102. Should we include the metadata topic in the
> > > > MetadataResponse?
> > > > > > If so,
> > > > > > > > > when it will be included and what will the metadata response
> > look
> > > > > > like?
> > > > > > > > >
> > > > > > > >
> > > > > > > > No, it won't be included in the metadata response sent back
> > from
> > > > the
> > > > > > brokers.
> > > > > > > >
> > > > > > > > > 103. "The active controller assigns the broker a new broker
> > > > epoch,
> > > > > > based on
> > > > > > > > > the latest committed offset in the log." This seems
> > inaccurate
> > > > since
> > > > > > the
> > > > > > > > > latest committed offset doesn't always advance on every log
> > > > append.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Given that the new broker epoch won't be visible until the
> > commit
> > > > has
> > > > > > happened, I have changed this to "the next available offset in the
> > log"
> > > > > > > >
> > > > > > > > > 104. REGISTERING(1) : It says "Otherwise, the broker moves
> > into
> > > > the
> > > > > > FENCED
> > > > > > > > > state.". It seems this should be RUNNING?
> > > > > > > > >
> > > > > > > > > 105. RUNNING: Should we require the broker to catch up to the
> > > > > > metadata log
> > > > > > > > > to get into this state?
> > > > > > > >
> > > > > > > > For 104 and 105, these sections have been reworked.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <
> > cmccabe@apache.org
> > > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > > > > > > > > Hi Colin,
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > > > > > > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have some questions about what happens to a broker
> > > > when it
> > > > > > becomes
> > > > > > > > > > > > > fenced (e.g. because it can't send a heartbeat
> > request to
> > > > > > keep its
> > > > > > > > > > > > > lease). The KIP says "When a broker is fenced, it
> > cannot
> > > > > > process any
> > > > > > > > > > > > > client requests. This prevents brokers which are not
> > > > > > receiving
> > > > > > > > > > > > > metadata updates or that are not receiving and
> > processing
> > > > > > them fast
> > > > > > > > > > > > > enough from causing issues to clients." And in the
> > > > > > description of the
> > > > > > > > > > > > > FENCED(4) state it likewise says "While in this
> > state,
> > > > the
> > > > > > broker
> > > > > > > > > > does
> > > > > > > > > > > > > not respond to client requests." It makes sense
> > that a
> > > > > > fenced broker
> > > > > > > > > > > > > should not accept producer requests -- I assume any
> > such
> > > > > > requests
> > > > > > > > > > > > > would result in NotLeaderOrFollowerException. But
> > what
> > > > > > about KIP-392
> > > > > > > > > > > > > (fetch from follower) consumer requests? It is
> > > > conceivable
> > > > > > that
> > > > > > > > > > these
> > > > > > > > > > > > > could continue. Related to that, would a fenced
> > broker
> > > > > > continue to
> > > > > > > > > > > > > fetch data for partitions where it thinks it is a
> > > > follower?
> > > > > > Even if
> > > > > > > > > > > > > it rejects consumer requests it might still continue
> > to
> > > > > > fetch as a
> > > > > > > > > > > > > follower. Might it be helpful to clarify both
> > decisions
> > > > > > here?
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > >
> > > > > > > > > > > > Good question. I think a fenced broker should
> > continue to
> > > > > > fetch on
> > > > > > > > > > > > partitions it was already fetching before it was
> > fenced,
> > > > > > unless it
> > > > > > > > > > hits a
> > > > > > > > > > > > problem. At that point it won't be able to continue,
> > > > since it
> > > > > > doesn't
> > > > > > > > > > have
> > > > > > > > > > > > the new metadata. For example, it won't know about
> > > > leadership
> > > > > > changes
> > > > > > > > > > in
> > > > > > > > > > > > the partitions it's fetching. The rationale for
> > > > continuing to
> > > > > > fetch
> > > > > > > > > > is to
> > > > > > > > > > > > try to avoid disruptions as much as possible.
> > > > > > > > > > > >
> > > > > > > > > > > > I don't think fenced brokers should accept client
> > requests.
> > > > > > The issue
> > > > > > > > > > is
> > > > > > > > > > > > that the fenced broker may or may not have any data it
> > is
> > > > > > supposed to
> > > > > > > > > > > > have. It may or may not have applied any configuration
> > > > > > changes, etc.
> > > > > > > > > > that
> > > > > > > > > > > > it is supposed to have applied. So it could get pretty
> > > > > > confusing, and
> > > > > > > > > > also
> > > > > > > > > > > > potentially waste the client's time.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > When fenced, how would the broker reply to a client
> > which did
> > > > > > make a
> > > > > > > > > > > request?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi Tom,
> > > > > > > > > >
> > > > > > > > > > The broker will respond with a retryable error in that
> > case.
> > > > Once
> > > > > > the
> > > > > > > > > > client has re-fetched its metadata, it will no longer see
> > the
> > > > > > fenced broker
> > > > > > > > > > as part of the cluster. I added a note to the KIP.
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Tom
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the reply. Just a couple of more comments below.
210. Since we are deprecating zookeeper.connection.timeout.ms, should we
add a new config to bound the time for a broker to connect to the
controller during starting up?
211. BrokerHeartbeat no longer has the state field in the request/response.
However, (a) the controller shutdown section still has "In its periodic
heartbeats, the broker asks the controller if it can transition into the
SHUTDOWN state. This motivates the controller to move all of the leaders
off of that broker. Once they are all moved, the controller responds to
the heartbeat with a nextState of SHUTDOWN."; (2) the description of
BrokerHeartbeat still references currentState and targetState.
Jun
On Fri, Dec 11, 2020 at 1:33 PM Colin McCabe <cm...@apache.org> wrote:
> On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the update. A few more follow up comments.
> >
>
> Hi Jun,
>
> Thanks again for the review.
>
> > 100. FailedReplicaRecord: Since this is reported by each broker
> > independently, perhaps we could use a more concise representation that
> has
> > a top level broker field, an array of topics, which has an array of
> > partitions.
> >
>
> The issue is that there is a size limit on the each record. Putting all
> of the partitions of a log directory into a single record would probably
> break that in many cases. Still, we can optimize a bit by having an array
> of partition IDs, since nearly all the time, we have more than one from the
> same topic.
>
> > 200. Sounds good. If we remove the broker-side fencing logic, do we plan
> to
> > still keep FENCED in broker state? Do we plan to expose the new states
> > through the existing BrokerState metric and if so, what are the values
> for
> > the new states?
> >
>
> No, we don't need FENCED any more. I have removed it from the KIP.
>
> The new states are very similar to the current ones, actually. There are
> no new states or removed ones. The main change in the broker state machine
> is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state has been renamed to
> RECOVERY. Also, unlike previously, the broker will always pass through
> RECOVERY (although it may only stay in this state for a very short amount
> of time).
>
> > 201. This may be fine too. Could we document what happens when the
> > broker.id/controller.id in metadata.properties don't match the broker
> > config when the broker starts up?
> >
>
> I added some documentation about this.
>
> > 204. There is still "The highest metadata offset which the broker has not
> > reached" referenced under BrokerRegistration.
> >
>
> It should be CurrentMetadataOffset. Fixed.
>
> > 206. Is that separate step needed given KIP-516? With KIP-516 (
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr
> ),
> > we don't need to wait for the topic data to be removed from all brokers
> > before removing the topic metadata. The combination of unmatching
> > topicId
> > or the missing topicId from the metadata is enough for the broker to
> > clean
> > up deleted topics asynchronously.
>
> It won't be needed once KIP-516 is adopted, but this hasn't been
> implemented yet.
>
> best,
> Colin
>
> >
> > Jun
> >
> >
> >
> >
> > On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the updated KIP. A few more comments below.
> > > >
> > >
> > > Hi Jun,
> > >
> > > Thanks again for the reviews.
> > >
> > > > 80.2 For deprecated configs, we need to include zookeeper.* and
> > > > broker.id.generation.enable.
> > > >
> > >
> > > Added.
> > >
> > > > 83.1 If a broker is down, does the controller keep the previously
> > > > registered broker epoch forever? If not, how long does the controller
> > > keep
> > > > it? What does the controller do when receiving a broker heartbeat
> request
> > > > with an unfound broker epoch?
> > > >
> > >
> > > Yes, the controller keeps the previous registration forever.
> > >
> > > Broker heartbeat requests with an incorrect broker epoch will be
> rejected
> > > with STALE_BROKER_EPOCH.
> > >
> > > > 100. Have you figured out if we need to add a new record type for
> > > reporting
> > > > partitions on failed disks?
> > > >
> > >
> > > I added FailedReplicaRecord to reflect the case where a JBOD directory
> has
> > > failed, leading to failed replicas.
> > >
> > > > 102. For debugging purposes, sometimes it's useful to read the
> metadata
> > > > topic using tools like console-consumer. Should we support that and
> if
> > > so,
> > > > how?
> > > >
> > >
> > > For now, we have the ability to read the metadata logs with the
> dump-logs
> > > tool. I think we will come up with some other tools in the future as
> we
> > > get experience.
> > >
> > > > 200. "brokers which are fenced will not appear in MetadataResponses.
> The
> > > > broker will not respond to these requests-- instead, it will simply
> > > > disconnect." If the controller is partitioned off from the brokers,
> this
> > > > design will cause every broker to stop accepting new client
> requests. In
> > > > contrast, if ZK is partitioned off, the existing behavior is that the
> > > > brokers can continue to work based on the last known metadata. So, I
> am
> > > not
> > > > sure if we should change the existing behavior because of the bigger
> > > impact
> > > > in the new one. Another option is to keep the existing behavior and
> > > expose
> > > > a metric for fenced brokers so that the operator could be alerted.
> > > >
> > >
> > > I'm skeptical about how well running without ZK currently works.
> However,
> > > I will move the broker-side fencing into a follow-up KIP. This KIP is
> > > already pretty large and there is no hard dependency on this. There
> may
> > > also be other ways of accomplishing the positive effects of what
> > > broker-side fencing, so more discussion is needed.
> > >
> > > > 201. I read Ron's comment, but I am still not sure the benefit of
> keeping
> > > > broker.id and controller.id in meta.properties. It seems that we are
> > > just
> > > > duplicating the same info in two places and have the additional
> burden of
> > > > making sure the values in the two places are consistent.
> > > >
> > >
> > > I think the reasoning is that having broker.id protects us against
> > > accidentally bringing up a broker with a disk from a different
> broker. I
> > > don't feel strongly about this but it seemed simpler to keep it.
> > >
> > > > 202. controller.connect.security.protocol: Is this needed since
> > > > controller.listener.names and listener.security.protocol.map imply
> the
> > > > security protocol already?
> > > >
> > >
> > > You're right, this isn't needed. I'll remove it.
> > >
> > > > 203. registration.heartbeat.interval.ms: It defaults to 2k. ZK uses
> 1/3
> > > of
> > > > the session timeout for heartbeat. So, given the default 18k for
> > > > registration.lease.timeout.ms, should we default
> > > > registration.heartbeat.interval.ms to 6k?
> > > >
> > >
> > > 6 seconds seems like a pretty long time between heartbeats. It might
> be
> > > useful to know when a broker is missing heartbeats, with less time than
> > > that. I provisionally set it to 3 seconds (we can always change
> later...)
> > >
> > > I also changed the name of these configurations to "
> > > broker.heartbeat.interval.ms" and "broker.registration.timeout.ms" to
> try
> > > to clarify them a bit.
> > >
> > > > 204. "The highest metadata offset which the broker has not reached."
> It
> > > > seems this should be "has reached".
> > > >
> > >
> > > I changed this to "one more than the highest metadata offset which the
> > > broker has reached."
> > >
> > > > 205. UnfenceBrokerRecord and UnregisterBrokerRecord: To me, they
> seem to
> > > be
> > > > the same. Do we need both?
> > > >
> > >
> > > Unregistration means that the broker has been removed from the cluster.
> > > That is different than unfencing, which marks the broker as active.
> > >
> > > > 206. TopicRecord: The Deleting field is used to indicate that the
> topic
> > > is
> > > > being deleted. I am wondering if this is really needed since
> RemoveTopic
> > > > already indicates the same thing.
> > > >
> > >
> > > RemoveTopic is the last step, that scrubs all metadata about the topic.
> > > In order to get to that last step, the topic data needs to removed
> from all
> > > brokers (after each broker notices that the topic is being deleted).
> > >
> > > best,
> > > Colin
> > >
> > > > Jun
> > > >
> > > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > > > > > Hi Colin. Thanks for the updates. It's now clear to me that
> brokers
> > > > > > keep their broker epoch for the life of their JVM -- they
> register
> > > > > > once, get their broker epoch in the response, and then never
> > > > > > re-register again. Brokers may get fenced, but they keep the
> same
> > > > > > broker epoch for the life of their JVM. The incarnation ID is
> also
> > > > > > kept for the life of the JVM but is generated by the broker
> itself
> > > > > > upon startup, and the combination of the two allows the
> Controller to
> > > > > > act idempotently if any previously-sent registration response
> gets
> > > > > > lost. Makes sense.
> > > > > >
> > > > >
> > > > > Thanks, Ron. That's a good summary.
> > > > >
> > > > > > One thing I wonder about is if it might be helpful for the
> broker to
> > > > > > send the Cluster ID as determined from its meta.properties file
> in
> > > its
> > > > > > registration request. Does it even make sense for the broker to
> > > > > > successfully register and enter the Fenced state if it has the
> wrong
> > > > > > Cluster ID?
> > > > >
> > > > > Yeah, that's a good idea. Let's have the broker pass its cluster
> ID in
> > > > > the registration RPC, and then registration can fail if the broker
> is
> > > > > configured for the wrong cluster.
> > > > >
> > > > > > The nextMetadatOffset value that the broker communicates
> > > > > > in its registration request only has meaning within the correct
> > > > > > cluster, so it feels to me that the Controller should have some
> way
> > > to
> > > > > > perform this sanity check. There is currently (pre-KIP 500) a
> check
> > > > > > in the broker to make sure its configured cluster ID matches the
> one
> > > > > > stored in ZooKeeper, and we will have to perform this validation
> > > > > > somewhere in the KIP-500 world. If the Controller doesn't do it
> > > > > > within the registration request then the broker will have to
> make a
> > > > > > metadata request to the Controller, retrieve the Cluster ID, and
> > > > > > perform the check itself. It feels to me that it might be
> better for
> > > > > > the Controller to just do it, and then the broker doesn't have to
> > > > > > worry about it anymore once it successfully registers.
> > > > > >
> > > > > > I also have a question about the broker.id value and
> > > meta.properties.
> > > > > > The KIP now says "In version 0 of meta.properties, there is a
> > > > > > broker.id field. Version 1 does not have this field. It is no
> > > longer
> > > > > > needed because we no longer support dynamic broker id
> assignment."
> > > > > > But then there is an example version 1 meta.properties file that
> > > shows
> > > > > > the broker.id value. I actually wonder if maybe the broker.id
> value
> > > > > > would be good to keep in the version 1 meta.properties file
> because
> > > it
> > > > > > currently (pre-KIP 500, version 0) acts as a sanity check to make
> > > sure
> > > > > > the broker is using the correct log directory. Similarly with
> the
> > > > > > controller.id value on controllers -- it would allow the same
> type
> > > of
> > > > > > sanity check for quorum controllers.
> > > > > >
> > > > >
> > > > > That's a good point. I will add broker.id back, and also add
> > > > > controller.id as a possibility.
> > > > >
> > > > > cheers,
> > > > > Colin
> > > > >
> > > > > >
> > > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <cmccabe@apache.org
> >
> > > wrote:
> > > > > > >
> > > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the reply. A few more comments.
> > > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > Thanks again for the reply. Sorry for the long hiatus. I was
> on
> > > > > vacation for a while.
> > > > > > >
> > > > > > > >
> > > > > > > > 55. There is still text that favors new broker registration.
> > > "When a
> > > > > broker
> > > > > > > > first starts up, when it is in the INITIAL state, it will
> always
> > > > > "win"
> > > > > > > > broker ID conflicts. However, once it is granted a lease, it
> > > > > transitions
> > > > > > > > out of the INITIAL state. Thereafter, it may lose subsequent
> > > > > conflicts if
> > > > > > > > its broker epoch is stale. (See KIP-380 for some background
> on
> > > > > broker
> > > > > > > > epoch.) The reason for favoring new processes is to
> accommodate
> > > the
> > > > > common
> > > > > > > > case where a process is killed with kill -9 and then
> restarted.
> > > We
> > > > > want it
> > > > > > > > to be able to reclaim its old ID quickly in this case."
> > > > > > > >
> > > > > > >
> > > > > > > Thanks for the reminder. I have clarified the language here.
> > > > > Hopefully now it is clear that we don't allow quick re-use of
> broker
> > > IDs.
> > > > > > >
> > > > > > > > 80.1 Sounds good. Could you document that listeners is a
> required
> > > > > config
> > > > > > > > now? It would also be useful to annotate other required
> configs.
> > > For
> > > > > > > > example, controller.connect should be required.
> > > > > > > >
> > > > > > >
> > > > > > > I added a note specifying that these are required.
> > > > > > >
> > > > > > > > 80.2 Could you list all deprecated existing configs? Another
> one
> > > is
> > > > > > > > control.plane.listener.name since the controller no longer
> sends
> > > > > > > > LeaderAndIsr, UpdateMetadata and StopReplica requests.
> > > > > > > >
> > > > > > >
> > > > > > > I added a section specifying some deprecated configs.
> > > > > > >
> > > > > > > > 83.1 It seems that the broker can transition from FENCED to
> > > RUNNING
> > > > > without
> > > > > > > > registering for a new broker epoch. I am not sure how this
> works.
> > > > > Once the
> > > > > > > > controller fences a broker, there is no need for the
> controller
> > > to
> > > > > keep the
> > > > > > > > boker epoch around. So, if the fenced broker's heartbeat
> request
> > > > > with the
> > > > > > > > existing broker epoch will be rejected, leading the broker
> back
> > > to
> > > > > the
> > > > > > > > FENCED state again.
> > > > > > > >
> > > > > > >
> > > > > > > The broker epoch refers to the broker registration. So we DO
> keep
> > > the
> > > > > broker epoch around even while the broker is fenced.
> > > > > > >
> > > > > > > The broker epoch changes only when there is a new broker
> > > > > registration. Fencing or unfencing the broker doesn't change the
> > > broker
> > > > > epoch.
> > > > > > >
> > > > > > > > 83.5 Good point on KIP-590. Then should we expose the
> controller
> > > for
> > > > > > > > debugging purposes? If not, we should deprecate the
> controllerID
> > > > > field in
> > > > > > > > MetadataResponse?
> > > > > > > >
> > > > > > >
> > > > > > > I think it's OK to expose it for now, with the proviso that it
> > > won't
> > > > > be reachable by clients.
> > > > > > >
> > > > > > > > 90. We rejected the shared ID with just one reason "This is
> not a
> > > > > good idea
> > > > > > > > because NetworkClient assumes a single ID space. So if
> there is
> > > > > both a
> > > > > > > > controller 1 and a broker 1, we don't have a way of picking
> the
> > > > > "right"
> > > > > > > > one." This doesn't seem to be a strong reason. For example,
> we
> > > could
> > > > > > > > address the NetworkClient issue with the node type as you
> pointed
> > > > > out or
> > > > > > > > using the negative value of a broker ID as the controller ID.
> > > > > > > >
> > > > > > >
> > > > > > > It would require a lot of code changes to support multiple
> types of
> > > > > node IDs. It's not clear to me that the end result would be
> better --
> > > I
> > > > > tend to think it would be worse, since it would be more complex.
> In a
> > > > > similar vein, using negative numbers seems dangerous, since we use
> > > > > negatives or -1 as "special values" in many places. For example,
> -1
> > > often
> > > > > represents "no such node."
> > > > > > >
> > > > > > > One important thing to keep in mind is that we want to be able
> to
> > > > > transition from a broker and a controller being co-located to them
> no
> > > > > longer being co-located. This is much easier to do when they have
> > > separate
> > > > > IDs.
> > > > > > >
> > > > > > > > 100. In KIP-589
> > > > > > > > <
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > > >,
> > > > > > > > the broker reports all offline replicas due to a disk
> failure to
> > > the
> > > > > > > > controller. It seems this information needs to be persisted
> to
> > > the
> > > > > > > > metadata
> > > > > > > > log. Do we have a corresponding record for that?
> > > > > > > >
> > > > > > >
> > > > > > > Hmm, I have to look into this a little bit more. We may need
> a new
> > > > > record type.
> > > > > > >
> > > > > > > > 101. Currently, StopReplica request has 2 modes, without
> deletion
> > > > > and with
> > > > > > > > deletion. The former is used for controlled shutdown and
> handling
> > > > > disk
> > > > > > > > failure, and causes the follower to stop. The latter is for
> topic
> > > > > deletion
> > > > > > > > and partition reassignment, and causes the replica to be
> deleted.
> > > > > Since we
> > > > > > > > are deprecating StopReplica, could we document what triggers
> the
> > > > > stopping
> > > > > > > > of a follower and the deleting of a replica now?
> > > > > > > >
> > > > > > >
> > > > > > > RemoveTopic triggers deletion. In general the functionality of
> > > > > StopReplica is subsumed by the metadata records.
> > > > > > >
> > > > > > > > 102. Should we include the metadata topic in the
> > > MetadataResponse?
> > > > > If so,
> > > > > > > > when it will be included and what will the metadata response
> look
> > > > > like?
> > > > > > > >
> > > > > > >
> > > > > > > No, it won't be included in the metadata response sent back
> from
> > > the
> > > > > brokers.
> > > > > > >
> > > > > > > > 103. "The active controller assigns the broker a new broker
> > > epoch,
> > > > > based on
> > > > > > > > the latest committed offset in the log." This seems
> inaccurate
> > > since
> > > > > the
> > > > > > > > latest committed offset doesn't always advance on every log
> > > append.
> > > > > > > >
> > > > > > >
> > > > > > > Given that the new broker epoch won't be visible until the
> commit
> > > has
> > > > > happened, I have changed this to "the next available offset in the
> log"
> > > > > > >
> > > > > > > > 104. REGISTERING(1) : It says "Otherwise, the broker moves
> into
> > > the
> > > > > FENCED
> > > > > > > > state.". It seems this should be RUNNING?
> > > > > > > >
> > > > > > > > 105. RUNNING: Should we require the broker to catch up to the
> > > > > metadata log
> > > > > > > > to get into this state?
> > > > > > >
> > > > > > > For 104 and 105, these sections have been reworked.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <
> cmccabe@apache.org
> > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > > > > > > > Hi Colin,
> > > > > > > > > >
> > > > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > > > > > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > I have some questions about what happens to a broker
> > > when it
> > > > > becomes
> > > > > > > > > > > > fenced (e.g. because it can't send a heartbeat
> request to
> > > > > keep its
> > > > > > > > > > > > lease). The KIP says "When a broker is fenced, it
> cannot
> > > > > process any
> > > > > > > > > > > > client requests. This prevents brokers which are not
> > > > > receiving
> > > > > > > > > > > > metadata updates or that are not receiving and
> processing
> > > > > them fast
> > > > > > > > > > > > enough from causing issues to clients." And in the
> > > > > description of the
> > > > > > > > > > > > FENCED(4) state it likewise says "While in this
> state,
> > > the
> > > > > broker
> > > > > > > > > does
> > > > > > > > > > > > not respond to client requests." It makes sense
> that a
> > > > > fenced broker
> > > > > > > > > > > > should not accept producer requests -- I assume any
> such
> > > > > requests
> > > > > > > > > > > > would result in NotLeaderOrFollowerException. But
> what
> > > > > about KIP-392
> > > > > > > > > > > > (fetch from follower) consumer requests? It is
> > > conceivable
> > > > > that
> > > > > > > > > these
> > > > > > > > > > > > could continue. Related to that, would a fenced
> broker
> > > > > continue to
> > > > > > > > > > > > fetch data for partitions where it thinks it is a
> > > follower?
> > > > > Even if
> > > > > > > > > > > > it rejects consumer requests it might still continue
> to
> > > > > fetch as a
> > > > > > > > > > > > follower. Might it be helpful to clarify both
> decisions
> > > > > here?
> > > > > > > > > > >
> > > > > > > > > > > Hi Ron,
> > > > > > > > > > >
> > > > > > > > > > > Good question. I think a fenced broker should
> continue to
> > > > > fetch on
> > > > > > > > > > > partitions it was already fetching before it was
> fenced,
> > > > > unless it
> > > > > > > > > hits a
> > > > > > > > > > > problem. At that point it won't be able to continue,
> > > since it
> > > > > doesn't
> > > > > > > > > have
> > > > > > > > > > > the new metadata. For example, it won't know about
> > > leadership
> > > > > changes
> > > > > > > > > in
> > > > > > > > > > > the partitions it's fetching. The rationale for
> > > continuing to
> > > > > fetch
> > > > > > > > > is to
> > > > > > > > > > > try to avoid disruptions as much as possible.
> > > > > > > > > > >
> > > > > > > > > > > I don't think fenced brokers should accept client
> requests.
> > > > > The issue
> > > > > > > > > is
> > > > > > > > > > > that the fenced broker may or may not have any data it
> is
> > > > > supposed to
> > > > > > > > > > > have. It may or may not have applied any configuration
> > > > > changes, etc.
> > > > > > > > > that
> > > > > > > > > > > it is supposed to have applied. So it could get pretty
> > > > > confusing, and
> > > > > > > > > also
> > > > > > > > > > > potentially waste the client's time.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > When fenced, how would the broker reply to a client
> which did
> > > > > make a
> > > > > > > > > > request?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Tom,
> > > > > > > > >
> > > > > > > > > The broker will respond with a retryable error in that
> case.
> > > Once
> > > > > the
> > > > > > > > > client has re-fetched its metadata, it will no longer see
> the
> > > > > fenced broker
> > > > > > > > > as part of the cluster. I added a note to the KIP.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Tom
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Wed, Dec 9, 2020, at 10:10, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the update. A few more follow up comments.
>
Hi Jun,
Thanks again for the review.
> 100. FailedReplicaRecord: Since this is reported by each broker
> independently, perhaps we could use a more concise representation that has
> a top level broker field, an array of topics, which has an array of
> partitions.
>
The issue is that there is a size limit on the each record. Putting all of the partitions of a log directory into a single record would probably break that in many cases. Still, we can optimize a bit by having an array of partition IDs, since nearly all the time, we have more than one from the same topic.
> 200. Sounds good. If we remove the broker-side fencing logic, do we plan to
> still keep FENCED in broker state? Do we plan to expose the new states
> through the existing BrokerState metric and if so, what are the values for
> the new states?
>
No, we don't need FENCED any more. I have removed it from the KIP.
The new states are very similar to the current ones, actually. There are no new states or removed ones. The main change in the broker state machine is that the RECOVERING_FROM_UNCLEAN_SHUTDOWN state has been renamed to RECOVERY. Also, unlike previously, the broker will always pass through RECOVERY (although it may only stay in this state for a very short amount of time).
> 201. This may be fine too. Could we document what happens when the
> broker.id/controller.id in metadata.properties don't match the broker
> config when the broker starts up?
>
I added some documentation about this.
> 204. There is still "The highest metadata offset which the broker has not
> reached" referenced under BrokerRegistration.
>
It should be CurrentMetadataOffset. Fixed.
> 206. Is that separate step needed given KIP-516? With KIP-516 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr),
> we don't need to wait for the topic data to be removed from all brokers
> before removing the topic metadata. The combination of unmatching
> topicId
> or the missing topicId from the metadata is enough for the broker to
> clean
> up deleted topics asynchronously.
It won't be needed once KIP-516 is adopted, but this hasn't been implemented yet.
best,
Colin
>
> Jun
>
>
>
>
> On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <cm...@apache.org> wrote:
>
> > On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the updated KIP. A few more comments below.
> > >
> >
> > Hi Jun,
> >
> > Thanks again for the reviews.
> >
> > > 80.2 For deprecated configs, we need to include zookeeper.* and
> > > broker.id.generation.enable.
> > >
> >
> > Added.
> >
> > > 83.1 If a broker is down, does the controller keep the previously
> > > registered broker epoch forever? If not, how long does the controller
> > keep
> > > it? What does the controller do when receiving a broker heartbeat request
> > > with an unfound broker epoch?
> > >
> >
> > Yes, the controller keeps the previous registration forever.
> >
> > Broker heartbeat requests with an incorrect broker epoch will be rejected
> > with STALE_BROKER_EPOCH.
> >
> > > 100. Have you figured out if we need to add a new record type for
> > reporting
> > > partitions on failed disks?
> > >
> >
> > I added FailedReplicaRecord to reflect the case where a JBOD directory has
> > failed, leading to failed replicas.
> >
> > > 102. For debugging purposes, sometimes it's useful to read the metadata
> > > topic using tools like console-consumer. Should we support that and if
> > so,
> > > how?
> > >
> >
> > For now, we have the ability to read the metadata logs with the dump-logs
> > tool. I think we will come up with some other tools in the future as we
> > get experience.
> >
> > > 200. "brokers which are fenced will not appear in MetadataResponses. The
> > > broker will not respond to these requests-- instead, it will simply
> > > disconnect." If the controller is partitioned off from the brokers, this
> > > design will cause every broker to stop accepting new client requests. In
> > > contrast, if ZK is partitioned off, the existing behavior is that the
> > > brokers can continue to work based on the last known metadata. So, I am
> > not
> > > sure if we should change the existing behavior because of the bigger
> > impact
> > > in the new one. Another option is to keep the existing behavior and
> > expose
> > > a metric for fenced brokers so that the operator could be alerted.
> > >
> >
> > I'm skeptical about how well running without ZK currently works. However,
> > I will move the broker-side fencing into a follow-up KIP. This KIP is
> > already pretty large and there is no hard dependency on this. There may
> > also be other ways of accomplishing the positive effects of what
> > broker-side fencing, so more discussion is needed.
> >
> > > 201. I read Ron's comment, but I am still not sure the benefit of keeping
> > > broker.id and controller.id in meta.properties. It seems that we are
> > just
> > > duplicating the same info in two places and have the additional burden of
> > > making sure the values in the two places are consistent.
> > >
> >
> > I think the reasoning is that having broker.id protects us against
> > accidentally bringing up a broker with a disk from a different broker. I
> > don't feel strongly about this but it seemed simpler to keep it.
> >
> > > 202. controller.connect.security.protocol: Is this needed since
> > > controller.listener.names and listener.security.protocol.map imply the
> > > security protocol already?
> > >
> >
> > You're right, this isn't needed. I'll remove it.
> >
> > > 203. registration.heartbeat.interval.ms: It defaults to 2k. ZK uses 1/3
> > of
> > > the session timeout for heartbeat. So, given the default 18k for
> > > registration.lease.timeout.ms, should we default
> > > registration.heartbeat.interval.ms to 6k?
> > >
> >
> > 6 seconds seems like a pretty long time between heartbeats. It might be
> > useful to know when a broker is missing heartbeats, with less time than
> > that. I provisionally set it to 3 seconds (we can always change later...)
> >
> > I also changed the name of these configurations to "
> > broker.heartbeat.interval.ms" and "broker.registration.timeout.ms" to try
> > to clarify them a bit.
> >
> > > 204. "The highest metadata offset which the broker has not reached." It
> > > seems this should be "has reached".
> > >
> >
> > I changed this to "one more than the highest metadata offset which the
> > broker has reached."
> >
> > > 205. UnfenceBrokerRecord and UnregisterBrokerRecord: To me, they seem to
> > be
> > > the same. Do we need both?
> > >
> >
> > Unregistration means that the broker has been removed from the cluster.
> > That is different than unfencing, which marks the broker as active.
> >
> > > 206. TopicRecord: The Deleting field is used to indicate that the topic
> > is
> > > being deleted. I am wondering if this is really needed since RemoveTopic
> > > already indicates the same thing.
> > >
> >
> > RemoveTopic is the last step, that scrubs all metadata about the topic.
> > In order to get to that last step, the topic data needs to removed from all
> > brokers (after each broker notices that the topic is being deleted).
> >
> > best,
> > Colin
> >
> > > Jun
> > >
> > > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > > > > Hi Colin. Thanks for the updates. It's now clear to me that brokers
> > > > > keep their broker epoch for the life of their JVM -- they register
> > > > > once, get their broker epoch in the response, and then never
> > > > > re-register again. Brokers may get fenced, but they keep the same
> > > > > broker epoch for the life of their JVM. The incarnation ID is also
> > > > > kept for the life of the JVM but is generated by the broker itself
> > > > > upon startup, and the combination of the two allows the Controller to
> > > > > act idempotently if any previously-sent registration response gets
> > > > > lost. Makes sense.
> > > > >
> > > >
> > > > Thanks, Ron. That's a good summary.
> > > >
> > > > > One thing I wonder about is if it might be helpful for the broker to
> > > > > send the Cluster ID as determined from its meta.properties file in
> > its
> > > > > registration request. Does it even make sense for the broker to
> > > > > successfully register and enter the Fenced state if it has the wrong
> > > > > Cluster ID?
> > > >
> > > > Yeah, that's a good idea. Let's have the broker pass its cluster ID in
> > > > the registration RPC, and then registration can fail if the broker is
> > > > configured for the wrong cluster.
> > > >
> > > > > The nextMetadatOffset value that the broker communicates
> > > > > in its registration request only has meaning within the correct
> > > > > cluster, so it feels to me that the Controller should have some way
> > to
> > > > > perform this sanity check. There is currently (pre-KIP 500) a check
> > > > > in the broker to make sure its configured cluster ID matches the one
> > > > > stored in ZooKeeper, and we will have to perform this validation
> > > > > somewhere in the KIP-500 world. If the Controller doesn't do it
> > > > > within the registration request then the broker will have to make a
> > > > > metadata request to the Controller, retrieve the Cluster ID, and
> > > > > perform the check itself. It feels to me that it might be better for
> > > > > the Controller to just do it, and then the broker doesn't have to
> > > > > worry about it anymore once it successfully registers.
> > > > >
> > > > > I also have a question about the broker.id value and
> > meta.properties.
> > > > > The KIP now says "In version 0 of meta.properties, there is a
> > > > > broker.id field. Version 1 does not have this field. It is no
> > longer
> > > > > needed because we no longer support dynamic broker id assignment."
> > > > > But then there is an example version 1 meta.properties file that
> > shows
> > > > > the broker.id value. I actually wonder if maybe the broker.id value
> > > > > would be good to keep in the version 1 meta.properties file because
> > it
> > > > > currently (pre-KIP 500, version 0) acts as a sanity check to make
> > sure
> > > > > the broker is using the correct log directory. Similarly with the
> > > > > controller.id value on controllers -- it would allow the same type
> > of
> > > > > sanity check for quorum controllers.
> > > > >
> > > >
> > > > That's a good point. I will add broker.id back, and also add
> > > > controller.id as a possibility.
> > > >
> > > > cheers,
> > > > Colin
> > > >
> > > > >
> > > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > > >
> > > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the reply. A few more comments.
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Thanks again for the reply. Sorry for the long hiatus. I was on
> > > > vacation for a while.
> > > > > >
> > > > > > >
> > > > > > > 55. There is still text that favors new broker registration.
> > "When a
> > > > broker
> > > > > > > first starts up, when it is in the INITIAL state, it will always
> > > > "win"
> > > > > > > broker ID conflicts. However, once it is granted a lease, it
> > > > transitions
> > > > > > > out of the INITIAL state. Thereafter, it may lose subsequent
> > > > conflicts if
> > > > > > > its broker epoch is stale. (See KIP-380 for some background on
> > > > broker
> > > > > > > epoch.) The reason for favoring new processes is to accommodate
> > the
> > > > common
> > > > > > > case where a process is killed with kill -9 and then restarted.
> > We
> > > > want it
> > > > > > > to be able to reclaim its old ID quickly in this case."
> > > > > > >
> > > > > >
> > > > > > Thanks for the reminder. I have clarified the language here.
> > > > Hopefully now it is clear that we don't allow quick re-use of broker
> > IDs.
> > > > > >
> > > > > > > 80.1 Sounds good. Could you document that listeners is a required
> > > > config
> > > > > > > now? It would also be useful to annotate other required configs.
> > For
> > > > > > > example, controller.connect should be required.
> > > > > > >
> > > > > >
> > > > > > I added a note specifying that these are required.
> > > > > >
> > > > > > > 80.2 Could you list all deprecated existing configs? Another one
> > is
> > > > > > > control.plane.listener.name since the controller no longer sends
> > > > > > > LeaderAndIsr, UpdateMetadata and StopReplica requests.
> > > > > > >
> > > > > >
> > > > > > I added a section specifying some deprecated configs.
> > > > > >
> > > > > > > 83.1 It seems that the broker can transition from FENCED to
> > RUNNING
> > > > without
> > > > > > > registering for a new broker epoch. I am not sure how this works.
> > > > Once the
> > > > > > > controller fences a broker, there is no need for the controller
> > to
> > > > keep the
> > > > > > > boker epoch around. So, if the fenced broker's heartbeat request
> > > > with the
> > > > > > > existing broker epoch will be rejected, leading the broker back
> > to
> > > > the
> > > > > > > FENCED state again.
> > > > > > >
> > > > > >
> > > > > > The broker epoch refers to the broker registration. So we DO keep
> > the
> > > > broker epoch around even while the broker is fenced.
> > > > > >
> > > > > > The broker epoch changes only when there is a new broker
> > > > registration. Fencing or unfencing the broker doesn't change the
> > broker
> > > > epoch.
> > > > > >
> > > > > > > 83.5 Good point on KIP-590. Then should we expose the controller
> > for
> > > > > > > debugging purposes? If not, we should deprecate the controllerID
> > > > field in
> > > > > > > MetadataResponse?
> > > > > > >
> > > > > >
> > > > > > I think it's OK to expose it for now, with the proviso that it
> > won't
> > > > be reachable by clients.
> > > > > >
> > > > > > > 90. We rejected the shared ID with just one reason "This is not a
> > > > good idea
> > > > > > > because NetworkClient assumes a single ID space. So if there is
> > > > both a
> > > > > > > controller 1 and a broker 1, we don't have a way of picking the
> > > > "right"
> > > > > > > one." This doesn't seem to be a strong reason. For example, we
> > could
> > > > > > > address the NetworkClient issue with the node type as you pointed
> > > > out or
> > > > > > > using the negative value of a broker ID as the controller ID.
> > > > > > >
> > > > > >
> > > > > > It would require a lot of code changes to support multiple types of
> > > > node IDs. It's not clear to me that the end result would be better --
> > I
> > > > tend to think it would be worse, since it would be more complex. In a
> > > > similar vein, using negative numbers seems dangerous, since we use
> > > > negatives or -1 as "special values" in many places. For example, -1
> > often
> > > > represents "no such node."
> > > > > >
> > > > > > One important thing to keep in mind is that we want to be able to
> > > > transition from a broker and a controller being co-located to them no
> > > > longer being co-located. This is much easier to do when they have
> > separate
> > > > IDs.
> > > > > >
> > > > > > > 100. In KIP-589
> > > > > > > <
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > > >,
> > > > > > > the broker reports all offline replicas due to a disk failure to
> > the
> > > > > > > controller. It seems this information needs to be persisted to
> > the
> > > > > > > metadata
> > > > > > > log. Do we have a corresponding record for that?
> > > > > > >
> > > > > >
> > > > > > Hmm, I have to look into this a little bit more. We may need a new
> > > > record type.
> > > > > >
> > > > > > > 101. Currently, StopReplica request has 2 modes, without deletion
> > > > and with
> > > > > > > deletion. The former is used for controlled shutdown and handling
> > > > disk
> > > > > > > failure, and causes the follower to stop. The latter is for topic
> > > > deletion
> > > > > > > and partition reassignment, and causes the replica to be deleted.
> > > > Since we
> > > > > > > are deprecating StopReplica, could we document what triggers the
> > > > stopping
> > > > > > > of a follower and the deleting of a replica now?
> > > > > > >
> > > > > >
> > > > > > RemoveTopic triggers deletion. In general the functionality of
> > > > StopReplica is subsumed by the metadata records.
> > > > > >
> > > > > > > 102. Should we include the metadata topic in the
> > MetadataResponse?
> > > > If so,
> > > > > > > when it will be included and what will the metadata response look
> > > > like?
> > > > > > >
> > > > > >
> > > > > > No, it won't be included in the metadata response sent back from
> > the
> > > > brokers.
> > > > > >
> > > > > > > 103. "The active controller assigns the broker a new broker
> > epoch,
> > > > based on
> > > > > > > the latest committed offset in the log." This seems inaccurate
> > since
> > > > the
> > > > > > > latest committed offset doesn't always advance on every log
> > append.
> > > > > > >
> > > > > >
> > > > > > Given that the new broker epoch won't be visible until the commit
> > has
> > > > happened, I have changed this to "the next available offset in the log"
> > > > > >
> > > > > > > 104. REGISTERING(1) : It says "Otherwise, the broker moves into
> > the
> > > > FENCED
> > > > > > > state.". It seems this should be RUNNING?
> > > > > > >
> > > > > > > 105. RUNNING: Should we require the broker to catch up to the
> > > > metadata log
> > > > > > > to get into this state?
> > > > > >
> > > > > > For 104 and 105, these sections have been reworked.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <cmccabe@apache.org
> > >
> > > > wrote:
> > > > > > >
> > > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > > > > > > Hi Colin,
> > > > > > > > >
> > > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > > > > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > > > > > > > >
> > > > > > > > > > > I have some questions about what happens to a broker
> > when it
> > > > becomes
> > > > > > > > > > > fenced (e.g. because it can't send a heartbeat request to
> > > > keep its
> > > > > > > > > > > lease). The KIP says "When a broker is fenced, it cannot
> > > > process any
> > > > > > > > > > > client requests. This prevents brokers which are not
> > > > receiving
> > > > > > > > > > > metadata updates or that are not receiving and processing
> > > > them fast
> > > > > > > > > > > enough from causing issues to clients." And in the
> > > > description of the
> > > > > > > > > > > FENCED(4) state it likewise says "While in this state,
> > the
> > > > broker
> > > > > > > > does
> > > > > > > > > > > not respond to client requests." It makes sense that a
> > > > fenced broker
> > > > > > > > > > > should not accept producer requests -- I assume any such
> > > > requests
> > > > > > > > > > > would result in NotLeaderOrFollowerException. But what
> > > > about KIP-392
> > > > > > > > > > > (fetch from follower) consumer requests? It is
> > conceivable
> > > > that
> > > > > > > > these
> > > > > > > > > > > could continue. Related to that, would a fenced broker
> > > > continue to
> > > > > > > > > > > fetch data for partitions where it thinks it is a
> > follower?
> > > > Even if
> > > > > > > > > > > it rejects consumer requests it might still continue to
> > > > fetch as a
> > > > > > > > > > > follower. Might it be helpful to clarify both decisions
> > > > here?
> > > > > > > > > >
> > > > > > > > > > Hi Ron,
> > > > > > > > > >
> > > > > > > > > > Good question. I think a fenced broker should continue to
> > > > fetch on
> > > > > > > > > > partitions it was already fetching before it was fenced,
> > > > unless it
> > > > > > > > hits a
> > > > > > > > > > problem. At that point it won't be able to continue,
> > since it
> > > > doesn't
> > > > > > > > have
> > > > > > > > > > the new metadata. For example, it won't know about
> > leadership
> > > > changes
> > > > > > > > in
> > > > > > > > > > the partitions it's fetching. The rationale for
> > continuing to
> > > > fetch
> > > > > > > > is to
> > > > > > > > > > try to avoid disruptions as much as possible.
> > > > > > > > > >
> > > > > > > > > > I don't think fenced brokers should accept client requests.
> > > > The issue
> > > > > > > > is
> > > > > > > > > > that the fenced broker may or may not have any data it is
> > > > supposed to
> > > > > > > > > > have. It may or may not have applied any configuration
> > > > changes, etc.
> > > > > > > > that
> > > > > > > > > > it is supposed to have applied. So it could get pretty
> > > > confusing, and
> > > > > > > > also
> > > > > > > > > > potentially waste the client's time.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > When fenced, how would the broker reply to a client which did
> > > > make a
> > > > > > > > > request?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi Tom,
> > > > > > > >
> > > > > > > > The broker will respond with a retryable error in that case.
> > Once
> > > > the
> > > > > > > > client has re-fetched its metadata, it will no longer see the
> > > > fenced broker
> > > > > > > > as part of the cluster. I added a note to the KIP.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Tom
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the update. A few more follow up comments.
100. FailedReplicaRecord: Since this is reported by each broker
independently, perhaps we could use a more concise representation that has
a top level broker field, an array of topics, which has an array of
partitions.
200. Sounds good. If we remove the broker-side fencing logic, do we plan to
still keep FENCED in broker state? Do we plan to expose the new states
through the existing BrokerState metric and if so, what are the values for
the new states?
201. This may be fine too. Could we document what happens when the
broker.id/controller.id in metadata.properties don't match the broker
config when the broker starts up?
204. There is still "The highest metadata offset which the broker has not
reached" referenced under BrokerRegistration.
206. Is that separate step needed given KIP-516? With KIP-516 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-LeaderAndIsr),
we don't need to wait for the topic data to be removed from all brokers
before removing the topic metadata. The combination of unmatching topicId
or the missing topicId from the metadata is enough for the broker to clean
up deleted topics asynchronously.
Jun
On Tue, Dec 8, 2020 at 5:27 PM Colin McCabe <cm...@apache.org> wrote:
> On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the updated KIP. A few more comments below.
> >
>
> Hi Jun,
>
> Thanks again for the reviews.
>
> > 80.2 For deprecated configs, we need to include zookeeper.* and
> > broker.id.generation.enable.
> >
>
> Added.
>
> > 83.1 If a broker is down, does the controller keep the previously
> > registered broker epoch forever? If not, how long does the controller
> keep
> > it? What does the controller do when receiving a broker heartbeat request
> > with an unfound broker epoch?
> >
>
> Yes, the controller keeps the previous registration forever.
>
> Broker heartbeat requests with an incorrect broker epoch will be rejected
> with STALE_BROKER_EPOCH.
>
> > 100. Have you figured out if we need to add a new record type for
> reporting
> > partitions on failed disks?
> >
>
> I added FailedReplicaRecord to reflect the case where a JBOD directory has
> failed, leading to failed replicas.
>
> > 102. For debugging purposes, sometimes it's useful to read the metadata
> > topic using tools like console-consumer. Should we support that and if
> so,
> > how?
> >
>
> For now, we have the ability to read the metadata logs with the dump-logs
> tool. I think we will come up with some other tools in the future as we
> get experience.
>
> > 200. "brokers which are fenced will not appear in MetadataResponses. The
> > broker will not respond to these requests-- instead, it will simply
> > disconnect." If the controller is partitioned off from the brokers, this
> > design will cause every broker to stop accepting new client requests. In
> > contrast, if ZK is partitioned off, the existing behavior is that the
> > brokers can continue to work based on the last known metadata. So, I am
> not
> > sure if we should change the existing behavior because of the bigger
> impact
> > in the new one. Another option is to keep the existing behavior and
> expose
> > a metric for fenced brokers so that the operator could be alerted.
> >
>
> I'm skeptical about how well running without ZK currently works. However,
> I will move the broker-side fencing into a follow-up KIP. This KIP is
> already pretty large and there is no hard dependency on this. There may
> also be other ways of accomplishing the positive effects of what
> broker-side fencing, so more discussion is needed.
>
> > 201. I read Ron's comment, but I am still not sure the benefit of keeping
> > broker.id and controller.id in meta.properties. It seems that we are
> just
> > duplicating the same info in two places and have the additional burden of
> > making sure the values in the two places are consistent.
> >
>
> I think the reasoning is that having broker.id protects us against
> accidentally bringing up a broker with a disk from a different broker. I
> don't feel strongly about this but it seemed simpler to keep it.
>
> > 202. controller.connect.security.protocol: Is this needed since
> > controller.listener.names and listener.security.protocol.map imply the
> > security protocol already?
> >
>
> You're right, this isn't needed. I'll remove it.
>
> > 203. registration.heartbeat.interval.ms: It defaults to 2k. ZK uses 1/3
> of
> > the session timeout for heartbeat. So, given the default 18k for
> > registration.lease.timeout.ms, should we default
> > registration.heartbeat.interval.ms to 6k?
> >
>
> 6 seconds seems like a pretty long time between heartbeats. It might be
> useful to know when a broker is missing heartbeats, with less time than
> that. I provisionally set it to 3 seconds (we can always change later...)
>
> I also changed the name of these configurations to "
> broker.heartbeat.interval.ms" and "broker.registration.timeout.ms" to try
> to clarify them a bit.
>
> > 204. "The highest metadata offset which the broker has not reached." It
> > seems this should be "has reached".
> >
>
> I changed this to "one more than the highest metadata offset which the
> broker has reached."
>
> > 205. UnfenceBrokerRecord and UnregisterBrokerRecord: To me, they seem to
> be
> > the same. Do we need both?
> >
>
> Unregistration means that the broker has been removed from the cluster.
> That is different than unfencing, which marks the broker as active.
>
> > 206. TopicRecord: The Deleting field is used to indicate that the topic
> is
> > being deleted. I am wondering if this is really needed since RemoveTopic
> > already indicates the same thing.
> >
>
> RemoveTopic is the last step, that scrubs all metadata about the topic.
> In order to get to that last step, the topic data needs to removed from all
> brokers (after each broker notices that the topic is being deleted).
>
> best,
> Colin
>
> > Jun
> >
> > On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > > > Hi Colin. Thanks for the updates. It's now clear to me that brokers
> > > > keep their broker epoch for the life of their JVM -- they register
> > > > once, get their broker epoch in the response, and then never
> > > > re-register again. Brokers may get fenced, but they keep the same
> > > > broker epoch for the life of their JVM. The incarnation ID is also
> > > > kept for the life of the JVM but is generated by the broker itself
> > > > upon startup, and the combination of the two allows the Controller to
> > > > act idempotently if any previously-sent registration response gets
> > > > lost. Makes sense.
> > > >
> > >
> > > Thanks, Ron. That's a good summary.
> > >
> > > > One thing I wonder about is if it might be helpful for the broker to
> > > > send the Cluster ID as determined from its meta.properties file in
> its
> > > > registration request. Does it even make sense for the broker to
> > > > successfully register and enter the Fenced state if it has the wrong
> > > > Cluster ID?
> > >
> > > Yeah, that's a good idea. Let's have the broker pass its cluster ID in
> > > the registration RPC, and then registration can fail if the broker is
> > > configured for the wrong cluster.
> > >
> > > > The nextMetadatOffset value that the broker communicates
> > > > in its registration request only has meaning within the correct
> > > > cluster, so it feels to me that the Controller should have some way
> to
> > > > perform this sanity check. There is currently (pre-KIP 500) a check
> > > > in the broker to make sure its configured cluster ID matches the one
> > > > stored in ZooKeeper, and we will have to perform this validation
> > > > somewhere in the KIP-500 world. If the Controller doesn't do it
> > > > within the registration request then the broker will have to make a
> > > > metadata request to the Controller, retrieve the Cluster ID, and
> > > > perform the check itself. It feels to me that it might be better for
> > > > the Controller to just do it, and then the broker doesn't have to
> > > > worry about it anymore once it successfully registers.
> > > >
> > > > I also have a question about the broker.id value and
> meta.properties.
> > > > The KIP now says "In version 0 of meta.properties, there is a
> > > > broker.id field. Version 1 does not have this field. It is no
> longer
> > > > needed because we no longer support dynamic broker id assignment."
> > > > But then there is an example version 1 meta.properties file that
> shows
> > > > the broker.id value. I actually wonder if maybe the broker.id value
> > > > would be good to keep in the version 1 meta.properties file because
> it
> > > > currently (pre-KIP 500, version 0) acts as a sanity check to make
> sure
> > > > the broker is using the correct log directory. Similarly with the
> > > > controller.id value on controllers -- it would allow the same type
> of
> > > > sanity check for quorum controllers.
> > > >
> > >
> > > That's a good point. I will add broker.id back, and also add
> > > controller.id as a possibility.
> > >
> > > cheers,
> > > Colin
> > >
> > > >
> > > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <cm...@apache.org>
> wrote:
> > > > >
> > > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the reply. A few more comments.
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > Thanks again for the reply. Sorry for the long hiatus. I was on
> > > vacation for a while.
> > > > >
> > > > > >
> > > > > > 55. There is still text that favors new broker registration.
> "When a
> > > broker
> > > > > > first starts up, when it is in the INITIAL state, it will always
> > > "win"
> > > > > > broker ID conflicts. However, once it is granted a lease, it
> > > transitions
> > > > > > out of the INITIAL state. Thereafter, it may lose subsequent
> > > conflicts if
> > > > > > its broker epoch is stale. (See KIP-380 for some background on
> > > broker
> > > > > > epoch.) The reason for favoring new processes is to accommodate
> the
> > > common
> > > > > > case where a process is killed with kill -9 and then restarted.
> We
> > > want it
> > > > > > to be able to reclaim its old ID quickly in this case."
> > > > > >
> > > > >
> > > > > Thanks for the reminder. I have clarified the language here.
> > > Hopefully now it is clear that we don't allow quick re-use of broker
> IDs.
> > > > >
> > > > > > 80.1 Sounds good. Could you document that listeners is a required
> > > config
> > > > > > now? It would also be useful to annotate other required configs.
> For
> > > > > > example, controller.connect should be required.
> > > > > >
> > > > >
> > > > > I added a note specifying that these are required.
> > > > >
> > > > > > 80.2 Could you list all deprecated existing configs? Another one
> is
> > > > > > control.plane.listener.name since the controller no longer sends
> > > > > > LeaderAndIsr, UpdateMetadata and StopReplica requests.
> > > > > >
> > > > >
> > > > > I added a section specifying some deprecated configs.
> > > > >
> > > > > > 83.1 It seems that the broker can transition from FENCED to
> RUNNING
> > > without
> > > > > > registering for a new broker epoch. I am not sure how this works.
> > > Once the
> > > > > > controller fences a broker, there is no need for the controller
> to
> > > keep the
> > > > > > boker epoch around. So, if the fenced broker's heartbeat request
> > > with the
> > > > > > existing broker epoch will be rejected, leading the broker back
> to
> > > the
> > > > > > FENCED state again.
> > > > > >
> > > > >
> > > > > The broker epoch refers to the broker registration. So we DO keep
> the
> > > broker epoch around even while the broker is fenced.
> > > > >
> > > > > The broker epoch changes only when there is a new broker
> > > registration. Fencing or unfencing the broker doesn't change the
> broker
> > > epoch.
> > > > >
> > > > > > 83.5 Good point on KIP-590. Then should we expose the controller
> for
> > > > > > debugging purposes? If not, we should deprecate the controllerID
> > > field in
> > > > > > MetadataResponse?
> > > > > >
> > > > >
> > > > > I think it's OK to expose it for now, with the proviso that it
> won't
> > > be reachable by clients.
> > > > >
> > > > > > 90. We rejected the shared ID with just one reason "This is not a
> > > good idea
> > > > > > because NetworkClient assumes a single ID space. So if there is
> > > both a
> > > > > > controller 1 and a broker 1, we don't have a way of picking the
> > > "right"
> > > > > > one." This doesn't seem to be a strong reason. For example, we
> could
> > > > > > address the NetworkClient issue with the node type as you pointed
> > > out or
> > > > > > using the negative value of a broker ID as the controller ID.
> > > > > >
> > > > >
> > > > > It would require a lot of code changes to support multiple types of
> > > node IDs. It's not clear to me that the end result would be better --
> I
> > > tend to think it would be worse, since it would be more complex. In a
> > > similar vein, using negative numbers seems dangerous, since we use
> > > negatives or -1 as "special values" in many places. For example, -1
> often
> > > represents "no such node."
> > > > >
> > > > > One important thing to keep in mind is that we want to be able to
> > > transition from a broker and a controller being co-located to them no
> > > longer being co-located. This is much easier to do when they have
> separate
> > > IDs.
> > > > >
> > > > > > 100. In KIP-589
> > > > > > <
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > > >,
> > > > > > the broker reports all offline replicas due to a disk failure to
> the
> > > > > > controller. It seems this information needs to be persisted to
> the
> > > > > > metadata
> > > > > > log. Do we have a corresponding record for that?
> > > > > >
> > > > >
> > > > > Hmm, I have to look into this a little bit more. We may need a new
> > > record type.
> > > > >
> > > > > > 101. Currently, StopReplica request has 2 modes, without deletion
> > > and with
> > > > > > deletion. The former is used for controlled shutdown and handling
> > > disk
> > > > > > failure, and causes the follower to stop. The latter is for topic
> > > deletion
> > > > > > and partition reassignment, and causes the replica to be deleted.
> > > Since we
> > > > > > are deprecating StopReplica, could we document what triggers the
> > > stopping
> > > > > > of a follower and the deleting of a replica now?
> > > > > >
> > > > >
> > > > > RemoveTopic triggers deletion. In general the functionality of
> > > StopReplica is subsumed by the metadata records.
> > > > >
> > > > > > 102. Should we include the metadata topic in the
> MetadataResponse?
> > > If so,
> > > > > > when it will be included and what will the metadata response look
> > > like?
> > > > > >
> > > > >
> > > > > No, it won't be included in the metadata response sent back from
> the
> > > brokers.
> > > > >
> > > > > > 103. "The active controller assigns the broker a new broker
> epoch,
> > > based on
> > > > > > the latest committed offset in the log." This seems inaccurate
> since
> > > the
> > > > > > latest committed offset doesn't always advance on every log
> append.
> > > > > >
> > > > >
> > > > > Given that the new broker epoch won't be visible until the commit
> has
> > > happened, I have changed this to "the next available offset in the log"
> > > > >
> > > > > > 104. REGISTERING(1) : It says "Otherwise, the broker moves into
> the
> > > FENCED
> > > > > > state.". It seems this should be RUNNING?
> > > > > >
> > > > > > 105. RUNNING: Should we require the broker to catch up to the
> > > metadata log
> > > > > > to get into this state?
> > > > >
> > > > > For 104 and 105, these sections have been reworked.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <cmccabe@apache.org
> >
> > > wrote:
> > > > > >
> > > > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > > > > > Hi Colin,
> > > > > > > >
> > > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > > > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > > > > > > >
> > > > > > > > > > I have some questions about what happens to a broker
> when it
> > > becomes
> > > > > > > > > > fenced (e.g. because it can't send a heartbeat request to
> > > keep its
> > > > > > > > > > lease). The KIP says "When a broker is fenced, it cannot
> > > process any
> > > > > > > > > > client requests. This prevents brokers which are not
> > > receiving
> > > > > > > > > > metadata updates or that are not receiving and processing
> > > them fast
> > > > > > > > > > enough from causing issues to clients." And in the
> > > description of the
> > > > > > > > > > FENCED(4) state it likewise says "While in this state,
> the
> > > broker
> > > > > > > does
> > > > > > > > > > not respond to client requests." It makes sense that a
> > > fenced broker
> > > > > > > > > > should not accept producer requests -- I assume any such
> > > requests
> > > > > > > > > > would result in NotLeaderOrFollowerException. But what
> > > about KIP-392
> > > > > > > > > > (fetch from follower) consumer requests? It is
> conceivable
> > > that
> > > > > > > these
> > > > > > > > > > could continue. Related to that, would a fenced broker
> > > continue to
> > > > > > > > > > fetch data for partitions where it thinks it is a
> follower?
> > > Even if
> > > > > > > > > > it rejects consumer requests it might still continue to
> > > fetch as a
> > > > > > > > > > follower. Might it be helpful to clarify both decisions
> > > here?
> > > > > > > > >
> > > > > > > > > Hi Ron,
> > > > > > > > >
> > > > > > > > > Good question. I think a fenced broker should continue to
> > > fetch on
> > > > > > > > > partitions it was already fetching before it was fenced,
> > > unless it
> > > > > > > hits a
> > > > > > > > > problem. At that point it won't be able to continue,
> since it
> > > doesn't
> > > > > > > have
> > > > > > > > > the new metadata. For example, it won't know about
> leadership
> > > changes
> > > > > > > in
> > > > > > > > > the partitions it's fetching. The rationale for
> continuing to
> > > fetch
> > > > > > > is to
> > > > > > > > > try to avoid disruptions as much as possible.
> > > > > > > > >
> > > > > > > > > I don't think fenced brokers should accept client requests.
> > > The issue
> > > > > > > is
> > > > > > > > > that the fenced broker may or may not have any data it is
> > > supposed to
> > > > > > > > > have. It may or may not have applied any configuration
> > > changes, etc.
> > > > > > > that
> > > > > > > > > it is supposed to have applied. So it could get pretty
> > > confusing, and
> > > > > > > also
> > > > > > > > > potentially waste the client's time.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > When fenced, how would the broker reply to a client which did
> > > make a
> > > > > > > > request?
> > > > > > > >
> > > > > > >
> > > > > > > Hi Tom,
> > > > > > >
> > > > > > > The broker will respond with a retryable error in that case.
> Once
> > > the
> > > > > > > client has re-fetched its metadata, it will no longer see the
> > > fenced broker
> > > > > > > as part of the cluster. I added a note to the KIP.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Tom
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Thu, Dec 3, 2020, at 16:37, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the updated KIP. A few more comments below.
>
Hi Jun,
Thanks again for the reviews.
> 80.2 For deprecated configs, we need to include zookeeper.* and
> broker.id.generation.enable.
>
Added.
> 83.1 If a broker is down, does the controller keep the previously
> registered broker epoch forever? If not, how long does the controller keep
> it? What does the controller do when receiving a broker heartbeat request
> with an unfound broker epoch?
>
Yes, the controller keeps the previous registration forever.
Broker heartbeat requests with an incorrect broker epoch will be rejected with STALE_BROKER_EPOCH.
> 100. Have you figured out if we need to add a new record type for reporting
> partitions on failed disks?
>
I added FailedReplicaRecord to reflect the case where a JBOD directory has failed, leading to failed replicas.
> 102. For debugging purposes, sometimes it's useful to read the metadata
> topic using tools like console-consumer. Should we support that and if so,
> how?
>
For now, we have the ability to read the metadata logs with the dump-logs tool. I think we will come up with some other tools in the future as we get experience.
> 200. "brokers which are fenced will not appear in MetadataResponses. The
> broker will not respond to these requests-- instead, it will simply
> disconnect." If the controller is partitioned off from the brokers, this
> design will cause every broker to stop accepting new client requests. In
> contrast, if ZK is partitioned off, the existing behavior is that the
> brokers can continue to work based on the last known metadata. So, I am not
> sure if we should change the existing behavior because of the bigger impact
> in the new one. Another option is to keep the existing behavior and expose
> a metric for fenced brokers so that the operator could be alerted.
>
I'm skeptical about how well running without ZK currently works. However, I will move the broker-side fencing into a follow-up KIP. This KIP is already pretty large and there is no hard dependency on this. There may also be other ways of accomplishing the positive effects of what broker-side fencing, so more discussion is needed.
> 201. I read Ron's comment, but I am still not sure the benefit of keeping
> broker.id and controller.id in meta.properties. It seems that we are just
> duplicating the same info in two places and have the additional burden of
> making sure the values in the two places are consistent.
>
I think the reasoning is that having broker.id protects us against accidentally bringing up a broker with a disk from a different broker. I don't feel strongly about this but it seemed simpler to keep it.
> 202. controller.connect.security.protocol: Is this needed since
> controller.listener.names and listener.security.protocol.map imply the
> security protocol already?
>
You're right, this isn't needed. I'll remove it.
> 203. registration.heartbeat.interval.ms: It defaults to 2k. ZK uses 1/3 of
> the session timeout for heartbeat. So, given the default 18k for
> registration.lease.timeout.ms, should we default
> registration.heartbeat.interval.ms to 6k?
>
6 seconds seems like a pretty long time between heartbeats. It might be useful to know when a broker is missing heartbeats, with less time than that. I provisionally set it to 3 seconds (we can always change later...)
I also changed the name of these configurations to "broker.heartbeat.interval.ms" and "broker.registration.timeout.ms" to try to clarify them a bit.
> 204. "The highest metadata offset which the broker has not reached." It
> seems this should be "has reached".
>
I changed this to "one more than the highest metadata offset which the broker has reached."
> 205. UnfenceBrokerRecord and UnregisterBrokerRecord: To me, they seem to be
> the same. Do we need both?
>
Unregistration means that the broker has been removed from the cluster. That is different than unfencing, which marks the broker as active.
> 206. TopicRecord: The Deleting field is used to indicate that the topic is
> being deleted. I am wondering if this is really needed since RemoveTopic
> already indicates the same thing.
>
RemoveTopic is the last step, that scrubs all metadata about the topic. In order to get to that last step, the topic data needs to removed from all brokers (after each broker notices that the topic is being deleted).
best,
Colin
> Jun
>
> On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <cm...@apache.org> wrote:
>
> > On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > > Hi Colin. Thanks for the updates. It's now clear to me that brokers
> > > keep their broker epoch for the life of their JVM -- they register
> > > once, get their broker epoch in the response, and then never
> > > re-register again. Brokers may get fenced, but they keep the same
> > > broker epoch for the life of their JVM. The incarnation ID is also
> > > kept for the life of the JVM but is generated by the broker itself
> > > upon startup, and the combination of the two allows the Controller to
> > > act idempotently if any previously-sent registration response gets
> > > lost. Makes sense.
> > >
> >
> > Thanks, Ron. That's a good summary.
> >
> > > One thing I wonder about is if it might be helpful for the broker to
> > > send the Cluster ID as determined from its meta.properties file in its
> > > registration request. Does it even make sense for the broker to
> > > successfully register and enter the Fenced state if it has the wrong
> > > Cluster ID?
> >
> > Yeah, that's a good idea. Let's have the broker pass its cluster ID in
> > the registration RPC, and then registration can fail if the broker is
> > configured for the wrong cluster.
> >
> > > The nextMetadatOffset value that the broker communicates
> > > in its registration request only has meaning within the correct
> > > cluster, so it feels to me that the Controller should have some way to
> > > perform this sanity check. There is currently (pre-KIP 500) a check
> > > in the broker to make sure its configured cluster ID matches the one
> > > stored in ZooKeeper, and we will have to perform this validation
> > > somewhere in the KIP-500 world. If the Controller doesn't do it
> > > within the registration request then the broker will have to make a
> > > metadata request to the Controller, retrieve the Cluster ID, and
> > > perform the check itself. It feels to me that it might be better for
> > > the Controller to just do it, and then the broker doesn't have to
> > > worry about it anymore once it successfully registers.
> > >
> > > I also have a question about the broker.id value and meta.properties.
> > > The KIP now says "In version 0 of meta.properties, there is a
> > > broker.id field. Version 1 does not have this field. It is no longer
> > > needed because we no longer support dynamic broker id assignment."
> > > But then there is an example version 1 meta.properties file that shows
> > > the broker.id value. I actually wonder if maybe the broker.id value
> > > would be good to keep in the version 1 meta.properties file because it
> > > currently (pre-KIP 500, version 0) acts as a sanity check to make sure
> > > the broker is using the correct log directory. Similarly with the
> > > controller.id value on controllers -- it would allow the same type of
> > > sanity check for quorum controllers.
> > >
> >
> > That's a good point. I will add broker.id back, and also add
> > controller.id as a possibility.
> >
> > cheers,
> > Colin
> >
> > >
> > > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <cm...@apache.org> wrote:
> > > >
> > > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the reply. A few more comments.
> > > >
> > > > Hi Jun,
> > > >
> > > > Thanks again for the reply. Sorry for the long hiatus. I was on
> > vacation for a while.
> > > >
> > > > >
> > > > > 55. There is still text that favors new broker registration. "When a
> > broker
> > > > > first starts up, when it is in the INITIAL state, it will always
> > "win"
> > > > > broker ID conflicts. However, once it is granted a lease, it
> > transitions
> > > > > out of the INITIAL state. Thereafter, it may lose subsequent
> > conflicts if
> > > > > its broker epoch is stale. (See KIP-380 for some background on
> > broker
> > > > > epoch.) The reason for favoring new processes is to accommodate the
> > common
> > > > > case where a process is killed with kill -9 and then restarted. We
> > want it
> > > > > to be able to reclaim its old ID quickly in this case."
> > > > >
> > > >
> > > > Thanks for the reminder. I have clarified the language here.
> > Hopefully now it is clear that we don't allow quick re-use of broker IDs.
> > > >
> > > > > 80.1 Sounds good. Could you document that listeners is a required
> > config
> > > > > now? It would also be useful to annotate other required configs. For
> > > > > example, controller.connect should be required.
> > > > >
> > > >
> > > > I added a note specifying that these are required.
> > > >
> > > > > 80.2 Could you list all deprecated existing configs? Another one is
> > > > > control.plane.listener.name since the controller no longer sends
> > > > > LeaderAndIsr, UpdateMetadata and StopReplica requests.
> > > > >
> > > >
> > > > I added a section specifying some deprecated configs.
> > > >
> > > > > 83.1 It seems that the broker can transition from FENCED to RUNNING
> > without
> > > > > registering for a new broker epoch. I am not sure how this works.
> > Once the
> > > > > controller fences a broker, there is no need for the controller to
> > keep the
> > > > > boker epoch around. So, if the fenced broker's heartbeat request
> > with the
> > > > > existing broker epoch will be rejected, leading the broker back to
> > the
> > > > > FENCED state again.
> > > > >
> > > >
> > > > The broker epoch refers to the broker registration. So we DO keep the
> > broker epoch around even while the broker is fenced.
> > > >
> > > > The broker epoch changes only when there is a new broker
> > registration. Fencing or unfencing the broker doesn't change the broker
> > epoch.
> > > >
> > > > > 83.5 Good point on KIP-590. Then should we expose the controller for
> > > > > debugging purposes? If not, we should deprecate the controllerID
> > field in
> > > > > MetadataResponse?
> > > > >
> > > >
> > > > I think it's OK to expose it for now, with the proviso that it won't
> > be reachable by clients.
> > > >
> > > > > 90. We rejected the shared ID with just one reason "This is not a
> > good idea
> > > > > because NetworkClient assumes a single ID space. So if there is
> > both a
> > > > > controller 1 and a broker 1, we don't have a way of picking the
> > "right"
> > > > > one." This doesn't seem to be a strong reason. For example, we could
> > > > > address the NetworkClient issue with the node type as you pointed
> > out or
> > > > > using the negative value of a broker ID as the controller ID.
> > > > >
> > > >
> > > > It would require a lot of code changes to support multiple types of
> > node IDs. It's not clear to me that the end result would be better -- I
> > tend to think it would be worse, since it would be more complex. In a
> > similar vein, using negative numbers seems dangerous, since we use
> > negatives or -1 as "special values" in many places. For example, -1 often
> > represents "no such node."
> > > >
> > > > One important thing to keep in mind is that we want to be able to
> > transition from a broker and a controller being co-located to them no
> > longer being co-located. This is much easier to do when they have separate
> > IDs.
> > > >
> > > > > 100. In KIP-589
> > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> > >,
> > > > > the broker reports all offline replicas due to a disk failure to the
> > > > > controller. It seems this information needs to be persisted to the
> > > > > metadata
> > > > > log. Do we have a corresponding record for that?
> > > > >
> > > >
> > > > Hmm, I have to look into this a little bit more. We may need a new
> > record type.
> > > >
> > > > > 101. Currently, StopReplica request has 2 modes, without deletion
> > and with
> > > > > deletion. The former is used for controlled shutdown and handling
> > disk
> > > > > failure, and causes the follower to stop. The latter is for topic
> > deletion
> > > > > and partition reassignment, and causes the replica to be deleted.
> > Since we
> > > > > are deprecating StopReplica, could we document what triggers the
> > stopping
> > > > > of a follower and the deleting of a replica now?
> > > > >
> > > >
> > > > RemoveTopic triggers deletion. In general the functionality of
> > StopReplica is subsumed by the metadata records.
> > > >
> > > > > 102. Should we include the metadata topic in the MetadataResponse?
> > If so,
> > > > > when it will be included and what will the metadata response look
> > like?
> > > > >
> > > >
> > > > No, it won't be included in the metadata response sent back from the
> > brokers.
> > > >
> > > > > 103. "The active controller assigns the broker a new broker epoch,
> > based on
> > > > > the latest committed offset in the log." This seems inaccurate since
> > the
> > > > > latest committed offset doesn't always advance on every log append.
> > > > >
> > > >
> > > > Given that the new broker epoch won't be visible until the commit has
> > happened, I have changed this to "the next available offset in the log"
> > > >
> > > > > 104. REGISTERING(1) : It says "Otherwise, the broker moves into the
> > FENCED
> > > > > state.". It seems this should be RUNNING?
> > > > >
> > > > > 105. RUNNING: Should we require the broker to catch up to the
> > metadata log
> > > > > to get into this state?
> > > >
> > > > For 104 and 105, these sections have been reworked.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > > > > Hi Colin,
> > > > > > >
> > > > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > > > > > >
> > > > > > > > > I have some questions about what happens to a broker when it
> > becomes
> > > > > > > > > fenced (e.g. because it can't send a heartbeat request to
> > keep its
> > > > > > > > > lease). The KIP says "When a broker is fenced, it cannot
> > process any
> > > > > > > > > client requests. This prevents brokers which are not
> > receiving
> > > > > > > > > metadata updates or that are not receiving and processing
> > them fast
> > > > > > > > > enough from causing issues to clients." And in the
> > description of the
> > > > > > > > > FENCED(4) state it likewise says "While in this state, the
> > broker
> > > > > > does
> > > > > > > > > not respond to client requests." It makes sense that a
> > fenced broker
> > > > > > > > > should not accept producer requests -- I assume any such
> > requests
> > > > > > > > > would result in NotLeaderOrFollowerException. But what
> > about KIP-392
> > > > > > > > > (fetch from follower) consumer requests? It is conceivable
> > that
> > > > > > these
> > > > > > > > > could continue. Related to that, would a fenced broker
> > continue to
> > > > > > > > > fetch data for partitions where it thinks it is a follower?
> > Even if
> > > > > > > > > it rejects consumer requests it might still continue to
> > fetch as a
> > > > > > > > > follower. Might it be helpful to clarify both decisions
> > here?
> > > > > > > >
> > > > > > > > Hi Ron,
> > > > > > > >
> > > > > > > > Good question. I think a fenced broker should continue to
> > fetch on
> > > > > > > > partitions it was already fetching before it was fenced,
> > unless it
> > > > > > hits a
> > > > > > > > problem. At that point it won't be able to continue, since it
> > doesn't
> > > > > > have
> > > > > > > > the new metadata. For example, it won't know about leadership
> > changes
> > > > > > in
> > > > > > > > the partitions it's fetching. The rationale for continuing to
> > fetch
> > > > > > is to
> > > > > > > > try to avoid disruptions as much as possible.
> > > > > > > >
> > > > > > > > I don't think fenced brokers should accept client requests.
> > The issue
> > > > > > is
> > > > > > > > that the fenced broker may or may not have any data it is
> > supposed to
> > > > > > > > have. It may or may not have applied any configuration
> > changes, etc.
> > > > > > that
> > > > > > > > it is supposed to have applied. So it could get pretty
> > confusing, and
> > > > > > also
> > > > > > > > potentially waste the client's time.
> > > > > > > >
> > > > > > > >
> > > > > > > When fenced, how would the broker reply to a client which did
> > make a
> > > > > > > request?
> > > > > > >
> > > > > >
> > > > > > Hi Tom,
> > > > > >
> > > > > > The broker will respond with a retryable error in that case. Once
> > the
> > > > > > client has re-fetched its metadata, it will no longer see the
> > fenced broker
> > > > > > as part of the cluster. I added a note to the KIP.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Tom
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the updated KIP. A few more comments below.
80.2 For deprecated configs, we need to include zookeeper.* and
broker.id.generation.enable.
83.1 If a broker is down, does the controller keep the previously
registered broker epoch forever? If not, how long does the controller keep
it? What does the controller do when receiving a broker heartbeat request
with an unfound broker epoch?
100. Have you figured out if we need to add a new record type for reporting
partitions on failed disks?
102. For debugging purposes, sometimes it's useful to read the metadata
topic using tools like console-consumer. Should we support that and if so,
how?
200. "brokers which are fenced will not appear in MetadataResponses. The
broker will not respond to these requests-- instead, it will simply
disconnect." If the controller is partitioned off from the brokers, this
design will cause every broker to stop accepting new client requests. In
contrast, if ZK is partitioned off, the existing behavior is that the
brokers can continue to work based on the last known metadata. So, I am not
sure if we should change the existing behavior because of the bigger impact
in the new one. Another option is to keep the existing behavior and expose
a metric for fenced brokers so that the operator could be alerted.
201. I read Ron's comment, but I am still not sure the benefit of keeping
broker.id and controller.id in meta.properties. It seems that we are just
duplicating the same info in two places and have the additional burden of
making sure the values in the two places are consistent.
202. controller.connect.security.protocol: Is this needed since
controller.listener.names and listener.security.protocol.map imply the
security protocol already?
203. registration.heartbeat.interval.ms: It defaults to 2k. ZK uses 1/3 of
the session timeout for heartbeat. So, given the default 18k for
registration.lease.timeout.ms, should we default
registration.heartbeat.interval.ms to 6k?
204. "The highest metadata offset which the broker has not reached." It
seems this should be "has reached".
205. UnfenceBrokerRecord and UnregisterBrokerRecord: To me, they seem to be
the same. Do we need both?
206. TopicRecord: The Deleting field is used to indicate that the topic is
being deleted. I am wondering if this is really needed since RemoveTopic
already indicates the same thing.
Jun
On Wed, Dec 2, 2020 at 2:50 PM Colin McCabe <cm...@apache.org> wrote:
> On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> > Hi Colin. Thanks for the updates. It's now clear to me that brokers
> > keep their broker epoch for the life of their JVM -- they register
> > once, get their broker epoch in the response, and then never
> > re-register again. Brokers may get fenced, but they keep the same
> > broker epoch for the life of their JVM. The incarnation ID is also
> > kept for the life of the JVM but is generated by the broker itself
> > upon startup, and the combination of the two allows the Controller to
> > act idempotently if any previously-sent registration response gets
> > lost. Makes sense.
> >
>
> Thanks, Ron. That's a good summary.
>
> > One thing I wonder about is if it might be helpful for the broker to
> > send the Cluster ID as determined from its meta.properties file in its
> > registration request. Does it even make sense for the broker to
> > successfully register and enter the Fenced state if it has the wrong
> > Cluster ID?
>
> Yeah, that's a good idea. Let's have the broker pass its cluster ID in
> the registration RPC, and then registration can fail if the broker is
> configured for the wrong cluster.
>
> > The nextMetadatOffset value that the broker communicates
> > in its registration request only has meaning within the correct
> > cluster, so it feels to me that the Controller should have some way to
> > perform this sanity check. There is currently (pre-KIP 500) a check
> > in the broker to make sure its configured cluster ID matches the one
> > stored in ZooKeeper, and we will have to perform this validation
> > somewhere in the KIP-500 world. If the Controller doesn't do it
> > within the registration request then the broker will have to make a
> > metadata request to the Controller, retrieve the Cluster ID, and
> > perform the check itself. It feels to me that it might be better for
> > the Controller to just do it, and then the broker doesn't have to
> > worry about it anymore once it successfully registers.
> >
> > I also have a question about the broker.id value and meta.properties.
> > The KIP now says "In version 0 of meta.properties, there is a
> > broker.id field. Version 1 does not have this field. It is no longer
> > needed because we no longer support dynamic broker id assignment."
> > But then there is an example version 1 meta.properties file that shows
> > the broker.id value. I actually wonder if maybe the broker.id value
> > would be good to keep in the version 1 meta.properties file because it
> > currently (pre-KIP 500, version 0) acts as a sanity check to make sure
> > the broker is using the correct log directory. Similarly with the
> > controller.id value on controllers -- it would allow the same type of
> > sanity check for quorum controllers.
> >
>
> That's a good point. I will add broker.id back, and also add
> controller.id as a possibility.
>
> cheers,
> Colin
>
> >
> > On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply. A few more comments.
> > >
> > > Hi Jun,
> > >
> > > Thanks again for the reply. Sorry for the long hiatus. I was on
> vacation for a while.
> > >
> > > >
> > > > 55. There is still text that favors new broker registration. "When a
> broker
> > > > first starts up, when it is in the INITIAL state, it will always
> "win"
> > > > broker ID conflicts. However, once it is granted a lease, it
> transitions
> > > > out of the INITIAL state. Thereafter, it may lose subsequent
> conflicts if
> > > > its broker epoch is stale. (See KIP-380 for some background on
> broker
> > > > epoch.) The reason for favoring new processes is to accommodate the
> common
> > > > case where a process is killed with kill -9 and then restarted. We
> want it
> > > > to be able to reclaim its old ID quickly in this case."
> > > >
> > >
> > > Thanks for the reminder. I have clarified the language here.
> Hopefully now it is clear that we don't allow quick re-use of broker IDs.
> > >
> > > > 80.1 Sounds good. Could you document that listeners is a required
> config
> > > > now? It would also be useful to annotate other required configs. For
> > > > example, controller.connect should be required.
> > > >
> > >
> > > I added a note specifying that these are required.
> > >
> > > > 80.2 Could you list all deprecated existing configs? Another one is
> > > > control.plane.listener.name since the controller no longer sends
> > > > LeaderAndIsr, UpdateMetadata and StopReplica requests.
> > > >
> > >
> > > I added a section specifying some deprecated configs.
> > >
> > > > 83.1 It seems that the broker can transition from FENCED to RUNNING
> without
> > > > registering for a new broker epoch. I am not sure how this works.
> Once the
> > > > controller fences a broker, there is no need for the controller to
> keep the
> > > > boker epoch around. So, if the fenced broker's heartbeat request
> with the
> > > > existing broker epoch will be rejected, leading the broker back to
> the
> > > > FENCED state again.
> > > >
> > >
> > > The broker epoch refers to the broker registration. So we DO keep the
> broker epoch around even while the broker is fenced.
> > >
> > > The broker epoch changes only when there is a new broker
> registration. Fencing or unfencing the broker doesn't change the broker
> epoch.
> > >
> > > > 83.5 Good point on KIP-590. Then should we expose the controller for
> > > > debugging purposes? If not, we should deprecate the controllerID
> field in
> > > > MetadataResponse?
> > > >
> > >
> > > I think it's OK to expose it for now, with the proviso that it won't
> be reachable by clients.
> > >
> > > > 90. We rejected the shared ID with just one reason "This is not a
> good idea
> > > > because NetworkClient assumes a single ID space. So if there is
> both a
> > > > controller 1 and a broker 1, we don't have a way of picking the
> "right"
> > > > one." This doesn't seem to be a strong reason. For example, we could
> > > > address the NetworkClient issue with the node type as you pointed
> out or
> > > > using the negative value of a broker ID as the controller ID.
> > > >
> > >
> > > It would require a lot of code changes to support multiple types of
> node IDs. It's not clear to me that the end result would be better -- I
> tend to think it would be worse, since it would be more complex. In a
> similar vein, using negative numbers seems dangerous, since we use
> negatives or -1 as "special values" in many places. For example, -1 often
> represents "no such node."
> > >
> > > One important thing to keep in mind is that we want to be able to
> transition from a broker and a controller being co-located to them no
> longer being co-located. This is much easier to do when they have separate
> IDs.
> > >
> > > > 100. In KIP-589
> > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller
> >,
> > > > the broker reports all offline replicas due to a disk failure to the
> > > > controller. It seems this information needs to be persisted to the
> > > > metadata
> > > > log. Do we have a corresponding record for that?
> > > >
> > >
> > > Hmm, I have to look into this a little bit more. We may need a new
> record type.
> > >
> > > > 101. Currently, StopReplica request has 2 modes, without deletion
> and with
> > > > deletion. The former is used for controlled shutdown and handling
> disk
> > > > failure, and causes the follower to stop. The latter is for topic
> deletion
> > > > and partition reassignment, and causes the replica to be deleted.
> Since we
> > > > are deprecating StopReplica, could we document what triggers the
> stopping
> > > > of a follower and the deleting of a replica now?
> > > >
> > >
> > > RemoveTopic triggers deletion. In general the functionality of
> StopReplica is subsumed by the metadata records.
> > >
> > > > 102. Should we include the metadata topic in the MetadataResponse?
> If so,
> > > > when it will be included and what will the metadata response look
> like?
> > > >
> > >
> > > No, it won't be included in the metadata response sent back from the
> brokers.
> > >
> > > > 103. "The active controller assigns the broker a new broker epoch,
> based on
> > > > the latest committed offset in the log." This seems inaccurate since
> the
> > > > latest committed offset doesn't always advance on every log append.
> > > >
> > >
> > > Given that the new broker epoch won't be visible until the commit has
> happened, I have changed this to "the next available offset in the log"
> > >
> > > > 104. REGISTERING(1) : It says "Otherwise, the broker moves into the
> FENCED
> > > > state.". It seems this should be RUNNING?
> > > >
> > > > 105. RUNNING: Should we require the broker to catch up to the
> metadata log
> > > > to get into this state?
> > >
> > > For 104 and 105, these sections have been reworked.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > >
> > > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > > > Hi Colin,
> > > > > >
> > > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > > > > >
> > > > > > > > I have some questions about what happens to a broker when it
> becomes
> > > > > > > > fenced (e.g. because it can't send a heartbeat request to
> keep its
> > > > > > > > lease). The KIP says "When a broker is fenced, it cannot
> process any
> > > > > > > > client requests. This prevents brokers which are not
> receiving
> > > > > > > > metadata updates or that are not receiving and processing
> them fast
> > > > > > > > enough from causing issues to clients." And in the
> description of the
> > > > > > > > FENCED(4) state it likewise says "While in this state, the
> broker
> > > > > does
> > > > > > > > not respond to client requests." It makes sense that a
> fenced broker
> > > > > > > > should not accept producer requests -- I assume any such
> requests
> > > > > > > > would result in NotLeaderOrFollowerException. But what
> about KIP-392
> > > > > > > > (fetch from follower) consumer requests? It is conceivable
> that
> > > > > these
> > > > > > > > could continue. Related to that, would a fenced broker
> continue to
> > > > > > > > fetch data for partitions where it thinks it is a follower?
> Even if
> > > > > > > > it rejects consumer requests it might still continue to
> fetch as a
> > > > > > > > follower. Might it be helpful to clarify both decisions
> here?
> > > > > > >
> > > > > > > Hi Ron,
> > > > > > >
> > > > > > > Good question. I think a fenced broker should continue to
> fetch on
> > > > > > > partitions it was already fetching before it was fenced,
> unless it
> > > > > hits a
> > > > > > > problem. At that point it won't be able to continue, since it
> doesn't
> > > > > have
> > > > > > > the new metadata. For example, it won't know about leadership
> changes
> > > > > in
> > > > > > > the partitions it's fetching. The rationale for continuing to
> fetch
> > > > > is to
> > > > > > > try to avoid disruptions as much as possible.
> > > > > > >
> > > > > > > I don't think fenced brokers should accept client requests.
> The issue
> > > > > is
> > > > > > > that the fenced broker may or may not have any data it is
> supposed to
> > > > > > > have. It may or may not have applied any configuration
> changes, etc.
> > > > > that
> > > > > > > it is supposed to have applied. So it could get pretty
> confusing, and
> > > > > also
> > > > > > > potentially waste the client's time.
> > > > > > >
> > > > > > >
> > > > > > When fenced, how would the broker reply to a client which did
> make a
> > > > > > request?
> > > > > >
> > > > >
> > > > > Hi Tom,
> > > > >
> > > > > The broker will respond with a retryable error in that case. Once
> the
> > > > > client has re-fetched its metadata, it will no longer see the
> fenced broker
> > > > > as part of the cluster. I added a note to the KIP.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Tom
> > > > > >
> > > > >
> > > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Wed, Dec 2, 2020, at 14:07, Ron Dagostino wrote:
> Hi Colin. Thanks for the updates. It's now clear to me that brokers
> keep their broker epoch for the life of their JVM -- they register
> once, get their broker epoch in the response, and then never
> re-register again. Brokers may get fenced, but they keep the same
> broker epoch for the life of their JVM. The incarnation ID is also
> kept for the life of the JVM but is generated by the broker itself
> upon startup, and the combination of the two allows the Controller to
> act idempotently if any previously-sent registration response gets
> lost. Makes sense.
>
Thanks, Ron. That's a good summary.
> One thing I wonder about is if it might be helpful for the broker to
> send the Cluster ID as determined from its meta.properties file in its
> registration request. Does it even make sense for the broker to
> successfully register and enter the Fenced state if it has the wrong
> Cluster ID?
Yeah, that's a good idea. Let's have the broker pass its cluster ID in the registration RPC, and then registration can fail if the broker is configured for the wrong cluster.
> The nextMetadatOffset value that the broker communicates
> in its registration request only has meaning within the correct
> cluster, so it feels to me that the Controller should have some way to
> perform this sanity check. There is currently (pre-KIP 500) a check
> in the broker to make sure its configured cluster ID matches the one
> stored in ZooKeeper, and we will have to perform this validation
> somewhere in the KIP-500 world. If the Controller doesn't do it
> within the registration request then the broker will have to make a
> metadata request to the Controller, retrieve the Cluster ID, and
> perform the check itself. It feels to me that it might be better for
> the Controller to just do it, and then the broker doesn't have to
> worry about it anymore once it successfully registers.
>
> I also have a question about the broker.id value and meta.properties.
> The KIP now says "In version 0 of meta.properties, there is a
> broker.id field. Version 1 does not have this field. It is no longer
> needed because we no longer support dynamic broker id assignment."
> But then there is an example version 1 meta.properties file that shows
> the broker.id value. I actually wonder if maybe the broker.id value
> would be good to keep in the version 1 meta.properties file because it
> currently (pre-KIP 500, version 0) acts as a sanity check to make sure
> the broker is using the correct log directory. Similarly with the
> controller.id value on controllers -- it would allow the same type of
> sanity check for quorum controllers.
>
That's a good point. I will add broker.id back, and also add controller.id as a possibility.
cheers,
Colin
>
> On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply. A few more comments.
> >
> > Hi Jun,
> >
> > Thanks again for the reply. Sorry for the long hiatus. I was on vacation for a while.
> >
> > >
> > > 55. There is still text that favors new broker registration. "When a broker
> > > first starts up, when it is in the INITIAL state, it will always "win"
> > > broker ID conflicts. However, once it is granted a lease, it transitions
> > > out of the INITIAL state. Thereafter, it may lose subsequent conflicts if
> > > its broker epoch is stale. (See KIP-380 for some background on broker
> > > epoch.) The reason for favoring new processes is to accommodate the common
> > > case where a process is killed with kill -9 and then restarted. We want it
> > > to be able to reclaim its old ID quickly in this case."
> > >
> >
> > Thanks for the reminder. I have clarified the language here. Hopefully now it is clear that we don't allow quick re-use of broker IDs.
> >
> > > 80.1 Sounds good. Could you document that listeners is a required config
> > > now? It would also be useful to annotate other required configs. For
> > > example, controller.connect should be required.
> > >
> >
> > I added a note specifying that these are required.
> >
> > > 80.2 Could you list all deprecated existing configs? Another one is
> > > control.plane.listener.name since the controller no longer sends
> > > LeaderAndIsr, UpdateMetadata and StopReplica requests.
> > >
> >
> > I added a section specifying some deprecated configs.
> >
> > > 83.1 It seems that the broker can transition from FENCED to RUNNING without
> > > registering for a new broker epoch. I am not sure how this works. Once the
> > > controller fences a broker, there is no need for the controller to keep the
> > > boker epoch around. So, if the fenced broker's heartbeat request with the
> > > existing broker epoch will be rejected, leading the broker back to the
> > > FENCED state again.
> > >
> >
> > The broker epoch refers to the broker registration. So we DO keep the broker epoch around even while the broker is fenced.
> >
> > The broker epoch changes only when there is a new broker registration. Fencing or unfencing the broker doesn't change the broker epoch.
> >
> > > 83.5 Good point on KIP-590. Then should we expose the controller for
> > > debugging purposes? If not, we should deprecate the controllerID field in
> > > MetadataResponse?
> > >
> >
> > I think it's OK to expose it for now, with the proviso that it won't be reachable by clients.
> >
> > > 90. We rejected the shared ID with just one reason "This is not a good idea
> > > because NetworkClient assumes a single ID space. So if there is both a
> > > controller 1 and a broker 1, we don't have a way of picking the "right"
> > > one." This doesn't seem to be a strong reason. For example, we could
> > > address the NetworkClient issue with the node type as you pointed out or
> > > using the negative value of a broker ID as the controller ID.
> > >
> >
> > It would require a lot of code changes to support multiple types of node IDs. It's not clear to me that the end result would be better -- I tend to think it would be worse, since it would be more complex. In a similar vein, using negative numbers seems dangerous, since we use negatives or -1 as "special values" in many places. For example, -1 often represents "no such node."
> >
> > One important thing to keep in mind is that we want to be able to transition from a broker and a controller being co-located to them no longer being co-located. This is much easier to do when they have separate IDs.
> >
> > > 100. In KIP-589
> > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller>,
> > > the broker reports all offline replicas due to a disk failure to the
> > > controller. It seems this information needs to be persisted to the
> > > metadata
> > > log. Do we have a corresponding record for that?
> > >
> >
> > Hmm, I have to look into this a little bit more. We may need a new record type.
> >
> > > 101. Currently, StopReplica request has 2 modes, without deletion and with
> > > deletion. The former is used for controlled shutdown and handling disk
> > > failure, and causes the follower to stop. The latter is for topic deletion
> > > and partition reassignment, and causes the replica to be deleted. Since we
> > > are deprecating StopReplica, could we document what triggers the stopping
> > > of a follower and the deleting of a replica now?
> > >
> >
> > RemoveTopic triggers deletion. In general the functionality of StopReplica is subsumed by the metadata records.
> >
> > > 102. Should we include the metadata topic in the MetadataResponse? If so,
> > > when it will be included and what will the metadata response look like?
> > >
> >
> > No, it won't be included in the metadata response sent back from the brokers.
> >
> > > 103. "The active controller assigns the broker a new broker epoch, based on
> > > the latest committed offset in the log." This seems inaccurate since the
> > > latest committed offset doesn't always advance on every log append.
> > >
> >
> > Given that the new broker epoch won't be visible until the commit has happened, I have changed this to "the next available offset in the log"
> >
> > > 104. REGISTERING(1) : It says "Otherwise, the broker moves into the FENCED
> > > state.". It seems this should be RUNNING?
> > >
> > > 105. RUNNING: Should we require the broker to catch up to the metadata log
> > > to get into this state?
> >
> > For 104 and 105, these sections have been reworked.
> >
> > best,
> > Colin
> >
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > > Hi Colin,
> > > > >
> > > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > > > >
> > > > > > > I have some questions about what happens to a broker when it becomes
> > > > > > > fenced (e.g. because it can't send a heartbeat request to keep its
> > > > > > > lease). The KIP says "When a broker is fenced, it cannot process any
> > > > > > > client requests. This prevents brokers which are not receiving
> > > > > > > metadata updates or that are not receiving and processing them fast
> > > > > > > enough from causing issues to clients." And in the description of the
> > > > > > > FENCED(4) state it likewise says "While in this state, the broker
> > > > does
> > > > > > > not respond to client requests." It makes sense that a fenced broker
> > > > > > > should not accept producer requests -- I assume any such requests
> > > > > > > would result in NotLeaderOrFollowerException. But what about KIP-392
> > > > > > > (fetch from follower) consumer requests? It is conceivable that
> > > > these
> > > > > > > could continue. Related to that, would a fenced broker continue to
> > > > > > > fetch data for partitions where it thinks it is a follower? Even if
> > > > > > > it rejects consumer requests it might still continue to fetch as a
> > > > > > > follower. Might it be helpful to clarify both decisions here?
> > > > > >
> > > > > > Hi Ron,
> > > > > >
> > > > > > Good question. I think a fenced broker should continue to fetch on
> > > > > > partitions it was already fetching before it was fenced, unless it
> > > > hits a
> > > > > > problem. At that point it won't be able to continue, since it doesn't
> > > > have
> > > > > > the new metadata. For example, it won't know about leadership changes
> > > > in
> > > > > > the partitions it's fetching. The rationale for continuing to fetch
> > > > is to
> > > > > > try to avoid disruptions as much as possible.
> > > > > >
> > > > > > I don't think fenced brokers should accept client requests. The issue
> > > > is
> > > > > > that the fenced broker may or may not have any data it is supposed to
> > > > > > have. It may or may not have applied any configuration changes, etc.
> > > > that
> > > > > > it is supposed to have applied. So it could get pretty confusing, and
> > > > also
> > > > > > potentially waste the client's time.
> > > > > >
> > > > > >
> > > > > When fenced, how would the broker reply to a client which did make a
> > > > > request?
> > > > >
> > > >
> > > > Hi Tom,
> > > >
> > > > The broker will respond with a retryable error in that case. Once the
> > > > client has re-fetched its metadata, it will no longer see the fenced broker
> > > > as part of the cluster. I added a note to the KIP.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Tom
> > > > >
> > > >
> > >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Colin. Thanks for the updates. It's now clear to me that brokers
keep their broker epoch for the life of their JVM -- they register
once, get their broker epoch in the response, and then never
re-register again. Brokers may get fenced, but they keep the same
broker epoch for the life of their JVM. The incarnation ID is also
kept for the life of the JVM but is generated by the broker itself
upon startup, and the combination of the two allows the Controller to
act idempotently if any previously-sent registration response gets
lost. Makes sense.
One thing I wonder about is if it might be helpful for the broker to
send the Cluster ID as determined from its meta.properties file in its
registration request. Does it even make sense for the broker to
successfully register and enter the Fenced state if it has the wrong
Cluster ID? The nextMetadatOffset value that the broker communicates
in its registration request only has meaning within the correct
cluster, so it feels to me that the Controller should have some way to
perform this sanity check. There is currently (pre-KIP 500) a check
in the broker to make sure its configured cluster ID matches the one
stored in ZooKeeper, and we will have to perform this validation
somewhere in the KIP-500 world. If the Controller doesn't do it
within the registration request then the broker will have to make a
metadata request to the Controller, retrieve the Cluster ID, and
perform the check itself. It feels to me that it might be better for
the Controller to just do it, and then the broker doesn't have to
worry about it anymore once it successfully registers.
I also have a question about the broker.id value and meta.properties.
The KIP now says "In version 0 of meta.properties, there is a
broker.id field. Version 1 does not have this field. It is no longer
needed because we no longer support dynamic broker id assignment."
But then there is an example version 1 meta.properties file that shows
the broker.id value. I actually wonder if maybe the broker.id value
would be good to keep in the version 1 meta.properties file because it
currently (pre-KIP 500, version 0) acts as a sanity check to make sure
the broker is using the correct log directory. Similarly with the
controller.id value on controllers -- it would allow the same type of
sanity check for quorum controllers.
Thanks again for your diligent leadership on this KIP.
Ron
On Mon, Nov 30, 2020 at 7:41 PM Colin McCabe <cm...@apache.org> wrote:
>
> On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the reply. A few more comments.
>
> Hi Jun,
>
> Thanks again for the reply. Sorry for the long hiatus. I was on vacation for a while.
>
> >
> > 55. There is still text that favors new broker registration. "When a broker
> > first starts up, when it is in the INITIAL state, it will always "win"
> > broker ID conflicts. However, once it is granted a lease, it transitions
> > out of the INITIAL state. Thereafter, it may lose subsequent conflicts if
> > its broker epoch is stale. (See KIP-380 for some background on broker
> > epoch.) The reason for favoring new processes is to accommodate the common
> > case where a process is killed with kill -9 and then restarted. We want it
> > to be able to reclaim its old ID quickly in this case."
> >
>
> Thanks for the reminder. I have clarified the language here. Hopefully now it is clear that we don't allow quick re-use of broker IDs.
>
> > 80.1 Sounds good. Could you document that listeners is a required config
> > now? It would also be useful to annotate other required configs. For
> > example, controller.connect should be required.
> >
>
> I added a note specifying that these are required.
>
> > 80.2 Could you list all deprecated existing configs? Another one is
> > control.plane.listener.name since the controller no longer sends
> > LeaderAndIsr, UpdateMetadata and StopReplica requests.
> >
>
> I added a section specifying some deprecated configs.
>
> > 83.1 It seems that the broker can transition from FENCED to RUNNING without
> > registering for a new broker epoch. I am not sure how this works. Once the
> > controller fences a broker, there is no need for the controller to keep the
> > boker epoch around. So, if the fenced broker's heartbeat request with the
> > existing broker epoch will be rejected, leading the broker back to the
> > FENCED state again.
> >
>
> The broker epoch refers to the broker registration. So we DO keep the broker epoch around even while the broker is fenced.
>
> The broker epoch changes only when there is a new broker registration. Fencing or unfencing the broker doesn't change the broker epoch.
>
> > 83.5 Good point on KIP-590. Then should we expose the controller for
> > debugging purposes? If not, we should deprecate the controllerID field in
> > MetadataResponse?
> >
>
> I think it's OK to expose it for now, with the proviso that it won't be reachable by clients.
>
> > 90. We rejected the shared ID with just one reason "This is not a good idea
> > because NetworkClient assumes a single ID space. So if there is both a
> > controller 1 and a broker 1, we don't have a way of picking the "right"
> > one." This doesn't seem to be a strong reason. For example, we could
> > address the NetworkClient issue with the node type as you pointed out or
> > using the negative value of a broker ID as the controller ID.
> >
>
> It would require a lot of code changes to support multiple types of node IDs. It's not clear to me that the end result would be better -- I tend to think it would be worse, since it would be more complex. In a similar vein, using negative numbers seems dangerous, since we use negatives or -1 as "special values" in many places. For example, -1 often represents "no such node."
>
> One important thing to keep in mind is that we want to be able to transition from a broker and a controller being co-located to them no longer being co-located. This is much easier to do when they have separate IDs.
>
> > 100. In KIP-589
> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller>,
> > the broker reports all offline replicas due to a disk failure to the
> > controller. It seems this information needs to be persisted to the
> > metadata
> > log. Do we have a corresponding record for that?
> >
>
> Hmm, I have to look into this a little bit more. We may need a new record type.
>
> > 101. Currently, StopReplica request has 2 modes, without deletion and with
> > deletion. The former is used for controlled shutdown and handling disk
> > failure, and causes the follower to stop. The latter is for topic deletion
> > and partition reassignment, and causes the replica to be deleted. Since we
> > are deprecating StopReplica, could we document what triggers the stopping
> > of a follower and the deleting of a replica now?
> >
>
> RemoveTopic triggers deletion. In general the functionality of StopReplica is subsumed by the metadata records.
>
> > 102. Should we include the metadata topic in the MetadataResponse? If so,
> > when it will be included and what will the metadata response look like?
> >
>
> No, it won't be included in the metadata response sent back from the brokers.
>
> > 103. "The active controller assigns the broker a new broker epoch, based on
> > the latest committed offset in the log." This seems inaccurate since the
> > latest committed offset doesn't always advance on every log append.
> >
>
> Given that the new broker epoch won't be visible until the commit has happened, I have changed this to "the next available offset in the log"
>
> > 104. REGISTERING(1) : It says "Otherwise, the broker moves into the FENCED
> > state.". It seems this should be RUNNING?
> >
> > 105. RUNNING: Should we require the broker to catch up to the metadata log
> > to get into this state?
>
> For 104 and 105, these sections have been reworked.
>
> best,
> Colin
>
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > > Hi Colin,
> > > >
> > > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > > >
> > > > > > I have some questions about what happens to a broker when it becomes
> > > > > > fenced (e.g. because it can't send a heartbeat request to keep its
> > > > > > lease). The KIP says "When a broker is fenced, it cannot process any
> > > > > > client requests. This prevents brokers which are not receiving
> > > > > > metadata updates or that are not receiving and processing them fast
> > > > > > enough from causing issues to clients." And in the description of the
> > > > > > FENCED(4) state it likewise says "While in this state, the broker
> > > does
> > > > > > not respond to client requests." It makes sense that a fenced broker
> > > > > > should not accept producer requests -- I assume any such requests
> > > > > > would result in NotLeaderOrFollowerException. But what about KIP-392
> > > > > > (fetch from follower) consumer requests? It is conceivable that
> > > these
> > > > > > could continue. Related to that, would a fenced broker continue to
> > > > > > fetch data for partitions where it thinks it is a follower? Even if
> > > > > > it rejects consumer requests it might still continue to fetch as a
> > > > > > follower. Might it be helpful to clarify both decisions here?
> > > > >
> > > > > Hi Ron,
> > > > >
> > > > > Good question. I think a fenced broker should continue to fetch on
> > > > > partitions it was already fetching before it was fenced, unless it
> > > hits a
> > > > > problem. At that point it won't be able to continue, since it doesn't
> > > have
> > > > > the new metadata. For example, it won't know about leadership changes
> > > in
> > > > > the partitions it's fetching. The rationale for continuing to fetch
> > > is to
> > > > > try to avoid disruptions as much as possible.
> > > > >
> > > > > I don't think fenced brokers should accept client requests. The issue
> > > is
> > > > > that the fenced broker may or may not have any data it is supposed to
> > > > > have. It may or may not have applied any configuration changes, etc.
> > > that
> > > > > it is supposed to have applied. So it could get pretty confusing, and
> > > also
> > > > > potentially waste the client's time.
> > > > >
> > > > >
> > > > When fenced, how would the broker reply to a client which did make a
> > > > request?
> > > >
> > >
> > > Hi Tom,
> > >
> > > The broker will respond with a retryable error in that case. Once the
> > > client has re-fetched its metadata, it will no longer see the fenced broker
> > > as part of the cluster. I added a note to the KIP.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > Thanks,
> > > >
> > > > Tom
> > > >
> > >
> >
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Fri, Oct 23, 2020, at 16:10, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the reply. A few more comments.
Hi Jun,
Thanks again for the reply. Sorry for the long hiatus. I was on vacation for a while.
>
> 55. There is still text that favors new broker registration. "When a broker
> first starts up, when it is in the INITIAL state, it will always "win"
> broker ID conflicts. However, once it is granted a lease, it transitions
> out of the INITIAL state. Thereafter, it may lose subsequent conflicts if
> its broker epoch is stale. (See KIP-380 for some background on broker
> epoch.) The reason for favoring new processes is to accommodate the common
> case where a process is killed with kill -9 and then restarted. We want it
> to be able to reclaim its old ID quickly in this case."
>
Thanks for the reminder. I have clarified the language here. Hopefully now it is clear that we don't allow quick re-use of broker IDs.
> 80.1 Sounds good. Could you document that listeners is a required config
> now? It would also be useful to annotate other required configs. For
> example, controller.connect should be required.
>
I added a note specifying that these are required.
> 80.2 Could you list all deprecated existing configs? Another one is
> control.plane.listener.name since the controller no longer sends
> LeaderAndIsr, UpdateMetadata and StopReplica requests.
>
I added a section specifying some deprecated configs.
> 83.1 It seems that the broker can transition from FENCED to RUNNING without
> registering for a new broker epoch. I am not sure how this works. Once the
> controller fences a broker, there is no need for the controller to keep the
> boker epoch around. So, if the fenced broker's heartbeat request with the
> existing broker epoch will be rejected, leading the broker back to the
> FENCED state again.
>
The broker epoch refers to the broker registration. So we DO keep the broker epoch around even while the broker is fenced.
The broker epoch changes only when there is a new broker registration. Fencing or unfencing the broker doesn't change the broker epoch.
> 83.5 Good point on KIP-590. Then should we expose the controller for
> debugging purposes? If not, we should deprecate the controllerID field in
> MetadataResponse?
>
I think it's OK to expose it for now, with the proviso that it won't be reachable by clients.
> 90. We rejected the shared ID with just one reason "This is not a good idea
> because NetworkClient assumes a single ID space. So if there is both a
> controller 1 and a broker 1, we don't have a way of picking the "right"
> one." This doesn't seem to be a strong reason. For example, we could
> address the NetworkClient issue with the node type as you pointed out or
> using the negative value of a broker ID as the controller ID.
>
It would require a lot of code changes to support multiple types of node IDs. It's not clear to me that the end result would be better -- I tend to think it would be worse, since it would be more complex. In a similar vein, using negative numbers seems dangerous, since we use negatives or -1 as "special values" in many places. For example, -1 often represents "no such node."
One important thing to keep in mind is that we want to be able to transition from a broker and a controller being co-located to them no longer being co-located. This is much easier to do when they have separate IDs.
> 100. In KIP-589
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller>,
> the broker reports all offline replicas due to a disk failure to the
> controller. It seems this information needs to be persisted to the
> metadata
> log. Do we have a corresponding record for that?
>
Hmm, I have to look into this a little bit more. We may need a new record type.
> 101. Currently, StopReplica request has 2 modes, without deletion and with
> deletion. The former is used for controlled shutdown and handling disk
> failure, and causes the follower to stop. The latter is for topic deletion
> and partition reassignment, and causes the replica to be deleted. Since we
> are deprecating StopReplica, could we document what triggers the stopping
> of a follower and the deleting of a replica now?
>
RemoveTopic triggers deletion. In general the functionality of StopReplica is subsumed by the metadata records.
> 102. Should we include the metadata topic in the MetadataResponse? If so,
> when it will be included and what will the metadata response look like?
>
No, it won't be included in the metadata response sent back from the brokers.
> 103. "The active controller assigns the broker a new broker epoch, based on
> the latest committed offset in the log." This seems inaccurate since the
> latest committed offset doesn't always advance on every log append.
>
Given that the new broker epoch won't be visible until the commit has happened, I have changed this to "the next available offset in the log"
> 104. REGISTERING(1) : It says "Otherwise, the broker moves into the FENCED
> state.". It seems this should be RUNNING?
>
> 105. RUNNING: Should we require the broker to catch up to the metadata log
> to get into this state?
For 104 and 105, these sections have been reworked.
best,
Colin
>
> Thanks,
>
> Jun
>
>
>
> On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <cm...@apache.org> wrote:
>
> > On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > > Hi Colin,
> > >
> > > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > > Hi Colin. Thanks for the hard work on this KIP.
> > > > >
> > > > > I have some questions about what happens to a broker when it becomes
> > > > > fenced (e.g. because it can't send a heartbeat request to keep its
> > > > > lease). The KIP says "When a broker is fenced, it cannot process any
> > > > > client requests. This prevents brokers which are not receiving
> > > > > metadata updates or that are not receiving and processing them fast
> > > > > enough from causing issues to clients." And in the description of the
> > > > > FENCED(4) state it likewise says "While in this state, the broker
> > does
> > > > > not respond to client requests." It makes sense that a fenced broker
> > > > > should not accept producer requests -- I assume any such requests
> > > > > would result in NotLeaderOrFollowerException. But what about KIP-392
> > > > > (fetch from follower) consumer requests? It is conceivable that
> > these
> > > > > could continue. Related to that, would a fenced broker continue to
> > > > > fetch data for partitions where it thinks it is a follower? Even if
> > > > > it rejects consumer requests it might still continue to fetch as a
> > > > > follower. Might it be helpful to clarify both decisions here?
> > > >
> > > > Hi Ron,
> > > >
> > > > Good question. I think a fenced broker should continue to fetch on
> > > > partitions it was already fetching before it was fenced, unless it
> > hits a
> > > > problem. At that point it won't be able to continue, since it doesn't
> > have
> > > > the new metadata. For example, it won't know about leadership changes
> > in
> > > > the partitions it's fetching. The rationale for continuing to fetch
> > is to
> > > > try to avoid disruptions as much as possible.
> > > >
> > > > I don't think fenced brokers should accept client requests. The issue
> > is
> > > > that the fenced broker may or may not have any data it is supposed to
> > > > have. It may or may not have applied any configuration changes, etc.
> > that
> > > > it is supposed to have applied. So it could get pretty confusing, and
> > also
> > > > potentially waste the client's time.
> > > >
> > > >
> > > When fenced, how would the broker reply to a client which did make a
> > > request?
> > >
> >
> > Hi Tom,
> >
> > The broker will respond with a retryable error in that case. Once the
> > client has re-fetched its metadata, it will no longer see the fenced broker
> > as part of the cluster. I added a note to the KIP.
> >
> > best,
> > Colin
> >
> > >
> > > Thanks,
> > >
> > > Tom
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the reply. A few more comments.
55. There is still text that favors new broker registration. "When a broker
first starts up, when it is in the INITIAL state, it will always "win"
broker ID conflicts. However, once it is granted a lease, it transitions
out of the INITIAL state. Thereafter, it may lose subsequent conflicts if
its broker epoch is stale. (See KIP-380 for some background on broker
epoch.) The reason for favoring new processes is to accommodate the common
case where a process is killed with kill -9 and then restarted. We want it
to be able to reclaim its old ID quickly in this case."
80.1 Sounds good. Could you document that listeners is a required config
now? It would also be useful to annotate other required configs. For
example, controller.connect should be required.
80.2 Could you list all deprecated existing configs? Another one is
control.plane.listener.name since the controller no longer sends
LeaderAndIsr, UpdateMetadata and StopReplica requests.
83.1 It seems that the broker can transition from FENCED to RUNNING without
registering for a new broker epoch. I am not sure how this works. Once the
controller fences a broker, there is no need for the controller to keep the
boker epoch around. So, if the fenced broker's heartbeat request with the
existing broker epoch will be rejected, leading the broker back to the
FENCED state again.
83.5 Good point on KIP-590. Then should we expose the controller for
debugging purposes? If not, we should deprecate the controllerID field in
MetadataResponse?
90. We rejected the shared ID with just one reason "This is not a good idea
because NetworkClient assumes a single ID space. So if there is both a
controller 1 and a broker 1, we don't have a way of picking the "right"
one." This doesn't seem to be a strong reason. For example, we could
address the NetworkClient issue with the node type as you pointed out or
using the negative value of a broker ID as the controller ID.
100. In KIP-589
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller>,
the broker reports all offline replicas due to a disk failure to the
controller. It seems this information needs to be persisted to the metadata
log. Do we have a corresponding record for that?
101. Currently, StopReplica request has 2 modes, without deletion and with
deletion. The former is used for controlled shutdown and handling disk
failure, and causes the follower to stop. The latter is for topic deletion
and partition reassignment, and causes the replica to be deleted. Since we
are deprecating StopReplica, could we document what triggers the stopping
of a follower and the deleting of a replica now?
102. Should we include the metadata topic in the MetadataResponse? If so,
when it will be included and what will the metadata response look like?
103. "The active controller assigns the broker a new broker epoch, based on
the latest committed offset in the log." This seems inaccurate since the
latest committed offset doesn't always advance on every log append.
104. REGISTERING(1) : It says "Otherwise, the broker moves into the FENCED
state.". It seems this should be RUNNING?
105. RUNNING: Should we require the broker to catch up to the metadata log
to get into this state?
Thanks,
Jun
On Fri, Oct 23, 2020 at 1:20 PM Colin McCabe <cm...@apache.org> wrote:
> On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> > Hi Colin,
> >
> > On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > > Hi Colin. Thanks for the hard work on this KIP.
> > > >
> > > > I have some questions about what happens to a broker when it becomes
> > > > fenced (e.g. because it can't send a heartbeat request to keep its
> > > > lease). The KIP says "When a broker is fenced, it cannot process any
> > > > client requests. This prevents brokers which are not receiving
> > > > metadata updates or that are not receiving and processing them fast
> > > > enough from causing issues to clients." And in the description of the
> > > > FENCED(4) state it likewise says "While in this state, the broker
> does
> > > > not respond to client requests." It makes sense that a fenced broker
> > > > should not accept producer requests -- I assume any such requests
> > > > would result in NotLeaderOrFollowerException. But what about KIP-392
> > > > (fetch from follower) consumer requests? It is conceivable that
> these
> > > > could continue. Related to that, would a fenced broker continue to
> > > > fetch data for partitions where it thinks it is a follower? Even if
> > > > it rejects consumer requests it might still continue to fetch as a
> > > > follower. Might it be helpful to clarify both decisions here?
> > >
> > > Hi Ron,
> > >
> > > Good question. I think a fenced broker should continue to fetch on
> > > partitions it was already fetching before it was fenced, unless it
> hits a
> > > problem. At that point it won't be able to continue, since it doesn't
> have
> > > the new metadata. For example, it won't know about leadership changes
> in
> > > the partitions it's fetching. The rationale for continuing to fetch
> is to
> > > try to avoid disruptions as much as possible.
> > >
> > > I don't think fenced brokers should accept client requests. The issue
> is
> > > that the fenced broker may or may not have any data it is supposed to
> > > have. It may or may not have applied any configuration changes, etc.
> that
> > > it is supposed to have applied. So it could get pretty confusing, and
> also
> > > potentially waste the client's time.
> > >
> > >
> > When fenced, how would the broker reply to a client which did make a
> > request?
> >
>
> Hi Tom,
>
> The broker will respond with a retryable error in that case. Once the
> client has re-fetched its metadata, it will no longer see the fenced broker
> as part of the cluster. I added a note to the KIP.
>
> best,
> Colin
>
> >
> > Thanks,
> >
> > Tom
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Wed, Oct 21, 2020, at 05:51, Tom Bentley wrote:
> Hi Colin,
>
> On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > > Hi Colin. Thanks for the hard work on this KIP.
> > >
> > > I have some questions about what happens to a broker when it becomes
> > > fenced (e.g. because it can't send a heartbeat request to keep its
> > > lease). The KIP says "When a broker is fenced, it cannot process any
> > > client requests. This prevents brokers which are not receiving
> > > metadata updates or that are not receiving and processing them fast
> > > enough from causing issues to clients." And in the description of the
> > > FENCED(4) state it likewise says "While in this state, the broker does
> > > not respond to client requests." It makes sense that a fenced broker
> > > should not accept producer requests -- I assume any such requests
> > > would result in NotLeaderOrFollowerException. But what about KIP-392
> > > (fetch from follower) consumer requests? It is conceivable that these
> > > could continue. Related to that, would a fenced broker continue to
> > > fetch data for partitions where it thinks it is a follower? Even if
> > > it rejects consumer requests it might still continue to fetch as a
> > > follower. Might it be helpful to clarify both decisions here?
> >
> > Hi Ron,
> >
> > Good question. I think a fenced broker should continue to fetch on
> > partitions it was already fetching before it was fenced, unless it hits a
> > problem. At that point it won't be able to continue, since it doesn't have
> > the new metadata. For example, it won't know about leadership changes in
> > the partitions it's fetching. The rationale for continuing to fetch is to
> > try to avoid disruptions as much as possible.
> >
> > I don't think fenced brokers should accept client requests. The issue is
> > that the fenced broker may or may not have any data it is supposed to
> > have. It may or may not have applied any configuration changes, etc. that
> > it is supposed to have applied. So it could get pretty confusing, and also
> > potentially waste the client's time.
> >
> >
> When fenced, how would the broker reply to a client which did make a
> request?
>
Hi Tom,
The broker will respond with a retryable error in that case. Once the client has re-fetched its metadata, it will no longer see the fenced broker as part of the cluster. I added a note to the KIP.
best,
Colin
>
> Thanks,
>
> Tom
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Tom Bentley <tb...@redhat.com>.
Hi Colin,
On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> > Hi Colin. Thanks for the hard work on this KIP.
> >
> > I have some questions about what happens to a broker when it becomes
> > fenced (e.g. because it can't send a heartbeat request to keep its
> > lease). The KIP says "When a broker is fenced, it cannot process any
> > client requests. This prevents brokers which are not receiving
> > metadata updates or that are not receiving and processing them fast
> > enough from causing issues to clients." And in the description of the
> > FENCED(4) state it likewise says "While in this state, the broker does
> > not respond to client requests." It makes sense that a fenced broker
> > should not accept producer requests -- I assume any such requests
> > would result in NotLeaderOrFollowerException. But what about KIP-392
> > (fetch from follower) consumer requests? It is conceivable that these
> > could continue. Related to that, would a fenced broker continue to
> > fetch data for partitions where it thinks it is a follower? Even if
> > it rejects consumer requests it might still continue to fetch as a
> > follower. Might it be helpful to clarify both decisions here?
>
> Hi Ron,
>
> Good question. I think a fenced broker should continue to fetch on
> partitions it was already fetching before it was fenced, unless it hits a
> problem. At that point it won't be able to continue, since it doesn't have
> the new metadata. For example, it won't know about leadership changes in
> the partitions it's fetching. The rationale for continuing to fetch is to
> try to avoid disruptions as much as possible.
>
> I don't think fenced brokers should accept client requests. The issue is
> that the fenced broker may or may not have any data it is supposed to
> have. It may or may not have applied any configuration changes, etc. that
> it is supposed to have applied. So it could get pretty confusing, and also
> potentially waste the client's time.
>
>
When fenced, how would the broker reply to a client which did make a
request?
Thanks,
Tom
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Mon, Oct 19, 2020, at 08:59, Ron Dagostino wrote:
> Hi Colin. Thanks for the hard work on this KIP.
>
> I have some questions about what happens to a broker when it becomes
> fenced (e.g. because it can't send a heartbeat request to keep its
> lease). The KIP says "When a broker is fenced, it cannot process any
> client requests. This prevents brokers which are not receiving
> metadata updates or that are not receiving and processing them fast
> enough from causing issues to clients." And in the description of the
> FENCED(4) state it likewise says "While in this state, the broker does
> not respond to client requests." It makes sense that a fenced broker
> should not accept producer requests -- I assume any such requests
> would result in NotLeaderOrFollowerException. But what about KIP-392
> (fetch from follower) consumer requests? It is conceivable that these
> could continue. Related to that, would a fenced broker continue to
> fetch data for partitions where it thinks it is a follower? Even if
> it rejects consumer requests it might still continue to fetch as a
> follower. Might it be helpful to clarify both decisions here?
Hi Ron,
Good question. I think a fenced broker should continue to fetch on partitions it was already fetching before it was fenced, unless it hits a problem. At that point it won't be able to continue, since it doesn't have the new metadata. For example, it won't know about leadership changes in the partitions it's fetching. The rationale for continuing to fetch is to try to avoid disruptions as much as possible.
I don't think fenced brokers should accept client requests. The issue is that the fenced broker may or may not have any data it is supposed to have. It may or may not have applied any configuration changes, etc. that it is supposed to have applied. So it could get pretty confusing, and also potentially waste the client's time.
>
> Also, for the ScramUserRecord, since we settled on the term
> "UserScramCredential" in KIP-554, might it be better named as
> "UserScramCredsentialRecord"? (Also, the name currently has a
> copy-paste error and is "DelegationTokenRecord").
>
Fixed, on both counts.
thanks,
Colin
> Ron
>
> On Tue, Oct 13, 2020 at 9:31 PM Jun Rao <ju...@confluent.io> wrote:
> >
> > Hi, Colin,
> >
> > Thanks for the reply. A few more comments below.
> >
> > 80.1 controller.listener.names only defines the name of the listener. The
> > actual listener including host/port/security_protocol is typically defined
> > in advertised_listners. Does that mean advertised_listners is a required
> > config now?
> >
> > 83.1 broker state machine: It seems that we should transition from FENCED
> > => INITIAL since only INITIAL generates new broker epoch?
> >
> > 83.5. It's true that the controller node doesn't serve metadata requests.
> > However, there are admin requests such as topic creation/deletion are sent
> > to the controller directly. So, it seems that the client needs to know
> > the controller host/port?
> >
> > 85. "I was hoping that we could avoid responding to requests when the
> > broker was fenced." This issue is that if we don't send a response, the
> > client won't know the reason and can't act properly.
> >
> > 88. CurMetadataOffset: I was thinking that we may want to
> > use CurMetadataOffset to compute the MetadataLag. Since HWM is exclusive,
> > it's more convenient if CurMetadataOffset is also exclusive.
> >
> > 90. It would be useful to add a rejected section on why separate controller
> > and broker id is preferred over just broker id. For example, the following
> > are some potential reasons. (a) We can guard duplicated brokerID, but it's
> > hard to guard against duplicated controllerId. (b) brokerID can be auto
> > assigned in the future, but controllerId is hard to be generated
> > automatically.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Oct 12, 2020 at 11:14 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Tue, Oct 6, 2020, at 16:09, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply. Made another pass of the KIP. A few more comments
> > > > below.
> > > >
> > >
> > > Hi Jun,
> > >
> > > Thanks for the review.
> > >
> > > > 55. We discussed earlier why the current behavior where we favor the
> > > > current broker registration is better. Have you given this more thought?
> > > >
> > >
> > > Yes, I think we should favor the current broker registration, as you
> > > suggested earlier.
> > >
> > > > 80. Config related.
> > > > 80.1 Currently, each broker only has the following 3 required configs. It
> > > > will be useful to document the required configs post KIP-500 (in both the
> > > > dedicated and shared controller mode).
> > > > broker.id
> > > > log.dirs
> > > > zookeeper.connect
> > >
> > > For the broker, these configs will be required:
> > >
> > > broker.id
> > > log.dirs
> > > process.roles
> > > controller.listener.names
> > > controller.connect
> > >
> > > For the controller, these configs will be required:
> > >
> > > controller.id
> > > log.dirs
> > > process.roles
> > > controller.listener.names
> > > controller.connect
> > >
> > > For broker+controller, it will be the union of these two, which
> > > essentially means we need both broker.id and controller.id, but all
> > > others are the same as standalone.
> > >
> > > > 80.2 It would be useful to document all deprecated configs post KIP-500.
> > > > For example, all zookeeper.* are obviously deprecated. But there could be
> > > > others. For example, since we don't plan to support auto broker id
> > > > generation, it seems broker.id.generation.enable is deprecated too.
> > > > 80.3 Could we make it clear that controller.connect replaces
> > > quorum.voters
> > > > in KIP-595?
> > >
> > > OK. I added a comment about this in the table.
> > >
> > > > 80.4 Could we document that broker.id is now optional?
> > >
> > > OK. I have added a line for broker.id.
> > >
> > > > 80.5 The KIP suggests that controller.id is optional on the controller
> > > > node. I am concerned that this can cause a bit of confusion in 2 aspects.
> > > > First, in the dedicated controller mode, controller.id is not optional
> > > > (since broker.id is now optional). Second, in the shared controller
> > > mode,
> > > > it may not be easy for the user to figure out the default value of
> > > > controller.id to set controller.connect properly.
> > >
> > > I got rid of the default value of controller.id. I agree it was a bit
> > > confusing to explain.
> > >
> > > > 80.6 Regarding the consistency of config names, our metrics already
> > > include
> > > > controller. So, prefixing all controller related configs with
> > > "controller"
> > > > may be more consistent. If we choose to do that, could we rename all new
> > > > configs here and in KIP-595 consistently?
> > > >
> > >
> > > I added the new names for the KIP-595 configurations to the configuration
> > > table. I think prefixing these configurations with "controller.quorum" is
> > > the best option since it makes it clear that they are related to the
> > > controller quorum.
> > >
> > > > 81. Metrics
> > > > 81.1 kafka.controller:type=KafkaController,name=MetadataSnapshotLag: This
> > > > is now redundant since KIP-630 already has
> > > > kafka.controller:type=KafkaController,name=SnapshotLag.
> > >
> > > Thanks for bringing up the metric name issue. I think it would be good to
> > > clarify that this is the lag for the metadata snapshot specifically. So I
> > > prefer the name MetadataSnapshotLag. I'll add a note to the table.
> > >
> > > > 81.2 Do we need both kafka.controller:type=KafkaServer,name=MetadataLag
> > > and
> > > > kafka.controller:type=KafkaController,name=MetadataLag since in the
> > > shared
> > > > controller mode, the metadata log is shared?
> > >
> > > Yes, both are needed. This metric is about what metadata logs we have
> > > applied, not about the Raft log's high water mark. If the broker or
> > > controller takes a while to apply the log messages (for example, because of
> > > lock contention or some other issue), they could fall significantly behind
> > > the high watermark.
> > >
> > > >
> > > > 82. Metadata records
> > > > 82.1 BrokerRecord: It needs to include supported features based on
> > > KIP-584.
> > >
> > > Added.
> > >
> > > > 82.2 FeatureLevel: The finalized feature in KIP-584 has the following
> > > > structure and needs to be reflected here.
> > > > {
> > > > "version": 0, // int32 -> Represents the version of the schema for the
> > > > data stored in the ZK node
> > > > "status": 1, // int32 -> Represents the status of the node
> > > > "features": {
> > > > "group_coordinator": { // string -> name of the feature
> > > > "min_version_level": 0, // int16 -> Represents the
> > > cluster-wide
> > > > finalized minimum version level (>=1) of this feature
> > > > "max_version_level": 3 // int16 -> Represents the
> > > cluster-wide
> > > > finalized maximum version level (>=1 and >= min_version_level) of this
> > > > feature
> > > > },
> > > > "consumer_offsets_topic_schema": {
> > > > "min_version_level": 0,
> > > > "max_version_level": 4
> > > > }
> > > > }
> > >
> > > Fair point. I will add both a minimum and maximum finalized feature level.
> > >
> > > >
> > > > 83. RPC
> > > > 83.1 BrokerHeartbeatRequest: Could you document when the controller
> > > assigns
> > > > a new broker epoch?
> > >
> > > The controller assigns a broker epoch if the heartbeat request was sent
> > > without an epoch (that is, it is in the INITIAL state), and the controller
> > > wants to add the new broker to the cluster. I added a note to the state
> > > machine.
> > >
> > > > 83.2 BrokerHeartbeatResponse: Do we need a new error code to indicate a
> > > > broker is fenced?
> > >
> > > I don't think we need a new error code, since targetState will just
> > > indicate that the broker should be fenced.
> > >
> > > > 83.3 What error/state indicates an unsuccessful controlled shutdown in
> > > > BrokerHeartbeatResponse? Currently, the controlled shutdown responses
> > > also
> > > > include a list of remaining partitions. That's mostly for debugging
> > > > purposes. Should we include it in BrokerHeartbeatResponse too?
> > >
> > > Once the controller wants the broker to shut down, it will set targetState
> > > to SHUTDOWN.
> > >
> > > We don't need to send a list of remaining partitions, since the broker is
> > > following the metadata log, and will get all its instructions from there.
> > >
> > > > 83.4 Could we document when Listeners, Features and Rack are expected to
> > > be
> > > > set in BrokerHeartbeatRequest?
> > >
> > > I added a line specifying that: The listeners, features, and rack fields
> > > only need to be set when the broker is in INITIAL state, and is requesting
> > > a new broker epoch.
> > >
> > > > 83.5 Currently, the metadata response only returns a list of brokers.
> > > > Should we have a separate field for the controller info or change the
> > > field
> > > > name to sth like nodes? Also, is the rack field relevant to the
> > > controller?
> > >
> > > As per KIP-590, we don't want the clients to communicate directly with the
> > > controller. So we weren't planning on including the controller's
> > > information in the MetadataResponse.
> > >
> > > We don't currently have rack-awareness for controllers, but I suppose we
> > > could in the future. I think we can leave that for later.
> > >
> > > >
> > > > 84. There are a few references to __kafka_metadata. However, KIP-595
> > > > uses __cluster_metadata.
> > >
> > > We agreed earlier to decide on the name of the metadata topic in this KIP,
> > > and to treat the KIP-595 name as provisional. Looking at it again, I think
> > > we should go with @metadata, in order to avoid colliding with topic names
> > > that already exist.
> > >
> > > >
> > > > 85. "When a broker is fenced, it cannot process any client requests. "
> > > > Should we send a new error code in the response to indicate that the
> > > broker
> > > > is fenced?
> > >
> > > I was hoping that we could avoid responding to requests when the broker
> > > was fenced. I suppose a new error code could work too, but then we also
> > > need a compatibility story for older clients.
> > >
> > > >
> > > > 86. "The controller can generate a new broker epoch by using the latest
> > > log
> > > > offset." Perhaps it's more accurate to say the new broker epoch is the
> > > > offset for the corresponding BrokerRecord in the metadata log.
> > >
> > > The idea here is slightly different. We want to use the current latest
> > > offset (prior to creating the BrokerRecord) as the new epoch. We can't use
> > > the offset of the BrokerRecord itself since we won't know what that is
> > > until we give the BrokerRecord to the Raft layer.
> > >
> > > >
> > > > 87. meta.properties:
> > > > 87.1 Could you document the format change to this file? Also, should we
> > > > bump up the version of the file?
> > >
> > > Hmm. The file is just a list of key=value pairs, one per line. So I'm
> > > not sure there is any format change per se, at least technically.
> > >
> > > Maybe we should bump up the version anyway, in order to prevent older
> > > pre-KIP-500 software from launching if the new stuff is present?
> > >
> > > > 87.2 How do we know a node is configured in KIP-500 mode?
> > > >
> > >
> > > A cluster configured in KIP-500 mode will contain the following line in
> > > the meta.properties file in each directory: kip.500.mode=enabled
> > >
> > > > 88. CurMetadataOffset: Should we use nextMetadataOffset since it's more
> > > > consistent with fetchOffset and HWM?
> > >
> > > I'm not sure. Drawing an analogy with HWM might be confusing, since
> > > CurMetadataOffset is not HWM. It's totally possible for the broker to have
> > > fetched up to offset 1000 but only applied up to offset 500.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, Oct 5, 2020 at 9:03 AM Colin McCabe <cm...@apache.org> wrote:
> > > >
> > > > > On Mon, Sep 28, 2020, at 11:41, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the reply. A few more comments below.
> > > > > >
> > > > > > 62.
> > > > > > 62.1 controller.listener.names: So, is this used for the controller
> > > or
> > > > > the
> > > > > > broker trying to connect to the controller?
> > > > > >
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > It's used by both. The broker tries to connect to controllers by using
> > > > > the first listener in this list. The controller uses the list to
> > > determine
> > > > > which listeners it should bind to.
> > > > >
> > > > > >
> > > > > > 62.2 If we want to take the approach to share the configs that are
> > > common
> > > > > > between the broker and the controller, should we share the id too?
> > > > > >
> > > > >
> > > > > On nodes where we are running both the controller and broker roles, we
> > > > > need two IDs: one for the controller, and one for the broker. So we
> > > can't
> > > > > share the same configuration.
> > > > >
> > > > > >
> > > > > > 62.3 We added some configs in KIP-595 prefixed with "quorum" and we
> > > plan
> > > > > to
> > > > > > add some controller specific configs prefixed with "controller".
> > > KIP-630
> > > > > > plans to add some other controller specific configs with no prefix.
> > > > > Should
> > > > > > we standardize all controller specific configs with the same prefix?
> > > > > >
> > > > >
> > > > > That's a good question. As Jose said elsewhere in the thread, let's
> > > use
> > > > > "metadata" as the prefix.
> > > > >
> > > > > >
> > > > > > 70. Could you explain the impact of process.roles a bit more? For
> > > > > example,
> > > > > > if process.roles=controller, does the node still maintain metadata
> > > cache
> > > > > as
> > > > > > described in KIP-630? Do we still return the host/port for those
> > > nodes in
> > > > > > the metadata response?
> > > > > >
> > > > >
> > > > > No, the node does not need any broker data structures if it is not a
> > > > > broker. Controllers also don't handle MetadataRequests. Clients will
> > > > > still go to the brokers for those.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Mon, Sep 28, 2020 at 9:24 AM Colin McCabe <cm...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > On Fri, Sep 25, 2020, at 17:35, Jun Rao wrote:
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the reply.
> > > > > > > >
> > > > > > > > 62. Thinking about this more, I am wondering what's our overall
> > > > > strategy
> > > > > > > > for configs shared between the broker and the controller. For
> > > > > example,
> > > > > > > both
> > > > > > > > the broker and the controller have to define listeners. So both
> > > need
> > > > > > > > configs like listeners/advertised.listeners. Both the new
> > > controller
> > > > > and
> > > > > > > > the broker replicate data. So both need to define some
> > > replication
> > > > > > > related
> > > > > > > > configurations (replica.fetch.min.bytes,
> > > replica.fetch.wait.max.ms,
> > > > > > > etc).
> > > > > > > > Should we let the new controller share those configs with
> > > brokers or
> > > > > > > should
> > > > > > > > we define new configs just for the controller? It seems that we
> > > want
> > > > > to
> > > > > > > > apply the same strategy consistently for all shared configs?
> > > > > > > >
> > > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > That's a good question. I think that we should share as many
> > > > > > > configurations as possible. There will be a few cases where this
> > > isn't
> > > > > > > possible, and we need to create a new configuration key that is
> > > > > > > controller-only, but I think that will be relatively rare.
> > > > > > >
> > > > > > > >
> > > > > > > > 63. If a node has process.roles set to controller, does one still
> > > > > need to
> > > > > > > > set broker.id on this node?
> > > > > > > >
> > > > > > >
> > > > > > > No, broker.id does not need to be set in that case.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Sep 25, 2020 at 2:17 PM Colin McCabe <cmccabe@apache.org
> > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Fri, Sep 25, 2020, at 10:17, Jun Rao wrote:
> > > > > > > > > > Hi, Colin,
> > > > > > > > > >
> > > > > > > > > > Thanks for the reply.
> > > > > > > > > >
> > > > > > > > > > 60. Yes, I think you are right. We probably need the
> > > controller
> > > > > id
> > > > > > > when a
> > > > > > > > > > broker starts up. A broker only stores the Raft leader id in
> > > the
> > > > > > > metadata
> > > > > > > > > > file. To do the initial fetch to the Raft leader, it needs to
> > > > > know
> > > > > > > the
> > > > > > > > > > host/port associated with the leader id.
> > > > > > > > > >
> > > > > > > > > > 62. It seems there are 2 parts to this : (1) which listener a
> > > > > client
> > > > > > > > > should
> > > > > > > > > > use to initiate a connection to the controller and (2) which
> > > > > listener
> > > > > > > > > > should a controller use to accept client requests. For (1),
> > > at
> > > > > any
> > > > > > > point
> > > > > > > > > of
> > > > > > > > > > time, a client only needs to use one listener. I think
> > > > > > > > > > controller.listener.name is meant for the client.
> > > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > controller.listener.names is also used by the controllers. In
> > > the
> > > > > case
> > > > > > > > > where we have a broker and a controller in the same JVM, we
> > > have a
> > > > > > > single
> > > > > > > > > config file. Then we need to know which listeners belong to
> > > the
> > > > > > > controller
> > > > > > > > > and which belong to the broker component. That's why it's a
> > > list.
> > > > > > > > >
> > > > > > > > > > So, a single value seems
> > > > > > > > > > to make more sense. Currently, we don't have a configuration
> > > for
> > > > > > > (2). We
> > > > > > > > > > could add a new one for that and support a list. I am
> > > wondering
> > > > > how
> > > > > > > > > useful
> > > > > > > > > > it will be. One example that I can think of is that we can
> > > reject
> > > > > > > > > > non-controller related requests if accepted on
> > > controller-only
> > > > > > > listeners.
> > > > > > > > > > However, we already support separate authentication for the
> > > > > > > controller
> > > > > > > > > > listener. So, not sure how useful it is.
> > > > > > > > >
> > > > > > > > > The controller always has a separate listener and does not
> > > share
> > > > > > > listeners
> > > > > > > > > with the broker. The main reason for this is to avoid giving
> > > > > clients
> > > > > > > the
> > > > > > > > > ability to launch a denial-of-service attack on the controller
> > > by
> > > > > > > flooding
> > > > > > > > > a broker port. A lot of times, these attacks are made
> > > > > unintentionally
> > > > > > > by
> > > > > > > > > poorly coded or configured clients. Additionally, broker
> > > ports can
> > > > > > > also be
> > > > > > > > > very busy with large fetch requests, and so on. It's just a
> > > bad
> > > > > > > > > configuration in general to try to overload the same port for
> > > the
> > > > > > > > > controller and broker, and we don't want to allow it. It
> > > would be
> > > > > a
> > > > > > > > > regression to go from the current system where control
> > > requests are
> > > > > > > safely
> > > > > > > > > isolated on a separate port, to one where they're not. It also
> > > > > makes
> > > > > > > the
> > > > > > > > > code and configuration a lot messier.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 63. (a) I think most users won't know controller.id
> > > defaults to
> > > > > > > > > broker.id +
> > > > > > > > > > 3000. So, it can be confusing for them to set up
> > > > > controller.connect.
> > > > > > > If
> > > > > > > > > > this is truly needed, it seems that it's less confusing to
> > > make
> > > > > > > > > > controller.id required.
> > > > > > > > > > (b) I am still trying to understand if we truly need to
> > > expose a
> > > > > > > > > > controller.id. What if we only expose broker.id and let
> > > > > > > > > controller.connect
> > > > > > > > > > just use broker.id? What will be missing?
> > > > > > > > >
> > > > > > > > > The controller has a separate ID from the broker, so knowing
> > > > > broker.id
> > > > > > > is
> > > > > > > > > not helpful here.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 24, 2020 at 10:55 PM Colin McCabe <
> > > > > cmccabe@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 24, 2020, at 16:24, Jun Rao wrote:
> > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the reply and the updated KIP. A few more
> > > comments
> > > > > > > below.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Jun,
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 53. It seems that you already incorporated the changes in
> > > > > > > KIP-516.
> > > > > > > > > With
> > > > > > > > > > > > topic ids, we don't need to wait for the topic's data to
> > > be
> > > > > > > deleted
> > > > > > > > > > > before
> > > > > > > > > > > > removing the topic metadata. If the topic is recreated,
> > > we
> > > > > can
> > > > > > > still
> > > > > > > > > > > delete
> > > > > > > > > > > > the data properly based on the topic id. So, it seems
> > > that
> > > > > we can
> > > > > > > > > remove
> > > > > > > > > > > > TopicRecord.Deleting.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the reply. What I was thinking of doing here
> > > was
> > > > > using
> > > > > > > > > topic
> > > > > > > > > > > IDs internally, but still using names externally. So the
> > > topic
> > > > > > > UUIDs
> > > > > > > > > are
> > > > > > > > > > > only for the purpose of associating topics with partitions
> > > --
> > > > > from
> > > > > > > the
> > > > > > > > > > > user's point of view, topics are still identified by names.
> > > > > > > > > > >
> > > > > > > > > > > You're right that KIP-516 will simplify things, but I'm not
> > > > > sure
> > > > > > > when
> > > > > > > > > that
> > > > > > > > > > > will land, so I wanted to avoid blocking the initial
> > > > > > > implementation of
> > > > > > > > > this
> > > > > > > > > > > KIP on it.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 55. It seems to me that the current behavior where we
> > > favor
> > > > > the
> > > > > > > > > current
> > > > > > > > > > > > broker registration is better. This is because
> > > uncontrolled
> > > > > > > broker
> > > > > > > > > > > shutdown
> > > > > > > > > > > > is rare and its impact is less severe since one just
> > > needs to
> > > > > > > wait
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > session timeout before restarting the broker. If a
> > > > > mis-configured
> > > > > > > > > broker
> > > > > > > > > > > > replaces an existing broker, the consequence is more
> > > severe
> > > > > > > since it
> > > > > > > > > can
> > > > > > > > > > > > cause the leader to be unavailable, a replica to be out
> > > of
> > > > > ISR,
> > > > > > > and
> > > > > > > > > add
> > > > > > > > > > > > more load on the leaders etc.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hmm, that's a good point. Let me check this a bit more
> > > before
> > > > > I
> > > > > > > change
> > > > > > > > > > > it, though.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 60. controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com
> > > :
> > > > > Do we
> > > > > > > > > need to
> > > > > > > > > > > > include the controller id before @? It seems that the
> > > > > host/port
> > > > > > > is
> > > > > > > > > enough
> > > > > > > > > > > > for establishing the connection. It would also be useful
> > > to
> > > > > make
> > > > > > > it
> > > > > > > > > clear
> > > > > > > > > > > > that controller.connect replaces quorum.voters in
> > > KIP-595.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I discussed this with Jason earlier, and he felt that the
> > > > > > > controller
> > > > > > > > > IDs
> > > > > > > > > > > were needed in this configuration key. It is certainly
> > > needed
> > > > > when
> > > > > > > > > > > configuring the controllers themselves, since they need to
> > > know
> > > > > > > each
> > > > > > > > > > > others' IDs.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 61. I am not sure that we need both controller.listeners
> > > and
> > > > > > > > > > > > controller.connect.security.protocol since the former
> > > > > implies the
> > > > > > > > > > > security
> > > > > > > > > > > > protocol. The reason that we have both
> > > > > > > inter.broker.listener.name
> > > > > > > > > and
> > > > > > > > > > > > inter.broker.security.protocol is because we had the
> > > latter
> > > > > > > first and
> > > > > > > > > > > later
> > > > > > > > > > > > realized that the former is more general.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > That's a good point. I've removed this from the KIP.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 62. I am still not sure that you need
> > > controller.listeners
> > > > > to be
> > > > > > > a
> > > > > > > > > list.
> > > > > > > > > > > > All listeners are already defined in listeners. To
> > > migrate
> > > > > from
> > > > > > > > > plaintext
> > > > > > > > > > > > to SSL, one can configure listeners to have both
> > > plaintext
> > > > > and
> > > > > > > SSL.
> > > > > > > > > After
> > > > > > > > > > > > that, one can just change controller.listeners from
> > > > > plaintext to
> > > > > > > SSL.
> > > > > > > > > > > This
> > > > > > > > > > > > is similar to how to change the listener for inter broker
> > > > > > > > > connections.
> > > > > > > > > > > > Also, controller.listener.name may be a more accurate
> > > name?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The issue that I see here is that if you are running with
> > > the
> > > > > > > > > controller
> > > > > > > > > > > and broker in the same JVM, if you define a few listeners
> > > in
> > > > > > > > > "listeners"
> > > > > > > > > > > they will get used as regular broker listeners, unless you
> > > put
> > > > > > > them in
> > > > > > > > > > > controller.listeners. Therefore, controller.listeners
> > > needs
> > > > > to be
> > > > > > > a
> > > > > > > > > list.
> > > > > > > > > > >
> > > > > > > > > > > controller.listener.names does sound like a better name,
> > > > > though...
> > > > > > > I've
> > > > > > > > > > > updated it to that.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 63. Regarding controller.id, I am trying to understand
> > > > > whether
> > > > > > > it's
> > > > > > > > > a
> > > > > > > > > > > > required configuration or an optional one. From the KIP,
> > > it
> > > > > > > sounds
> > > > > > > > > like
> > > > > > > > > > > > controller.id is optional. Then, if this is unset, it's
> > > not
> > > > > > > clear
> > > > > > > > > how
> > > > > > > > > > > the
> > > > > > > > > > > > user will obtain the controller id for setting
> > > > > > > controller.connect.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > If you specify broker.id but not controller.id, then
> > > > > controller.id
> > > > > > > > > will
> > > > > > > > > > > just be set to broker.id + 3000. This was intended to
> > > make
> > > > > some
> > > > > > > > > > > configurations a little bit more concise to write. You
> > > still
> > > > > do
> > > > > > > have
> > > > > > > > > to
> > > > > > > > > > > know the controller IDs when configuring the brokers,
> > > though.
> > > > > If
> > > > > > > this
> > > > > > > > > > > seems confusing then I can just make it mandatory.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 64. KIP-516 adds a flag in LeaderAndIsrRequest to
> > > indicate
> > > > > > > whether it
> > > > > > > > > > > > includes all partitions or not. This will be used to
> > > clean up
> > > > > > > strayed
> > > > > > > > > > > logs.
> > > > > > > > > > > > I was thinking how we will do the same thing once the
> > > > > controller
> > > > > > > > > moves to
> > > > > > > > > > > > Raft based. One way to do that is on broker startup, it
> > > gets
> > > > > the
> > > > > > > HWM
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > metadata log from the Raft leader and waits until its
> > > > > metadata
> > > > > > > > > catches up
> > > > > > > > > > > > to HWM. At that point, any local log not seen in the
> > > metadata
> > > > > > > can be
> > > > > > > > > > > > removed. Since the Fetch response returns the HWM, there
> > > > > seems
> > > > > > > to be
> > > > > > > > > > > enough
> > > > > > > > > > > > APIs to achieve this.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > That's a very good point. I added a note about this under
> > > > > Broker
> > > > > > > > > Startup.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Jun
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 24, 2020 at 1:07 PM Colin McCabe <
> > > > > colin@cmccabe.xyz>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Sep 21, 2020, at 18:13, Jun Rao wrote:
> > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Sorry for the late reply. A few more comments below.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for taking another look.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 50. Configurations
> > > > > > > > > > > > > > 50.1 controller.listeners: It seems that a controller
> > > > > just
> > > > > > > needs
> > > > > > > > > one
> > > > > > > > > > > > > > listener. Why do we need to have a list here? Also,
> > > > > could you
> > > > > > > > > > > provide an
> > > > > > > > > > > > > > example of how this is set and what's its
> > > relationship
> > > > > with
> > > > > > > > > existing
> > > > > > > > > > > > > > configs such as "security.inter.broker.protocol" and
> > > "
> > > > > > > > > > > > > > inter.broker.listener.name"?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I agree that most administrators will want to run with
> > > > > only one
> > > > > > > > > > > controller
> > > > > > > > > > > > > listener. However, just as with brokers, it is nice to
> > > > > have
> > > > > > > the
> > > > > > > > > > > option to
> > > > > > > > > > > > > expose multiple ports.
> > > > > > > > > > > > >
> > > > > > > > > > > > > One reason why you might want multiple ports is if you
> > > were
> > > > > > > doing a
> > > > > > > > > > > > > migration from plaintext to SSL. You could add new SSL
> > > > > > > listeners
> > > > > > > > > but
> > > > > > > > > > > > > continue to expose the PLAINTEXT listeners. Then you
> > > could
> > > > > > > add the
> > > > > > > > > > > new SSL
> > > > > > > > > > > > > controller config to each broker and roll each broker.
> > > > > Then
> > > > > > > you
> > > > > > > > > could
> > > > > > > > > > > > > migrate from PLAINTEXT to SSL with no downtime.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Here's an example configuration for the controller:
> > > > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > > 1@controller1.example.com:9093,
> > > 2@controller2.example.com
> > > > > > > > > > > > > controller.listeners=CONTROLLER
> > > > > > > > > > > > > listeners=CONTROLLER://controller0.example.com:9093
> > > > > > > > > > > > > listener.security.protocol.map=CONTROLLER:SSL
> > > > > > > > > > > > >
> > > > > > > > > > > > > Here's an example configuration for the broker:
> > > > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > > 1@controller1.example.com:9093,
> > > 2@controller2.example.com
> > > > > > > > > > > > > controller.connect.security.protocol=SSL
> > > > > > > > > > > > >
> > > > > > > > > > > > > security.inter.broker.protocol or
> > > > > inter.broker.listener.name
> > > > > > > do
> > > > > > > > > not
> > > > > > > > > > > > > affect how the broker communicates with the controller.
> > > > > Those
> > > > > > > > > > > > > configurations relate to how brokers communicate with
> > > each
> > > > > > > other,
> > > > > > > > > but
> > > > > > > > > > > the
> > > > > > > > > > > > > controller is not a broker.
> > > > > > > > > > > > >
> > > > > > > > > > > > > (Note that I just added
> > > > > controller.connect.security.protocol
> > > > > > > to the
> > > > > > > > > > > KIP --
> > > > > > > > > > > > > I had forgotten to put it in earlier)
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 50.2 registration.heartbeat.interval.ms and
> > > > > > > > > > > > > registration.lease.timeout.ms.
> > > > > > > > > > > > > > Should we match their default value with the
> > > > > corresponding
> > > > > > > > > default
> > > > > > > > > > > for
> > > > > > > > > > > > > ZK?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Fair enough. I'll set them to the values of the
> > > > > > > > > > > zookeeper.sync.time.ms
> > > > > > > > > > > > > and zookeeper.connection.timeout.ms configurations.
> > > I do
> > > > > > > think we
> > > > > > > > > > > should
> > > > > > > > > > > > > experiment later on to see what works well here, but
> > > the ZK
> > > > > > > values
> > > > > > > > > are
> > > > > > > > > > > at
> > > > > > > > > > > > > least a starting point.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 50.3 controller.connect: Could you provide an
> > > example? I
> > > > > am
> > > > > > > > > > > wondering how
> > > > > > > > > > > > > > it differs from quorum.voters=1@kafka-1:9092,
> > > 2@kafka-2
> > > > > > > :9092,
> > > > > > > > > > > 3@kafka-3
> > > > > > > > > > > > > > :9092.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > controller.connect is intended to be the new name for
> > > > > > > > > quorum.voters.
> > > > > > > > > > > > > During the vote for KIP-595, we sort of agreed to
> > > defer the
> > > > > > > > > discussion
> > > > > > > > > > > > > about what this configuration should be called. I
> > > proposed
> > > > > > > this
> > > > > > > > > new
> > > > > > > > > > > name
> > > > > > > > > > > > > because it makes it clear what the configuration is for
> > > > > (how to
> > > > > > > > > > > connect to
> > > > > > > > > > > > > the controllers).
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 50.4 controller.id: I am still not sure how this is
> > > > > being
> > > > > > > used.
> > > > > > > > > > > Could
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > explain this in more detail?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > The controller ID needs to be set on each controller.
> > > It
> > > > > also
> > > > > > > > > appears
> > > > > > > > > > > in
> > > > > > > > > > > > > controller.connect as the thing before the "at" sign.
> > > Its
> > > > > > > > > function is
> > > > > > > > > > > > > pretty similar to broker.id.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Broker IDs and controller IDs exist in the same ID
> > > space,
> > > > > so
> > > > > > > you
> > > > > > > > > can't
> > > > > > > > > > > > > have a broker and a controller that use the same ID.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 51. BrokerHeartbeat: It seems a bit wasteful to
> > > include
> > > > > > > > > Listeners in
> > > > > > > > > > > > > every
> > > > > > > > > > > > > > heartbeat request since it typically doesn't change.
> > > > > Could we
> > > > > > > > > make
> > > > > > > > > > > that
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > optional field?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ok.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 52. KIP-584 adds a new ZK node /features. Should we
> > > add a
> > > > > > > > > > > corresponding
> > > > > > > > > > > > > > metadata record?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Good point. I added FeatureLevelRecord.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 53. TopicRecord and DeleteTopic: Both DeleteTopic and
> > > > > > > > > > > > > TopicRecord.Deleting
> > > > > > > > > > > > > > indicate topic deletion. Could we outline the flow
> > > when
> > > > > each
> > > > > > > > > will be
> > > > > > > > > > > set?
> > > > > > > > > > > > > > In particular, which one indicates the intention to
> > > > > delete
> > > > > > > and
> > > > > > > > > which
> > > > > > > > > > > one
> > > > > > > > > > > > > > indicates the completion of the deletion.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > TopicRecord.Deleting is set when we intend to delete
> > > the
> > > > > topic.
> > > > > > > > > > > > >
> > > > > > > > > > > > > DeleteTopic removes the topic completely. I will
> > > rename
> > > > > > > > > DeleteTopic to
> > > > > > > > > > > > > RemoveTopic to make this clearer.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 54. "The controller can generate a new broker epoch
> > > by
> > > > > using
> > > > > > > the
> > > > > > > > > > > latest
> > > > > > > > > > > > > log
> > > > > > > > > > > > > > offset." Which offset is that? Is it the offset of
> > > the
> > > > > > > metadata
> > > > > > > > > > > topic for
> > > > > > > > > > > > > > the corresponding BrokerRecord? Is it guaranteed to
> > > be
> > > > > > > different
> > > > > > > > > on
> > > > > > > > > > > each
> > > > > > > > > > > > > > broker restart?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes.... a new broker epoch implies that there has been
> > > a
> > > > > new
> > > > > > > record
> > > > > > > > > > > > > created in the metadata log. Therefore the last
> > > committed
> > > > > > > offset
> > > > > > > > > must
> > > > > > > > > > > be
> > > > > > > > > > > > > different.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 55. "Thereafter, it may lose subsequent conflicts if
> > > its
> > > > > > > broker
> > > > > > > > > > > epoch is
> > > > > > > > > > > > > > stale. (See KIP-380 for some background on broker
> > > > > epoch.)
> > > > > > > The
> > > > > > > > > > > reason
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > favoring new processes is to accommodate the common
> > > case
> > > > > > > where a
> > > > > > > > > > > process
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > killed with kill -9 and then restarted. " Are you
> > > saying
> > > > > > > that if
> > > > > > > > > > > there is
> > > > > > > > > > > > > > an active broker registered in the controller, a new
> > > > > broker
> > > > > > > > > heartbeat
> > > > > > > > > > > > > > request with the INITIAL state will fence the current
> > > > > broker
> > > > > > > > > > > session? Not
> > > > > > > > > > > > > > sure about this. For example, this will allow a
> > > broker
> > > > > with
> > > > > > > > > > > incorrectly
> > > > > > > > > > > > > > configured broker id to kill an existing valid
> > > broker.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, a new broker with an incorrectly configured
> > > broker id
> > > > > > > could
> > > > > > > > > fence
> > > > > > > > > > > an
> > > > > > > > > > > > > existing valid broker.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The goal of the conflict resolution process described
> > > here
> > > > > is
> > > > > > > to
> > > > > > > > > avoid
> > > > > > > > > > > > > having two brokers go back and forth in claiming the
> > > > > broker id.
> > > > > > > > > > > Choosing
> > > > > > > > > > > > > the newest was just an easy way to do that.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Choosing the oldest is another possibility, and more
> > > inline
> > > > > > > with
> > > > > > > > > what
> > > > > > > > > > > > > ZooKeeper does today. The problem with that is that it
> > > > > would
> > > > > > > not
> > > > > > > > > be
> > > > > > > > > > > > > possible to re-create the broker if the old instance
> > > died
> > > > > > > suddenly,
> > > > > > > > > > > except
> > > > > > > > > > > > > by waiting for the full lease timeout.
> > > > > > > > > > > > >
> > > > > > > > > > > > > It might be possible to do a little better here by
> > > > > breaking the
> > > > > > > > > lease
> > > > > > > > > > > if
> > > > > > > > > > > > > the TCP connection from the broker drops. But that
> > > > > requires
> > > > > > > > > crossing
> > > > > > > > > > > some
> > > > > > > > > > > > > layers of abstraction in the Kafka networking stack.
> > > > > There's
> > > > > > > also
> > > > > > > > > the
> > > > > > > > > > > > > possibility of false negatives or positives. For
> > > example,
> > > > > TCP
> > > > > > > > > > > connections
> > > > > > > > > > > > > sometimes just drop even if the node is still there.
> > > Or
> > > > > > > sometimes
> > > > > > > > > the
> > > > > > > > > > > > > networking stack keeps a TCP connection alive for a
> > > while
> > > > > when
> > > > > > > the
> > > > > > > > > > > other
> > > > > > > > > > > > > end is gone.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 56. kafka-storage.sh:
> > > > > > > > > > > > > > 56.1 In the info mode, what other information does it
> > > > > show in
> > > > > > > > > > > addition to
> > > > > > > > > > > > > > "kip.500.mode=enabled"?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think it should show a list of all the storage
> > > > > directories,
> > > > > > > and
> > > > > > > > > > > whether
> > > > > > > > > > > > > each one is formatted. In addition, it should show
> > > whether
> > > > > > > > > > > kip.500.mode is
> > > > > > > > > > > > > enabled, and what the cluster id is.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 56.2 Should the format mode take the config file as
> > > the
> > > > > > > input too
> > > > > > > > > > > like
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > info mode?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, that's a good idea.
> > > > > > > > > > > > >
> > > > > > > > > > > > > best,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 17, 2020 at 4:12 PM Colin McCabe <
> > > > > > > cmccabe@apache.org
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > That's a fair point. I have moved the lease
> > > duration
> > > > > to
> > > > > > > the
> > > > > > > > > broker
> > > > > > > > > > > > > > > heartbeat response. That way lease durations can
> > > be
> > > > > > > changed
> > > > > > > > > just
> > > > > > > > > > > be
> > > > > > > > > > > > > > > reconfiguring the controllers.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Wed, Sep 16, 2020, at 07:40, Unmesh Joshi wrote:
> > > > > > > > > > > > > > > > Thanks Colin, the changes look good to me. One
> > > small
> > > > > > > thing.
> > > > > > > > > > > > > > > > registration.lease.timeout.ms is the
> > > configuration
> > > > > on
> > > > > > > the
> > > > > > > > > > > controller
> > > > > > > > > > > > > > > side.
> > > > > > > > > > > > > > > > It will be good to comment on how brokers know
> > > about
> > > > > it,
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > able
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > send LeaseDurationMs
> > > > > > > > > > > > > > > > in the heartbeat request,
> > > > > > > > > > > > > > > > or else it can be added in the heartbeat
> > > response for
> > > > > > > > > brokers to
> > > > > > > > > > > know
> > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Sep 11, 2020 at 10:32 PM Colin McCabe <
> > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think you're right that we should use a
> > > duration
> > > > > here
> > > > > > > > > rather
> > > > > > > > > > > > > than a
> > > > > > > > > > > > > > > > > time. As you said, the clock on the controller
> > > > > will
> > > > > > > > > probably
> > > > > > > > > > > not
> > > > > > > > > > > > > > > match the
> > > > > > > > > > > > > > > > > one on the broker. I have updated the KIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It's important to keep in mind that
> > > messages
> > > > > may be
> > > > > > > > > > > delayed in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > > > happens, we
> > > > > > > > > > > will
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > > time specified in the request to determine
> > > if
> > > > > the
> > > > > > > > > request
> > > > > > > > > > > is
> > > > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > > > I am assuming that there will be a single TCP
> > > > > > > connection
> > > > > > > > > > > > > maintained
> > > > > > > > > > > > > > > > > between
> > > > > > > > > > > > > > > > > > broker and active controller. So, there
> > > won't be
> > > > > any
> > > > > > > out
> > > > > > > > > of
> > > > > > > > > > > order
> > > > > > > > > > > > > > > > > requests?
> > > > > > > > > > > > > > > > > > There will be a scenario of broker GC pause,
> > > > > which
> > > > > > > might
> > > > > > > > > > > cause
> > > > > > > > > > > > > > > connection
> > > > > > > > > > > > > > > > > > timeout and broker might need to reestablish
> > > the
> > > > > > > > > connection.
> > > > > > > > > > > If
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > pause
> > > > > > > > > > > > > > > > > > is too long, lease will expire and the
> > > heartbeat
> > > > > sent
> > > > > > > > > after
> > > > > > > > > > > the
> > > > > > > > > > > > > pause
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > be treated as a new registration (similar to
> > > > > restart
> > > > > > > > > case),
> > > > > > > > > > > and
> > > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > > epoch
> > > > > > > > > > > > > > > > > > number will be assigned to the broker.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I agree with the end of this paragraph, but not
> > > > > with
> > > > > > > the
> > > > > > > > > start
> > > > > > > > > > > :)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There can be out-of-order requests, since the
> > > > > broker
> > > > > > > will
> > > > > > > > > > > simply
> > > > > > > > > > > > > use a
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > TCP connection if the old one has problems.
> > > This
> > > > > can
> > > > > > > > > happen
> > > > > > > > > > > for a
> > > > > > > > > > > > > > > variety
> > > > > > > > > > > > > > > > > of reasons. I don't think GC pauses are the
> > > most
> > > > > > > common
> > > > > > > > > > > reason for
> > > > > > > > > > > > > > > this to
> > > > > > > > > > > > > > > > > happen. It's more common to see issues issues
> > > in
> > > > > the
> > > > > > > > > network
> > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > result connections getting dropped from time to
> > > > > time.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > So we have to assume that messages may arrive
> > > out
> > > > > of
> > > > > > > > > order, and
> > > > > > > > > > > > > > > possibly
> > > > > > > > > > > > > > > > > be delayed. I added a note that heartbeat
> > > requests
> > > > > > > should
> > > > > > > > > be
> > > > > > > > > > > > > ignored
> > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > the metadata log offset they contain is smaller
> > > > > than a
> > > > > > > > > previous
> > > > > > > > > > > > > > > heartbeat.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > When the active controller fails, the new
> > > active
> > > > > > > > > controller
> > > > > > > > > > > > > needs to
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > sure that it considers all the known brokers
> > > as
> > > > > alive
> > > > > > > > > till
> > > > > > > > > > > the
> > > > > > > > > > > > > lease
> > > > > > > > > > > > > > > > > > expiration interval. Because
> > > > > > > > > registration.lease.timeout.ms,
> > > > > > > > > > > is
> > > > > > > > > > > > > > > > > configured
> > > > > > > > > > > > > > > > > > on the controller, the new active controller
> > > will
> > > > > > > extend
> > > > > > > > > all
> > > > > > > > > > > the
> > > > > > > > > > > > > > > leases
> > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > registration.lease.timeout.ms. I see that it
> > > > > won't
> > > > > > > need
> > > > > > > > > last
> > > > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Agreed.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Sat, Sep 5, 2020 at 1:28 AM Colin McCabe <
> > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Colin wrote:
> > > > > > > > > > > > > > > > > > > > > The reason for including
> > > LeaseStartTimeMs
> > > > > in
> > > > > > > the
> > > > > > > > > > > request
> > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > > > > that the time required to communicate
> > > with
> > > > > the
> > > > > > > > > > > controller
> > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > included in
> > > > > > > > > > > > > > > > > > > > > the lease time. Since requests can
> > > > > > > potentially be
> > > > > > > > > > > delayed
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > > > > for a long time, this is important.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020, at 05:58, Unmesh
> > > Joshi
> > > > > wrote:
> > > > > > > > > > > > > > > > > > > > The network time will be added anyway,
> > > > > because
> > > > > > > the
> > > > > > > > > lease
> > > > > > > > > > > > > timer
> > > > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > > > active controller will start only after
> > > the
> > > > > > > heartbeat
> > > > > > > > > > > request
> > > > > > > > > > > > > > > > > reaches the
> > > > > > > > > > > > > > > > > > > > server.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > If the start time is not specified in the
> > > > > request,
> > > > > > > > > then the
> > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > > > is excluded from the heartbeat time.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Here's an example:
> > > > > > > > > > > > > > > > > > > Let's say broker A sends a heartbeat at
> > > time
> > > > > 100,
> > > > > > > and
> > > > > > > > > it
> > > > > > > > > > > > > arrives
> > > > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > > controller at time 200, and the lease
> > > duration
> > > > > is
> > > > > > > 1000.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The controller looks at the start time in
> > > the
> > > > > > > request,
> > > > > > > > > > > which is
> > > > > > > > > > > > > > > 100,
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > adds 1000 to it, getting 1100. On the
> > > other
> > > > > hand,
> > > > > > > if
> > > > > > > > > start
> > > > > > > > > > > > > time
> > > > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > > > specified in the request, then the
> > > expiration
> > > > > will
> > > > > > > be
> > > > > > > > > at
> > > > > > > > > > > time
> > > > > > > > > > > > > 1200.
> > > > > > > > > > > > > > > > > > > That is what I mean by "the network time is
> > > > > > > included
> > > > > > > > > in the
> > > > > > > > > > > > > > > expiration
> > > > > > > > > > > > > > > > > > > time."
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > And I think, some assumption about
> > > network
> > > > > round
> > > > > > > trip
> > > > > > > > > > > time is
> > > > > > > > > > > > > > > > > > > > needed anyway to decide on the frequency
> > > of
> > > > > the
> > > > > > > > > > > heartbeat (
> > > > > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms), and
> > > > > lease
> > > > > > > > > timeout (
> > > > > > > > > > > > > > > > > > > > registration.lease.timeout.ms). So I
> > > think
> > > > > just
> > > > > > > > > having a
> > > > > > > > > > > > > > > leaseTTL
> > > > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > > > request is easier to understand and
> > > > > implement.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It's important to keep in mind that
> > > messages
> > > > > may be
> > > > > > > > > > > delayed in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > > > happens, we
> > > > > > > > > > > will
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > > time specified in the request to determine
> > > if
> > > > > the
> > > > > > > > > request
> > > > > > > > > > > is
> > > > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> > > the
> > > > > > > > > controller
> > > > > > > > > > > side
> > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > reset in the case of controller
> > > failover.
> > > > > The
> > > > > > > > > > > alternative
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > be to
> > > > > > > > > > > > > > > > > > > > > track the lease as hard state rather
> > > than
> > > > > soft
> > > > > > > > > state,
> > > > > > > > > > > but I
> > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > is not really needed, and would result
> > > in
> > > > > more
> > > > > > > log
> > > > > > > > > > > entries.
> > > > > > > > > > > > > > > > > > > > My interpretation of the mention of
> > > > > BrokerRecord
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > KIP
> > > > > > > > > > > > > was
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > record exists in the Raft log.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > BrokerRecord does exist in the Raft log,
> > > but
> > > > > does
> > > > > > > not
> > > > > > > > > > > include
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > > heartbeat time.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > By soft state, do you mean the broker
> > > > > > > > > > > > > > > > > > > > records exist only on the active leader
> > > and
> > > > > will
> > > > > > > not
> > > > > > > > > be
> > > > > > > > > > > > > > > replicated
> > > > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > > > raft log? If the live brokers list is
> > > > > maintained
> > > > > > > > > only on
> > > > > > > > > > > the
> > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > controller (raft leader), then, in case
> > > of
> > > > > leader
> > > > > > > > > > > failure,
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > will be
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > window where the new leader does not know
> > > > > about
> > > > > > > the
> > > > > > > > > live
> > > > > > > > > > > > > brokers,
> > > > > > > > > > > > > > > > > till
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > brokers establish the leases again.
> > > > > > > > > > > > > > > > > > > > I think it will be safer to have leases
> > > as a
> > > > > hard
> > > > > > > > > state
> > > > > > > > > > > > > managed
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > standard
> > > > > > > > > > > > > > > > > > > > Raft replication.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Leases are short, so the need to
> > > re-establish
> > > > > them
> > > > > > > > > after a
> > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > failover doesn't seem like a big problem.
> > > But
> > > > > > > this is
> > > > > > > > > > > > > something
> > > > > > > > > > > > > > > we can
> > > > > > > > > > > > > > > > > > > tweak if it becomes an issue. One option
> > > > > would be
> > > > > > > to
> > > > > > > > > have
> > > > > > > > > > > a
> > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > > which is only used by the controller nodes
> > > for
> > > > > this
> > > > > > > > > (since,
> > > > > > > > > > > > > after
> > > > > > > > > > > > > > > all,
> > > > > > > > > > > > > > > > > > > brokers don't care about registration
> > > > > renewals).
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Or am I misunderstanding something? (I
> > > assume
> > > > > > > that
> > > > > > > > > with
> > > > > > > > > > > soft
> > > > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > mean something like zookeeper local
> > > sessions
> > > > > > > > > > > > > > > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1147
> > > > > > > > > .)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > > > think
> > > > > > > it
> > > > > > > > > makes
> > > > > > > > > > > > > sense
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> > > becomes
> > > > > > > very
> > > > > > > > > messy.
> > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > > > understand your question about
> > > duplicate
> > > > > > > broker ID
> > > > > > > > > > > > > detection,
> > > > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > > There's a section in the KIP about this
> > > -- is
> > > > > > > there a
> > > > > > > > > > > detail
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This is an implementation detail that
> > > doesn't
> > > > > need
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > in the
> > > > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I assumed broker leases are implemented
> > > as a
> > > > > hard
> > > > > > > > > state.
> > > > > > > > > > > In
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > case, to
> > > > > > > > > > > > > > > > > > > > check for broker id conflict, we need to
> > > > > check
> > > > > > > the
> > > > > > > > > broker
> > > > > > > > > > > > > ids at
> > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > places
> > > > > > > > > > > > > > > > > > > > 1. Pending broker registrations (which
> > > are
> > > > > yet
> > > > > > > to be
> > > > > > > > > > > > > committed)
> > > > > > > > > > > > > > > 2.
> > > > > > > > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > > > > > > committed broker registrations.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020 at 5:42 PM Colin
> > > McCabe
> > > > > <
> > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020, at 01:12, Unmesh
> > > > > Joshi
> > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >>>Can you repeat your questions
> > > about
> > > > > broker
> > > > > > > > > leases?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >>>>The LeaseStartTimeMs is expected
> > > to
> > > > > be
> > > > > > > the
> > > > > > > > > > > broker's
> > > > > > > > > > > > > > > > > > > > > > 'System.currentTimeMillis()' at the
> > > > > point of
> > > > > > > the
> > > > > > > > > > > > > request. The
> > > > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > > > controller will add its lease period
> > > to
> > > > > this
> > > > > > > in
> > > > > > > > > order
> > > > > > > > > > > > > >>>>to
> > > > > > > > > > > > > > > > > compute
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > LeaseEndTimeMs.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think the use of LeaseStartTimeMs
> > > and
> > > > > > > > > > > LeaseEndTimeMs
> > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > > confusing. Monotonic Clock
> > > > > > > (System.nanoTime) on
> > > > > > > > > the
> > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > should be used to track leases.
> > > > > > > > > > > > > > > > > > > > > > (For example,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-1616https://github.com/etcd-io/etcd/pull/6888/commits/e7f4010ccaf28b6ce64fe514d25a4b2fa459d114
> > > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Then we will not need
> > > LeaseStartTimeMs?
> > > > > > > > > > > > > > > > > > > > > > Instead of LeaseStartTimeMs, can we
> > > call
> > > > > it
> > > > > > > > > > > LeaseTTL? The
> > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > can then calculate LeaseEndTime =
> > > > > > > > > System.nanoTime() +
> > > > > > > > > > > > > > > LeaseTTL.
> > > > > > > > > > > > > > > > > > > > > > In this case we might just drop
> > > > > > > LeaseEndTimeMs
> > > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > response, as
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > broker already knows about the TTL
> > > and
> > > > > can
> > > > > > > send
> > > > > > > > > > > > > heartbeats at
> > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > fraction
> > > > > > > > > > > > > > > > > > > > > > of TTL, say every TTL/4
> > > > > milliseconds.(elapsed
> > > > > > > > > time
> > > > > > > > > > > on the
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > measured
> > > > > > > > > > > > > > > > > > > > > > by System.nanoTime)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I agree that the monotonic clock is
> > > > > probably a
> > > > > > > > > better
> > > > > > > > > > > idea
> > > > > > > > > > > > > > > here.
> > > > > > > > > > > > > > > > > It is
> > > > > > > > > > > > > > > > > > > > > good to be robust against wall clock
> > > > > changes,
> > > > > > > > > although
> > > > > > > > > > > I
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > > > > > which had them might suffer other
> > > issues.
> > > > > I
> > > > > > > will
> > > > > > > > > > > change
> > > > > > > > > > > > > it to
> > > > > > > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > monotonic clock.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The reason for including
> > > LeaseStartTimeMs
> > > > > in
> > > > > > > the
> > > > > > > > > > > request
> > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > the time required to communicate with
> > > the
> > > > > > > > > controller
> > > > > > > > > > > gets
> > > > > > > > > > > > > > > included
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > lease time. Since requests can
> > > > > potentially be
> > > > > > > > > delayed
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > > > > > > long time, this is important.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I have a prototype built to
> > > demonstrate
> > > > > this
> > > > > > > as
> > > > > > > > > > > > > following:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The Kip631Controller itself depends
> > > on a
> > > > > > > > > Consensus
> > > > > > > > > > > > > module, to
> > > > > > > > > > > > > > > > > > > demonstrate
> > > > > > > > > > > > > > > > > > > > > > how possible interactions with the
> > > > > consensus
> > > > > > > > > module
> > > > > > > > > > > will
> > > > > > > > > > > > > look
> > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > > (The Consensus can be pluggable
> > > really,
> > > > > > > with an
> > > > > > > > > API
> > > > > > > > > > > to
> > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > > > replicated log upto HighWaterMark)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > It has an implementation of
> > > LeaseTracker
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/LeaderLeaseTracker.scala
> > > > > > > > > > > > > > > > > > > > > > to demonstrate LeaseTracker's
> > > interaction
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > consensus
> > > > > > > > > > > > > > > > > module.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The implementation has the following
> > > > > aspects:
> > > > > > > > > > > > > > > > > > > > > > 1. The lease tracking happens only
> > > on the
> > > > > > > active
> > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > (raft
> > > > > > > > > > > > > > > > > > > > > > leader)
> > > > > > > > > > > > > > > > > > > > > > 2. Once the lease expires, it needs
> > > to
> > > > > > > propose
> > > > > > > > > and
> > > > > > > > > > > > > commit a
> > > > > > > > > > > > > > > > > > > FenceBroker
> > > > > > > > > > > > > > > > > > > > > > record for that lease.
> > > > > > > > > > > > > > > > > > > > > > 3. In case of active controller
> > > failure,
> > > > > the
> > > > > > > > > lease
> > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > tracked by
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > newly raft leader. The new raft
> > > leader
> > > > > > > starts the
> > > > > > > > > > > lease
> > > > > > > > > > > > > timer
> > > > > > > > > > > > > > > > > again,
> > > > > > > > > > > > > > > > > > > (as
> > > > > > > > > > > > > > > > > > > > > > implemented in onBecomingLeader
> > > method of
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > > > in effect extending the lease by the
> > > time
> > > > > > > spent
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > election
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > whatever time was elapsed on the old
> > > > > leader.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> > > the
> > > > > > > > > controller
> > > > > > > > > > > side
> > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > reset
> > > > > > > > > > > > > > > > > > > > > in the case of controller failover.
> > > The
> > > > > > > > > alternative
> > > > > > > > > > > would
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > track
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > lease as hard state rather than soft
> > > state,
> > > > > > > but I
> > > > > > > > > think
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > > > > > > needed, and would result in more log
> > > > > entries.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > There are working tests for this
> > > > > > > implementation
> > > > > > > > > here.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > > > > and an end to end test here
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/ProducerConsumerKIP500Test.scala
> > > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >>'m not sure what you mean by
> > > > > > > "de-duplication
> > > > > > > > > of the
> > > > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > > > Apologies for using the confusing
> > > term
> > > > > > > > > > > deduplication. I
> > > > > > > > > > > > > meant
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > id
> > > > > > > > > > > > > > > > > > > > > > conflict.
> > > > > > > > > > > > > > > > > > > > > > As you can see in the prototype
> > > > > > > handleRequest of
> > > > > > > > > > > > > > > KIP631Controller
> > > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > > > the duplicate broker id needs to be
> > > > > detected
> > > > > > > > > before
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > BrokerRecord
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > submitted to the raft module.
> > > > > > > > > > > > > > > > > > > > > > Also as implemented in the
> > > prototype, the
> > > > > > > > > > > > > KIP631Controller is
> > > > > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > > > > > > threaded, handling requests one at a
> > > > > time.
> > > > > > > (an
> > > > > > > > > > > example of
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://martinfowler.com/articles/patterns-of-distributed-systems/singular-update-queue.html
> > > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > > > think
> > > > > > > it
> > > > > > > > > makes
> > > > > > > > > > > > > sense
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> > > becomes
> > > > > > > very
> > > > > > > > > messy.
> > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > > > understand your question about
> > > duplicate
> > > > > > > broker ID
> > > > > > > > > > > > > detection,
> > > > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > > > There's a section in the KIP about
> > > this --
> > > > > is
> > > > > > > > > there a
> > > > > > > > > > > > > detail we
> > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 10:49 AM
> > > Colin
> > > > > > > McCabe <
> > > > > > > > > > > > > > > colin@cmccabe.xyz
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2020, at 19:36,
> > > Unmesh
> > > > > > > Joshi
> > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > There were a few of questions I
> > > had..
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for the response.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1. Were my comments on the broker
> > > > > lease
> > > > > > > > > > > > > implementation
> > > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > > prototype) appropriate and do we
> > > > > need to
> > > > > > > > > change
> > > > > > > > > > > the
> > > > > > > > > > > > > KIP
> > > > > > > > > > > > > > > > > > > > > > > > description accordingly?.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Can you repeat your questions about
> > > > > broker
> > > > > > > > > leases?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2. How will broker epochs be
> > > > > generated?
> > > > > > > I am
> > > > > > > > > > > > > assuming it
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > committed log offset (like zxid?)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > There isn't any need to use a log
> > > > > offset.
> > > > > > > We
> > > > > > > > > can
> > > > > > > > > > > just
> > > > > > > > > > > > > > > look at
> > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > in-memory hash table and see what
> > > the
> > > > > > > latest
> > > > > > > > > > > number is,
> > > > > > > > > > > > > > > and add
> > > > > > > > > > > > > > > > > > > one, to
> > > > > > > > > > > > > > > > > > > > > > > generate a new broker epoch.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3. How will producer registration
> > > > > > > happen? I
> > > > > > > > > am
> > > > > > > > > > > > > assuming
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > should be
> > > > > > > > > > > > > > > > > > > > > > > > similar to broker registration,
> > > with
> > > > > a
> > > > > > > > > similar
> > > > > > > > > > > way to
> > > > > > > > > > > > > > > > > generate
> > > > > > > > > > > > > > > > > > > > > producer
> > > > > > > > > > > > > > > > > > > > > > > id.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > For the EOS stuff, we will need a
> > > few
> > > > > new
> > > > > > > RPCs
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > controller.
> > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > > we should do that in a follow-on
> > > KIP,
> > > > > > > though,
> > > > > > > > > since
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > one is
> > > > > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > > > > > pretty big.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 4. Because we expose Raft log to
> > > all
> > > > > the
> > > > > > > > > > > brokers, any
> > > > > > > > > > > > > > > > > > > de-duplication
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > broker needs to happen before the
> > > > > > > requests
> > > > > > > > > are
> > > > > > > > > > > > > proposed
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > Raft.
> > > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > the controller needs to be single
> > > > > > > threaded,
> > > > > > > > > and
> > > > > > > > > > > > > should do
> > > > > > > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > > > > > against the in-process or pending
> > > > > > > requests
> > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > > > final
> > > > > > > > > > > > > > > > > state. I
> > > > > > > > > > > > > > > > > > > > > read a
> > > > > > > > > > > > > > > > > > > > > > > > mention of this, in the
> > > responses in
> > > > > this
> > > > > > > > > > > > > thread.Will it
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > useful to
> > > > > > > > > > > > > > > > > > > > > > > > mention this in the KIP?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I'm not sure what you mean by
> > > > > > > "de-duplication
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 4:50 AM
> > > Colin
> > > > > > > McCabe
> > > > > > > > > <
> > > > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I'm thinking of calling a vote
> > > on
> > > > > > > KIP-631
> > > > > > > > > on
> > > > > > > > > > > > > Monday.
> > > > > > > > > > > > > > > Let
> > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > know
> > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > > there's any more comments I
> > > should
> > > > > > > address
> > > > > > > > > > > before I
> > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > vote.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020, at 05:39,
> > > > > Unmesh
> > > > > > > > > Joshi
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > >>Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > >>Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks. I will be adding
> > > more to
> > > > > the
> > > > > > > > > > > prototype
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > > > > > and collaborate.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020 at
> > > 12:28 AM
> > > > > > > Colin
> > > > > > > > > > > McCabe <
> > > > > > > > > > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Jose,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > That'a s good point that I
> > > > > hadn't
> > > > > > > > > > > considered.
> > > > > > > > > > > > > It's
> > > > > > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > > > a separate leader change
> > > > > message,
> > > > > > > as
> > > > > > > > > you
> > > > > > > > > > > > > mentioned.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2020, at
> > > 11:56,
> > > > > Jose
> > > > > > > > > Garcia
> > > > > > > > > > > > > Sancio
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Very cool prototype!
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > The KIP proposes a record
> > > > > called
> > > > > > > > > > > IsrChange
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > includes the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > partition, topic, isr,
> > > > > leader and
> > > > > > > > > leader
> > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > > During
> > > > > > > > > > > > > > > > > > > normal
> > > > > > > > > > > > > > > > > > > > > > > > > > > > operation ISR changes do
> > > not
> > > > > > > result
> > > > > > > > > in
> > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > > > > Similarly,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > leader changes do not
> > > > > necessarily
> > > > > > > > > > > involve ISR
> > > > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > > > implementation that uses
> > > ZK
> > > > > > > modeled
> > > > > > > > > them
> > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. All of this
> > > information is
> > > > > > > stored
> > > > > > > > > in
> > > > > > > > > > > one
> > > > > > > > > > > > > > > znode.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. ZK's optimistic lock
> > > > > requires
> > > > > > > > > that you
> > > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > > the new
> > > > > > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. The change to that
> > > znode
> > > > > was
> > > > > > > being
> > > > > > > > > > > > > performed
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > > > and the leader.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > None of these reasons are
> > > > > true in
> > > > > > > > > > > KIP-500.
> > > > > > > > > > > > > Have
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > considered
> > > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > > > > two different records?
> > > For
> > > > > > > example
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. IsrChange record which
> > > > > > > includes
> > > > > > > > > topic,
> > > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > > isr
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. LeaderChange record
> > > which
> > > > > > > includes
> > > > > > > > > > > topic,
> > > > > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > leader epoch.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > I suspect that making
> > > this
> > > > > change
> > > > > > > > > will
> > > > > > > > > > > also
> > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > message AlterIsrRequest
> > > > > > > introduced in
> > > > > > > > > > > > > KIP-497:
> > > > > > > > > > > > > > > Add
> > > > > > > > > > > > > > > > > > > > > inter-broker
> > > > > > > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > alter ISR.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > > > > > > -Jose
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Colin. Thanks for the hard work on this KIP.
I have some questions about what happens to a broker when it becomes
fenced (e.g. because it can't send a heartbeat request to keep its
lease). The KIP says "When a broker is fenced, it cannot process any
client requests. This prevents brokers which are not receiving
metadata updates or that are not receiving and processing them fast
enough from causing issues to clients." And in the description of the
FENCED(4) state it likewise says "While in this state, the broker does
not respond to client requests." It makes sense that a fenced broker
should not accept producer requests -- I assume any such requests
would result in NotLeaderOrFollowerException. But what about KIP-392
(fetch from follower) consumer requests? It is conceivable that these
could continue. Related to that, would a fenced broker continue to
fetch data for partitions where it thinks it is a follower? Even if
it rejects consumer requests it might still continue to fetch as a
follower. Might it be helpful to clarify both decisions here?
Also, for the ScramUserRecord, since we settled on the term
"UserScramCredential" in KIP-554, might it be better named as
"UserScramCredsentialRecord"? (Also, the name currently has a
copy-paste error and is "DelegationTokenRecord").
Ron
On Tue, Oct 13, 2020 at 9:31 PM Jun Rao <ju...@confluent.io> wrote:
>
> Hi, Colin,
>
> Thanks for the reply. A few more comments below.
>
> 80.1 controller.listener.names only defines the name of the listener. The
> actual listener including host/port/security_protocol is typically defined
> in advertised_listners. Does that mean advertised_listners is a required
> config now?
>
> 83.1 broker state machine: It seems that we should transition from FENCED
> => INITIAL since only INITIAL generates new broker epoch?
>
> 83.5. It's true that the controller node doesn't serve metadata requests.
> However, there are admin requests such as topic creation/deletion are sent
> to the controller directly. So, it seems that the client needs to know
> the controller host/port?
>
> 85. "I was hoping that we could avoid responding to requests when the
> broker was fenced." This issue is that if we don't send a response, the
> client won't know the reason and can't act properly.
>
> 88. CurMetadataOffset: I was thinking that we may want to
> use CurMetadataOffset to compute the MetadataLag. Since HWM is exclusive,
> it's more convenient if CurMetadataOffset is also exclusive.
>
> 90. It would be useful to add a rejected section on why separate controller
> and broker id is preferred over just broker id. For example, the following
> are some potential reasons. (a) We can guard duplicated brokerID, but it's
> hard to guard against duplicated controllerId. (b) brokerID can be auto
> assigned in the future, but controllerId is hard to be generated
> automatically.
>
> Thanks,
>
> Jun
>
> On Mon, Oct 12, 2020 at 11:14 AM Colin McCabe <cm...@apache.org> wrote:
>
> > On Tue, Oct 6, 2020, at 16:09, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply. Made another pass of the KIP. A few more comments
> > > below.
> > >
> >
> > Hi Jun,
> >
> > Thanks for the review.
> >
> > > 55. We discussed earlier why the current behavior where we favor the
> > > current broker registration is better. Have you given this more thought?
> > >
> >
> > Yes, I think we should favor the current broker registration, as you
> > suggested earlier.
> >
> > > 80. Config related.
> > > 80.1 Currently, each broker only has the following 3 required configs. It
> > > will be useful to document the required configs post KIP-500 (in both the
> > > dedicated and shared controller mode).
> > > broker.id
> > > log.dirs
> > > zookeeper.connect
> >
> > For the broker, these configs will be required:
> >
> > broker.id
> > log.dirs
> > process.roles
> > controller.listener.names
> > controller.connect
> >
> > For the controller, these configs will be required:
> >
> > controller.id
> > log.dirs
> > process.roles
> > controller.listener.names
> > controller.connect
> >
> > For broker+controller, it will be the union of these two, which
> > essentially means we need both broker.id and controller.id, but all
> > others are the same as standalone.
> >
> > > 80.2 It would be useful to document all deprecated configs post KIP-500.
> > > For example, all zookeeper.* are obviously deprecated. But there could be
> > > others. For example, since we don't plan to support auto broker id
> > > generation, it seems broker.id.generation.enable is deprecated too.
> > > 80.3 Could we make it clear that controller.connect replaces
> > quorum.voters
> > > in KIP-595?
> >
> > OK. I added a comment about this in the table.
> >
> > > 80.4 Could we document that broker.id is now optional?
> >
> > OK. I have added a line for broker.id.
> >
> > > 80.5 The KIP suggests that controller.id is optional on the controller
> > > node. I am concerned that this can cause a bit of confusion in 2 aspects.
> > > First, in the dedicated controller mode, controller.id is not optional
> > > (since broker.id is now optional). Second, in the shared controller
> > mode,
> > > it may not be easy for the user to figure out the default value of
> > > controller.id to set controller.connect properly.
> >
> > I got rid of the default value of controller.id. I agree it was a bit
> > confusing to explain.
> >
> > > 80.6 Regarding the consistency of config names, our metrics already
> > include
> > > controller. So, prefixing all controller related configs with
> > "controller"
> > > may be more consistent. If we choose to do that, could we rename all new
> > > configs here and in KIP-595 consistently?
> > >
> >
> > I added the new names for the KIP-595 configurations to the configuration
> > table. I think prefixing these configurations with "controller.quorum" is
> > the best option since it makes it clear that they are related to the
> > controller quorum.
> >
> > > 81. Metrics
> > > 81.1 kafka.controller:type=KafkaController,name=MetadataSnapshotLag: This
> > > is now redundant since KIP-630 already has
> > > kafka.controller:type=KafkaController,name=SnapshotLag.
> >
> > Thanks for bringing up the metric name issue. I think it would be good to
> > clarify that this is the lag for the metadata snapshot specifically. So I
> > prefer the name MetadataSnapshotLag. I'll add a note to the table.
> >
> > > 81.2 Do we need both kafka.controller:type=KafkaServer,name=MetadataLag
> > and
> > > kafka.controller:type=KafkaController,name=MetadataLag since in the
> > shared
> > > controller mode, the metadata log is shared?
> >
> > Yes, both are needed. This metric is about what metadata logs we have
> > applied, not about the Raft log's high water mark. If the broker or
> > controller takes a while to apply the log messages (for example, because of
> > lock contention or some other issue), they could fall significantly behind
> > the high watermark.
> >
> > >
> > > 82. Metadata records
> > > 82.1 BrokerRecord: It needs to include supported features based on
> > KIP-584.
> >
> > Added.
> >
> > > 82.2 FeatureLevel: The finalized feature in KIP-584 has the following
> > > structure and needs to be reflected here.
> > > {
> > > "version": 0, // int32 -> Represents the version of the schema for the
> > > data stored in the ZK node
> > > "status": 1, // int32 -> Represents the status of the node
> > > "features": {
> > > "group_coordinator": { // string -> name of the feature
> > > "min_version_level": 0, // int16 -> Represents the
> > cluster-wide
> > > finalized minimum version level (>=1) of this feature
> > > "max_version_level": 3 // int16 -> Represents the
> > cluster-wide
> > > finalized maximum version level (>=1 and >= min_version_level) of this
> > > feature
> > > },
> > > "consumer_offsets_topic_schema": {
> > > "min_version_level": 0,
> > > "max_version_level": 4
> > > }
> > > }
> >
> > Fair point. I will add both a minimum and maximum finalized feature level.
> >
> > >
> > > 83. RPC
> > > 83.1 BrokerHeartbeatRequest: Could you document when the controller
> > assigns
> > > a new broker epoch?
> >
> > The controller assigns a broker epoch if the heartbeat request was sent
> > without an epoch (that is, it is in the INITIAL state), and the controller
> > wants to add the new broker to the cluster. I added a note to the state
> > machine.
> >
> > > 83.2 BrokerHeartbeatResponse: Do we need a new error code to indicate a
> > > broker is fenced?
> >
> > I don't think we need a new error code, since targetState will just
> > indicate that the broker should be fenced.
> >
> > > 83.3 What error/state indicates an unsuccessful controlled shutdown in
> > > BrokerHeartbeatResponse? Currently, the controlled shutdown responses
> > also
> > > include a list of remaining partitions. That's mostly for debugging
> > > purposes. Should we include it in BrokerHeartbeatResponse too?
> >
> > Once the controller wants the broker to shut down, it will set targetState
> > to SHUTDOWN.
> >
> > We don't need to send a list of remaining partitions, since the broker is
> > following the metadata log, and will get all its instructions from there.
> >
> > > 83.4 Could we document when Listeners, Features and Rack are expected to
> > be
> > > set in BrokerHeartbeatRequest?
> >
> > I added a line specifying that: The listeners, features, and rack fields
> > only need to be set when the broker is in INITIAL state, and is requesting
> > a new broker epoch.
> >
> > > 83.5 Currently, the metadata response only returns a list of brokers.
> > > Should we have a separate field for the controller info or change the
> > field
> > > name to sth like nodes? Also, is the rack field relevant to the
> > controller?
> >
> > As per KIP-590, we don't want the clients to communicate directly with the
> > controller. So we weren't planning on including the controller's
> > information in the MetadataResponse.
> >
> > We don't currently have rack-awareness for controllers, but I suppose we
> > could in the future. I think we can leave that for later.
> >
> > >
> > > 84. There are a few references to __kafka_metadata. However, KIP-595
> > > uses __cluster_metadata.
> >
> > We agreed earlier to decide on the name of the metadata topic in this KIP,
> > and to treat the KIP-595 name as provisional. Looking at it again, I think
> > we should go with @metadata, in order to avoid colliding with topic names
> > that already exist.
> >
> > >
> > > 85. "When a broker is fenced, it cannot process any client requests. "
> > > Should we send a new error code in the response to indicate that the
> > broker
> > > is fenced?
> >
> > I was hoping that we could avoid responding to requests when the broker
> > was fenced. I suppose a new error code could work too, but then we also
> > need a compatibility story for older clients.
> >
> > >
> > > 86. "The controller can generate a new broker epoch by using the latest
> > log
> > > offset." Perhaps it's more accurate to say the new broker epoch is the
> > > offset for the corresponding BrokerRecord in the metadata log.
> >
> > The idea here is slightly different. We want to use the current latest
> > offset (prior to creating the BrokerRecord) as the new epoch. We can't use
> > the offset of the BrokerRecord itself since we won't know what that is
> > until we give the BrokerRecord to the Raft layer.
> >
> > >
> > > 87. meta.properties:
> > > 87.1 Could you document the format change to this file? Also, should we
> > > bump up the version of the file?
> >
> > Hmm. The file is just a list of key=value pairs, one per line. So I'm
> > not sure there is any format change per se, at least technically.
> >
> > Maybe we should bump up the version anyway, in order to prevent older
> > pre-KIP-500 software from launching if the new stuff is present?
> >
> > > 87.2 How do we know a node is configured in KIP-500 mode?
> > >
> >
> > A cluster configured in KIP-500 mode will contain the following line in
> > the meta.properties file in each directory: kip.500.mode=enabled
> >
> > > 88. CurMetadataOffset: Should we use nextMetadataOffset since it's more
> > > consistent with fetchOffset and HWM?
> >
> > I'm not sure. Drawing an analogy with HWM might be confusing, since
> > CurMetadataOffset is not HWM. It's totally possible for the broker to have
> > fetched up to offset 1000 but only applied up to offset 500.
> >
> > best,
> > Colin
> >
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Oct 5, 2020 at 9:03 AM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Mon, Sep 28, 2020, at 11:41, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the reply. A few more comments below.
> > > > >
> > > > > 62.
> > > > > 62.1 controller.listener.names: So, is this used for the controller
> > or
> > > > the
> > > > > broker trying to connect to the controller?
> > > > >
> > > >
> > > > Hi Jun,
> > > >
> > > > It's used by both. The broker tries to connect to controllers by using
> > > > the first listener in this list. The controller uses the list to
> > determine
> > > > which listeners it should bind to.
> > > >
> > > > >
> > > > > 62.2 If we want to take the approach to share the configs that are
> > common
> > > > > between the broker and the controller, should we share the id too?
> > > > >
> > > >
> > > > On nodes where we are running both the controller and broker roles, we
> > > > need two IDs: one for the controller, and one for the broker. So we
> > can't
> > > > share the same configuration.
> > > >
> > > > >
> > > > > 62.3 We added some configs in KIP-595 prefixed with "quorum" and we
> > plan
> > > > to
> > > > > add some controller specific configs prefixed with "controller".
> > KIP-630
> > > > > plans to add some other controller specific configs with no prefix.
> > > > Should
> > > > > we standardize all controller specific configs with the same prefix?
> > > > >
> > > >
> > > > That's a good question. As Jose said elsewhere in the thread, let's
> > use
> > > > "metadata" as the prefix.
> > > >
> > > > >
> > > > > 70. Could you explain the impact of process.roles a bit more? For
> > > > example,
> > > > > if process.roles=controller, does the node still maintain metadata
> > cache
> > > > as
> > > > > described in KIP-630? Do we still return the host/port for those
> > nodes in
> > > > > the metadata response?
> > > > >
> > > >
> > > > No, the node does not need any broker data structures if it is not a
> > > > broker. Controllers also don't handle MetadataRequests. Clients will
> > > > still go to the brokers for those.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Sep 28, 2020 at 9:24 AM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > On Fri, Sep 25, 2020, at 17:35, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the reply.
> > > > > > >
> > > > > > > 62. Thinking about this more, I am wondering what's our overall
> > > > strategy
> > > > > > > for configs shared between the broker and the controller. For
> > > > example,
> > > > > > both
> > > > > > > the broker and the controller have to define listeners. So both
> > need
> > > > > > > configs like listeners/advertised.listeners. Both the new
> > controller
> > > > and
> > > > > > > the broker replicate data. So both need to define some
> > replication
> > > > > > related
> > > > > > > configurations (replica.fetch.min.bytes,
> > replica.fetch.wait.max.ms,
> > > > > > etc).
> > > > > > > Should we let the new controller share those configs with
> > brokers or
> > > > > > should
> > > > > > > we define new configs just for the controller? It seems that we
> > want
> > > > to
> > > > > > > apply the same strategy consistently for all shared configs?
> > > > > > >
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > That's a good question. I think that we should share as many
> > > > > > configurations as possible. There will be a few cases where this
> > isn't
> > > > > > possible, and we need to create a new configuration key that is
> > > > > > controller-only, but I think that will be relatively rare.
> > > > > >
> > > > > > >
> > > > > > > 63. If a node has process.roles set to controller, does one still
> > > > need to
> > > > > > > set broker.id on this node?
> > > > > > >
> > > > > >
> > > > > > No, broker.id does not need to be set in that case.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Sep 25, 2020 at 2:17 PM Colin McCabe <cmccabe@apache.org
> > >
> > > > wrote:
> > > > > > >
> > > > > > > > On Fri, Sep 25, 2020, at 10:17, Jun Rao wrote:
> > > > > > > > > Hi, Colin,
> > > > > > > > >
> > > > > > > > > Thanks for the reply.
> > > > > > > > >
> > > > > > > > > 60. Yes, I think you are right. We probably need the
> > controller
> > > > id
> > > > > > when a
> > > > > > > > > broker starts up. A broker only stores the Raft leader id in
> > the
> > > > > > metadata
> > > > > > > > > file. To do the initial fetch to the Raft leader, it needs to
> > > > know
> > > > > > the
> > > > > > > > > host/port associated with the leader id.
> > > > > > > > >
> > > > > > > > > 62. It seems there are 2 parts to this : (1) which listener a
> > > > client
> > > > > > > > should
> > > > > > > > > use to initiate a connection to the controller and (2) which
> > > > listener
> > > > > > > > > should a controller use to accept client requests. For (1),
> > at
> > > > any
> > > > > > point
> > > > > > > > of
> > > > > > > > > time, a client only needs to use one listener. I think
> > > > > > > > > controller.listener.name is meant for the client.
> > > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > controller.listener.names is also used by the controllers. In
> > the
> > > > case
> > > > > > > > where we have a broker and a controller in the same JVM, we
> > have a
> > > > > > single
> > > > > > > > config file. Then we need to know which listeners belong to
> > the
> > > > > > controller
> > > > > > > > and which belong to the broker component. That's why it's a
> > list.
> > > > > > > >
> > > > > > > > > So, a single value seems
> > > > > > > > > to make more sense. Currently, we don't have a configuration
> > for
> > > > > > (2). We
> > > > > > > > > could add a new one for that and support a list. I am
> > wondering
> > > > how
> > > > > > > > useful
> > > > > > > > > it will be. One example that I can think of is that we can
> > reject
> > > > > > > > > non-controller related requests if accepted on
> > controller-only
> > > > > > listeners.
> > > > > > > > > However, we already support separate authentication for the
> > > > > > controller
> > > > > > > > > listener. So, not sure how useful it is.
> > > > > > > >
> > > > > > > > The controller always has a separate listener and does not
> > share
> > > > > > listeners
> > > > > > > > with the broker. The main reason for this is to avoid giving
> > > > clients
> > > > > > the
> > > > > > > > ability to launch a denial-of-service attack on the controller
> > by
> > > > > > flooding
> > > > > > > > a broker port. A lot of times, these attacks are made
> > > > unintentionally
> > > > > > by
> > > > > > > > poorly coded or configured clients. Additionally, broker
> > ports can
> > > > > > also be
> > > > > > > > very busy with large fetch requests, and so on. It's just a
> > bad
> > > > > > > > configuration in general to try to overload the same port for
> > the
> > > > > > > > controller and broker, and we don't want to allow it. It
> > would be
> > > > a
> > > > > > > > regression to go from the current system where control
> > requests are
> > > > > > safely
> > > > > > > > isolated on a separate port, to one where they're not. It also
> > > > makes
> > > > > > the
> > > > > > > > code and configuration a lot messier.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 63. (a) I think most users won't know controller.id
> > defaults to
> > > > > > > > broker.id +
> > > > > > > > > 3000. So, it can be confusing for them to set up
> > > > controller.connect.
> > > > > > If
> > > > > > > > > this is truly needed, it seems that it's less confusing to
> > make
> > > > > > > > > controller.id required.
> > > > > > > > > (b) I am still trying to understand if we truly need to
> > expose a
> > > > > > > > > controller.id. What if we only expose broker.id and let
> > > > > > > > controller.connect
> > > > > > > > > just use broker.id? What will be missing?
> > > > > > > >
> > > > > > > > The controller has a separate ID from the broker, so knowing
> > > > broker.id
> > > > > > is
> > > > > > > > not helpful here.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Thu, Sep 24, 2020 at 10:55 PM Colin McCabe <
> > > > cmccabe@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Thu, Sep 24, 2020, at 16:24, Jun Rao wrote:
> > > > > > > > > > > Hi, Colin,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the reply and the updated KIP. A few more
> > comments
> > > > > > below.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi Jun,
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 53. It seems that you already incorporated the changes in
> > > > > > KIP-516.
> > > > > > > > With
> > > > > > > > > > > topic ids, we don't need to wait for the topic's data to
> > be
> > > > > > deleted
> > > > > > > > > > before
> > > > > > > > > > > removing the topic metadata. If the topic is recreated,
> > we
> > > > can
> > > > > > still
> > > > > > > > > > delete
> > > > > > > > > > > the data properly based on the topic id. So, it seems
> > that
> > > > we can
> > > > > > > > remove
> > > > > > > > > > > TopicRecord.Deleting.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks for the reply. What I was thinking of doing here
> > was
> > > > using
> > > > > > > > topic
> > > > > > > > > > IDs internally, but still using names externally. So the
> > topic
> > > > > > UUIDs
> > > > > > > > are
> > > > > > > > > > only for the purpose of associating topics with partitions
> > --
> > > > from
> > > > > > the
> > > > > > > > > > user's point of view, topics are still identified by names.
> > > > > > > > > >
> > > > > > > > > > You're right that KIP-516 will simplify things, but I'm not
> > > > sure
> > > > > > when
> > > > > > > > that
> > > > > > > > > > will land, so I wanted to avoid blocking the initial
> > > > > > implementation of
> > > > > > > > this
> > > > > > > > > > KIP on it.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 55. It seems to me that the current behavior where we
> > favor
> > > > the
> > > > > > > > current
> > > > > > > > > > > broker registration is better. This is because
> > uncontrolled
> > > > > > broker
> > > > > > > > > > shutdown
> > > > > > > > > > > is rare and its impact is less severe since one just
> > needs to
> > > > > > wait
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > session timeout before restarting the broker. If a
> > > > mis-configured
> > > > > > > > broker
> > > > > > > > > > > replaces an existing broker, the consequence is more
> > severe
> > > > > > since it
> > > > > > > > can
> > > > > > > > > > > cause the leader to be unavailable, a replica to be out
> > of
> > > > ISR,
> > > > > > and
> > > > > > > > add
> > > > > > > > > > > more load on the leaders etc.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hmm, that's a good point. Let me check this a bit more
> > before
> > > > I
> > > > > > change
> > > > > > > > > > it, though.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 60. controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com
> > :
> > > > Do we
> > > > > > > > need to
> > > > > > > > > > > include the controller id before @? It seems that the
> > > > host/port
> > > > > > is
> > > > > > > > enough
> > > > > > > > > > > for establishing the connection. It would also be useful
> > to
> > > > make
> > > > > > it
> > > > > > > > clear
> > > > > > > > > > > that controller.connect replaces quorum.voters in
> > KIP-595.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I discussed this with Jason earlier, and he felt that the
> > > > > > controller
> > > > > > > > IDs
> > > > > > > > > > were needed in this configuration key. It is certainly
> > needed
> > > > when
> > > > > > > > > > configuring the controllers themselves, since they need to
> > know
> > > > > > each
> > > > > > > > > > others' IDs.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 61. I am not sure that we need both controller.listeners
> > and
> > > > > > > > > > > controller.connect.security.protocol since the former
> > > > implies the
> > > > > > > > > > security
> > > > > > > > > > > protocol. The reason that we have both
> > > > > > inter.broker.listener.name
> > > > > > > > and
> > > > > > > > > > > inter.broker.security.protocol is because we had the
> > latter
> > > > > > first and
> > > > > > > > > > later
> > > > > > > > > > > realized that the former is more general.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > That's a good point. I've removed this from the KIP.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 62. I am still not sure that you need
> > controller.listeners
> > > > to be
> > > > > > a
> > > > > > > > list.
> > > > > > > > > > > All listeners are already defined in listeners. To
> > migrate
> > > > from
> > > > > > > > plaintext
> > > > > > > > > > > to SSL, one can configure listeners to have both
> > plaintext
> > > > and
> > > > > > SSL.
> > > > > > > > After
> > > > > > > > > > > that, one can just change controller.listeners from
> > > > plaintext to
> > > > > > SSL.
> > > > > > > > > > This
> > > > > > > > > > > is similar to how to change the listener for inter broker
> > > > > > > > connections.
> > > > > > > > > > > Also, controller.listener.name may be a more accurate
> > name?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The issue that I see here is that if you are running with
> > the
> > > > > > > > controller
> > > > > > > > > > and broker in the same JVM, if you define a few listeners
> > in
> > > > > > > > "listeners"
> > > > > > > > > > they will get used as regular broker listeners, unless you
> > put
> > > > > > them in
> > > > > > > > > > controller.listeners. Therefore, controller.listeners
> > needs
> > > > to be
> > > > > > a
> > > > > > > > list.
> > > > > > > > > >
> > > > > > > > > > controller.listener.names does sound like a better name,
> > > > though...
> > > > > > I've
> > > > > > > > > > updated it to that.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 63. Regarding controller.id, I am trying to understand
> > > > whether
> > > > > > it's
> > > > > > > > a
> > > > > > > > > > > required configuration or an optional one. From the KIP,
> > it
> > > > > > sounds
> > > > > > > > like
> > > > > > > > > > > controller.id is optional. Then, if this is unset, it's
> > not
> > > > > > clear
> > > > > > > > how
> > > > > > > > > > the
> > > > > > > > > > > user will obtain the controller id for setting
> > > > > > controller.connect.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > If you specify broker.id but not controller.id, then
> > > > controller.id
> > > > > > > > will
> > > > > > > > > > just be set to broker.id + 3000. This was intended to
> > make
> > > > some
> > > > > > > > > > configurations a little bit more concise to write. You
> > still
> > > > do
> > > > > > have
> > > > > > > > to
> > > > > > > > > > know the controller IDs when configuring the brokers,
> > though.
> > > > If
> > > > > > this
> > > > > > > > > > seems confusing then I can just make it mandatory.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 64. KIP-516 adds a flag in LeaderAndIsrRequest to
> > indicate
> > > > > > whether it
> > > > > > > > > > > includes all partitions or not. This will be used to
> > clean up
> > > > > > strayed
> > > > > > > > > > logs.
> > > > > > > > > > > I was thinking how we will do the same thing once the
> > > > controller
> > > > > > > > moves to
> > > > > > > > > > > Raft based. One way to do that is on broker startup, it
> > gets
> > > > the
> > > > > > HWM
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > metadata log from the Raft leader and waits until its
> > > > metadata
> > > > > > > > catches up
> > > > > > > > > > > to HWM. At that point, any local log not seen in the
> > metadata
> > > > > > can be
> > > > > > > > > > > removed. Since the Fetch response returns the HWM, there
> > > > seems
> > > > > > to be
> > > > > > > > > > enough
> > > > > > > > > > > APIs to achieve this.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > That's a very good point. I added a note about this under
> > > > Broker
> > > > > > > > Startup.
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 24, 2020 at 1:07 PM Colin McCabe <
> > > > colin@cmccabe.xyz>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > On Mon, Sep 21, 2020, at 18:13, Jun Rao wrote:
> > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Sorry for the late reply. A few more comments below.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for taking another look.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 50. Configurations
> > > > > > > > > > > > > 50.1 controller.listeners: It seems that a controller
> > > > just
> > > > > > needs
> > > > > > > > one
> > > > > > > > > > > > > listener. Why do we need to have a list here? Also,
> > > > could you
> > > > > > > > > > provide an
> > > > > > > > > > > > > example of how this is set and what's its
> > relationship
> > > > with
> > > > > > > > existing
> > > > > > > > > > > > > configs such as "security.inter.broker.protocol" and
> > "
> > > > > > > > > > > > > inter.broker.listener.name"?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I agree that most administrators will want to run with
> > > > only one
> > > > > > > > > > controller
> > > > > > > > > > > > listener. However, just as with brokers, it is nice to
> > > > have
> > > > > > the
> > > > > > > > > > option to
> > > > > > > > > > > > expose multiple ports.
> > > > > > > > > > > >
> > > > > > > > > > > > One reason why you might want multiple ports is if you
> > were
> > > > > > doing a
> > > > > > > > > > > > migration from plaintext to SSL. You could add new SSL
> > > > > > listeners
> > > > > > > > but
> > > > > > > > > > > > continue to expose the PLAINTEXT listeners. Then you
> > could
> > > > > > add the
> > > > > > > > > > new SSL
> > > > > > > > > > > > controller config to each broker and roll each broker.
> > > > Then
> > > > > > you
> > > > > > > > could
> > > > > > > > > > > > migrate from PLAINTEXT to SSL with no downtime.
> > > > > > > > > > > >
> > > > > > > > > > > > Here's an example configuration for the controller:
> > > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > 1@controller1.example.com:9093,
> > 2@controller2.example.com
> > > > > > > > > > > > controller.listeners=CONTROLLER
> > > > > > > > > > > > listeners=CONTROLLER://controller0.example.com:9093
> > > > > > > > > > > > listener.security.protocol.map=CONTROLLER:SSL
> > > > > > > > > > > >
> > > > > > > > > > > > Here's an example configuration for the broker:
> > > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > 1@controller1.example.com:9093,
> > 2@controller2.example.com
> > > > > > > > > > > > controller.connect.security.protocol=SSL
> > > > > > > > > > > >
> > > > > > > > > > > > security.inter.broker.protocol or
> > > > inter.broker.listener.name
> > > > > > do
> > > > > > > > not
> > > > > > > > > > > > affect how the broker communicates with the controller.
> > > > Those
> > > > > > > > > > > > configurations relate to how brokers communicate with
> > each
> > > > > > other,
> > > > > > > > but
> > > > > > > > > > the
> > > > > > > > > > > > controller is not a broker.
> > > > > > > > > > > >
> > > > > > > > > > > > (Note that I just added
> > > > controller.connect.security.protocol
> > > > > > to the
> > > > > > > > > > KIP --
> > > > > > > > > > > > I had forgotten to put it in earlier)
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 50.2 registration.heartbeat.interval.ms and
> > > > > > > > > > > > registration.lease.timeout.ms.
> > > > > > > > > > > > > Should we match their default value with the
> > > > corresponding
> > > > > > > > default
> > > > > > > > > > for
> > > > > > > > > > > > ZK?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Fair enough. I'll set them to the values of the
> > > > > > > > > > zookeeper.sync.time.ms
> > > > > > > > > > > > and zookeeper.connection.timeout.ms configurations.
> > I do
> > > > > > think we
> > > > > > > > > > should
> > > > > > > > > > > > experiment later on to see what works well here, but
> > the ZK
> > > > > > values
> > > > > > > > are
> > > > > > > > > > at
> > > > > > > > > > > > least a starting point.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 50.3 controller.connect: Could you provide an
> > example? I
> > > > am
> > > > > > > > > > wondering how
> > > > > > > > > > > > > it differs from quorum.voters=1@kafka-1:9092,
> > 2@kafka-2
> > > > > > :9092,
> > > > > > > > > > 3@kafka-3
> > > > > > > > > > > > > :9092.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > controller.connect is intended to be the new name for
> > > > > > > > quorum.voters.
> > > > > > > > > > > > During the vote for KIP-595, we sort of agreed to
> > defer the
> > > > > > > > discussion
> > > > > > > > > > > > about what this configuration should be called. I
> > proposed
> > > > > > this
> > > > > > > > new
> > > > > > > > > > name
> > > > > > > > > > > > because it makes it clear what the configuration is for
> > > > (how to
> > > > > > > > > > connect to
> > > > > > > > > > > > the controllers).
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 50.4 controller.id: I am still not sure how this is
> > > > being
> > > > > > used.
> > > > > > > > > > Could
> > > > > > > > > > > > you
> > > > > > > > > > > > > explain this in more detail?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > The controller ID needs to be set on each controller.
> > It
> > > > also
> > > > > > > > appears
> > > > > > > > > > in
> > > > > > > > > > > > controller.connect as the thing before the "at" sign.
> > Its
> > > > > > > > function is
> > > > > > > > > > > > pretty similar to broker.id.
> > > > > > > > > > > >
> > > > > > > > > > > > Broker IDs and controller IDs exist in the same ID
> > space,
> > > > so
> > > > > > you
> > > > > > > > can't
> > > > > > > > > > > > have a broker and a controller that use the same ID.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 51. BrokerHeartbeat: It seems a bit wasteful to
> > include
> > > > > > > > Listeners in
> > > > > > > > > > > > every
> > > > > > > > > > > > > heartbeat request since it typically doesn't change.
> > > > Could we
> > > > > > > > make
> > > > > > > > > > that
> > > > > > > > > > > > an
> > > > > > > > > > > > > optional field?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Ok.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 52. KIP-584 adds a new ZK node /features. Should we
> > add a
> > > > > > > > > > corresponding
> > > > > > > > > > > > > metadata record?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Good point. I added FeatureLevelRecord.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 53. TopicRecord and DeleteTopic: Both DeleteTopic and
> > > > > > > > > > > > TopicRecord.Deleting
> > > > > > > > > > > > > indicate topic deletion. Could we outline the flow
> > when
> > > > each
> > > > > > > > will be
> > > > > > > > > > set?
> > > > > > > > > > > > > In particular, which one indicates the intention to
> > > > delete
> > > > > > and
> > > > > > > > which
> > > > > > > > > > one
> > > > > > > > > > > > > indicates the completion of the deletion.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > TopicRecord.Deleting is set when we intend to delete
> > the
> > > > topic.
> > > > > > > > > > > >
> > > > > > > > > > > > DeleteTopic removes the topic completely. I will
> > rename
> > > > > > > > DeleteTopic to
> > > > > > > > > > > > RemoveTopic to make this clearer.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 54. "The controller can generate a new broker epoch
> > by
> > > > using
> > > > > > the
> > > > > > > > > > latest
> > > > > > > > > > > > log
> > > > > > > > > > > > > offset." Which offset is that? Is it the offset of
> > the
> > > > > > metadata
> > > > > > > > > > topic for
> > > > > > > > > > > > > the corresponding BrokerRecord? Is it guaranteed to
> > be
> > > > > > different
> > > > > > > > on
> > > > > > > > > > each
> > > > > > > > > > > > > broker restart?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Yes.... a new broker epoch implies that there has been
> > a
> > > > new
> > > > > > record
> > > > > > > > > > > > created in the metadata log. Therefore the last
> > committed
> > > > > > offset
> > > > > > > > must
> > > > > > > > > > be
> > > > > > > > > > > > different.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 55. "Thereafter, it may lose subsequent conflicts if
> > its
> > > > > > broker
> > > > > > > > > > epoch is
> > > > > > > > > > > > > stale. (See KIP-380 for some background on broker
> > > > epoch.)
> > > > > > The
> > > > > > > > > > reason
> > > > > > > > > > > > for
> > > > > > > > > > > > > favoring new processes is to accommodate the common
> > case
> > > > > > where a
> > > > > > > > > > process
> > > > > > > > > > > > is
> > > > > > > > > > > > > killed with kill -9 and then restarted. " Are you
> > saying
> > > > > > that if
> > > > > > > > > > there is
> > > > > > > > > > > > > an active broker registered in the controller, a new
> > > > broker
> > > > > > > > heartbeat
> > > > > > > > > > > > > request with the INITIAL state will fence the current
> > > > broker
> > > > > > > > > > session? Not
> > > > > > > > > > > > > sure about this. For example, this will allow a
> > broker
> > > > with
> > > > > > > > > > incorrectly
> > > > > > > > > > > > > configured broker id to kill an existing valid
> > broker.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, a new broker with an incorrectly configured
> > broker id
> > > > > > could
> > > > > > > > fence
> > > > > > > > > > an
> > > > > > > > > > > > existing valid broker.
> > > > > > > > > > > >
> > > > > > > > > > > > The goal of the conflict resolution process described
> > here
> > > > is
> > > > > > to
> > > > > > > > avoid
> > > > > > > > > > > > having two brokers go back and forth in claiming the
> > > > broker id.
> > > > > > > > > > Choosing
> > > > > > > > > > > > the newest was just an easy way to do that.
> > > > > > > > > > > >
> > > > > > > > > > > > Choosing the oldest is another possibility, and more
> > inline
> > > > > > with
> > > > > > > > what
> > > > > > > > > > > > ZooKeeper does today. The problem with that is that it
> > > > would
> > > > > > not
> > > > > > > > be
> > > > > > > > > > > > possible to re-create the broker if the old instance
> > died
> > > > > > suddenly,
> > > > > > > > > > except
> > > > > > > > > > > > by waiting for the full lease timeout.
> > > > > > > > > > > >
> > > > > > > > > > > > It might be possible to do a little better here by
> > > > breaking the
> > > > > > > > lease
> > > > > > > > > > if
> > > > > > > > > > > > the TCP connection from the broker drops. But that
> > > > requires
> > > > > > > > crossing
> > > > > > > > > > some
> > > > > > > > > > > > layers of abstraction in the Kafka networking stack.
> > > > There's
> > > > > > also
> > > > > > > > the
> > > > > > > > > > > > possibility of false negatives or positives. For
> > example,
> > > > TCP
> > > > > > > > > > connections
> > > > > > > > > > > > sometimes just drop even if the node is still there.
> > Or
> > > > > > sometimes
> > > > > > > > the
> > > > > > > > > > > > networking stack keeps a TCP connection alive for a
> > while
> > > > when
> > > > > > the
> > > > > > > > > > other
> > > > > > > > > > > > end is gone.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 56. kafka-storage.sh:
> > > > > > > > > > > > > 56.1 In the info mode, what other information does it
> > > > show in
> > > > > > > > > > addition to
> > > > > > > > > > > > > "kip.500.mode=enabled"?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I think it should show a list of all the storage
> > > > directories,
> > > > > > and
> > > > > > > > > > whether
> > > > > > > > > > > > each one is formatted. In addition, it should show
> > whether
> > > > > > > > > > kip.500.mode is
> > > > > > > > > > > > enabled, and what the cluster id is.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 56.2 Should the format mode take the config file as
> > the
> > > > > > input too
> > > > > > > > > > like
> > > > > > > > > > > > the
> > > > > > > > > > > > > info mode?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, that's a good idea.
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jun
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 17, 2020 at 4:12 PM Colin McCabe <
> > > > > > cmccabe@apache.org
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > That's a fair point. I have moved the lease
> > duration
> > > > to
> > > > > > the
> > > > > > > > broker
> > > > > > > > > > > > > > heartbeat response. That way lease durations can
> > be
> > > > > > changed
> > > > > > > > just
> > > > > > > > > > be
> > > > > > > > > > > > > > reconfiguring the controllers.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, Sep 16, 2020, at 07:40, Unmesh Joshi wrote:
> > > > > > > > > > > > > > > Thanks Colin, the changes look good to me. One
> > small
> > > > > > thing.
> > > > > > > > > > > > > > > registration.lease.timeout.ms is the
> > configuration
> > > > on
> > > > > > the
> > > > > > > > > > controller
> > > > > > > > > > > > > > side.
> > > > > > > > > > > > > > > It will be good to comment on how brokers know
> > about
> > > > it,
> > > > > > to
> > > > > > > > be
> > > > > > > > > > able
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > send LeaseDurationMs
> > > > > > > > > > > > > > > in the heartbeat request,
> > > > > > > > > > > > > > > or else it can be added in the heartbeat
> > response for
> > > > > > > > brokers to
> > > > > > > > > > know
> > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Sep 11, 2020 at 10:32 PM Colin McCabe <
> > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think you're right that we should use a
> > duration
> > > > here
> > > > > > > > rather
> > > > > > > > > > > > than a
> > > > > > > > > > > > > > > > time. As you said, the clock on the controller
> > > > will
> > > > > > > > probably
> > > > > > > > > > not
> > > > > > > > > > > > > > match the
> > > > > > > > > > > > > > > > one on the broker. I have updated the KIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It's important to keep in mind that
> > messages
> > > > may be
> > > > > > > > > > delayed in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > > happens, we
> > > > > > > > > > will
> > > > > > > > > > > > use
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > time specified in the request to determine
> > if
> > > > the
> > > > > > > > request
> > > > > > > > > > is
> > > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > > I am assuming that there will be a single TCP
> > > > > > connection
> > > > > > > > > > > > maintained
> > > > > > > > > > > > > > > > between
> > > > > > > > > > > > > > > > > broker and active controller. So, there
> > won't be
> > > > any
> > > > > > out
> > > > > > > > of
> > > > > > > > > > order
> > > > > > > > > > > > > > > > requests?
> > > > > > > > > > > > > > > > > There will be a scenario of broker GC pause,
> > > > which
> > > > > > might
> > > > > > > > > > cause
> > > > > > > > > > > > > > connection
> > > > > > > > > > > > > > > > > timeout and broker might need to reestablish
> > the
> > > > > > > > connection.
> > > > > > > > > > If
> > > > > > > > > > > > the
> > > > > > > > > > > > > > pause
> > > > > > > > > > > > > > > > > is too long, lease will expire and the
> > heartbeat
> > > > sent
> > > > > > > > after
> > > > > > > > > > the
> > > > > > > > > > > > pause
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > be treated as a new registration (similar to
> > > > restart
> > > > > > > > case),
> > > > > > > > > > and
> > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > epoch
> > > > > > > > > > > > > > > > > number will be assigned to the broker.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I agree with the end of this paragraph, but not
> > > > with
> > > > > > the
> > > > > > > > start
> > > > > > > > > > :)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There can be out-of-order requests, since the
> > > > broker
> > > > > > will
> > > > > > > > > > simply
> > > > > > > > > > > > use a
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > TCP connection if the old one has problems.
> > This
> > > > can
> > > > > > > > happen
> > > > > > > > > > for a
> > > > > > > > > > > > > > variety
> > > > > > > > > > > > > > > > of reasons. I don't think GC pauses are the
> > most
> > > > > > common
> > > > > > > > > > reason for
> > > > > > > > > > > > > > this to
> > > > > > > > > > > > > > > > happen. It's more common to see issues issues
> > in
> > > > the
> > > > > > > > network
> > > > > > > > > > > > itself
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > result connections getting dropped from time to
> > > > time.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > So we have to assume that messages may arrive
> > out
> > > > of
> > > > > > > > order, and
> > > > > > > > > > > > > > possibly
> > > > > > > > > > > > > > > > be delayed. I added a note that heartbeat
> > requests
> > > > > > should
> > > > > > > > be
> > > > > > > > > > > > ignored
> > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > the metadata log offset they contain is smaller
> > > > than a
> > > > > > > > previous
> > > > > > > > > > > > > > heartbeat.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > When the active controller fails, the new
> > active
> > > > > > > > controller
> > > > > > > > > > > > needs to
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > sure that it considers all the known brokers
> > as
> > > > alive
> > > > > > > > till
> > > > > > > > > > the
> > > > > > > > > > > > lease
> > > > > > > > > > > > > > > > > expiration interval. Because
> > > > > > > > registration.lease.timeout.ms,
> > > > > > > > > > is
> > > > > > > > > > > > > > > > configured
> > > > > > > > > > > > > > > > > on the controller, the new active controller
> > will
> > > > > > extend
> > > > > > > > all
> > > > > > > > > > the
> > > > > > > > > > > > > > leases
> > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > registration.lease.timeout.ms. I see that it
> > > > won't
> > > > > > need
> > > > > > > > last
> > > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Agreed.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Sat, Sep 5, 2020 at 1:28 AM Colin McCabe <
> > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Colin wrote:
> > > > > > > > > > > > > > > > > > > > The reason for including
> > LeaseStartTimeMs
> > > > in
> > > > > > the
> > > > > > > > > > request
> > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > > > that the time required to communicate
> > with
> > > > the
> > > > > > > > > > controller
> > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > included in
> > > > > > > > > > > > > > > > > > > > the lease time. Since requests can
> > > > > > potentially be
> > > > > > > > > > delayed
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > > > for a long time, this is important.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020, at 05:58, Unmesh
> > Joshi
> > > > wrote:
> > > > > > > > > > > > > > > > > > > The network time will be added anyway,
> > > > because
> > > > > > the
> > > > > > > > lease
> > > > > > > > > > > > timer
> > > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > > active controller will start only after
> > the
> > > > > > heartbeat
> > > > > > > > > > request
> > > > > > > > > > > > > > > > reaches the
> > > > > > > > > > > > > > > > > > > server.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > If the start time is not specified in the
> > > > request,
> > > > > > > > then the
> > > > > > > > > > > > network
> > > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > > is excluded from the heartbeat time.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Here's an example:
> > > > > > > > > > > > > > > > > > Let's say broker A sends a heartbeat at
> > time
> > > > 100,
> > > > > > and
> > > > > > > > it
> > > > > > > > > > > > arrives
> > > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > controller at time 200, and the lease
> > duration
> > > > is
> > > > > > 1000.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The controller looks at the start time in
> > the
> > > > > > request,
> > > > > > > > > > which is
> > > > > > > > > > > > > > 100,
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > adds 1000 to it, getting 1100. On the
> > other
> > > > hand,
> > > > > > if
> > > > > > > > start
> > > > > > > > > > > > time
> > > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > > specified in the request, then the
> > expiration
> > > > will
> > > > > > be
> > > > > > > > at
> > > > > > > > > > time
> > > > > > > > > > > > 1200.
> > > > > > > > > > > > > > > > > > That is what I mean by "the network time is
> > > > > > included
> > > > > > > > in the
> > > > > > > > > > > > > > expiration
> > > > > > > > > > > > > > > > > > time."
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > And I think, some assumption about
> > network
> > > > round
> > > > > > trip
> > > > > > > > > > time is
> > > > > > > > > > > > > > > > > > > needed anyway to decide on the frequency
> > of
> > > > the
> > > > > > > > > > heartbeat (
> > > > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms), and
> > > > lease
> > > > > > > > timeout (
> > > > > > > > > > > > > > > > > > > registration.lease.timeout.ms). So I
> > think
> > > > just
> > > > > > > > having a
> > > > > > > > > > > > > > leaseTTL
> > > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > > request is easier to understand and
> > > > implement.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It's important to keep in mind that
> > messages
> > > > may be
> > > > > > > > > > delayed in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > > happens, we
> > > > > > > > > > will
> > > > > > > > > > > > use
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > time specified in the request to determine
> > if
> > > > the
> > > > > > > > request
> > > > > > > > > > is
> > > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> > the
> > > > > > > > controller
> > > > > > > > > > side
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > reset in the case of controller
> > failover.
> > > > The
> > > > > > > > > > alternative
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > be to
> > > > > > > > > > > > > > > > > > > > track the lease as hard state rather
> > than
> > > > soft
> > > > > > > > state,
> > > > > > > > > > but I
> > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > is not really needed, and would result
> > in
> > > > more
> > > > > > log
> > > > > > > > > > entries.
> > > > > > > > > > > > > > > > > > > My interpretation of the mention of
> > > > BrokerRecord
> > > > > > in
> > > > > > > > the
> > > > > > > > > > KIP
> > > > > > > > > > > > was
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > record exists in the Raft log.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > BrokerRecord does exist in the Raft log,
> > but
> > > > does
> > > > > > not
> > > > > > > > > > include
> > > > > > > > > > > > the
> > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > heartbeat time.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > By soft state, do you mean the broker
> > > > > > > > > > > > > > > > > > > records exist only on the active leader
> > and
> > > > will
> > > > > > not
> > > > > > > > be
> > > > > > > > > > > > > > replicated
> > > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > > raft log? If the live brokers list is
> > > > maintained
> > > > > > > > only on
> > > > > > > > > > the
> > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > controller (raft leader), then, in case
> > of
> > > > leader
> > > > > > > > > > failure,
> > > > > > > > > > > > there
> > > > > > > > > > > > > > > > will be
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > window where the new leader does not know
> > > > about
> > > > > > the
> > > > > > > > live
> > > > > > > > > > > > brokers,
> > > > > > > > > > > > > > > > till
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > brokers establish the leases again.
> > > > > > > > > > > > > > > > > > > I think it will be safer to have leases
> > as a
> > > > hard
> > > > > > > > state
> > > > > > > > > > > > managed
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > standard
> > > > > > > > > > > > > > > > > > > Raft replication.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Leases are short, so the need to
> > re-establish
> > > > them
> > > > > > > > after a
> > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > failover doesn't seem like a big problem.
> > But
> > > > > > this is
> > > > > > > > > > > > something
> > > > > > > > > > > > > > we can
> > > > > > > > > > > > > > > > > > tweak if it becomes an issue. One option
> > > > would be
> > > > > > to
> > > > > > > > have
> > > > > > > > > > a
> > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > which is only used by the controller nodes
> > for
> > > > this
> > > > > > > > (since,
> > > > > > > > > > > > after
> > > > > > > > > > > > > > all,
> > > > > > > > > > > > > > > > > > brokers don't care about registration
> > > > renewals).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Or am I misunderstanding something? (I
> > assume
> > > > > > that
> > > > > > > > with
> > > > > > > > > > soft
> > > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > mean something like zookeeper local
> > sessions
> > > > > > > > > > > > > > > > > > >
> > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1147
> > > > > > > > .)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > > think
> > > > > > it
> > > > > > > > makes
> > > > > > > > > > > > sense
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> > becomes
> > > > > > very
> > > > > > > > messy.
> > > > > > > > > > > > I'm
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > > understand your question about
> > duplicate
> > > > > > broker ID
> > > > > > > > > > > > detection,
> > > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > There's a section in the KIP about this
> > -- is
> > > > > > there a
> > > > > > > > > > detail
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This is an implementation detail that
> > doesn't
> > > > need
> > > > > > to
> > > > > > > > be
> > > > > > > > > > in the
> > > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I assumed broker leases are implemented
> > as a
> > > > hard
> > > > > > > > state.
> > > > > > > > > > In
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > case, to
> > > > > > > > > > > > > > > > > > > check for broker id conflict, we need to
> > > > check
> > > > > > the
> > > > > > > > broker
> > > > > > > > > > > > ids at
> > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > places
> > > > > > > > > > > > > > > > > > > 1. Pending broker registrations (which
> > are
> > > > yet
> > > > > > to be
> > > > > > > > > > > > committed)
> > > > > > > > > > > > > > 2.
> > > > > > > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > > > > > committed broker registrations.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020 at 5:42 PM Colin
> > McCabe
> > > > <
> > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020, at 01:12, Unmesh
> > > > Joshi
> > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >>>Can you repeat your questions
> > about
> > > > broker
> > > > > > > > leases?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >>>>The LeaseStartTimeMs is expected
> > to
> > > > be
> > > > > > the
> > > > > > > > > > broker's
> > > > > > > > > > > > > > > > > > > > > 'System.currentTimeMillis()' at the
> > > > point of
> > > > > > the
> > > > > > > > > > > > request. The
> > > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > > controller will add its lease period
> > to
> > > > this
> > > > > > in
> > > > > > > > order
> > > > > > > > > > > > >>>>to
> > > > > > > > > > > > > > > > compute
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > LeaseEndTimeMs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think the use of LeaseStartTimeMs
> > and
> > > > > > > > > > LeaseEndTimeMs
> > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > confusing. Monotonic Clock
> > > > > > (System.nanoTime) on
> > > > > > > > the
> > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > should be used to track leases.
> > > > > > > > > > > > > > > > > > > > > (For example,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1616https://github.com/etcd-io/etcd/pull/6888/commits/e7f4010ccaf28b6ce64fe514d25a4b2fa459d114
> > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Then we will not need
> > LeaseStartTimeMs?
> > > > > > > > > > > > > > > > > > > > > Instead of LeaseStartTimeMs, can we
> > call
> > > > it
> > > > > > > > > > LeaseTTL? The
> > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > can then calculate LeaseEndTime =
> > > > > > > > System.nanoTime() +
> > > > > > > > > > > > > > LeaseTTL.
> > > > > > > > > > > > > > > > > > > > > In this case we might just drop
> > > > > > LeaseEndTimeMs
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > > > > > response, as
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > broker already knows about the TTL
> > and
> > > > can
> > > > > > send
> > > > > > > > > > > > heartbeats at
> > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > fraction
> > > > > > > > > > > > > > > > > > > > > of TTL, say every TTL/4
> > > > milliseconds.(elapsed
> > > > > > > > time
> > > > > > > > > > on the
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > measured
> > > > > > > > > > > > > > > > > > > > > by System.nanoTime)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I agree that the monotonic clock is
> > > > probably a
> > > > > > > > better
> > > > > > > > > > idea
> > > > > > > > > > > > > > here.
> > > > > > > > > > > > > > > > It is
> > > > > > > > > > > > > > > > > > > > good to be robust against wall clock
> > > > changes,
> > > > > > > > although
> > > > > > > > > > I
> > > > > > > > > > > > think
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > > > > which had them might suffer other
> > issues.
> > > > I
> > > > > > will
> > > > > > > > > > change
> > > > > > > > > > > > it to
> > > > > > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > monotonic clock.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The reason for including
> > LeaseStartTimeMs
> > > > in
> > > > > > the
> > > > > > > > > > request
> > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > the time required to communicate with
> > the
> > > > > > > > controller
> > > > > > > > > > gets
> > > > > > > > > > > > > > included
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > lease time. Since requests can
> > > > potentially be
> > > > > > > > delayed
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > > > > > long time, this is important.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I have a prototype built to
> > demonstrate
> > > > this
> > > > > > as
> > > > > > > > > > > > following:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The Kip631Controller itself depends
> > on a
> > > > > > > > Consensus
> > > > > > > > > > > > module, to
> > > > > > > > > > > > > > > > > > demonstrate
> > > > > > > > > > > > > > > > > > > > > how possible interactions with the
> > > > consensus
> > > > > > > > module
> > > > > > > > > > will
> > > > > > > > > > > > look
> > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > (The Consensus can be pluggable
> > really,
> > > > > > with an
> > > > > > > > API
> > > > > > > > > > to
> > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > > replicated log upto HighWaterMark)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > It has an implementation of
> > LeaseTracker
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/LeaderLeaseTracker.scala
> > > > > > > > > > > > > > > > > > > > > to demonstrate LeaseTracker's
> > interaction
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > > consensus
> > > > > > > > > > > > > > > > module.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The implementation has the following
> > > > aspects:
> > > > > > > > > > > > > > > > > > > > > 1. The lease tracking happens only
> > on the
> > > > > > active
> > > > > > > > > > > > controller
> > > > > > > > > > > > > > (raft
> > > > > > > > > > > > > > > > > > > > > leader)
> > > > > > > > > > > > > > > > > > > > > 2. Once the lease expires, it needs
> > to
> > > > > > propose
> > > > > > > > and
> > > > > > > > > > > > commit a
> > > > > > > > > > > > > > > > > > FenceBroker
> > > > > > > > > > > > > > > > > > > > > record for that lease.
> > > > > > > > > > > > > > > > > > > > > 3. In case of active controller
> > failure,
> > > > the
> > > > > > > > lease
> > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > tracked by
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > newly raft leader. The new raft
> > leader
> > > > > > starts the
> > > > > > > > > > lease
> > > > > > > > > > > > timer
> > > > > > > > > > > > > > > > again,
> > > > > > > > > > > > > > > > > > (as
> > > > > > > > > > > > > > > > > > > > > implemented in onBecomingLeader
> > method of
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > > in effect extending the lease by the
> > time
> > > > > > spent
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > election
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > whatever time was elapsed on the old
> > > > leader.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> > the
> > > > > > > > controller
> > > > > > > > > > side
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > reset
> > > > > > > > > > > > > > > > > > > > in the case of controller failover.
> > The
> > > > > > > > alternative
> > > > > > > > > > would
> > > > > > > > > > > > be
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > track
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > lease as hard state rather than soft
> > state,
> > > > > > but I
> > > > > > > > think
> > > > > > > > > > > > that
> > > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > > > > > needed, and would result in more log
> > > > entries.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > There are working tests for this
> > > > > > implementation
> > > > > > > > here.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > > > and an end to end test here
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/ProducerConsumerKIP500Test.scala
> > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >>'m not sure what you mean by
> > > > > > "de-duplication
> > > > > > > > of the
> > > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > > Apologies for using the confusing
> > term
> > > > > > > > > > deduplication. I
> > > > > > > > > > > > meant
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > id
> > > > > > > > > > > > > > > > > > > > > conflict.
> > > > > > > > > > > > > > > > > > > > > As you can see in the prototype
> > > > > > handleRequest of
> > > > > > > > > > > > > > KIP631Controller
> > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > > the duplicate broker id needs to be
> > > > detected
> > > > > > > > before
> > > > > > > > > > the
> > > > > > > > > > > > > > > > BrokerRecord
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > submitted to the raft module.
> > > > > > > > > > > > > > > > > > > > > Also as implemented in the
> > prototype, the
> > > > > > > > > > > > KIP631Controller is
> > > > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > > > > > threaded, handling requests one at a
> > > > time.
> > > > > > (an
> > > > > > > > > > example of
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://martinfowler.com/articles/patterns-of-distributed-systems/singular-update-queue.html
> > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > > think
> > > > > > it
> > > > > > > > makes
> > > > > > > > > > > > sense
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> > becomes
> > > > > > very
> > > > > > > > messy.
> > > > > > > > > > > > I'm
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > > understand your question about
> > duplicate
> > > > > > broker ID
> > > > > > > > > > > > detection,
> > > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > > There's a section in the KIP about
> > this --
> > > > is
> > > > > > > > there a
> > > > > > > > > > > > detail we
> > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 10:49 AM
> > Colin
> > > > > > McCabe <
> > > > > > > > > > > > > > colin@cmccabe.xyz
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2020, at 19:36,
> > Unmesh
> > > > > > Joshi
> > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > There were a few of questions I
> > had..
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for the response.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. Were my comments on the broker
> > > > lease
> > > > > > > > > > > > implementation
> > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > prototype) appropriate and do we
> > > > need to
> > > > > > > > change
> > > > > > > > > > the
> > > > > > > > > > > > KIP
> > > > > > > > > > > > > > > > > > > > > > > description accordingly?.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Can you repeat your questions about
> > > > broker
> > > > > > > > leases?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. How will broker epochs be
> > > > generated?
> > > > > > I am
> > > > > > > > > > > > assuming it
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > committed log offset (like zxid?)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > There isn't any need to use a log
> > > > offset.
> > > > > > We
> > > > > > > > can
> > > > > > > > > > just
> > > > > > > > > > > > > > look at
> > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > in-memory hash table and see what
> > the
> > > > > > latest
> > > > > > > > > > number is,
> > > > > > > > > > > > > > and add
> > > > > > > > > > > > > > > > > > one, to
> > > > > > > > > > > > > > > > > > > > > > generate a new broker epoch.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3. How will producer registration
> > > > > > happen? I
> > > > > > > > am
> > > > > > > > > > > > assuming
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > should be
> > > > > > > > > > > > > > > > > > > > > > > similar to broker registration,
> > with
> > > > a
> > > > > > > > similar
> > > > > > > > > > way to
> > > > > > > > > > > > > > > > generate
> > > > > > > > > > > > > > > > > > > > producer
> > > > > > > > > > > > > > > > > > > > > > id.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > For the EOS stuff, we will need a
> > few
> > > > new
> > > > > > RPCs
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > > > controller.
> > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > we should do that in a follow-on
> > KIP,
> > > > > > though,
> > > > > > > > since
> > > > > > > > > > > > this
> > > > > > > > > > > > > > one is
> > > > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > > > > pretty big.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 4. Because we expose Raft log to
> > all
> > > > the
> > > > > > > > > > brokers, any
> > > > > > > > > > > > > > > > > > de-duplication
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > broker needs to happen before the
> > > > > > requests
> > > > > > > > are
> > > > > > > > > > > > proposed
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > Raft.
> > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > the controller needs to be single
> > > > > > threaded,
> > > > > > > > and
> > > > > > > > > > > > should do
> > > > > > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > > > > against the in-process or pending
> > > > > > requests
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > > final
> > > > > > > > > > > > > > > > state. I
> > > > > > > > > > > > > > > > > > > > read a
> > > > > > > > > > > > > > > > > > > > > > > mention of this, in the
> > responses in
> > > > this
> > > > > > > > > > > > thread.Will it
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > useful to
> > > > > > > > > > > > > > > > > > > > > > > mention this in the KIP?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I'm not sure what you mean by
> > > > > > "de-duplication
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 4:50 AM
> > Colin
> > > > > > McCabe
> > > > > > > > <
> > > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I'm thinking of calling a vote
> > on
> > > > > > KIP-631
> > > > > > > > on
> > > > > > > > > > > > Monday.
> > > > > > > > > > > > > > Let
> > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > know
> > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > there's any more comments I
> > should
> > > > > > address
> > > > > > > > > > before I
> > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > vote.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020, at 05:39,
> > > > Unmesh
> > > > > > > > Joshi
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >>Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > >>Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks. I will be adding
> > more to
> > > > the
> > > > > > > > > > prototype
> > > > > > > > > > > > and
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > > > > and collaborate.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020 at
> > 12:28 AM
> > > > > > Colin
> > > > > > > > > > McCabe <
> > > > > > > > > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Jose,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > That'a s good point that I
> > > > hadn't
> > > > > > > > > > considered.
> > > > > > > > > > > > It's
> > > > > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > > a separate leader change
> > > > message,
> > > > > > as
> > > > > > > > you
> > > > > > > > > > > > mentioned.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2020, at
> > 11:56,
> > > > Jose
> > > > > > > > Garcia
> > > > > > > > > > > > Sancio
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Very cool prototype!
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > The KIP proposes a record
> > > > called
> > > > > > > > > > IsrChange
> > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > includes the
> > > > > > > > > > > > > > > > > > > > > > > > > > > partition, topic, isr,
> > > > leader and
> > > > > > > > leader
> > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > During
> > > > > > > > > > > > > > > > > > normal
> > > > > > > > > > > > > > > > > > > > > > > > > > > operation ISR changes do
> > not
> > > > > > result
> > > > > > > > in
> > > > > > > > > > leader
> > > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > > > Similarly,
> > > > > > > > > > > > > > > > > > > > > > > > > > > leader changes do not
> > > > necessarily
> > > > > > > > > > involve ISR
> > > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > > implementation that uses
> > ZK
> > > > > > modeled
> > > > > > > > them
> > > > > > > > > > > > together
> > > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. All of this
> > information is
> > > > > > stored
> > > > > > > > in
> > > > > > > > > > one
> > > > > > > > > > > > > > znode.
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. ZK's optimistic lock
> > > > requires
> > > > > > > > that you
> > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > the new
> > > > > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > > > > > > > > 3. The change to that
> > znode
> > > > was
> > > > > > being
> > > > > > > > > > > > performed
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > > and the leader.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > None of these reasons are
> > > > true in
> > > > > > > > > > KIP-500.
> > > > > > > > > > > > Have
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > considered
> > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > > > two different records?
> > For
> > > > > > example
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. IsrChange record which
> > > > > > includes
> > > > > > > > topic,
> > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > isr
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. LeaderChange record
> > which
> > > > > > includes
> > > > > > > > > > topic,
> > > > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > leader epoch.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I suspect that making
> > this
> > > > change
> > > > > > > > will
> > > > > > > > > > also
> > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > message AlterIsrRequest
> > > > > > introduced in
> > > > > > > > > > > > KIP-497:
> > > > > > > > > > > > > > Add
> > > > > > > > > > > > > > > > > > > > inter-broker
> > > > > > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > alter ISR.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > > > > > -Jose
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
Hi Ron,
Thanks for the questions.
Talking about "broker state" can get confusing because there's lots of different concepts that we refer to as "broker state." For example, there is a JMX metric that we expose, whose values are given in BrokerStates.scala.
However, for the purpose of this discussion, the important broker state is registration state. And registration state is currently very simple: you're either in the cluster, or you're not. If you're in, you have an active ZooKeeper connection, and you have a znode under /brokers/ which has your connection information. If you're out, you do not. Registration is a thing that happens between the broker and ZooKeeper-- nobody else is involved.
Controlled shutdown is just a path that allows you to transition from "in the cluster" to "out of the cluster" more gracefully. It doesn't have a different destination state than just using kill -9 on the process.
The intention in KIP-500 is to have three registration states. I added a paragraph to KIP-631 about this:
> In the post-KIP-500 world, there will be three broker registration states:
> unregistered, registered but fenced, and registered and active. Just like
> today, unregistered means that there is no registration information and no
> way to reach the broker. It is effectively not part of the cluster. In
> the two registered states, in contrast, contact information is available.
> However, in the "registered but fenced" state, the contact information
> might no longer be valid. For example, if a broker crashes and is not
> restarted, it will end up in "registered but fenced" state.
I also added a new Record, UnfenceBroker, which I think addresses your earlier point that we should not confuse unfencing with registering.
best,
Colin
It is not necessary to communicate
On Thu, Oct 22, 2020, at 10:48, Ron Dagostino wrote:
> Hi again, Colin. Related to the issue of how to communicate fenced
> vs. not fenced (and how to communicate or imply Offline vs not
> Offline), do we need to communicate in the log that a broker has
> gracefully shut down? We do distinguish between a broker being
> unavailable due to a controlled shutdown vs. having died -- one that
> dies appears in the metadata as being Offline, whereas one that
> gracefully shuts down does not. I don't see a way to imply this
> gracefully-shutdown state, though I may be missing it.
>
> Ron
>
> On Thu, Oct 22, 2020 at 1:32 PM Ron Dagostino <rn...@gmail.com> wrote:
> >
> > HI Colin. A FencedBrokerRecord appears in the metadata log when a
> > broker is fenced. What appears in the metadata log to indicate that a
> > broker is no longer fenced? Does a BrokerRecord appear? That seems
> > to communicate a bunch of unnecessary data in this context (endpoints,
> > features, rack). If that is the current plan, might it be better to
> > include a Boolean value within FencedBrokerRecord and specify true
> > when the broker becomes fenced and false when it is no longer fenced?
> >
> > Also, is a fenced broker considered Offline for partition metadata
> > purposes? I think so but would like to confirm.
> >
> > Ron
> >
> > On Wed, Oct 21, 2020 at 9:05 AM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > On Tue, Oct 13, 2020, at 18:30, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply. A few more comments below.
> > > >
> > > > 80.1 controller.listener.names only defines the name of the listener. The
> > > > actual listener including host/port/security_protocol is typically defined
> > > > in advertised_listners. Does that mean advertised_listners is a required
> > > > config now?
> > > >
> > >
> > > Hi Jun,
> > >
> > > Thanks for the re-review.
> > >
> > > The controller listeners are not advertised to clients. So I think they should go in listeners, rather than advertised.listeners.
> > >
> > > I agree that this makes listeners a required configuration. At very least, it is required to have the controller listener in there.
> > >
> > > >
> > > > 83.1 broker state machine: It seems that we should transition from FENCED
> > > > => INITIAL since only INITIAL generates new broker epoch?
> > > >
> > >
> > > I changed the broker state machine a bit-- take a look. In the new state machine, the FENCED state can re-register.
> > >
> > > >
> > > > 83.5. It's true that the controller node doesn't serve metadata requests.
> > > > However, there are admin requests such as topic creation/deletion are sent
> > > > to the controller directly. So, it seems that the client needs to know
> > > > the controller host/port?
> > > >
> > >
> > > The client sends admin requests to a random broker, which forwards them to the controller. This is described in KIP-590.
> > >
> > > >
> > > > 85. "I was hoping that we could avoid responding to requests when the
> > > > broker was fenced." This issue is that if we don't send a response, the
> > > > client won't know the reason and can't act properly.
> > > >
> > >
> > > Sorry, when I said "does not respond" I meant it would return an error.
> > >
> > > >
> > > > 88. CurMetadataOffset: I was thinking that we may want to
> > > > use CurMetadataOffset to compute the MetadataLag. Since HWM is exclusive,
> > > > it's more convenient if CurMetadataOffset is also exclusive.
> > > >
> > >
> > > OK, I will change it to NextMetadataOffset and make it exclusive.
> > >
> > > > 90. It would be useful to add a rejected section on why separate controller
> > > > and broker id is preferred over just broker id. For example, the following
> > > > are some potential reasons. (a) We can guard duplicated brokerID, but it's
> > > > hard to guard against duplicated controllerId. (b) brokerID can be auto
> > > > assigned in the future, but controllerId is hard to be generated
> > > > automatically.
> > > >
> > >
> > > OK. I added a rejected alternatives section about sharing IDs between multiple nodes.
> > >
> > > Colin
> > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Oct 12, 2020 at 11:14 AM Colin McCabe <cm...@apache.org> wrote:
> > > >
> > > > > On Tue, Oct 6, 2020, at 16:09, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the reply. Made another pass of the KIP. A few more comments
> > > > > > below.
> > > > > >
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > Thanks for the review.
> > > > >
> > > > > > 55. We discussed earlier why the current behavior where we favor the
> > > > > > current broker registration is better. Have you given this more thought?
> > > > > >
> > > > >
> > > > > Yes, I think we should favor the current broker registration, as you
> > > > > suggested earlier.
> > > > >
> > > > > > 80. Config related.
> > > > > > 80.1 Currently, each broker only has the following 3 required configs. It
> > > > > > will be useful to document the required configs post KIP-500 (in both the
> > > > > > dedicated and shared controller mode).
> > > > > > broker.id
> > > > > > log.dirs
> > > > > > zookeeper.connect
> > > > >
> > > > > For the broker, these configs will be required:
> > > > >
> > > > > broker.id
> > > > > log.dirs
> > > > > process.roles
> > > > > controller.listener.names
> > > > > controller.connect
> > > > >
> > > > > For the controller, these configs will be required:
> > > > >
> > > > > controller.id
> > > > > log.dirs
> > > > > process.roles
> > > > > controller.listener.names
> > > > > controller.connect
> > > > >
> > > > > For broker+controller, it will be the union of these two, which
> > > > > essentially means we need both broker.id and controller.id, but all
> > > > > others are the same as standalone.
> > > > >
> > > > > > 80.2 It would be useful to document all deprecated configs post KIP-500.
> > > > > > For example, all zookeeper.* are obviously deprecated. But there could be
> > > > > > others. For example, since we don't plan to support auto broker id
> > > > > > generation, it seems broker.id.generation.enable is deprecated too.
> > > > > > 80.3 Could we make it clear that controller.connect replaces
> > > > > quorum.voters
> > > > > > in KIP-595?
> > > > >
> > > > > OK. I added a comment about this in the table.
> > > > >
> > > > > > 80.4 Could we document that broker.id is now optional?
> > > > >
> > > > > OK. I have added a line for broker.id.
> > > > >
> > > > > > 80.5 The KIP suggests that controller.id is optional on the controller
> > > > > > node. I am concerned that this can cause a bit of confusion in 2 aspects.
> > > > > > First, in the dedicated controller mode, controller.id is not optional
> > > > > > (since broker.id is now optional). Second, in the shared controller
> > > > > mode,
> > > > > > it may not be easy for the user to figure out the default value of
> > > > > > controller.id to set controller.connect properly.
> > > > >
> > > > > I got rid of the default value of controller.id. I agree it was a bit
> > > > > confusing to explain.
> > > > >
> > > > > > 80.6 Regarding the consistency of config names, our metrics already
> > > > > include
> > > > > > controller. So, prefixing all controller related configs with
> > > > > "controller"
> > > > > > may be more consistent. If we choose to do that, could we rename all new
> > > > > > configs here and in KIP-595 consistently?
> > > > > >
> > > > >
> > > > > I added the new names for the KIP-595 configurations to the configuration
> > > > > table. I think prefixing these configurations with "controller.quorum" is
> > > > > the best option since it makes it clear that they are related to the
> > > > > controller quorum.
> > > > >
> > > > > > 81. Metrics
> > > > > > 81.1 kafka.controller:type=KafkaController,name=MetadataSnapshotLag: This
> > > > > > is now redundant since KIP-630 already has
> > > > > > kafka.controller:type=KafkaController,name=SnapshotLag.
> > > > >
> > > > > Thanks for bringing up the metric name issue. I think it would be good to
> > > > > clarify that this is the lag for the metadata snapshot specifically. So I
> > > > > prefer the name MetadataSnapshotLag. I'll add a note to the table.
> > > > >
> > > > > > 81.2 Do we need both kafka.controller:type=KafkaServer,name=MetadataLag
> > > > > and
> > > > > > kafka.controller:type=KafkaController,name=MetadataLag since in the
> > > > > shared
> > > > > > controller mode, the metadata log is shared?
> > > > >
> > > > > Yes, both are needed. This metric is about what metadata logs we have
> > > > > applied, not about the Raft log's high water mark. If the broker or
> > > > > controller takes a while to apply the log messages (for example, because of
> > > > > lock contention or some other issue), they could fall significantly behind
> > > > > the high watermark.
> > > > >
> > > > > >
> > > > > > 82. Metadata records
> > > > > > 82.1 BrokerRecord: It needs to include supported features based on
> > > > > KIP-584.
> > > > >
> > > > > Added.
> > > > >
> > > > > > 82.2 FeatureLevel: The finalized feature in KIP-584 has the following
> > > > > > structure and needs to be reflected here.
> > > > > > {
> > > > > > "version": 0, // int32 -> Represents the version of the schema for the
> > > > > > data stored in the ZK node
> > > > > > "status": 1, // int32 -> Represents the status of the node
> > > > > > "features": {
> > > > > > "group_coordinator": { // string -> name of the feature
> > > > > > "min_version_level": 0, // int16 -> Represents the
> > > > > cluster-wide
> > > > > > finalized minimum version level (>=1) of this feature
> > > > > > "max_version_level": 3 // int16 -> Represents the
> > > > > cluster-wide
> > > > > > finalized maximum version level (>=1 and >= min_version_level) of this
> > > > > > feature
> > > > > > },
> > > > > > "consumer_offsets_topic_schema": {
> > > > > > "min_version_level": 0,
> > > > > > "max_version_level": 4
> > > > > > }
> > > > > > }
> > > > >
> > > > > Fair point. I will add both a minimum and maximum finalized feature level.
> > > > >
> > > > > >
> > > > > > 83. RPC
> > > > > > 83.1 BrokerHeartbeatRequest: Could you document when the controller
> > > > > assigns
> > > > > > a new broker epoch?
> > > > >
> > > > > The controller assigns a broker epoch if the heartbeat request was sent
> > > > > without an epoch (that is, it is in the INITIAL state), and the controller
> > > > > wants to add the new broker to the cluster. I added a note to the state
> > > > > machine.
> > > > >
> > > > > > 83.2 BrokerHeartbeatResponse: Do we need a new error code to indicate a
> > > > > > broker is fenced?
> > > > >
> > > > > I don't think we need a new error code, since targetState will just
> > > > > indicate that the broker should be fenced.
> > > > >
> > > > > > 83.3 What error/state indicates an unsuccessful controlled shutdown in
> > > > > > BrokerHeartbeatResponse? Currently, the controlled shutdown responses
> > > > > also
> > > > > > include a list of remaining partitions. That's mostly for debugging
> > > > > > purposes. Should we include it in BrokerHeartbeatResponse too?
> > > > >
> > > > > Once the controller wants the broker to shut down, it will set targetState
> > > > > to SHUTDOWN.
> > > > >
> > > > > We don't need to send a list of remaining partitions, since the broker is
> > > > > following the metadata log, and will get all its instructions from there.
> > > > >
> > > > > > 83.4 Could we document when Listeners, Features and Rack are expected to
> > > > > be
> > > > > > set in BrokerHeartbeatRequest?
> > > > >
> > > > > I added a line specifying that: The listeners, features, and rack fields
> > > > > only need to be set when the broker is in INITIAL state, and is requesting
> > > > > a new broker epoch.
> > > > >
> > > > > > 83.5 Currently, the metadata response only returns a list of brokers.
> > > > > > Should we have a separate field for the controller info or change the
> > > > > field
> > > > > > name to sth like nodes? Also, is the rack field relevant to the
> > > > > controller?
> > > > >
> > > > > As per KIP-590, we don't want the clients to communicate directly with the
> > > > > controller. So we weren't planning on including the controller's
> > > > > information in the MetadataResponse.
> > > > >
> > > > > We don't currently have rack-awareness for controllers, but I suppose we
> > > > > could in the future. I think we can leave that for later.
> > > > >
> > > > > >
> > > > > > 84. There are a few references to __kafka_metadata. However, KIP-595
> > > > > > uses __cluster_metadata.
> > > > >
> > > > > We agreed earlier to decide on the name of the metadata topic in this KIP,
> > > > > and to treat the KIP-595 name as provisional. Looking at it again, I think
> > > > > we should go with @metadata, in order to avoid colliding with topic names
> > > > > that already exist.
> > > > >
> > > > > >
> > > > > > 85. "When a broker is fenced, it cannot process any client requests. "
> > > > > > Should we send a new error code in the response to indicate that the
> > > > > broker
> > > > > > is fenced?
> > > > >
> > > > > I was hoping that we could avoid responding to requests when the broker
> > > > > was fenced. I suppose a new error code could work too, but then we also
> > > > > need a compatibility story for older clients.
> > > > >
> > > > > >
> > > > > > 86. "The controller can generate a new broker epoch by using the latest
> > > > > log
> > > > > > offset." Perhaps it's more accurate to say the new broker epoch is the
> > > > > > offset for the corresponding BrokerRecord in the metadata log.
> > > > >
> > > > > The idea here is slightly different. We want to use the current latest
> > > > > offset (prior to creating the BrokerRecord) as the new epoch. We can't use
> > > > > the offset of the BrokerRecord itself since we won't know what that is
> > > > > until we give the BrokerRecord to the Raft layer.
> > > > >
> > > > > >
> > > > > > 87. meta.properties:
> > > > > > 87.1 Could you document the format change to this file? Also, should we
> > > > > > bump up the version of the file?
> > > > >
> > > > > Hmm. The file is just a list of key=value pairs, one per line. So I'm
> > > > > not sure there is any format change per se, at least technically.
> > > > >
> > > > > Maybe we should bump up the version anyway, in order to prevent older
> > > > > pre-KIP-500 software from launching if the new stuff is present?
> > > > >
> > > > > > 87.2 How do we know a node is configured in KIP-500 mode?
> > > > > >
> > > > >
> > > > > A cluster configured in KIP-500 mode will contain the following line in
> > > > > the meta.properties file in each directory: kip.500.mode=enabled
> > > > >
> > > > > > 88. CurMetadataOffset: Should we use nextMetadataOffset since it's more
> > > > > > consistent with fetchOffset and HWM?
> > > > >
> > > > > I'm not sure. Drawing an analogy with HWM might be confusing, since
> > > > > CurMetadataOffset is not HWM. It's totally possible for the broker to have
> > > > > fetched up to offset 1000 but only applied up to offset 500.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 5, 2020 at 9:03 AM Colin McCabe <cm...@apache.org> wrote:
> > > > > >
> > > > > > > On Mon, Sep 28, 2020, at 11:41, Jun Rao wrote:
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the reply. A few more comments below.
> > > > > > > >
> > > > > > > > 62.
> > > > > > > > 62.1 controller.listener.names: So, is this used for the controller
> > > > > or
> > > > > > > the
> > > > > > > > broker trying to connect to the controller?
> > > > > > > >
> > > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > It's used by both. The broker tries to connect to controllers by using
> > > > > > > the first listener in this list. The controller uses the list to
> > > > > determine
> > > > > > > which listeners it should bind to.
> > > > > > >
> > > > > > > >
> > > > > > > > 62.2 If we want to take the approach to share the configs that are
> > > > > common
> > > > > > > > between the broker and the controller, should we share the id too?
> > > > > > > >
> > > > > > >
> > > > > > > On nodes where we are running both the controller and broker roles, we
> > > > > > > need two IDs: one for the controller, and one for the broker. So we
> > > > > can't
> > > > > > > share the same configuration.
> > > > > > >
> > > > > > > >
> > > > > > > > 62.3 We added some configs in KIP-595 prefixed with "quorum" and we
> > > > > plan
> > > > > > > to
> > > > > > > > add some controller specific configs prefixed with "controller".
> > > > > KIP-630
> > > > > > > > plans to add some other controller specific configs with no prefix.
> > > > > > > Should
> > > > > > > > we standardize all controller specific configs with the same prefix?
> > > > > > > >
> > > > > > >
> > > > > > > That's a good question. As Jose said elsewhere in the thread, let's
> > > > > use
> > > > > > > "metadata" as the prefix.
> > > > > > >
> > > > > > > >
> > > > > > > > 70. Could you explain the impact of process.roles a bit more? For
> > > > > > > example,
> > > > > > > > if process.roles=controller, does the node still maintain metadata
> > > > > cache
> > > > > > > as
> > > > > > > > described in KIP-630? Do we still return the host/port for those
> > > > > nodes in
> > > > > > > > the metadata response?
> > > > > > > >
> > > > > > >
> > > > > > > No, the node does not need any broker data structures if it is not a
> > > > > > > broker. Controllers also don't handle MetadataRequests. Clients will
> > > > > > > still go to the brokers for those.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Mon, Sep 28, 2020 at 9:24 AM Colin McCabe <cm...@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Fri, Sep 25, 2020, at 17:35, Jun Rao wrote:
> > > > > > > > > > Hi, Colin,
> > > > > > > > > >
> > > > > > > > > > Thanks for the reply.
> > > > > > > > > >
> > > > > > > > > > 62. Thinking about this more, I am wondering what's our overall
> > > > > > > strategy
> > > > > > > > > > for configs shared between the broker and the controller. For
> > > > > > > example,
> > > > > > > > > both
> > > > > > > > > > the broker and the controller have to define listeners. So both
> > > > > need
> > > > > > > > > > configs like listeners/advertised.listeners. Both the new
> > > > > controller
> > > > > > > and
> > > > > > > > > > the broker replicate data. So both need to define some
> > > > > replication
> > > > > > > > > related
> > > > > > > > > > configurations (replica.fetch.min.bytes,
> > > > > replica.fetch.wait.max.ms,
> > > > > > > > > etc).
> > > > > > > > > > Should we let the new controller share those configs with
> > > > > brokers or
> > > > > > > > > should
> > > > > > > > > > we define new configs just for the controller? It seems that we
> > > > > want
> > > > > > > to
> > > > > > > > > > apply the same strategy consistently for all shared configs?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > That's a good question. I think that we should share as many
> > > > > > > > > configurations as possible. There will be a few cases where this
> > > > > isn't
> > > > > > > > > possible, and we need to create a new configuration key that is
> > > > > > > > > controller-only, but I think that will be relatively rare.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 63. If a node has process.roles set to controller, does one still
> > > > > > > need to
> > > > > > > > > > set broker.id on this node?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > No, broker.id does not need to be set in that case.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 25, 2020 at 2:17 PM Colin McCabe <cmccabe@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > On Fri, Sep 25, 2020, at 10:17, Jun Rao wrote:
> > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the reply.
> > > > > > > > > > > >
> > > > > > > > > > > > 60. Yes, I think you are right. We probably need the
> > > > > controller
> > > > > > > id
> > > > > > > > > when a
> > > > > > > > > > > > broker starts up. A broker only stores the Raft leader id in
> > > > > the
> > > > > > > > > metadata
> > > > > > > > > > > > file. To do the initial fetch to the Raft leader, it needs to
> > > > > > > know
> > > > > > > > > the
> > > > > > > > > > > > host/port associated with the leader id.
> > > > > > > > > > > >
> > > > > > > > > > > > 62. It seems there are 2 parts to this : (1) which listener a
> > > > > > > client
> > > > > > > > > > > should
> > > > > > > > > > > > use to initiate a connection to the controller and (2) which
> > > > > > > listener
> > > > > > > > > > > > should a controller use to accept client requests. For (1),
> > > > > at
> > > > > > > any
> > > > > > > > > point
> > > > > > > > > > > of
> > > > > > > > > > > > time, a client only needs to use one listener. I think
> > > > > > > > > > > > controller.listener.name is meant for the client.
> > > > > > > > > > >
> > > > > > > > > > > Hi Jun,
> > > > > > > > > > >
> > > > > > > > > > > controller.listener.names is also used by the controllers. In
> > > > > the
> > > > > > > case
> > > > > > > > > > > where we have a broker and a controller in the same JVM, we
> > > > > have a
> > > > > > > > > single
> > > > > > > > > > > config file. Then we need to know which listeners belong to
> > > > > the
> > > > > > > > > controller
> > > > > > > > > > > and which belong to the broker component. That's why it's a
> > > > > list.
> > > > > > > > > > >
> > > > > > > > > > > > So, a single value seems
> > > > > > > > > > > > to make more sense. Currently, we don't have a configuration
> > > > > for
> > > > > > > > > (2). We
> > > > > > > > > > > > could add a new one for that and support a list. I am
> > > > > wondering
> > > > > > > how
> > > > > > > > > > > useful
> > > > > > > > > > > > it will be. One example that I can think of is that we can
> > > > > reject
> > > > > > > > > > > > non-controller related requests if accepted on
> > > > > controller-only
> > > > > > > > > listeners.
> > > > > > > > > > > > However, we already support separate authentication for the
> > > > > > > > > controller
> > > > > > > > > > > > listener. So, not sure how useful it is.
> > > > > > > > > > >
> > > > > > > > > > > The controller always has a separate listener and does not
> > > > > share
> > > > > > > > > listeners
> > > > > > > > > > > with the broker. The main reason for this is to avoid giving
> > > > > > > clients
> > > > > > > > > the
> > > > > > > > > > > ability to launch a denial-of-service attack on the controller
> > > > > by
> > > > > > > > > flooding
> > > > > > > > > > > a broker port. A lot of times, these attacks are made
> > > > > > > unintentionally
> > > > > > > > > by
> > > > > > > > > > > poorly coded or configured clients. Additionally, broker
> > > > > ports can
> > > > > > > > > also be
> > > > > > > > > > > very busy with large fetch requests, and so on. It's just a
> > > > > bad
> > > > > > > > > > > configuration in general to try to overload the same port for
> > > > > the
> > > > > > > > > > > controller and broker, and we don't want to allow it. It
> > > > > would be
> > > > > > > a
> > > > > > > > > > > regression to go from the current system where control
> > > > > requests are
> > > > > > > > > safely
> > > > > > > > > > > isolated on a separate port, to one where they're not. It also
> > > > > > > makes
> > > > > > > > > the
> > > > > > > > > > > code and configuration a lot messier.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 63. (a) I think most users won't know controller.id
> > > > > defaults to
> > > > > > > > > > > broker.id +
> > > > > > > > > > > > 3000. So, it can be confusing for them to set up
> > > > > > > controller.connect.
> > > > > > > > > If
> > > > > > > > > > > > this is truly needed, it seems that it's less confusing to
> > > > > make
> > > > > > > > > > > > controller.id required.
> > > > > > > > > > > > (b) I am still trying to understand if we truly need to
> > > > > expose a
> > > > > > > > > > > > controller.id. What if we only expose broker.id and let
> > > > > > > > > > > controller.connect
> > > > > > > > > > > > just use broker.id? What will be missing?
> > > > > > > > > > >
> > > > > > > > > > > The controller has a separate ID from the broker, so knowing
> > > > > > > broker.id
> > > > > > > > > is
> > > > > > > > > > > not helpful here.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jun
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 24, 2020 at 10:55 PM Colin McCabe <
> > > > > > > cmccabe@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 24, 2020, at 16:24, Jun Rao wrote:
> > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the reply and the updated KIP. A few more
> > > > > comments
> > > > > > > > > below.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 53. It seems that you already incorporated the changes in
> > > > > > > > > KIP-516.
> > > > > > > > > > > With
> > > > > > > > > > > > > > topic ids, we don't need to wait for the topic's data to
> > > > > be
> > > > > > > > > deleted
> > > > > > > > > > > > > before
> > > > > > > > > > > > > > removing the topic metadata. If the topic is recreated,
> > > > > we
> > > > > > > can
> > > > > > > > > still
> > > > > > > > > > > > > delete
> > > > > > > > > > > > > > the data properly based on the topic id. So, it seems
> > > > > that
> > > > > > > we can
> > > > > > > > > > > remove
> > > > > > > > > > > > > > TopicRecord.Deleting.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the reply. What I was thinking of doing here
> > > > > was
> > > > > > > using
> > > > > > > > > > > topic
> > > > > > > > > > > > > IDs internally, but still using names externally. So the
> > > > > topic
> > > > > > > > > UUIDs
> > > > > > > > > > > are
> > > > > > > > > > > > > only for the purpose of associating topics with partitions
> > > > > --
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > > > user's point of view, topics are still identified by names.
> > > > > > > > > > > > >
> > > > > > > > > > > > > You're right that KIP-516 will simplify things, but I'm not
> > > > > > > sure
> > > > > > > > > when
> > > > > > > > > > > that
> > > > > > > > > > > > > will land, so I wanted to avoid blocking the initial
> > > > > > > > > implementation of
> > > > > > > > > > > this
> > > > > > > > > > > > > KIP on it.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 55. It seems to me that the current behavior where we
> > > > > favor
> > > > > > > the
> > > > > > > > > > > current
> > > > > > > > > > > > > > broker registration is better. This is because
> > > > > uncontrolled
> > > > > > > > > broker
> > > > > > > > > > > > > shutdown
> > > > > > > > > > > > > > is rare and its impact is less severe since one just
> > > > > needs to
> > > > > > > > > wait
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > session timeout before restarting the broker. If a
> > > > > > > mis-configured
> > > > > > > > > > > broker
> > > > > > > > > > > > > > replaces an existing broker, the consequence is more
> > > > > severe
> > > > > > > > > since it
> > > > > > > > > > > can
> > > > > > > > > > > > > > cause the leader to be unavailable, a replica to be out
> > > > > of
> > > > > > > ISR,
> > > > > > > > > and
> > > > > > > > > > > add
> > > > > > > > > > > > > > more load on the leaders etc.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hmm, that's a good point. Let me check this a bit more
> > > > > before
> > > > > > > I
> > > > > > > > > change
> > > > > > > > > > > > > it, though.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 60. controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com
> > > > > :
> > > > > > > Do we
> > > > > > > > > > > need to
> > > > > > > > > > > > > > include the controller id before @? It seems that the
> > > > > > > host/port
> > > > > > > > > is
> > > > > > > > > > > enough
> > > > > > > > > > > > > > for establishing the connection. It would also be useful
> > > > > to
> > > > > > > make
> > > > > > > > > it
> > > > > > > > > > > clear
> > > > > > > > > > > > > > that controller.connect replaces quorum.voters in
> > > > > KIP-595.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I discussed this with Jason earlier, and he felt that the
> > > > > > > > > controller
> > > > > > > > > > > IDs
> > > > > > > > > > > > > were needed in this configuration key. It is certainly
> > > > > needed
> > > > > > > when
> > > > > > > > > > > > > configuring the controllers themselves, since they need to
> > > > > know
> > > > > > > > > each
> > > > > > > > > > > > > others' IDs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 61. I am not sure that we need both controller.listeners
> > > > > and
> > > > > > > > > > > > > > controller.connect.security.protocol since the former
> > > > > > > implies the
> > > > > > > > > > > > > security
> > > > > > > > > > > > > > protocol. The reason that we have both
> > > > > > > > > inter.broker.listener.name
> > > > > > > > > > > and
> > > > > > > > > > > > > > inter.broker.security.protocol is because we had the
> > > > > latter
> > > > > > > > > first and
> > > > > > > > > > > > > later
> > > > > > > > > > > > > > realized that the former is more general.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > That's a good point. I've removed this from the KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 62. I am still not sure that you need
> > > > > controller.listeners
> > > > > > > to be
> > > > > > > > > a
> > > > > > > > > > > list.
> > > > > > > > > > > > > > All listeners are already defined in listeners. To
> > > > > migrate
> > > > > > > from
> > > > > > > > > > > plaintext
> > > > > > > > > > > > > > to SSL, one can configure listeners to have both
> > > > > plaintext
> > > > > > > and
> > > > > > > > > SSL.
> > > > > > > > > > > After
> > > > > > > > > > > > > > that, one can just change controller.listeners from
> > > > > > > plaintext to
> > > > > > > > > SSL.
> > > > > > > > > > > > > This
> > > > > > > > > > > > > > is similar to how to change the listener for inter broker
> > > > > > > > > > > connections.
> > > > > > > > > > > > > > Also, controller.listener.name may be a more accurate
> > > > > name?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > The issue that I see here is that if you are running with
> > > > > the
> > > > > > > > > > > controller
> > > > > > > > > > > > > and broker in the same JVM, if you define a few listeners
> > > > > in
> > > > > > > > > > > "listeners"
> > > > > > > > > > > > > they will get used as regular broker listeners, unless you
> > > > > put
> > > > > > > > > them in
> > > > > > > > > > > > > controller.listeners. Therefore, controller.listeners
> > > > > needs
> > > > > > > to be
> > > > > > > > > a
> > > > > > > > > > > list.
> > > > > > > > > > > > >
> > > > > > > > > > > > > controller.listener.names does sound like a better name,
> > > > > > > though...
> > > > > > > > > I've
> > > > > > > > > > > > > updated it to that.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 63. Regarding controller.id, I am trying to understand
> > > > > > > whether
> > > > > > > > > it's
> > > > > > > > > > > a
> > > > > > > > > > > > > > required configuration or an optional one. From the KIP,
> > > > > it
> > > > > > > > > sounds
> > > > > > > > > > > like
> > > > > > > > > > > > > > controller.id is optional. Then, if this is unset, it's
> > > > > not
> > > > > > > > > clear
> > > > > > > > > > > how
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > user will obtain the controller id for setting
> > > > > > > > > controller.connect.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > If you specify broker.id but not controller.id, then
> > > > > > > controller.id
> > > > > > > > > > > will
> > > > > > > > > > > > > just be set to broker.id + 3000. This was intended to
> > > > > make
> > > > > > > some
> > > > > > > > > > > > > configurations a little bit more concise to write. You
> > > > > still
> > > > > > > do
> > > > > > > > > have
> > > > > > > > > > > to
> > > > > > > > > > > > > know the controller IDs when configuring the brokers,
> > > > > though.
> > > > > > > If
> > > > > > > > > this
> > > > > > > > > > > > > seems confusing then I can just make it mandatory.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 64. KIP-516 adds a flag in LeaderAndIsrRequest to
> > > > > indicate
> > > > > > > > > whether it
> > > > > > > > > > > > > > includes all partitions or not. This will be used to
> > > > > clean up
> > > > > > > > > strayed
> > > > > > > > > > > > > logs.
> > > > > > > > > > > > > > I was thinking how we will do the same thing once the
> > > > > > > controller
> > > > > > > > > > > moves to
> > > > > > > > > > > > > > Raft based. One way to do that is on broker startup, it
> > > > > gets
> > > > > > > the
> > > > > > > > > HWM
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > metadata log from the Raft leader and waits until its
> > > > > > > metadata
> > > > > > > > > > > catches up
> > > > > > > > > > > > > > to HWM. At that point, any local log not seen in the
> > > > > metadata
> > > > > > > > > can be
> > > > > > > > > > > > > > removed. Since the Fetch response returns the HWM, there
> > > > > > > seems
> > > > > > > > > to be
> > > > > > > > > > > > > enough
> > > > > > > > > > > > > > APIs to achieve this.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > That's a very good point. I added a note about this under
> > > > > > > Broker
> > > > > > > > > > > Startup.
> > > > > > > > > > > > >
> > > > > > > > > > > > > best,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 24, 2020 at 1:07 PM Colin McCabe <
> > > > > > > colin@cmccabe.xyz>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Sep 21, 2020, at 18:13, Jun Rao wrote:
> > > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Sorry for the late reply. A few more comments below.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for taking another look.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 50. Configurations
> > > > > > > > > > > > > > > > 50.1 controller.listeners: It seems that a controller
> > > > > > > just
> > > > > > > > > needs
> > > > > > > > > > > one
> > > > > > > > > > > > > > > > listener. Why do we need to have a list here? Also,
> > > > > > > could you
> > > > > > > > > > > > > provide an
> > > > > > > > > > > > > > > > example of how this is set and what's its
> > > > > relationship
> > > > > > > with
> > > > > > > > > > > existing
> > > > > > > > > > > > > > > > configs such as "security.inter.broker.protocol" and
> > > > > "
> > > > > > > > > > > > > > > > inter.broker.listener.name"?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I agree that most administrators will want to run with
> > > > > > > only one
> > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > listener. However, just as with brokers, it is nice to
> > > > > > > have
> > > > > > > > > the
> > > > > > > > > > > > > option to
> > > > > > > > > > > > > > > expose multiple ports.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > One reason why you might want multiple ports is if you
> > > > > were
> > > > > > > > > doing a
> > > > > > > > > > > > > > > migration from plaintext to SSL. You could add new SSL
> > > > > > > > > listeners
> > > > > > > > > > > but
> > > > > > > > > > > > > > > continue to expose the PLAINTEXT listeners. Then you
> > > > > could
> > > > > > > > > add the
> > > > > > > > > > > > > new SSL
> > > > > > > > > > > > > > > controller config to each broker and roll each broker.
> > > > > > > Then
> > > > > > > > > you
> > > > > > > > > > > could
> > > > > > > > > > > > > > > migrate from PLAINTEXT to SSL with no downtime.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Here's an example configuration for the controller:
> > > > > > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > > > > 1@controller1.example.com:9093,
> > > > > 2@controller2.example.com
> > > > > > > > > > > > > > > controller.listeners=CONTROLLER
> > > > > > > > > > > > > > > listeners=CONTROLLER://controller0.example.com:9093
> > > > > > > > > > > > > > > listener.security.protocol.map=CONTROLLER:SSL
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Here's an example configuration for the broker:
> > > > > > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > > > > 1@controller1.example.com:9093,
> > > > > 2@controller2.example.com
> > > > > > > > > > > > > > > controller.connect.security.protocol=SSL
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > security.inter.broker.protocol or
> > > > > > > inter.broker.listener.name
> > > > > > > > > do
> > > > > > > > > > > not
> > > > > > > > > > > > > > > affect how the broker communicates with the controller.
> > > > > > > Those
> > > > > > > > > > > > > > > configurations relate to how brokers communicate with
> > > > > each
> > > > > > > > > other,
> > > > > > > > > > > but
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > controller is not a broker.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > (Note that I just added
> > > > > > > controller.connect.security.protocol
> > > > > > > > > to the
> > > > > > > > > > > > > KIP --
> > > > > > > > > > > > > > > I had forgotten to put it in earlier)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 50.2 registration.heartbeat.interval.ms and
> > > > > > > > > > > > > > > registration.lease.timeout.ms.
> > > > > > > > > > > > > > > > Should we match their default value with the
> > > > > > > corresponding
> > > > > > > > > > > default
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > ZK?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Fair enough. I'll set them to the values of the
> > > > > > > > > > > > > zookeeper.sync.time.ms
> > > > > > > > > > > > > > > and zookeeper.connection.timeout.ms configurations.
> > > > > I do
> > > > > > > > > think we
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > > experiment later on to see what works well here, but
> > > > > the ZK
> > > > > > > > > values
> > > > > > > > > > > are
> > > > > > > > > > > > > at
> > > > > > > > > > > > > > > least a starting point.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 50.3 controller.connect: Could you provide an
> > > > > example? I
> > > > > > > am
> > > > > > > > > > > > > wondering how
> > > > > > > > > > > > > > > > it differs from quorum.voters=1@kafka-1:9092,
> > > > > 2@kafka-2
> > > > > > > > > :9092,
> > > > > > > > > > > > > 3@kafka-3
> > > > > > > > > > > > > > > > :9092.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > controller.connect is intended to be the new name for
> > > > > > > > > > > quorum.voters.
> > > > > > > > > > > > > > > During the vote for KIP-595, we sort of agreed to
> > > > > defer the
> > > > > > > > > > > discussion
> > > > > > > > > > > > > > > about what this configuration should be called. I
> > > > > proposed
> > > > > > > > > this
> > > > > > > > > > > new
> > > > > > > > > > > > > name
> > > > > > > > > > > > > > > because it makes it clear what the configuration is for
> > > > > > > (how to
> > > > > > > > > > > > > connect to
> > > > > > > > > > > > > > > the controllers).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 50.4 controller.id: I am still not sure how this is
> > > > > > > being
> > > > > > > > > used.
> > > > > > > > > > > > > Could
> > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > explain this in more detail?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The controller ID needs to be set on each controller.
> > > > > It
> > > > > > > also
> > > > > > > > > > > appears
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > controller.connect as the thing before the "at" sign.
> > > > > Its
> > > > > > > > > > > function is
> > > > > > > > > > > > > > > pretty similar to broker.id.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Broker IDs and controller IDs exist in the same ID
> > > > > space,
> > > > > > > so
> > > > > > > > > you
> > > > > > > > > > > can't
> > > > > > > > > > > > > > > have a broker and a controller that use the same ID.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 51. BrokerHeartbeat: It seems a bit wasteful to
> > > > > include
> > > > > > > > > > > Listeners in
> > > > > > > > > > > > > > > every
> > > > > > > > > > > > > > > > heartbeat request since it typically doesn't change.
> > > > > > > Could we
> > > > > > > > > > > make
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > optional field?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Ok.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 52. KIP-584 adds a new ZK node /features. Should we
> > > > > add a
> > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > metadata record?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Good point. I added FeatureLevelRecord.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 53. TopicRecord and DeleteTopic: Both DeleteTopic and
> > > > > > > > > > > > > > > TopicRecord.Deleting
> > > > > > > > > > > > > > > > indicate topic deletion. Could we outline the flow
> > > > > when
> > > > > > > each
> > > > > > > > > > > will be
> > > > > > > > > > > > > set?
> > > > > > > > > > > > > > > > In particular, which one indicates the intention to
> > > > > > > delete
> > > > > > > > > and
> > > > > > > > > > > which
> > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > indicates the completion of the deletion.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > TopicRecord.Deleting is set when we intend to delete
> > > > > the
> > > > > > > topic.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > DeleteTopic removes the topic completely. I will
> > > > > rename
> > > > > > > > > > > DeleteTopic to
> > > > > > > > > > > > > > > RemoveTopic to make this clearer.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 54. "The controller can generate a new broker epoch
> > > > > by
> > > > > > > using
> > > > > > > > > the
> > > > > > > > > > > > > latest
> > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > offset." Which offset is that? Is it the offset of
> > > > > the
> > > > > > > > > metadata
> > > > > > > > > > > > > topic for
> > > > > > > > > > > > > > > > the corresponding BrokerRecord? Is it guaranteed to
> > > > > be
> > > > > > > > > different
> > > > > > > > > > > on
> > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > broker restart?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Yes.... a new broker epoch implies that there has been
> > > > > a
> > > > > > > new
> > > > > > > > > record
> > > > > > > > > > > > > > > created in the metadata log. Therefore the last
> > > > > committed
> > > > > > > > > offset
> > > > > > > > > > > must
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > different.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 55. "Thereafter, it may lose subsequent conflicts if
> > > > > its
> > > > > > > > > broker
> > > > > > > > > > > > > epoch is
> > > > > > > > > > > > > > > > stale. (See KIP-380 for some background on broker
> > > > > > > epoch.)
> > > > > > > > > The
> > > > > > > > > > > > > reason
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > favoring new processes is to accommodate the common
> > > > > case
> > > > > > > > > where a
> > > > > > > > > > > > > process
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > killed with kill -9 and then restarted. " Are you
> > > > > saying
> > > > > > > > > that if
> > > > > > > > > > > > > there is
> > > > > > > > > > > > > > > > an active broker registered in the controller, a new
> > > > > > > broker
> > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > request with the INITIAL state will fence the current
> > > > > > > broker
> > > > > > > > > > > > > session? Not
> > > > > > > > > > > > > > > > sure about this. For example, this will allow a
> > > > > broker
> > > > > > > with
> > > > > > > > > > > > > incorrectly
> > > > > > > > > > > > > > > > configured broker id to kill an existing valid
> > > > > broker.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Yes, a new broker with an incorrectly configured
> > > > > broker id
> > > > > > > > > could
> > > > > > > > > > > fence
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > existing valid broker.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The goal of the conflict resolution process described
> > > > > here
> > > > > > > is
> > > > > > > > > to
> > > > > > > > > > > avoid
> > > > > > > > > > > > > > > having two brokers go back and forth in claiming the
> > > > > > > broker id.
> > > > > > > > > > > > > Choosing
> > > > > > > > > > > > > > > the newest was just an easy way to do that.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Choosing the oldest is another possibility, and more
> > > > > inline
> > > > > > > > > with
> > > > > > > > > > > what
> > > > > > > > > > > > > > > ZooKeeper does today. The problem with that is that it
> > > > > > > would
> > > > > > > > > not
> > > > > > > > > > > be
> > > > > > > > > > > > > > > possible to re-create the broker if the old instance
> > > > > died
> > > > > > > > > suddenly,
> > > > > > > > > > > > > except
> > > > > > > > > > > > > > > by waiting for the full lease timeout.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It might be possible to do a little better here by
> > > > > > > breaking the
> > > > > > > > > > > lease
> > > > > > > > > > > > > if
> > > > > > > > > > > > > > > the TCP connection from the broker drops. But that
> > > > > > > requires
> > > > > > > > > > > crossing
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > > layers of abstraction in the Kafka networking stack.
> > > > > > > There's
> > > > > > > > > also
> > > > > > > > > > > the
> > > > > > > > > > > > > > > possibility of false negatives or positives. For
> > > > > example,
> > > > > > > TCP
> > > > > > > > > > > > > connections
> > > > > > > > > > > > > > > sometimes just drop even if the node is still there.
> > > > > Or
> > > > > > > > > sometimes
> > > > > > > > > > > the
> > > > > > > > > > > > > > > networking stack keeps a TCP connection alive for a
> > > > > while
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > > > > other
> > > > > > > > > > > > > > > end is gone.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 56. kafka-storage.sh:
> > > > > > > > > > > > > > > > 56.1 In the info mode, what other information does it
> > > > > > > show in
> > > > > > > > > > > > > addition to
> > > > > > > > > > > > > > > > "kip.500.mode=enabled"?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think it should show a list of all the storage
> > > > > > > directories,
> > > > > > > > > and
> > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > each one is formatted. In addition, it should show
> > > > > whether
> > > > > > > > > > > > > kip.500.mode is
> > > > > > > > > > > > > > > enabled, and what the cluster id is.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 56.2 Should the format mode take the config file as
> > > > > the
> > > > > > > > > input too
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > info mode?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Yes, that's a good idea.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 17, 2020 at 4:12 PM Colin McCabe <
> > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > That's a fair point. I have moved the lease
> > > > > duration
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > heartbeat response. That way lease durations can
> > > > > be
> > > > > > > > > changed
> > > > > > > > > > > just
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > reconfiguring the controllers.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Wed, Sep 16, 2020, at 07:40, Unmesh Joshi wrote:
> > > > > > > > > > > > > > > > > > Thanks Colin, the changes look good to me. One
> > > > > small
> > > > > > > > > thing.
> > > > > > > > > > > > > > > > > > registration.lease.timeout.ms is the
> > > > > configuration
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > side.
> > > > > > > > > > > > > > > > > > It will be good to comment on how brokers know
> > > > > about
> > > > > > > it,
> > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > > able
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > send LeaseDurationMs
> > > > > > > > > > > > > > > > > > in the heartbeat request,
> > > > > > > > > > > > > > > > > > or else it can be added in the heartbeat
> > > > > response for
> > > > > > > > > > > brokers to
> > > > > > > > > > > > > know
> > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Sep 11, 2020 at 10:32 PM Colin McCabe <
> > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think you're right that we should use a
> > > > > duration
> > > > > > > here
> > > > > > > > > > > rather
> > > > > > > > > > > > > > > than a
> > > > > > > > > > > > > > > > > > > time. As you said, the clock on the controller
> > > > > > > will
> > > > > > > > > > > probably
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > match the
> > > > > > > > > > > > > > > > > > > one on the broker. I have updated the KIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > It's important to keep in mind that
> > > > > messages
> > > > > > > may be
> > > > > > > > > > > > > delayed in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > > > > > happens, we
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > > > > time specified in the request to determine
> > > > > if
> > > > > > > the
> > > > > > > > > > > request
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > > > > > I am assuming that there will be a single TCP
> > > > > > > > > connection
> > > > > > > > > > > > > > > maintained
> > > > > > > > > > > > > > > > > > > between
> > > > > > > > > > > > > > > > > > > > broker and active controller. So, there
> > > > > won't be
> > > > > > > any
> > > > > > > > > out
> > > > > > > > > > > of
> > > > > > > > > > > > > order
> > > > > > > > > > > > > > > > > > > requests?
> > > > > > > > > > > > > > > > > > > > There will be a scenario of broker GC pause,
> > > > > > > which
> > > > > > > > > might
> > > > > > > > > > > > > cause
> > > > > > > > > > > > > > > > > connection
> > > > > > > > > > > > > > > > > > > > timeout and broker might need to reestablish
> > > > > the
> > > > > > > > > > > connection.
> > > > > > > > > > > > > If
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > pause
> > > > > > > > > > > > > > > > > > > > is too long, lease will expire and the
> > > > > heartbeat
> > > > > > > sent
> > > > > > > > > > > after
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > pause
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > be treated as a new registration (similar to
> > > > > > > restart
> > > > > > > > > > > case),
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > > > > epoch
> > > > > > > > > > > > > > > > > > > > number will be assigned to the broker.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I agree with the end of this paragraph, but not
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > start
> > > > > > > > > > > > > :)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There can be out-of-order requests, since the
> > > > > > > broker
> > > > > > > > > will
> > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > use a
> > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > TCP connection if the old one has problems.
> > > > > This
> > > > > > > can
> > > > > > > > > > > happen
> > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > > variety
> > > > > > > > > > > > > > > > > > > of reasons. I don't think GC pauses are the
> > > > > most
> > > > > > > > > common
> > > > > > > > > > > > > reason for
> > > > > > > > > > > > > > > > > this to
> > > > > > > > > > > > > > > > > > > happen. It's more common to see issues issues
> > > > > in
> > > > > > > the
> > > > > > > > > > > network
> > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > result connections getting dropped from time to
> > > > > > > time.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > So we have to assume that messages may arrive
> > > > > out
> > > > > > > of
> > > > > > > > > > > order, and
> > > > > > > > > > > > > > > > > possibly
> > > > > > > > > > > > > > > > > > > be delayed. I added a note that heartbeat
> > > > > requests
> > > > > > > > > should
> > > > > > > > > > > be
> > > > > > > > > > > > > > > ignored
> > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > the metadata log offset they contain is smaller
> > > > > > > than a
> > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > heartbeat.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > When the active controller fails, the new
> > > > > active
> > > > > > > > > > > controller
> > > > > > > > > > > > > > > needs to
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > sure that it considers all the known brokers
> > > > > as
> > > > > > > alive
> > > > > > > > > > > till
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > lease
> > > > > > > > > > > > > > > > > > > > expiration interval. Because
> > > > > > > > > > > registration.lease.timeout.ms,
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > configured
> > > > > > > > > > > > > > > > > > > > on the controller, the new active controller
> > > > > will
> > > > > > > > > extend
> > > > > > > > > > > all
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > leases
> > > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > registration.lease.timeout.ms. I see that it
> > > > > > > won't
> > > > > > > > > need
> > > > > > > > > > > last
> > > > > > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Agreed.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Sat, Sep 5, 2020 at 1:28 AM Colin McCabe <
> > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Colin wrote:
> > > > > > > > > > > > > > > > > > > > > > > The reason for including
> > > > > LeaseStartTimeMs
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > request
> > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > > > > > > that the time required to communicate
> > > > > with
> > > > > > > the
> > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > > > included in
> > > > > > > > > > > > > > > > > > > > > > > the lease time. Since requests can
> > > > > > > > > potentially be
> > > > > > > > > > > > > delayed
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > > > > > > for a long time, this is important.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020, at 05:58, Unmesh
> > > > > Joshi
> > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > The network time will be added anyway,
> > > > > > > because
> > > > > > > > > the
> > > > > > > > > > > lease
> > > > > > > > > > > > > > > timer
> > > > > > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > > > > > active controller will start only after
> > > > > the
> > > > > > > > > heartbeat
> > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > > reaches the
> > > > > > > > > > > > > > > > > > > > > > server.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > If the start time is not specified in the
> > > > > > > request,
> > > > > > > > > > > then the
> > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > > > > > is excluded from the heartbeat time.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Here's an example:
> > > > > > > > > > > > > > > > > > > > > Let's say broker A sends a heartbeat at
> > > > > time
> > > > > > > 100,
> > > > > > > > > and
> > > > > > > > > > > it
> > > > > > > > > > > > > > > arrives
> > > > > > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > > > > controller at time 200, and the lease
> > > > > duration
> > > > > > > is
> > > > > > > > > 1000.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The controller looks at the start time in
> > > > > the
> > > > > > > > > request,
> > > > > > > > > > > > > which is
> > > > > > > > > > > > > > > > > 100,
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > adds 1000 to it, getting 1100. On the
> > > > > other
> > > > > > > hand,
> > > > > > > > > if
> > > > > > > > > > > start
> > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > > > > > specified in the request, then the
> > > > > expiration
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > at
> > > > > > > > > > > > > time
> > > > > > > > > > > > > > > 1200.
> > > > > > > > > > > > > > > > > > > > > That is what I mean by "the network time is
> > > > > > > > > included
> > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > expiration
> > > > > > > > > > > > > > > > > > > > > time."
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > And I think, some assumption about
> > > > > network
> > > > > > > round
> > > > > > > > > trip
> > > > > > > > > > > > > time is
> > > > > > > > > > > > > > > > > > > > > > needed anyway to decide on the frequency
> > > > > of
> > > > > > > the
> > > > > > > > > > > > > heartbeat (
> > > > > > > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms), and
> > > > > > > lease
> > > > > > > > > > > timeout (
> > > > > > > > > > > > > > > > > > > > > > registration.lease.timeout.ms). So I
> > > > > think
> > > > > > > just
> > > > > > > > > > > having a
> > > > > > > > > > > > > > > > > leaseTTL
> > > > > > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > > > > > request is easier to understand and
> > > > > > > implement.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > It's important to keep in mind that
> > > > > messages
> > > > > > > may be
> > > > > > > > > > > > > delayed in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > > > > > happens, we
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > > > > time specified in the request to determine
> > > > > if
> > > > > > > the
> > > > > > > > > > > request
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> > > > > the
> > > > > > > > > > > controller
> > > > > > > > > > > > > side
> > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > reset in the case of controller
> > > > > failover.
> > > > > > > The
> > > > > > > > > > > > > alternative
> > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > be to
> > > > > > > > > > > > > > > > > > > > > > > track the lease as hard state rather
> > > > > than
> > > > > > > soft
> > > > > > > > > > > state,
> > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > is not really needed, and would result
> > > > > in
> > > > > > > more
> > > > > > > > > log
> > > > > > > > > > > > > entries.
> > > > > > > > > > > > > > > > > > > > > > My interpretation of the mention of
> > > > > > > BrokerRecord
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > KIP
> > > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > record exists in the Raft log.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > BrokerRecord does exist in the Raft log,
> > > > > but
> > > > > > > does
> > > > > > > > > not
> > > > > > > > > > > > > include
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > > > > heartbeat time.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > By soft state, do you mean the broker
> > > > > > > > > > > > > > > > > > > > > > records exist only on the active leader
> > > > > and
> > > > > > > will
> > > > > > > > > not
> > > > > > > > > > > be
> > > > > > > > > > > > > > > > > replicated
> > > > > > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > > > > > raft log? If the live brokers list is
> > > > > > > maintained
> > > > > > > > > > > only on
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > > > controller (raft leader), then, in case
> > > > > of
> > > > > > > leader
> > > > > > > > > > > > > failure,
> > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > will be
> > > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > window where the new leader does not know
> > > > > > > about
> > > > > > > > > the
> > > > > > > > > > > live
> > > > > > > > > > > > > > > brokers,
> > > > > > > > > > > > > > > > > > > till
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > brokers establish the leases again.
> > > > > > > > > > > > > > > > > > > > > > I think it will be safer to have leases
> > > > > as a
> > > > > > > hard
> > > > > > > > > > > state
> > > > > > > > > > > > > > > managed
> > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > standard
> > > > > > > > > > > > > > > > > > > > > > Raft replication.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Leases are short, so the need to
> > > > > re-establish
> > > > > > > them
> > > > > > > > > > > after a
> > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > failover doesn't seem like a big problem.
> > > > > But
> > > > > > > > > this is
> > > > > > > > > > > > > > > something
> > > > > > > > > > > > > > > > > we can
> > > > > > > > > > > > > > > > > > > > > tweak if it becomes an issue. One option
> > > > > > > would be
> > > > > > > > > to
> > > > > > > > > > > have
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > > > > which is only used by the controller nodes
> > > > > for
> > > > > > > this
> > > > > > > > > > > (since,
> > > > > > > > > > > > > > > after
> > > > > > > > > > > > > > > > > all,
> > > > > > > > > > > > > > > > > > > > > brokers don't care about registration
> > > > > > > renewals).
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Or am I misunderstanding something? (I
> > > > > assume
> > > > > > > > > that
> > > > > > > > > > > with
> > > > > > > > > > > > > soft
> > > > > > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > > mean something like zookeeper local
> > > > > sessions
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1147
> > > > > > > > > > > .)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > > > > > think
> > > > > > > > > it
> > > > > > > > > > > makes
> > > > > > > > > > > > > > > sense
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> > > > > becomes
> > > > > > > > > very
> > > > > > > > > > > messy.
> > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > > > > > understand your question about
> > > > > duplicate
> > > > > > > > > broker ID
> > > > > > > > > > > > > > > detection,
> > > > > > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > > > > There's a section in the KIP about this
> > > > > -- is
> > > > > > > > > there a
> > > > > > > > > > > > > detail
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This is an implementation detail that
> > > > > doesn't
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I assumed broker leases are implemented
> > > > > as a
> > > > > > > hard
> > > > > > > > > > > state.
> > > > > > > > > > > > > In
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > case, to
> > > > > > > > > > > > > > > > > > > > > > check for broker id conflict, we need to
> > > > > > > check
> > > > > > > > > the
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > ids at
> > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > > places
> > > > > > > > > > > > > > > > > > > > > > 1. Pending broker registrations (which
> > > > > are
> > > > > > > yet
> > > > > > > > > to be
> > > > > > > > > > > > > > > committed)
> > > > > > > > > > > > > > > > > 2.
> > > > > > > > > > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > > > > > > > > committed broker registrations.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020 at 5:42 PM Colin
> > > > > McCabe
> > > > > > > <
> > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020, at 01:12, Unmesh
> > > > > > > Joshi
> > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >>>Can you repeat your questions
> > > > > about
> > > > > > > broker
> > > > > > > > > > > leases?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >>>>The LeaseStartTimeMs is expected
> > > > > to
> > > > > > > be
> > > > > > > > > the
> > > > > > > > > > > > > broker's
> > > > > > > > > > > > > > > > > > > > > > > > 'System.currentTimeMillis()' at the
> > > > > > > point of
> > > > > > > > > the
> > > > > > > > > > > > > > > request. The
> > > > > > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > > > > > controller will add its lease period
> > > > > to
> > > > > > > this
> > > > > > > > > in
> > > > > > > > > > > order
> > > > > > > > > > > > > > > >>>>to
> > > > > > > > > > > > > > > > > > > compute
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > LeaseEndTimeMs.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think the use of LeaseStartTimeMs
> > > > > and
> > > > > > > > > > > > > LeaseEndTimeMs
> > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > > > > confusing. Monotonic Clock
> > > > > > > > > (System.nanoTime) on
> > > > > > > > > > > the
> > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > should be used to track leases.
> > > > > > > > > > > > > > > > > > > > > > > > (For example,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1616https://github.com/etcd-io/etcd/pull/6888/commits/e7f4010ccaf28b6ce64fe514d25a4b2fa459d114
> > > > > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Then we will not need
> > > > > LeaseStartTimeMs?
> > > > > > > > > > > > > > > > > > > > > > > > Instead of LeaseStartTimeMs, can we
> > > > > call
> > > > > > > it
> > > > > > > > > > > > > LeaseTTL? The
> > > > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > can then calculate LeaseEndTime =
> > > > > > > > > > > System.nanoTime() +
> > > > > > > > > > > > > > > > > LeaseTTL.
> > > > > > > > > > > > > > > > > > > > > > > > In this case we might just drop
> > > > > > > > > LeaseEndTimeMs
> > > > > > > > > > > from
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > response, as
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > broker already knows about the TTL
> > > > > and
> > > > > > > can
> > > > > > > > > send
> > > > > > > > > > > > > > > heartbeats at
> > > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > > > fraction
> > > > > > > > > > > > > > > > > > > > > > > > of TTL, say every TTL/4
> > > > > > > milliseconds.(elapsed
> > > > > > > > > > > time
> > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > measured
> > > > > > > > > > > > > > > > > > > > > > > > by System.nanoTime)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I agree that the monotonic clock is
> > > > > > > probably a
> > > > > > > > > > > better
> > > > > > > > > > > > > idea
> > > > > > > > > > > > > > > > > here.
> > > > > > > > > > > > > > > > > > > It is
> > > > > > > > > > > > > > > > > > > > > > > good to be robust against wall clock
> > > > > > > changes,
> > > > > > > > > > > although
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > > > > > > > which had them might suffer other
> > > > > issues.
> > > > > > > I
> > > > > > > > > will
> > > > > > > > > > > > > change
> > > > > > > > > > > > > > > it to
> > > > > > > > > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > monotonic clock.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The reason for including
> > > > > LeaseStartTimeMs
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > request
> > > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > the time required to communicate with
> > > > > the
> > > > > > > > > > > controller
> > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > included
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > lease time. Since requests can
> > > > > > > potentially be
> > > > > > > > > > > delayed
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > > > > > > > > long time, this is important.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I have a prototype built to
> > > > > demonstrate
> > > > > > > this
> > > > > > > > > as
> > > > > > > > > > > > > > > following:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The Kip631Controller itself depends
> > > > > on a
> > > > > > > > > > > Consensus
> > > > > > > > > > > > > > > module, to
> > > > > > > > > > > > > > > > > > > > > demonstrate
> > > > > > > > > > > > > > > > > > > > > > > > how possible interactions with the
> > > > > > > consensus
> > > > > > > > > > > module
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > look
> > > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > > > > (The Consensus can be pluggable
> > > > > really,
> > > > > > > > > with an
> > > > > > > > > > > API
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > > > > > replicated log upto HighWaterMark)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > It has an implementation of
> > > > > LeaseTracker
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/LeaderLeaseTracker.scala
> > > > > > > > > > > > > > > > > > > > > > > > to demonstrate LeaseTracker's
> > > > > interaction
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > > consensus
> > > > > > > > > > > > > > > > > > > module.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The implementation has the following
> > > > > > > aspects:
> > > > > > > > > > > > > > > > > > > > > > > > 1. The lease tracking happens only
> > > > > on the
> > > > > > > > > active
> > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > (raft
> > > > > > > > > > > > > > > > > > > > > > > > leader)
> > > > > > > > > > > > > > > > > > > > > > > > 2. Once the lease expires, it needs
> > > > > to
> > > > > > > > > propose
> > > > > > > > > > > and
> > > > > > > > > > > > > > > commit a
> > > > > > > > > > > > > > > > > > > > > FenceBroker
> > > > > > > > > > > > > > > > > > > > > > > > record for that lease.
> > > > > > > > > > > > > > > > > > > > > > > > 3. In case of active controller
> > > > > failure,
> > > > > > > the
> > > > > > > > > > > lease
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > tracked by
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > newly raft leader. The new raft
> > > > > leader
> > > > > > > > > starts the
> > > > > > > > > > > > > lease
> > > > > > > > > > > > > > > timer
> > > > > > > > > > > > > > > > > > > again,
> > > > > > > > > > > > > > > > > > > > > (as
> > > > > > > > > > > > > > > > > > > > > > > > implemented in onBecomingLeader
> > > > > method of
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > > > > > in effect extending the lease by the
> > > > > time
> > > > > > > > > spent
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > > > election
> > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > whatever time was elapsed on the old
> > > > > > > leader.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> > > > > the
> > > > > > > > > > > controller
> > > > > > > > > > > > > side
> > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > reset
> > > > > > > > > > > > > > > > > > > > > > > in the case of controller failover.
> > > > > The
> > > > > > > > > > > alternative
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > track
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > lease as hard state rather than soft
> > > > > state,
> > > > > > > > > but I
> > > > > > > > > > > think
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > > > > > > > > needed, and would result in more log
> > > > > > > entries.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > There are working tests for this
> > > > > > > > > implementation
> > > > > > > > > > > here.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > > > > > > and an end to end test here
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/ProducerConsumerKIP500Test.scala
> > > > > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >>'m not sure what you mean by
> > > > > > > > > "de-duplication
> > > > > > > > > > > of the
> > > > > > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > > > > > Apologies for using the confusing
> > > > > term
> > > > > > > > > > > > > deduplication. I
> > > > > > > > > > > > > > > meant
> > > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > > id
> > > > > > > > > > > > > > > > > > > > > > > > conflict.
> > > > > > > > > > > > > > > > > > > > > > > > As you can see in the prototype
> > > > > > > > > handleRequest of
> > > > > > > > > > > > > > > > > KIP631Controller
> > > > > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > > > > > the duplicate broker id needs to be
> > > > > > > detected
> > > > > > > > > > > before
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > BrokerRecord
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > submitted to the raft module.
> > > > > > > > > > > > > > > > > > > > > > > > Also as implemented in the
> > > > > prototype, the
> > > > > > > > > > > > > > > KIP631Controller is
> > > > > > > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > > > > > > > > threaded, handling requests one at a
> > > > > > > time.
> > > > > > > > > (an
> > > > > > > > > > > > > example of
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > https://martinfowler.com/articles/patterns-of-distributed-systems/singular-update-queue.html
> > > > > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > > > > > think
> > > > > > > > > it
> > > > > > > > > > > makes
> > > > > > > > > > > > > > > sense
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> > > > > becomes
> > > > > > > > > very
> > > > > > > > > > > messy.
> > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > > > > > understand your question about
> > > > > duplicate
> > > > > > > > > broker ID
> > > > > > > > > > > > > > > detection,
> > > > > > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > > > > > There's a section in the KIP about
> > > > > this --
> > > > > > > is
> > > > > > > > > > > there a
> > > > > > > > > > > > > > > detail we
> > > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 10:49 AM
> > > > > Colin
> > > > > > > > > McCabe <
> > > > > > > > > > > > > > > > > colin@cmccabe.xyz
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2020, at 19:36,
> > > > > Unmesh
> > > > > > > > > Joshi
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > There were a few of questions I
> > > > > had..
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the response.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. Were my comments on the broker
> > > > > > > lease
> > > > > > > > > > > > > > > implementation
> > > > > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > > > > prototype) appropriate and do we
> > > > > > > need to
> > > > > > > > > > > change
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > KIP
> > > > > > > > > > > > > > > > > > > > > > > > > > description accordingly?.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Can you repeat your questions about
> > > > > > > broker
> > > > > > > > > > > leases?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. How will broker epochs be
> > > > > > > generated?
> > > > > > > > > I am
> > > > > > > > > > > > > > > assuming it
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > committed log offset (like zxid?)
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > There isn't any need to use a log
> > > > > > > offset.
> > > > > > > > > We
> > > > > > > > > > > can
> > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > look at
> > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > > > in-memory hash table and see what
> > > > > the
> > > > > > > > > latest
> > > > > > > > > > > > > number is,
> > > > > > > > > > > > > > > > > and add
> > > > > > > > > > > > > > > > > > > > > one, to
> > > > > > > > > > > > > > > > > > > > > > > > > generate a new broker epoch.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 3. How will producer registration
> > > > > > > > > happen? I
> > > > > > > > > > > am
> > > > > > > > > > > > > > > assuming
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > should be
> > > > > > > > > > > > > > > > > > > > > > > > > > similar to broker registration,
> > > > > with
> > > > > > > a
> > > > > > > > > > > similar
> > > > > > > > > > > > > way to
> > > > > > > > > > > > > > > > > > > generate
> > > > > > > > > > > > > > > > > > > > > > > producer
> > > > > > > > > > > > > > > > > > > > > > > > > id.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > For the EOS stuff, we will need a
> > > > > few
> > > > > > > new
> > > > > > > > > RPCs
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > controller.
> > > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > > > > we should do that in a follow-on
> > > > > KIP,
> > > > > > > > > though,
> > > > > > > > > > > since
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > one is
> > > > > > > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > > > > > > > pretty big.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 4. Because we expose Raft log to
> > > > > all
> > > > > > > the
> > > > > > > > > > > > > brokers, any
> > > > > > > > > > > > > > > > > > > > > de-duplication
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > broker needs to happen before the
> > > > > > > > > requests
> > > > > > > > > > > are
> > > > > > > > > > > > > > > proposed
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > Raft.
> > > > > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > > > the controller needs to be single
> > > > > > > > > threaded,
> > > > > > > > > > > and
> > > > > > > > > > > > > > > should do
> > > > > > > > > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > > > > > > > against the in-process or pending
> > > > > > > > > requests
> > > > > > > > > > > and
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > final
> > > > > > > > > > > > > > > > > > > state. I
> > > > > > > > > > > > > > > > > > > > > > > read a
> > > > > > > > > > > > > > > > > > > > > > > > > > mention of this, in the
> > > > > responses in
> > > > > > > this
> > > > > > > > > > > > > > > thread.Will it
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > useful to
> > > > > > > > > > > > > > > > > > > > > > > > > > mention this in the KIP?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I'm not sure what you mean by
> > > > > > > > > "de-duplication
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 4:50 AM
> > > > > Colin
> > > > > > > > > McCabe
> > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I'm thinking of calling a vote
> > > > > on
> > > > > > > > > KIP-631
> > > > > > > > > > > on
> > > > > > > > > > > > > > > Monday.
> > > > > > > > > > > > > > > > > Let
> > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > know
> > > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > > > > there's any more comments I
> > > > > should
> > > > > > > > > address
> > > > > > > > > > > > > before I
> > > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > vote.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020, at 05:39,
> > > > > > > Unmesh
> > > > > > > > > > > Joshi
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks. I will be adding
> > > > > more to
> > > > > > > the
> > > > > > > > > > > > > prototype
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > > > > > > > and collaborate.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020 at
> > > > > 12:28 AM
> > > > > > > > > Colin
> > > > > > > > > > > > > McCabe <
> > > > > > > > > > > > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Jose,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > That'a s good point that I
> > > > > > > hadn't
> > > > > > > > > > > > > considered.
> > > > > > > > > > > > > > > It's
> > > > > > > > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > a separate leader change
> > > > > > > message,
> > > > > > > > > as
> > > > > > > > > > > you
> > > > > > > > > > > > > > > mentioned.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2020, at
> > > > > 11:56,
> > > > > > > Jose
> > > > > > > > > > > Garcia
> > > > > > > > > > > > > > > Sancio
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Very cool prototype!
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The KIP proposes a record
> > > > > > > called
> > > > > > > > > > > > > IsrChange
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > > > includes the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > partition, topic, isr,
> > > > > > > leader and
> > > > > > > > > > > leader
> > > > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > > > > During
> > > > > > > > > > > > > > > > > > > > > normal
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > operation ISR changes do
> > > > > not
> > > > > > > > > result
> > > > > > > > > > > in
> > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > > > > > > Similarly,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > leader changes do not
> > > > > > > necessarily
> > > > > > > > > > > > > involve ISR
> > > > > > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > implementation that uses
> > > > > ZK
> > > > > > > > > modeled
> > > > > > > > > > > them
> > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. All of this
> > > > > information is
> > > > > > > > > stored
> > > > > > > > > > > in
> > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > > znode.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. ZK's optimistic lock
> > > > > > > requires
> > > > > > > > > > > that you
> > > > > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > > > > the new
> > > > > > > > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. The change to that
> > > > > znode
> > > > > > > was
> > > > > > > > > being
> > > > > > > > > > > > > > > performed
> > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > and the leader.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > None of these reasons are
> > > > > > > true in
> > > > > > > > > > > > > KIP-500.
> > > > > > > > > > > > > > > Have
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > considered
> > > > > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > two different records?
> > > > > For
> > > > > > > > > example
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. IsrChange record which
> > > > > > > > > includes
> > > > > > > > > > > topic,
> > > > > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > > > > isr
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. LeaderChange record
> > > > > which
> > > > > > > > > includes
> > > > > > > > > > > > > topic,
> > > > > > > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > leader epoch.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I suspect that making
> > > > > this
> > > > > > > change
> > > > > > > > > > > will
> > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > message AlterIsrRequest
> > > > > > > > > introduced in
> > > > > > > > > > > > > > > KIP-497:
> > > > > > > > > > > > > > > > > Add
> > > > > > > > > > > > > > > > > > > > > > > inter-broker
> > > > > > > > > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > alter ISR.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Jose
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Ron Dagostino <rn...@gmail.com>.
Hi again, Colin. Related to the issue of how to communicate fenced
vs. not fenced (and how to communicate or imply Offline vs not
Offline), do we need to communicate in the log that a broker has
gracefully shut down? We do distinguish between a broker being
unavailable due to a controlled shutdown vs. having died -- one that
dies appears in the metadata as being Offline, whereas one that
gracefully shuts down does not. I don't see a way to imply this
gracefully-shutdown state, though I may be missing it.
Ron
On Thu, Oct 22, 2020 at 1:32 PM Ron Dagostino <rn...@gmail.com> wrote:
>
> HI Colin. A FencedBrokerRecord appears in the metadata log when a
> broker is fenced. What appears in the metadata log to indicate that a
> broker is no longer fenced? Does a BrokerRecord appear? That seems
> to communicate a bunch of unnecessary data in this context (endpoints,
> features, rack). If that is the current plan, might it be better to
> include a Boolean value within FencedBrokerRecord and specify true
> when the broker becomes fenced and false when it is no longer fenced?
>
> Also, is a fenced broker considered Offline for partition metadata
> purposes? I think so but would like to confirm.
>
> Ron
>
> On Wed, Oct 21, 2020 at 9:05 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > On Tue, Oct 13, 2020, at 18:30, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply. A few more comments below.
> > >
> > > 80.1 controller.listener.names only defines the name of the listener. The
> > > actual listener including host/port/security_protocol is typically defined
> > > in advertised_listners. Does that mean advertised_listners is a required
> > > config now?
> > >
> >
> > Hi Jun,
> >
> > Thanks for the re-review.
> >
> > The controller listeners are not advertised to clients. So I think they should go in listeners, rather than advertised.listeners.
> >
> > I agree that this makes listeners a required configuration. At very least, it is required to have the controller listener in there.
> >
> > >
> > > 83.1 broker state machine: It seems that we should transition from FENCED
> > > => INITIAL since only INITIAL generates new broker epoch?
> > >
> >
> > I changed the broker state machine a bit-- take a look. In the new state machine, the FENCED state can re-register.
> >
> > >
> > > 83.5. It's true that the controller node doesn't serve metadata requests.
> > > However, there are admin requests such as topic creation/deletion are sent
> > > to the controller directly. So, it seems that the client needs to know
> > > the controller host/port?
> > >
> >
> > The client sends admin requests to a random broker, which forwards them to the controller. This is described in KIP-590.
> >
> > >
> > > 85. "I was hoping that we could avoid responding to requests when the
> > > broker was fenced." This issue is that if we don't send a response, the
> > > client won't know the reason and can't act properly.
> > >
> >
> > Sorry, when I said "does not respond" I meant it would return an error.
> >
> > >
> > > 88. CurMetadataOffset: I was thinking that we may want to
> > > use CurMetadataOffset to compute the MetadataLag. Since HWM is exclusive,
> > > it's more convenient if CurMetadataOffset is also exclusive.
> > >
> >
> > OK, I will change it to NextMetadataOffset and make it exclusive.
> >
> > > 90. It would be useful to add a rejected section on why separate controller
> > > and broker id is preferred over just broker id. For example, the following
> > > are some potential reasons. (a) We can guard duplicated brokerID, but it's
> > > hard to guard against duplicated controllerId. (b) brokerID can be auto
> > > assigned in the future, but controllerId is hard to be generated
> > > automatically.
> > >
> >
> > OK. I added a rejected alternatives section about sharing IDs between multiple nodes.
> >
> > Colin
> >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Oct 12, 2020 at 11:14 AM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Tue, Oct 6, 2020, at 16:09, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the reply. Made another pass of the KIP. A few more comments
> > > > > below.
> > > > >
> > > >
> > > > Hi Jun,
> > > >
> > > > Thanks for the review.
> > > >
> > > > > 55. We discussed earlier why the current behavior where we favor the
> > > > > current broker registration is better. Have you given this more thought?
> > > > >
> > > >
> > > > Yes, I think we should favor the current broker registration, as you
> > > > suggested earlier.
> > > >
> > > > > 80. Config related.
> > > > > 80.1 Currently, each broker only has the following 3 required configs. It
> > > > > will be useful to document the required configs post KIP-500 (in both the
> > > > > dedicated and shared controller mode).
> > > > > broker.id
> > > > > log.dirs
> > > > > zookeeper.connect
> > > >
> > > > For the broker, these configs will be required:
> > > >
> > > > broker.id
> > > > log.dirs
> > > > process.roles
> > > > controller.listener.names
> > > > controller.connect
> > > >
> > > > For the controller, these configs will be required:
> > > >
> > > > controller.id
> > > > log.dirs
> > > > process.roles
> > > > controller.listener.names
> > > > controller.connect
> > > >
> > > > For broker+controller, it will be the union of these two, which
> > > > essentially means we need both broker.id and controller.id, but all
> > > > others are the same as standalone.
> > > >
> > > > > 80.2 It would be useful to document all deprecated configs post KIP-500.
> > > > > For example, all zookeeper.* are obviously deprecated. But there could be
> > > > > others. For example, since we don't plan to support auto broker id
> > > > > generation, it seems broker.id.generation.enable is deprecated too.
> > > > > 80.3 Could we make it clear that controller.connect replaces
> > > > quorum.voters
> > > > > in KIP-595?
> > > >
> > > > OK. I added a comment about this in the table.
> > > >
> > > > > 80.4 Could we document that broker.id is now optional?
> > > >
> > > > OK. I have added a line for broker.id.
> > > >
> > > > > 80.5 The KIP suggests that controller.id is optional on the controller
> > > > > node. I am concerned that this can cause a bit of confusion in 2 aspects.
> > > > > First, in the dedicated controller mode, controller.id is not optional
> > > > > (since broker.id is now optional). Second, in the shared controller
> > > > mode,
> > > > > it may not be easy for the user to figure out the default value of
> > > > > controller.id to set controller.connect properly.
> > > >
> > > > I got rid of the default value of controller.id. I agree it was a bit
> > > > confusing to explain.
> > > >
> > > > > 80.6 Regarding the consistency of config names, our metrics already
> > > > include
> > > > > controller. So, prefixing all controller related configs with
> > > > "controller"
> > > > > may be more consistent. If we choose to do that, could we rename all new
> > > > > configs here and in KIP-595 consistently?
> > > > >
> > > >
> > > > I added the new names for the KIP-595 configurations to the configuration
> > > > table. I think prefixing these configurations with "controller.quorum" is
> > > > the best option since it makes it clear that they are related to the
> > > > controller quorum.
> > > >
> > > > > 81. Metrics
> > > > > 81.1 kafka.controller:type=KafkaController,name=MetadataSnapshotLag: This
> > > > > is now redundant since KIP-630 already has
> > > > > kafka.controller:type=KafkaController,name=SnapshotLag.
> > > >
> > > > Thanks for bringing up the metric name issue. I think it would be good to
> > > > clarify that this is the lag for the metadata snapshot specifically. So I
> > > > prefer the name MetadataSnapshotLag. I'll add a note to the table.
> > > >
> > > > > 81.2 Do we need both kafka.controller:type=KafkaServer,name=MetadataLag
> > > > and
> > > > > kafka.controller:type=KafkaController,name=MetadataLag since in the
> > > > shared
> > > > > controller mode, the metadata log is shared?
> > > >
> > > > Yes, both are needed. This metric is about what metadata logs we have
> > > > applied, not about the Raft log's high water mark. If the broker or
> > > > controller takes a while to apply the log messages (for example, because of
> > > > lock contention or some other issue), they could fall significantly behind
> > > > the high watermark.
> > > >
> > > > >
> > > > > 82. Metadata records
> > > > > 82.1 BrokerRecord: It needs to include supported features based on
> > > > KIP-584.
> > > >
> > > > Added.
> > > >
> > > > > 82.2 FeatureLevel: The finalized feature in KIP-584 has the following
> > > > > structure and needs to be reflected here.
> > > > > {
> > > > > "version": 0, // int32 -> Represents the version of the schema for the
> > > > > data stored in the ZK node
> > > > > "status": 1, // int32 -> Represents the status of the node
> > > > > "features": {
> > > > > "group_coordinator": { // string -> name of the feature
> > > > > "min_version_level": 0, // int16 -> Represents the
> > > > cluster-wide
> > > > > finalized minimum version level (>=1) of this feature
> > > > > "max_version_level": 3 // int16 -> Represents the
> > > > cluster-wide
> > > > > finalized maximum version level (>=1 and >= min_version_level) of this
> > > > > feature
> > > > > },
> > > > > "consumer_offsets_topic_schema": {
> > > > > "min_version_level": 0,
> > > > > "max_version_level": 4
> > > > > }
> > > > > }
> > > >
> > > > Fair point. I will add both a minimum and maximum finalized feature level.
> > > >
> > > > >
> > > > > 83. RPC
> > > > > 83.1 BrokerHeartbeatRequest: Could you document when the controller
> > > > assigns
> > > > > a new broker epoch?
> > > >
> > > > The controller assigns a broker epoch if the heartbeat request was sent
> > > > without an epoch (that is, it is in the INITIAL state), and the controller
> > > > wants to add the new broker to the cluster. I added a note to the state
> > > > machine.
> > > >
> > > > > 83.2 BrokerHeartbeatResponse: Do we need a new error code to indicate a
> > > > > broker is fenced?
> > > >
> > > > I don't think we need a new error code, since targetState will just
> > > > indicate that the broker should be fenced.
> > > >
> > > > > 83.3 What error/state indicates an unsuccessful controlled shutdown in
> > > > > BrokerHeartbeatResponse? Currently, the controlled shutdown responses
> > > > also
> > > > > include a list of remaining partitions. That's mostly for debugging
> > > > > purposes. Should we include it in BrokerHeartbeatResponse too?
> > > >
> > > > Once the controller wants the broker to shut down, it will set targetState
> > > > to SHUTDOWN.
> > > >
> > > > We don't need to send a list of remaining partitions, since the broker is
> > > > following the metadata log, and will get all its instructions from there.
> > > >
> > > > > 83.4 Could we document when Listeners, Features and Rack are expected to
> > > > be
> > > > > set in BrokerHeartbeatRequest?
> > > >
> > > > I added a line specifying that: The listeners, features, and rack fields
> > > > only need to be set when the broker is in INITIAL state, and is requesting
> > > > a new broker epoch.
> > > >
> > > > > 83.5 Currently, the metadata response only returns a list of brokers.
> > > > > Should we have a separate field for the controller info or change the
> > > > field
> > > > > name to sth like nodes? Also, is the rack field relevant to the
> > > > controller?
> > > >
> > > > As per KIP-590, we don't want the clients to communicate directly with the
> > > > controller. So we weren't planning on including the controller's
> > > > information in the MetadataResponse.
> > > >
> > > > We don't currently have rack-awareness for controllers, but I suppose we
> > > > could in the future. I think we can leave that for later.
> > > >
> > > > >
> > > > > 84. There are a few references to __kafka_metadata. However, KIP-595
> > > > > uses __cluster_metadata.
> > > >
> > > > We agreed earlier to decide on the name of the metadata topic in this KIP,
> > > > and to treat the KIP-595 name as provisional. Looking at it again, I think
> > > > we should go with @metadata, in order to avoid colliding with topic names
> > > > that already exist.
> > > >
> > > > >
> > > > > 85. "When a broker is fenced, it cannot process any client requests. "
> > > > > Should we send a new error code in the response to indicate that the
> > > > broker
> > > > > is fenced?
> > > >
> > > > I was hoping that we could avoid responding to requests when the broker
> > > > was fenced. I suppose a new error code could work too, but then we also
> > > > need a compatibility story for older clients.
> > > >
> > > > >
> > > > > 86. "The controller can generate a new broker epoch by using the latest
> > > > log
> > > > > offset." Perhaps it's more accurate to say the new broker epoch is the
> > > > > offset for the corresponding BrokerRecord in the metadata log.
> > > >
> > > > The idea here is slightly different. We want to use the current latest
> > > > offset (prior to creating the BrokerRecord) as the new epoch. We can't use
> > > > the offset of the BrokerRecord itself since we won't know what that is
> > > > until we give the BrokerRecord to the Raft layer.
> > > >
> > > > >
> > > > > 87. meta.properties:
> > > > > 87.1 Could you document the format change to this file? Also, should we
> > > > > bump up the version of the file?
> > > >
> > > > Hmm. The file is just a list of key=value pairs, one per line. So I'm
> > > > not sure there is any format change per se, at least technically.
> > > >
> > > > Maybe we should bump up the version anyway, in order to prevent older
> > > > pre-KIP-500 software from launching if the new stuff is present?
> > > >
> > > > > 87.2 How do we know a node is configured in KIP-500 mode?
> > > > >
> > > >
> > > > A cluster configured in KIP-500 mode will contain the following line in
> > > > the meta.properties file in each directory: kip.500.mode=enabled
> > > >
> > > > > 88. CurMetadataOffset: Should we use nextMetadataOffset since it's more
> > > > > consistent with fetchOffset and HWM?
> > > >
> > > > I'm not sure. Drawing an analogy with HWM might be confusing, since
> > > > CurMetadataOffset is not HWM. It's totally possible for the broker to have
> > > > fetched up to offset 1000 but only applied up to offset 500.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Mon, Oct 5, 2020 at 9:03 AM Colin McCabe <cm...@apache.org> wrote:
> > > > >
> > > > > > On Mon, Sep 28, 2020, at 11:41, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the reply. A few more comments below.
> > > > > > >
> > > > > > > 62.
> > > > > > > 62.1 controller.listener.names: So, is this used for the controller
> > > > or
> > > > > > the
> > > > > > > broker trying to connect to the controller?
> > > > > > >
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > It's used by both. The broker tries to connect to controllers by using
> > > > > > the first listener in this list. The controller uses the list to
> > > > determine
> > > > > > which listeners it should bind to.
> > > > > >
> > > > > > >
> > > > > > > 62.2 If we want to take the approach to share the configs that are
> > > > common
> > > > > > > between the broker and the controller, should we share the id too?
> > > > > > >
> > > > > >
> > > > > > On nodes where we are running both the controller and broker roles, we
> > > > > > need two IDs: one for the controller, and one for the broker. So we
> > > > can't
> > > > > > share the same configuration.
> > > > > >
> > > > > > >
> > > > > > > 62.3 We added some configs in KIP-595 prefixed with "quorum" and we
> > > > plan
> > > > > > to
> > > > > > > add some controller specific configs prefixed with "controller".
> > > > KIP-630
> > > > > > > plans to add some other controller specific configs with no prefix.
> > > > > > Should
> > > > > > > we standardize all controller specific configs with the same prefix?
> > > > > > >
> > > > > >
> > > > > > That's a good question. As Jose said elsewhere in the thread, let's
> > > > use
> > > > > > "metadata" as the prefix.
> > > > > >
> > > > > > >
> > > > > > > 70. Could you explain the impact of process.roles a bit more? For
> > > > > > example,
> > > > > > > if process.roles=controller, does the node still maintain metadata
> > > > cache
> > > > > > as
> > > > > > > described in KIP-630? Do we still return the host/port for those
> > > > nodes in
> > > > > > > the metadata response?
> > > > > > >
> > > > > >
> > > > > > No, the node does not need any broker data structures if it is not a
> > > > > > broker. Controllers also don't handle MetadataRequests. Clients will
> > > > > > still go to the brokers for those.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Mon, Sep 28, 2020 at 9:24 AM Colin McCabe <cm...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > On Fri, Sep 25, 2020, at 17:35, Jun Rao wrote:
> > > > > > > > > Hi, Colin,
> > > > > > > > >
> > > > > > > > > Thanks for the reply.
> > > > > > > > >
> > > > > > > > > 62. Thinking about this more, I am wondering what's our overall
> > > > > > strategy
> > > > > > > > > for configs shared between the broker and the controller. For
> > > > > > example,
> > > > > > > > both
> > > > > > > > > the broker and the controller have to define listeners. So both
> > > > need
> > > > > > > > > configs like listeners/advertised.listeners. Both the new
> > > > controller
> > > > > > and
> > > > > > > > > the broker replicate data. So both need to define some
> > > > replication
> > > > > > > > related
> > > > > > > > > configurations (replica.fetch.min.bytes,
> > > > replica.fetch.wait.max.ms,
> > > > > > > > etc).
> > > > > > > > > Should we let the new controller share those configs with
> > > > brokers or
> > > > > > > > should
> > > > > > > > > we define new configs just for the controller? It seems that we
> > > > want
> > > > > > to
> > > > > > > > > apply the same strategy consistently for all shared configs?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > That's a good question. I think that we should share as many
> > > > > > > > configurations as possible. There will be a few cases where this
> > > > isn't
> > > > > > > > possible, and we need to create a new configuration key that is
> > > > > > > > controller-only, but I think that will be relatively rare.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 63. If a node has process.roles set to controller, does one still
> > > > > > need to
> > > > > > > > > set broker.id on this node?
> > > > > > > > >
> > > > > > > >
> > > > > > > > No, broker.id does not need to be set in that case.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Sep 25, 2020 at 2:17 PM Colin McCabe <cmccabe@apache.org
> > > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Fri, Sep 25, 2020, at 10:17, Jun Rao wrote:
> > > > > > > > > > > Hi, Colin,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the reply.
> > > > > > > > > > >
> > > > > > > > > > > 60. Yes, I think you are right. We probably need the
> > > > controller
> > > > > > id
> > > > > > > > when a
> > > > > > > > > > > broker starts up. A broker only stores the Raft leader id in
> > > > the
> > > > > > > > metadata
> > > > > > > > > > > file. To do the initial fetch to the Raft leader, it needs to
> > > > > > know
> > > > > > > > the
> > > > > > > > > > > host/port associated with the leader id.
> > > > > > > > > > >
> > > > > > > > > > > 62. It seems there are 2 parts to this : (1) which listener a
> > > > > > client
> > > > > > > > > > should
> > > > > > > > > > > use to initiate a connection to the controller and (2) which
> > > > > > listener
> > > > > > > > > > > should a controller use to accept client requests. For (1),
> > > > at
> > > > > > any
> > > > > > > > point
> > > > > > > > > > of
> > > > > > > > > > > time, a client only needs to use one listener. I think
> > > > > > > > > > > controller.listener.name is meant for the client.
> > > > > > > > > >
> > > > > > > > > > Hi Jun,
> > > > > > > > > >
> > > > > > > > > > controller.listener.names is also used by the controllers. In
> > > > the
> > > > > > case
> > > > > > > > > > where we have a broker and a controller in the same JVM, we
> > > > have a
> > > > > > > > single
> > > > > > > > > > config file. Then we need to know which listeners belong to
> > > > the
> > > > > > > > controller
> > > > > > > > > > and which belong to the broker component. That's why it's a
> > > > list.
> > > > > > > > > >
> > > > > > > > > > > So, a single value seems
> > > > > > > > > > > to make more sense. Currently, we don't have a configuration
> > > > for
> > > > > > > > (2). We
> > > > > > > > > > > could add a new one for that and support a list. I am
> > > > wondering
> > > > > > how
> > > > > > > > > > useful
> > > > > > > > > > > it will be. One example that I can think of is that we can
> > > > reject
> > > > > > > > > > > non-controller related requests if accepted on
> > > > controller-only
> > > > > > > > listeners.
> > > > > > > > > > > However, we already support separate authentication for the
> > > > > > > > controller
> > > > > > > > > > > listener. So, not sure how useful it is.
> > > > > > > > > >
> > > > > > > > > > The controller always has a separate listener and does not
> > > > share
> > > > > > > > listeners
> > > > > > > > > > with the broker. The main reason for this is to avoid giving
> > > > > > clients
> > > > > > > > the
> > > > > > > > > > ability to launch a denial-of-service attack on the controller
> > > > by
> > > > > > > > flooding
> > > > > > > > > > a broker port. A lot of times, these attacks are made
> > > > > > unintentionally
> > > > > > > > by
> > > > > > > > > > poorly coded or configured clients. Additionally, broker
> > > > ports can
> > > > > > > > also be
> > > > > > > > > > very busy with large fetch requests, and so on. It's just a
> > > > bad
> > > > > > > > > > configuration in general to try to overload the same port for
> > > > the
> > > > > > > > > > controller and broker, and we don't want to allow it. It
> > > > would be
> > > > > > a
> > > > > > > > > > regression to go from the current system where control
> > > > requests are
> > > > > > > > safely
> > > > > > > > > > isolated on a separate port, to one where they're not. It also
> > > > > > makes
> > > > > > > > the
> > > > > > > > > > code and configuration a lot messier.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 63. (a) I think most users won't know controller.id
> > > > defaults to
> > > > > > > > > > broker.id +
> > > > > > > > > > > 3000. So, it can be confusing for them to set up
> > > > > > controller.connect.
> > > > > > > > If
> > > > > > > > > > > this is truly needed, it seems that it's less confusing to
> > > > make
> > > > > > > > > > > controller.id required.
> > > > > > > > > > > (b) I am still trying to understand if we truly need to
> > > > expose a
> > > > > > > > > > > controller.id. What if we only expose broker.id and let
> > > > > > > > > > controller.connect
> > > > > > > > > > > just use broker.id? What will be missing?
> > > > > > > > > >
> > > > > > > > > > The controller has a separate ID from the broker, so knowing
> > > > > > broker.id
> > > > > > > > is
> > > > > > > > > > not helpful here.
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 24, 2020 at 10:55 PM Colin McCabe <
> > > > > > cmccabe@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 24, 2020, at 16:24, Jun Rao wrote:
> > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the reply and the updated KIP. A few more
> > > > comments
> > > > > > > > below.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 53. It seems that you already incorporated the changes in
> > > > > > > > KIP-516.
> > > > > > > > > > With
> > > > > > > > > > > > > topic ids, we don't need to wait for the topic's data to
> > > > be
> > > > > > > > deleted
> > > > > > > > > > > > before
> > > > > > > > > > > > > removing the topic metadata. If the topic is recreated,
> > > > we
> > > > > > can
> > > > > > > > still
> > > > > > > > > > > > delete
> > > > > > > > > > > > > the data properly based on the topic id. So, it seems
> > > > that
> > > > > > we can
> > > > > > > > > > remove
> > > > > > > > > > > > > TopicRecord.Deleting.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the reply. What I was thinking of doing here
> > > > was
> > > > > > using
> > > > > > > > > > topic
> > > > > > > > > > > > IDs internally, but still using names externally. So the
> > > > topic
> > > > > > > > UUIDs
> > > > > > > > > > are
> > > > > > > > > > > > only for the purpose of associating topics with partitions
> > > > --
> > > > > > from
> > > > > > > > the
> > > > > > > > > > > > user's point of view, topics are still identified by names.
> > > > > > > > > > > >
> > > > > > > > > > > > You're right that KIP-516 will simplify things, but I'm not
> > > > > > sure
> > > > > > > > when
> > > > > > > > > > that
> > > > > > > > > > > > will land, so I wanted to avoid blocking the initial
> > > > > > > > implementation of
> > > > > > > > > > this
> > > > > > > > > > > > KIP on it.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 55. It seems to me that the current behavior where we
> > > > favor
> > > > > > the
> > > > > > > > > > current
> > > > > > > > > > > > > broker registration is better. This is because
> > > > uncontrolled
> > > > > > > > broker
> > > > > > > > > > > > shutdown
> > > > > > > > > > > > > is rare and its impact is less severe since one just
> > > > needs to
> > > > > > > > wait
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > session timeout before restarting the broker. If a
> > > > > > mis-configured
> > > > > > > > > > broker
> > > > > > > > > > > > > replaces an existing broker, the consequence is more
> > > > severe
> > > > > > > > since it
> > > > > > > > > > can
> > > > > > > > > > > > > cause the leader to be unavailable, a replica to be out
> > > > of
> > > > > > ISR,
> > > > > > > > and
> > > > > > > > > > add
> > > > > > > > > > > > > more load on the leaders etc.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hmm, that's a good point. Let me check this a bit more
> > > > before
> > > > > > I
> > > > > > > > change
> > > > > > > > > > > > it, though.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 60. controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com
> > > > :
> > > > > > Do we
> > > > > > > > > > need to
> > > > > > > > > > > > > include the controller id before @? It seems that the
> > > > > > host/port
> > > > > > > > is
> > > > > > > > > > enough
> > > > > > > > > > > > > for establishing the connection. It would also be useful
> > > > to
> > > > > > make
> > > > > > > > it
> > > > > > > > > > clear
> > > > > > > > > > > > > that controller.connect replaces quorum.voters in
> > > > KIP-595.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I discussed this with Jason earlier, and he felt that the
> > > > > > > > controller
> > > > > > > > > > IDs
> > > > > > > > > > > > were needed in this configuration key. It is certainly
> > > > needed
> > > > > > when
> > > > > > > > > > > > configuring the controllers themselves, since they need to
> > > > know
> > > > > > > > each
> > > > > > > > > > > > others' IDs.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 61. I am not sure that we need both controller.listeners
> > > > and
> > > > > > > > > > > > > controller.connect.security.protocol since the former
> > > > > > implies the
> > > > > > > > > > > > security
> > > > > > > > > > > > > protocol. The reason that we have both
> > > > > > > > inter.broker.listener.name
> > > > > > > > > > and
> > > > > > > > > > > > > inter.broker.security.protocol is because we had the
> > > > latter
> > > > > > > > first and
> > > > > > > > > > > > later
> > > > > > > > > > > > > realized that the former is more general.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > That's a good point. I've removed this from the KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 62. I am still not sure that you need
> > > > controller.listeners
> > > > > > to be
> > > > > > > > a
> > > > > > > > > > list.
> > > > > > > > > > > > > All listeners are already defined in listeners. To
> > > > migrate
> > > > > > from
> > > > > > > > > > plaintext
> > > > > > > > > > > > > to SSL, one can configure listeners to have both
> > > > plaintext
> > > > > > and
> > > > > > > > SSL.
> > > > > > > > > > After
> > > > > > > > > > > > > that, one can just change controller.listeners from
> > > > > > plaintext to
> > > > > > > > SSL.
> > > > > > > > > > > > This
> > > > > > > > > > > > > is similar to how to change the listener for inter broker
> > > > > > > > > > connections.
> > > > > > > > > > > > > Also, controller.listener.name may be a more accurate
> > > > name?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > The issue that I see here is that if you are running with
> > > > the
> > > > > > > > > > controller
> > > > > > > > > > > > and broker in the same JVM, if you define a few listeners
> > > > in
> > > > > > > > > > "listeners"
> > > > > > > > > > > > they will get used as regular broker listeners, unless you
> > > > put
> > > > > > > > them in
> > > > > > > > > > > > controller.listeners. Therefore, controller.listeners
> > > > needs
> > > > > > to be
> > > > > > > > a
> > > > > > > > > > list.
> > > > > > > > > > > >
> > > > > > > > > > > > controller.listener.names does sound like a better name,
> > > > > > though...
> > > > > > > > I've
> > > > > > > > > > > > updated it to that.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 63. Regarding controller.id, I am trying to understand
> > > > > > whether
> > > > > > > > it's
> > > > > > > > > > a
> > > > > > > > > > > > > required configuration or an optional one. From the KIP,
> > > > it
> > > > > > > > sounds
> > > > > > > > > > like
> > > > > > > > > > > > > controller.id is optional. Then, if this is unset, it's
> > > > not
> > > > > > > > clear
> > > > > > > > > > how
> > > > > > > > > > > > the
> > > > > > > > > > > > > user will obtain the controller id for setting
> > > > > > > > controller.connect.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > If you specify broker.id but not controller.id, then
> > > > > > controller.id
> > > > > > > > > > will
> > > > > > > > > > > > just be set to broker.id + 3000. This was intended to
> > > > make
> > > > > > some
> > > > > > > > > > > > configurations a little bit more concise to write. You
> > > > still
> > > > > > do
> > > > > > > > have
> > > > > > > > > > to
> > > > > > > > > > > > know the controller IDs when configuring the brokers,
> > > > though.
> > > > > > If
> > > > > > > > this
> > > > > > > > > > > > seems confusing then I can just make it mandatory.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 64. KIP-516 adds a flag in LeaderAndIsrRequest to
> > > > indicate
> > > > > > > > whether it
> > > > > > > > > > > > > includes all partitions or not. This will be used to
> > > > clean up
> > > > > > > > strayed
> > > > > > > > > > > > logs.
> > > > > > > > > > > > > I was thinking how we will do the same thing once the
> > > > > > controller
> > > > > > > > > > moves to
> > > > > > > > > > > > > Raft based. One way to do that is on broker startup, it
> > > > gets
> > > > > > the
> > > > > > > > HWM
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > metadata log from the Raft leader and waits until its
> > > > > > metadata
> > > > > > > > > > catches up
> > > > > > > > > > > > > to HWM. At that point, any local log not seen in the
> > > > metadata
> > > > > > > > can be
> > > > > > > > > > > > > removed. Since the Fetch response returns the HWM, there
> > > > > > seems
> > > > > > > > to be
> > > > > > > > > > > > enough
> > > > > > > > > > > > > APIs to achieve this.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > That's a very good point. I added a note about this under
> > > > > > Broker
> > > > > > > > > > Startup.
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jun
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 24, 2020 at 1:07 PM Colin McCabe <
> > > > > > colin@cmccabe.xyz>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Sep 21, 2020, at 18:13, Jun Rao wrote:
> > > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Sorry for the late reply. A few more comments below.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for taking another look.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 50. Configurations
> > > > > > > > > > > > > > > 50.1 controller.listeners: It seems that a controller
> > > > > > just
> > > > > > > > needs
> > > > > > > > > > one
> > > > > > > > > > > > > > > listener. Why do we need to have a list here? Also,
> > > > > > could you
> > > > > > > > > > > > provide an
> > > > > > > > > > > > > > > example of how this is set and what's its
> > > > relationship
> > > > > > with
> > > > > > > > > > existing
> > > > > > > > > > > > > > > configs such as "security.inter.broker.protocol" and
> > > > "
> > > > > > > > > > > > > > > inter.broker.listener.name"?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I agree that most administrators will want to run with
> > > > > > only one
> > > > > > > > > > > > controller
> > > > > > > > > > > > > > listener. However, just as with brokers, it is nice to
> > > > > > have
> > > > > > > > the
> > > > > > > > > > > > option to
> > > > > > > > > > > > > > expose multiple ports.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > One reason why you might want multiple ports is if you
> > > > were
> > > > > > > > doing a
> > > > > > > > > > > > > > migration from plaintext to SSL. You could add new SSL
> > > > > > > > listeners
> > > > > > > > > > but
> > > > > > > > > > > > > > continue to expose the PLAINTEXT listeners. Then you
> > > > could
> > > > > > > > add the
> > > > > > > > > > > > new SSL
> > > > > > > > > > > > > > controller config to each broker and roll each broker.
> > > > > > Then
> > > > > > > > you
> > > > > > > > > > could
> > > > > > > > > > > > > > migrate from PLAINTEXT to SSL with no downtime.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Here's an example configuration for the controller:
> > > > > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > > > 1@controller1.example.com:9093,
> > > > 2@controller2.example.com
> > > > > > > > > > > > > > controller.listeners=CONTROLLER
> > > > > > > > > > > > > > listeners=CONTROLLER://controller0.example.com:9093
> > > > > > > > > > > > > > listener.security.protocol.map=CONTROLLER:SSL
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Here's an example configuration for the broker:
> > > > > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > > > 1@controller1.example.com:9093,
> > > > 2@controller2.example.com
> > > > > > > > > > > > > > controller.connect.security.protocol=SSL
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > security.inter.broker.protocol or
> > > > > > inter.broker.listener.name
> > > > > > > > do
> > > > > > > > > > not
> > > > > > > > > > > > > > affect how the broker communicates with the controller.
> > > > > > Those
> > > > > > > > > > > > > > configurations relate to how brokers communicate with
> > > > each
> > > > > > > > other,
> > > > > > > > > > but
> > > > > > > > > > > > the
> > > > > > > > > > > > > > controller is not a broker.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > (Note that I just added
> > > > > > controller.connect.security.protocol
> > > > > > > > to the
> > > > > > > > > > > > KIP --
> > > > > > > > > > > > > > I had forgotten to put it in earlier)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 50.2 registration.heartbeat.interval.ms and
> > > > > > > > > > > > > > registration.lease.timeout.ms.
> > > > > > > > > > > > > > > Should we match their default value with the
> > > > > > corresponding
> > > > > > > > > > default
> > > > > > > > > > > > for
> > > > > > > > > > > > > > ZK?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Fair enough. I'll set them to the values of the
> > > > > > > > > > > > zookeeper.sync.time.ms
> > > > > > > > > > > > > > and zookeeper.connection.timeout.ms configurations.
> > > > I do
> > > > > > > > think we
> > > > > > > > > > > > should
> > > > > > > > > > > > > > experiment later on to see what works well here, but
> > > > the ZK
> > > > > > > > values
> > > > > > > > > > are
> > > > > > > > > > > > at
> > > > > > > > > > > > > > least a starting point.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 50.3 controller.connect: Could you provide an
> > > > example? I
> > > > > > am
> > > > > > > > > > > > wondering how
> > > > > > > > > > > > > > > it differs from quorum.voters=1@kafka-1:9092,
> > > > 2@kafka-2
> > > > > > > > :9092,
> > > > > > > > > > > > 3@kafka-3
> > > > > > > > > > > > > > > :9092.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > controller.connect is intended to be the new name for
> > > > > > > > > > quorum.voters.
> > > > > > > > > > > > > > During the vote for KIP-595, we sort of agreed to
> > > > defer the
> > > > > > > > > > discussion
> > > > > > > > > > > > > > about what this configuration should be called. I
> > > > proposed
> > > > > > > > this
> > > > > > > > > > new
> > > > > > > > > > > > name
> > > > > > > > > > > > > > because it makes it clear what the configuration is for
> > > > > > (how to
> > > > > > > > > > > > connect to
> > > > > > > > > > > > > > the controllers).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 50.4 controller.id: I am still not sure how this is
> > > > > > being
> > > > > > > > used.
> > > > > > > > > > > > Could
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > explain this in more detail?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The controller ID needs to be set on each controller.
> > > > It
> > > > > > also
> > > > > > > > > > appears
> > > > > > > > > > > > in
> > > > > > > > > > > > > > controller.connect as the thing before the "at" sign.
> > > > Its
> > > > > > > > > > function is
> > > > > > > > > > > > > > pretty similar to broker.id.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Broker IDs and controller IDs exist in the same ID
> > > > space,
> > > > > > so
> > > > > > > > you
> > > > > > > > > > can't
> > > > > > > > > > > > > > have a broker and a controller that use the same ID.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 51. BrokerHeartbeat: It seems a bit wasteful to
> > > > include
> > > > > > > > > > Listeners in
> > > > > > > > > > > > > > every
> > > > > > > > > > > > > > > heartbeat request since it typically doesn't change.
> > > > > > Could we
> > > > > > > > > > make
> > > > > > > > > > > > that
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > optional field?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Ok.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 52. KIP-584 adds a new ZK node /features. Should we
> > > > add a
> > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > metadata record?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Good point. I added FeatureLevelRecord.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 53. TopicRecord and DeleteTopic: Both DeleteTopic and
> > > > > > > > > > > > > > TopicRecord.Deleting
> > > > > > > > > > > > > > > indicate topic deletion. Could we outline the flow
> > > > when
> > > > > > each
> > > > > > > > > > will be
> > > > > > > > > > > > set?
> > > > > > > > > > > > > > > In particular, which one indicates the intention to
> > > > > > delete
> > > > > > > > and
> > > > > > > > > > which
> > > > > > > > > > > > one
> > > > > > > > > > > > > > > indicates the completion of the deletion.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > TopicRecord.Deleting is set when we intend to delete
> > > > the
> > > > > > topic.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > DeleteTopic removes the topic completely. I will
> > > > rename
> > > > > > > > > > DeleteTopic to
> > > > > > > > > > > > > > RemoveTopic to make this clearer.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 54. "The controller can generate a new broker epoch
> > > > by
> > > > > > using
> > > > > > > > the
> > > > > > > > > > > > latest
> > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > offset." Which offset is that? Is it the offset of
> > > > the
> > > > > > > > metadata
> > > > > > > > > > > > topic for
> > > > > > > > > > > > > > > the corresponding BrokerRecord? Is it guaranteed to
> > > > be
> > > > > > > > different
> > > > > > > > > > on
> > > > > > > > > > > > each
> > > > > > > > > > > > > > > broker restart?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes.... a new broker epoch implies that there has been
> > > > a
> > > > > > new
> > > > > > > > record
> > > > > > > > > > > > > > created in the metadata log. Therefore the last
> > > > committed
> > > > > > > > offset
> > > > > > > > > > must
> > > > > > > > > > > > be
> > > > > > > > > > > > > > different.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 55. "Thereafter, it may lose subsequent conflicts if
> > > > its
> > > > > > > > broker
> > > > > > > > > > > > epoch is
> > > > > > > > > > > > > > > stale. (See KIP-380 for some background on broker
> > > > > > epoch.)
> > > > > > > > The
> > > > > > > > > > > > reason
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > favoring new processes is to accommodate the common
> > > > case
> > > > > > > > where a
> > > > > > > > > > > > process
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > killed with kill -9 and then restarted. " Are you
> > > > saying
> > > > > > > > that if
> > > > > > > > > > > > there is
> > > > > > > > > > > > > > > an active broker registered in the controller, a new
> > > > > > broker
> > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > request with the INITIAL state will fence the current
> > > > > > broker
> > > > > > > > > > > > session? Not
> > > > > > > > > > > > > > > sure about this. For example, this will allow a
> > > > broker
> > > > > > with
> > > > > > > > > > > > incorrectly
> > > > > > > > > > > > > > > configured broker id to kill an existing valid
> > > > broker.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes, a new broker with an incorrectly configured
> > > > broker id
> > > > > > > > could
> > > > > > > > > > fence
> > > > > > > > > > > > an
> > > > > > > > > > > > > > existing valid broker.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The goal of the conflict resolution process described
> > > > here
> > > > > > is
> > > > > > > > to
> > > > > > > > > > avoid
> > > > > > > > > > > > > > having two brokers go back and forth in claiming the
> > > > > > broker id.
> > > > > > > > > > > > Choosing
> > > > > > > > > > > > > > the newest was just an easy way to do that.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Choosing the oldest is another possibility, and more
> > > > inline
> > > > > > > > with
> > > > > > > > > > what
> > > > > > > > > > > > > > ZooKeeper does today. The problem with that is that it
> > > > > > would
> > > > > > > > not
> > > > > > > > > > be
> > > > > > > > > > > > > > possible to re-create the broker if the old instance
> > > > died
> > > > > > > > suddenly,
> > > > > > > > > > > > except
> > > > > > > > > > > > > > by waiting for the full lease timeout.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It might be possible to do a little better here by
> > > > > > breaking the
> > > > > > > > > > lease
> > > > > > > > > > > > if
> > > > > > > > > > > > > > the TCP connection from the broker drops. But that
> > > > > > requires
> > > > > > > > > > crossing
> > > > > > > > > > > > some
> > > > > > > > > > > > > > layers of abstraction in the Kafka networking stack.
> > > > > > There's
> > > > > > > > also
> > > > > > > > > > the
> > > > > > > > > > > > > > possibility of false negatives or positives. For
> > > > example,
> > > > > > TCP
> > > > > > > > > > > > connections
> > > > > > > > > > > > > > sometimes just drop even if the node is still there.
> > > > Or
> > > > > > > > sometimes
> > > > > > > > > > the
> > > > > > > > > > > > > > networking stack keeps a TCP connection alive for a
> > > > while
> > > > > > when
> > > > > > > > the
> > > > > > > > > > > > other
> > > > > > > > > > > > > > end is gone.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 56. kafka-storage.sh:
> > > > > > > > > > > > > > > 56.1 In the info mode, what other information does it
> > > > > > show in
> > > > > > > > > > > > addition to
> > > > > > > > > > > > > > > "kip.500.mode=enabled"?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it should show a list of all the storage
> > > > > > directories,
> > > > > > > > and
> > > > > > > > > > > > whether
> > > > > > > > > > > > > > each one is formatted. In addition, it should show
> > > > whether
> > > > > > > > > > > > kip.500.mode is
> > > > > > > > > > > > > > enabled, and what the cluster id is.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 56.2 Should the format mode take the config file as
> > > > the
> > > > > > > > input too
> > > > > > > > > > > > like
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > info mode?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes, that's a good idea.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 17, 2020 at 4:12 PM Colin McCabe <
> > > > > > > > cmccabe@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > That's a fair point. I have moved the lease
> > > > duration
> > > > > > to
> > > > > > > > the
> > > > > > > > > > broker
> > > > > > > > > > > > > > > > heartbeat response. That way lease durations can
> > > > be
> > > > > > > > changed
> > > > > > > > > > just
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > reconfiguring the controllers.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Wed, Sep 16, 2020, at 07:40, Unmesh Joshi wrote:
> > > > > > > > > > > > > > > > > Thanks Colin, the changes look good to me. One
> > > > small
> > > > > > > > thing.
> > > > > > > > > > > > > > > > > registration.lease.timeout.ms is the
> > > > configuration
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > side.
> > > > > > > > > > > > > > > > > It will be good to comment on how brokers know
> > > > about
> > > > > > it,
> > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > > able
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > send LeaseDurationMs
> > > > > > > > > > > > > > > > > in the heartbeat request,
> > > > > > > > > > > > > > > > > or else it can be added in the heartbeat
> > > > response for
> > > > > > > > > > brokers to
> > > > > > > > > > > > know
> > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Sep 11, 2020 at 10:32 PM Colin McCabe <
> > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think you're right that we should use a
> > > > duration
> > > > > > here
> > > > > > > > > > rather
> > > > > > > > > > > > > > than a
> > > > > > > > > > > > > > > > > > time. As you said, the clock on the controller
> > > > > > will
> > > > > > > > > > probably
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > > match the
> > > > > > > > > > > > > > > > > > one on the broker. I have updated the KIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > It's important to keep in mind that
> > > > messages
> > > > > > may be
> > > > > > > > > > > > delayed in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > > > > happens, we
> > > > > > > > > > > > will
> > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > > > time specified in the request to determine
> > > > if
> > > > > > the
> > > > > > > > > > request
> > > > > > > > > > > > is
> > > > > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > > > > I am assuming that there will be a single TCP
> > > > > > > > connection
> > > > > > > > > > > > > > maintained
> > > > > > > > > > > > > > > > > > between
> > > > > > > > > > > > > > > > > > > broker and active controller. So, there
> > > > won't be
> > > > > > any
> > > > > > > > out
> > > > > > > > > > of
> > > > > > > > > > > > order
> > > > > > > > > > > > > > > > > > requests?
> > > > > > > > > > > > > > > > > > > There will be a scenario of broker GC pause,
> > > > > > which
> > > > > > > > might
> > > > > > > > > > > > cause
> > > > > > > > > > > > > > > > connection
> > > > > > > > > > > > > > > > > > > timeout and broker might need to reestablish
> > > > the
> > > > > > > > > > connection.
> > > > > > > > > > > > If
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > pause
> > > > > > > > > > > > > > > > > > > is too long, lease will expire and the
> > > > heartbeat
> > > > > > sent
> > > > > > > > > > after
> > > > > > > > > > > > the
> > > > > > > > > > > > > > pause
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > be treated as a new registration (similar to
> > > > > > restart
> > > > > > > > > > case),
> > > > > > > > > > > > and
> > > > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > > > epoch
> > > > > > > > > > > > > > > > > > > number will be assigned to the broker.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I agree with the end of this paragraph, but not
> > > > > > with
> > > > > > > > the
> > > > > > > > > > start
> > > > > > > > > > > > :)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > There can be out-of-order requests, since the
> > > > > > broker
> > > > > > > > will
> > > > > > > > > > > > simply
> > > > > > > > > > > > > > use a
> > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > TCP connection if the old one has problems.
> > > > This
> > > > > > can
> > > > > > > > > > happen
> > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > variety
> > > > > > > > > > > > > > > > > > of reasons. I don't think GC pauses are the
> > > > most
> > > > > > > > common
> > > > > > > > > > > > reason for
> > > > > > > > > > > > > > > > this to
> > > > > > > > > > > > > > > > > > happen. It's more common to see issues issues
> > > > in
> > > > > > the
> > > > > > > > > > network
> > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > result connections getting dropped from time to
> > > > > > time.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > So we have to assume that messages may arrive
> > > > out
> > > > > > of
> > > > > > > > > > order, and
> > > > > > > > > > > > > > > > possibly
> > > > > > > > > > > > > > > > > > be delayed. I added a note that heartbeat
> > > > requests
> > > > > > > > should
> > > > > > > > > > be
> > > > > > > > > > > > > > ignored
> > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > the metadata log offset they contain is smaller
> > > > > > than a
> > > > > > > > > > previous
> > > > > > > > > > > > > > > > heartbeat.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > When the active controller fails, the new
> > > > active
> > > > > > > > > > controller
> > > > > > > > > > > > > > needs to
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > sure that it considers all the known brokers
> > > > as
> > > > > > alive
> > > > > > > > > > till
> > > > > > > > > > > > the
> > > > > > > > > > > > > > lease
> > > > > > > > > > > > > > > > > > > expiration interval. Because
> > > > > > > > > > registration.lease.timeout.ms,
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > configured
> > > > > > > > > > > > > > > > > > > on the controller, the new active controller
> > > > will
> > > > > > > > extend
> > > > > > > > > > all
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > leases
> > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > registration.lease.timeout.ms. I see that it
> > > > > > won't
> > > > > > > > need
> > > > > > > > > > last
> > > > > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Agreed.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Sat, Sep 5, 2020 at 1:28 AM Colin McCabe <
> > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Colin wrote:
> > > > > > > > > > > > > > > > > > > > > > The reason for including
> > > > LeaseStartTimeMs
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > request
> > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > > > > > that the time required to communicate
> > > > with
> > > > > > the
> > > > > > > > > > > > controller
> > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > > included in
> > > > > > > > > > > > > > > > > > > > > > the lease time. Since requests can
> > > > > > > > potentially be
> > > > > > > > > > > > delayed
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > > > > > for a long time, this is important.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020, at 05:58, Unmesh
> > > > Joshi
> > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > The network time will be added anyway,
> > > > > > because
> > > > > > > > the
> > > > > > > > > > lease
> > > > > > > > > > > > > > timer
> > > > > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > > > > active controller will start only after
> > > > the
> > > > > > > > heartbeat
> > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > reaches the
> > > > > > > > > > > > > > > > > > > > > server.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > If the start time is not specified in the
> > > > > > request,
> > > > > > > > > > then the
> > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > > > > is excluded from the heartbeat time.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Here's an example:
> > > > > > > > > > > > > > > > > > > > Let's say broker A sends a heartbeat at
> > > > time
> > > > > > 100,
> > > > > > > > and
> > > > > > > > > > it
> > > > > > > > > > > > > > arrives
> > > > > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > > > controller at time 200, and the lease
> > > > duration
> > > > > > is
> > > > > > > > 1000.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The controller looks at the start time in
> > > > the
> > > > > > > > request,
> > > > > > > > > > > > which is
> > > > > > > > > > > > > > > > 100,
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > adds 1000 to it, getting 1100. On the
> > > > other
> > > > > > hand,
> > > > > > > > if
> > > > > > > > > > start
> > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > > > > specified in the request, then the
> > > > expiration
> > > > > > will
> > > > > > > > be
> > > > > > > > > > at
> > > > > > > > > > > > time
> > > > > > > > > > > > > > 1200.
> > > > > > > > > > > > > > > > > > > > That is what I mean by "the network time is
> > > > > > > > included
> > > > > > > > > > in the
> > > > > > > > > > > > > > > > expiration
> > > > > > > > > > > > > > > > > > > > time."
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > And I think, some assumption about
> > > > network
> > > > > > round
> > > > > > > > trip
> > > > > > > > > > > > time is
> > > > > > > > > > > > > > > > > > > > > needed anyway to decide on the frequency
> > > > of
> > > > > > the
> > > > > > > > > > > > heartbeat (
> > > > > > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms), and
> > > > > > lease
> > > > > > > > > > timeout (
> > > > > > > > > > > > > > > > > > > > > registration.lease.timeout.ms). So I
> > > > think
> > > > > > just
> > > > > > > > > > having a
> > > > > > > > > > > > > > > > leaseTTL
> > > > > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > > > > request is easier to understand and
> > > > > > implement.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > It's important to keep in mind that
> > > > messages
> > > > > > may be
> > > > > > > > > > > > delayed in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > > > > happens, we
> > > > > > > > > > > > will
> > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > > > time specified in the request to determine
> > > > if
> > > > > > the
> > > > > > > > > > request
> > > > > > > > > > > > is
> > > > > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> > > > the
> > > > > > > > > > controller
> > > > > > > > > > > > side
> > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > reset in the case of controller
> > > > failover.
> > > > > > The
> > > > > > > > > > > > alternative
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > be to
> > > > > > > > > > > > > > > > > > > > > > track the lease as hard state rather
> > > > than
> > > > > > soft
> > > > > > > > > > state,
> > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > is not really needed, and would result
> > > > in
> > > > > > more
> > > > > > > > log
> > > > > > > > > > > > entries.
> > > > > > > > > > > > > > > > > > > > > My interpretation of the mention of
> > > > > > BrokerRecord
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > KIP
> > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > record exists in the Raft log.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > BrokerRecord does exist in the Raft log,
> > > > but
> > > > > > does
> > > > > > > > not
> > > > > > > > > > > > include
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > > > heartbeat time.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > By soft state, do you mean the broker
> > > > > > > > > > > > > > > > > > > > > records exist only on the active leader
> > > > and
> > > > > > will
> > > > > > > > not
> > > > > > > > > > be
> > > > > > > > > > > > > > > > replicated
> > > > > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > > > > raft log? If the live brokers list is
> > > > > > maintained
> > > > > > > > > > only on
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > > controller (raft leader), then, in case
> > > > of
> > > > > > leader
> > > > > > > > > > > > failure,
> > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > will be
> > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > window where the new leader does not know
> > > > > > about
> > > > > > > > the
> > > > > > > > > > live
> > > > > > > > > > > > > > brokers,
> > > > > > > > > > > > > > > > > > till
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > brokers establish the leases again.
> > > > > > > > > > > > > > > > > > > > > I think it will be safer to have leases
> > > > as a
> > > > > > hard
> > > > > > > > > > state
> > > > > > > > > > > > > > managed
> > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > standard
> > > > > > > > > > > > > > > > > > > > > Raft replication.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Leases are short, so the need to
> > > > re-establish
> > > > > > them
> > > > > > > > > > after a
> > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > failover doesn't seem like a big problem.
> > > > But
> > > > > > > > this is
> > > > > > > > > > > > > > something
> > > > > > > > > > > > > > > > we can
> > > > > > > > > > > > > > > > > > > > tweak if it becomes an issue. One option
> > > > > > would be
> > > > > > > > to
> > > > > > > > > > have
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > > > which is only used by the controller nodes
> > > > for
> > > > > > this
> > > > > > > > > > (since,
> > > > > > > > > > > > > > after
> > > > > > > > > > > > > > > > all,
> > > > > > > > > > > > > > > > > > > > brokers don't care about registration
> > > > > > renewals).
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Or am I misunderstanding something? (I
> > > > assume
> > > > > > > > that
> > > > > > > > > > with
> > > > > > > > > > > > soft
> > > > > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > mean something like zookeeper local
> > > > sessions
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1147
> > > > > > > > > > .)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > > > > think
> > > > > > > > it
> > > > > > > > > > makes
> > > > > > > > > > > > > > sense
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> > > > becomes
> > > > > > > > very
> > > > > > > > > > messy.
> > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > > > > understand your question about
> > > > duplicate
> > > > > > > > broker ID
> > > > > > > > > > > > > > detection,
> > > > > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > > > There's a section in the KIP about this
> > > > -- is
> > > > > > > > there a
> > > > > > > > > > > > detail
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This is an implementation detail that
> > > > doesn't
> > > > > > need
> > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I assumed broker leases are implemented
> > > > as a
> > > > > > hard
> > > > > > > > > > state.
> > > > > > > > > > > > In
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > case, to
> > > > > > > > > > > > > > > > > > > > > check for broker id conflict, we need to
> > > > > > check
> > > > > > > > the
> > > > > > > > > > broker
> > > > > > > > > > > > > > ids at
> > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > places
> > > > > > > > > > > > > > > > > > > > > 1. Pending broker registrations (which
> > > > are
> > > > > > yet
> > > > > > > > to be
> > > > > > > > > > > > > > committed)
> > > > > > > > > > > > > > > > 2.
> > > > > > > > > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > > > > > > > committed broker registrations.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020 at 5:42 PM Colin
> > > > McCabe
> > > > > > <
> > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020, at 01:12, Unmesh
> > > > > > Joshi
> > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >>>Can you repeat your questions
> > > > about
> > > > > > broker
> > > > > > > > > > leases?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >>>>The LeaseStartTimeMs is expected
> > > > to
> > > > > > be
> > > > > > > > the
> > > > > > > > > > > > broker's
> > > > > > > > > > > > > > > > > > > > > > > 'System.currentTimeMillis()' at the
> > > > > > point of
> > > > > > > > the
> > > > > > > > > > > > > > request. The
> > > > > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > > > > controller will add its lease period
> > > > to
> > > > > > this
> > > > > > > > in
> > > > > > > > > > order
> > > > > > > > > > > > > > >>>>to
> > > > > > > > > > > > > > > > > > compute
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > LeaseEndTimeMs.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think the use of LeaseStartTimeMs
> > > > and
> > > > > > > > > > > > LeaseEndTimeMs
> > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > > > confusing. Monotonic Clock
> > > > > > > > (System.nanoTime) on
> > > > > > > > > > the
> > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > should be used to track leases.
> > > > > > > > > > > > > > > > > > > > > > > (For example,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1616https://github.com/etcd-io/etcd/pull/6888/commits/e7f4010ccaf28b6ce64fe514d25a4b2fa459d114
> > > > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Then we will not need
> > > > LeaseStartTimeMs?
> > > > > > > > > > > > > > > > > > > > > > > Instead of LeaseStartTimeMs, can we
> > > > call
> > > > > > it
> > > > > > > > > > > > LeaseTTL? The
> > > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > can then calculate LeaseEndTime =
> > > > > > > > > > System.nanoTime() +
> > > > > > > > > > > > > > > > LeaseTTL.
> > > > > > > > > > > > > > > > > > > > > > > In this case we might just drop
> > > > > > > > LeaseEndTimeMs
> > > > > > > > > > from
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > response, as
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > broker already knows about the TTL
> > > > and
> > > > > > can
> > > > > > > > send
> > > > > > > > > > > > > > heartbeats at
> > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > > fraction
> > > > > > > > > > > > > > > > > > > > > > > of TTL, say every TTL/4
> > > > > > milliseconds.(elapsed
> > > > > > > > > > time
> > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > measured
> > > > > > > > > > > > > > > > > > > > > > > by System.nanoTime)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I agree that the monotonic clock is
> > > > > > probably a
> > > > > > > > > > better
> > > > > > > > > > > > idea
> > > > > > > > > > > > > > > > here.
> > > > > > > > > > > > > > > > > > It is
> > > > > > > > > > > > > > > > > > > > > > good to be robust against wall clock
> > > > > > changes,
> > > > > > > > > > although
> > > > > > > > > > > > I
> > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > > > > > > which had them might suffer other
> > > > issues.
> > > > > > I
> > > > > > > > will
> > > > > > > > > > > > change
> > > > > > > > > > > > > > it to
> > > > > > > > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > monotonic clock.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The reason for including
> > > > LeaseStartTimeMs
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > request
> > > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > the time required to communicate with
> > > > the
> > > > > > > > > > controller
> > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > included
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > lease time. Since requests can
> > > > > > potentially be
> > > > > > > > > > delayed
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > > > > > > > long time, this is important.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I have a prototype built to
> > > > demonstrate
> > > > > > this
> > > > > > > > as
> > > > > > > > > > > > > > following:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The Kip631Controller itself depends
> > > > on a
> > > > > > > > > > Consensus
> > > > > > > > > > > > > > module, to
> > > > > > > > > > > > > > > > > > > > demonstrate
> > > > > > > > > > > > > > > > > > > > > > > how possible interactions with the
> > > > > > consensus
> > > > > > > > > > module
> > > > > > > > > > > > will
> > > > > > > > > > > > > > look
> > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > > > (The Consensus can be pluggable
> > > > really,
> > > > > > > > with an
> > > > > > > > > > API
> > > > > > > > > > > > to
> > > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > > > > replicated log upto HighWaterMark)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > It has an implementation of
> > > > LeaseTracker
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/LeaderLeaseTracker.scala
> > > > > > > > > > > > > > > > > > > > > > > to demonstrate LeaseTracker's
> > > > interaction
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > > consensus
> > > > > > > > > > > > > > > > > > module.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The implementation has the following
> > > > > > aspects:
> > > > > > > > > > > > > > > > > > > > > > > 1. The lease tracking happens only
> > > > on the
> > > > > > > > active
> > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > (raft
> > > > > > > > > > > > > > > > > > > > > > > leader)
> > > > > > > > > > > > > > > > > > > > > > > 2. Once the lease expires, it needs
> > > > to
> > > > > > > > propose
> > > > > > > > > > and
> > > > > > > > > > > > > > commit a
> > > > > > > > > > > > > > > > > > > > FenceBroker
> > > > > > > > > > > > > > > > > > > > > > > record for that lease.
> > > > > > > > > > > > > > > > > > > > > > > 3. In case of active controller
> > > > failure,
> > > > > > the
> > > > > > > > > > lease
> > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > tracked by
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > newly raft leader. The new raft
> > > > leader
> > > > > > > > starts the
> > > > > > > > > > > > lease
> > > > > > > > > > > > > > timer
> > > > > > > > > > > > > > > > > > again,
> > > > > > > > > > > > > > > > > > > > (as
> > > > > > > > > > > > > > > > > > > > > > > implemented in onBecomingLeader
> > > > method of
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > > > > in effect extending the lease by the
> > > > time
> > > > > > > > spent
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > > election
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > whatever time was elapsed on the old
> > > > > > leader.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> > > > the
> > > > > > > > > > controller
> > > > > > > > > > > > side
> > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > reset
> > > > > > > > > > > > > > > > > > > > > > in the case of controller failover.
> > > > The
> > > > > > > > > > alternative
> > > > > > > > > > > > would
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > track
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > lease as hard state rather than soft
> > > > state,
> > > > > > > > but I
> > > > > > > > > > think
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > > > > > > > needed, and would result in more log
> > > > > > entries.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > There are working tests for this
> > > > > > > > implementation
> > > > > > > > > > here.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > > > > > and an end to end test here
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/ProducerConsumerKIP500Test.scala
> > > > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >>'m not sure what you mean by
> > > > > > > > "de-duplication
> > > > > > > > > > of the
> > > > > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > > > > Apologies for using the confusing
> > > > term
> > > > > > > > > > > > deduplication. I
> > > > > > > > > > > > > > meant
> > > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > > id
> > > > > > > > > > > > > > > > > > > > > > > conflict.
> > > > > > > > > > > > > > > > > > > > > > > As you can see in the prototype
> > > > > > > > handleRequest of
> > > > > > > > > > > > > > > > KIP631Controller
> > > > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > > > > the duplicate broker id needs to be
> > > > > > detected
> > > > > > > > > > before
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > BrokerRecord
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > submitted to the raft module.
> > > > > > > > > > > > > > > > > > > > > > > Also as implemented in the
> > > > prototype, the
> > > > > > > > > > > > > > KIP631Controller is
> > > > > > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > > > > > > > threaded, handling requests one at a
> > > > > > time.
> > > > > > > > (an
> > > > > > > > > > > > example of
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > https://martinfowler.com/articles/patterns-of-distributed-systems/singular-update-queue.html
> > > > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > > > > think
> > > > > > > > it
> > > > > > > > > > makes
> > > > > > > > > > > > > > sense
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> > > > becomes
> > > > > > > > very
> > > > > > > > > > messy.
> > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > > > > understand your question about
> > > > duplicate
> > > > > > > > broker ID
> > > > > > > > > > > > > > detection,
> > > > > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > > > > There's a section in the KIP about
> > > > this --
> > > > > > is
> > > > > > > > > > there a
> > > > > > > > > > > > > > detail we
> > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 10:49 AM
> > > > Colin
> > > > > > > > McCabe <
> > > > > > > > > > > > > > > > colin@cmccabe.xyz
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2020, at 19:36,
> > > > Unmesh
> > > > > > > > Joshi
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > There were a few of questions I
> > > > had..
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for the response.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. Were my comments on the broker
> > > > > > lease
> > > > > > > > > > > > > > implementation
> > > > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > > > prototype) appropriate and do we
> > > > > > need to
> > > > > > > > > > change
> > > > > > > > > > > > the
> > > > > > > > > > > > > > KIP
> > > > > > > > > > > > > > > > > > > > > > > > > description accordingly?.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Can you repeat your questions about
> > > > > > broker
> > > > > > > > > > leases?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. How will broker epochs be
> > > > > > generated?
> > > > > > > > I am
> > > > > > > > > > > > > > assuming it
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > committed log offset (like zxid?)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > There isn't any need to use a log
> > > > > > offset.
> > > > > > > > We
> > > > > > > > > > can
> > > > > > > > > > > > just
> > > > > > > > > > > > > > > > look at
> > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > > in-memory hash table and see what
> > > > the
> > > > > > > > latest
> > > > > > > > > > > > number is,
> > > > > > > > > > > > > > > > and add
> > > > > > > > > > > > > > > > > > > > one, to
> > > > > > > > > > > > > > > > > > > > > > > > generate a new broker epoch.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3. How will producer registration
> > > > > > > > happen? I
> > > > > > > > > > am
> > > > > > > > > > > > > > assuming
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > should be
> > > > > > > > > > > > > > > > > > > > > > > > > similar to broker registration,
> > > > with
> > > > > > a
> > > > > > > > > > similar
> > > > > > > > > > > > way to
> > > > > > > > > > > > > > > > > > generate
> > > > > > > > > > > > > > > > > > > > > > producer
> > > > > > > > > > > > > > > > > > > > > > > > id.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > For the EOS stuff, we will need a
> > > > few
> > > > > > new
> > > > > > > > RPCs
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > controller.
> > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > > > we should do that in a follow-on
> > > > KIP,
> > > > > > > > though,
> > > > > > > > > > since
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > one is
> > > > > > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > > > > > > pretty big.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 4. Because we expose Raft log to
> > > > all
> > > > > > the
> > > > > > > > > > > > brokers, any
> > > > > > > > > > > > > > > > > > > > de-duplication
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > broker needs to happen before the
> > > > > > > > requests
> > > > > > > > > > are
> > > > > > > > > > > > > > proposed
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > Raft.
> > > > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > > the controller needs to be single
> > > > > > > > threaded,
> > > > > > > > > > and
> > > > > > > > > > > > > > should do
> > > > > > > > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > > > > > > against the in-process or pending
> > > > > > > > requests
> > > > > > > > > > and
> > > > > > > > > > > > the
> > > > > > > > > > > > > > final
> > > > > > > > > > > > > > > > > > state. I
> > > > > > > > > > > > > > > > > > > > > > read a
> > > > > > > > > > > > > > > > > > > > > > > > > mention of this, in the
> > > > responses in
> > > > > > this
> > > > > > > > > > > > > > thread.Will it
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > useful to
> > > > > > > > > > > > > > > > > > > > > > > > > mention this in the KIP?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I'm not sure what you mean by
> > > > > > > > "de-duplication
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 4:50 AM
> > > > Colin
> > > > > > > > McCabe
> > > > > > > > > > <
> > > > > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I'm thinking of calling a vote
> > > > on
> > > > > > > > KIP-631
> > > > > > > > > > on
> > > > > > > > > > > > > > Monday.
> > > > > > > > > > > > > > > > Let
> > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > know
> > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > > > there's any more comments I
> > > > should
> > > > > > > > address
> > > > > > > > > > > > before I
> > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > vote.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020, at 05:39,
> > > > > > Unmesh
> > > > > > > > > > Joshi
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks. I will be adding
> > > > more to
> > > > > > the
> > > > > > > > > > > > prototype
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > > > > > > and collaborate.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020 at
> > > > 12:28 AM
> > > > > > > > Colin
> > > > > > > > > > > > McCabe <
> > > > > > > > > > > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Jose,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > That'a s good point that I
> > > > > > hadn't
> > > > > > > > > > > > considered.
> > > > > > > > > > > > > > It's
> > > > > > > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > > > > a separate leader change
> > > > > > message,
> > > > > > > > as
> > > > > > > > > > you
> > > > > > > > > > > > > > mentioned.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2020, at
> > > > 11:56,
> > > > > > Jose
> > > > > > > > > > Garcia
> > > > > > > > > > > > > > Sancio
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Very cool prototype!
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > The KIP proposes a record
> > > > > > called
> > > > > > > > > > > > IsrChange
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > > includes the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > partition, topic, isr,
> > > > > > leader and
> > > > > > > > > > leader
> > > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > > > During
> > > > > > > > > > > > > > > > > > > > normal
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > operation ISR changes do
> > > > not
> > > > > > > > result
> > > > > > > > > > in
> > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > > > > > Similarly,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > leader changes do not
> > > > > > necessarily
> > > > > > > > > > > > involve ISR
> > > > > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > implementation that uses
> > > > ZK
> > > > > > > > modeled
> > > > > > > > > > them
> > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. All of this
> > > > information is
> > > > > > > > stored
> > > > > > > > > > in
> > > > > > > > > > > > one
> > > > > > > > > > > > > > > > znode.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. ZK's optimistic lock
> > > > > > requires
> > > > > > > > > > that you
> > > > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > > > the new
> > > > > > > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. The change to that
> > > > znode
> > > > > > was
> > > > > > > > being
> > > > > > > > > > > > > > performed
> > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > and the leader.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > None of these reasons are
> > > > > > true in
> > > > > > > > > > > > KIP-500.
> > > > > > > > > > > > > > Have
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > considered
> > > > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > two different records?
> > > > For
> > > > > > > > example
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. IsrChange record which
> > > > > > > > includes
> > > > > > > > > > topic,
> > > > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > > > isr
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. LeaderChange record
> > > > which
> > > > > > > > includes
> > > > > > > > > > > > topic,
> > > > > > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > > leader epoch.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > I suspect that making
> > > > this
> > > > > > change
> > > > > > > > > > will
> > > > > > > > > > > > also
> > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > message AlterIsrRequest
> > > > > > > > introduced in
> > > > > > > > > > > > > > KIP-497:
> > > > > > > > > > > > > > > > Add
> > > > > > > > > > > > > > > > > > > > > > inter-broker
> > > > > > > > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > alter ISR.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Jose
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Ron Dagostino <rn...@gmail.com>.
HI Colin. A FencedBrokerRecord appears in the metadata log when a
broker is fenced. What appears in the metadata log to indicate that a
broker is no longer fenced? Does a BrokerRecord appear? That seems
to communicate a bunch of unnecessary data in this context (endpoints,
features, rack). If that is the current plan, might it be better to
include a Boolean value within FencedBrokerRecord and specify true
when the broker becomes fenced and false when it is no longer fenced?
Also, is a fenced broker considered Offline for partition metadata
purposes? I think so but would like to confirm.
Ron
On Wed, Oct 21, 2020 at 9:05 AM Colin McCabe <cm...@apache.org> wrote:
>
> On Tue, Oct 13, 2020, at 18:30, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the reply. A few more comments below.
> >
> > 80.1 controller.listener.names only defines the name of the listener. The
> > actual listener including host/port/security_protocol is typically defined
> > in advertised_listners. Does that mean advertised_listners is a required
> > config now?
> >
>
> Hi Jun,
>
> Thanks for the re-review.
>
> The controller listeners are not advertised to clients. So I think they should go in listeners, rather than advertised.listeners.
>
> I agree that this makes listeners a required configuration. At very least, it is required to have the controller listener in there.
>
> >
> > 83.1 broker state machine: It seems that we should transition from FENCED
> > => INITIAL since only INITIAL generates new broker epoch?
> >
>
> I changed the broker state machine a bit-- take a look. In the new state machine, the FENCED state can re-register.
>
> >
> > 83.5. It's true that the controller node doesn't serve metadata requests.
> > However, there are admin requests such as topic creation/deletion are sent
> > to the controller directly. So, it seems that the client needs to know
> > the controller host/port?
> >
>
> The client sends admin requests to a random broker, which forwards them to the controller. This is described in KIP-590.
>
> >
> > 85. "I was hoping that we could avoid responding to requests when the
> > broker was fenced." This issue is that if we don't send a response, the
> > client won't know the reason and can't act properly.
> >
>
> Sorry, when I said "does not respond" I meant it would return an error.
>
> >
> > 88. CurMetadataOffset: I was thinking that we may want to
> > use CurMetadataOffset to compute the MetadataLag. Since HWM is exclusive,
> > it's more convenient if CurMetadataOffset is also exclusive.
> >
>
> OK, I will change it to NextMetadataOffset and make it exclusive.
>
> > 90. It would be useful to add a rejected section on why separate controller
> > and broker id is preferred over just broker id. For example, the following
> > are some potential reasons. (a) We can guard duplicated brokerID, but it's
> > hard to guard against duplicated controllerId. (b) brokerID can be auto
> > assigned in the future, but controllerId is hard to be generated
> > automatically.
> >
>
> OK. I added a rejected alternatives section about sharing IDs between multiple nodes.
>
> Colin
>
> > Thanks,
> >
> > Jun
> >
> > On Mon, Oct 12, 2020 at 11:14 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Tue, Oct 6, 2020, at 16:09, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply. Made another pass of the KIP. A few more comments
> > > > below.
> > > >
> > >
> > > Hi Jun,
> > >
> > > Thanks for the review.
> > >
> > > > 55. We discussed earlier why the current behavior where we favor the
> > > > current broker registration is better. Have you given this more thought?
> > > >
> > >
> > > Yes, I think we should favor the current broker registration, as you
> > > suggested earlier.
> > >
> > > > 80. Config related.
> > > > 80.1 Currently, each broker only has the following 3 required configs. It
> > > > will be useful to document the required configs post KIP-500 (in both the
> > > > dedicated and shared controller mode).
> > > > broker.id
> > > > log.dirs
> > > > zookeeper.connect
> > >
> > > For the broker, these configs will be required:
> > >
> > > broker.id
> > > log.dirs
> > > process.roles
> > > controller.listener.names
> > > controller.connect
> > >
> > > For the controller, these configs will be required:
> > >
> > > controller.id
> > > log.dirs
> > > process.roles
> > > controller.listener.names
> > > controller.connect
> > >
> > > For broker+controller, it will be the union of these two, which
> > > essentially means we need both broker.id and controller.id, but all
> > > others are the same as standalone.
> > >
> > > > 80.2 It would be useful to document all deprecated configs post KIP-500.
> > > > For example, all zookeeper.* are obviously deprecated. But there could be
> > > > others. For example, since we don't plan to support auto broker id
> > > > generation, it seems broker.id.generation.enable is deprecated too.
> > > > 80.3 Could we make it clear that controller.connect replaces
> > > quorum.voters
> > > > in KIP-595?
> > >
> > > OK. I added a comment about this in the table.
> > >
> > > > 80.4 Could we document that broker.id is now optional?
> > >
> > > OK. I have added a line for broker.id.
> > >
> > > > 80.5 The KIP suggests that controller.id is optional on the controller
> > > > node. I am concerned that this can cause a bit of confusion in 2 aspects.
> > > > First, in the dedicated controller mode, controller.id is not optional
> > > > (since broker.id is now optional). Second, in the shared controller
> > > mode,
> > > > it may not be easy for the user to figure out the default value of
> > > > controller.id to set controller.connect properly.
> > >
> > > I got rid of the default value of controller.id. I agree it was a bit
> > > confusing to explain.
> > >
> > > > 80.6 Regarding the consistency of config names, our metrics already
> > > include
> > > > controller. So, prefixing all controller related configs with
> > > "controller"
> > > > may be more consistent. If we choose to do that, could we rename all new
> > > > configs here and in KIP-595 consistently?
> > > >
> > >
> > > I added the new names for the KIP-595 configurations to the configuration
> > > table. I think prefixing these configurations with "controller.quorum" is
> > > the best option since it makes it clear that they are related to the
> > > controller quorum.
> > >
> > > > 81. Metrics
> > > > 81.1 kafka.controller:type=KafkaController,name=MetadataSnapshotLag: This
> > > > is now redundant since KIP-630 already has
> > > > kafka.controller:type=KafkaController,name=SnapshotLag.
> > >
> > > Thanks for bringing up the metric name issue. I think it would be good to
> > > clarify that this is the lag for the metadata snapshot specifically. So I
> > > prefer the name MetadataSnapshotLag. I'll add a note to the table.
> > >
> > > > 81.2 Do we need both kafka.controller:type=KafkaServer,name=MetadataLag
> > > and
> > > > kafka.controller:type=KafkaController,name=MetadataLag since in the
> > > shared
> > > > controller mode, the metadata log is shared?
> > >
> > > Yes, both are needed. This metric is about what metadata logs we have
> > > applied, not about the Raft log's high water mark. If the broker or
> > > controller takes a while to apply the log messages (for example, because of
> > > lock contention or some other issue), they could fall significantly behind
> > > the high watermark.
> > >
> > > >
> > > > 82. Metadata records
> > > > 82.1 BrokerRecord: It needs to include supported features based on
> > > KIP-584.
> > >
> > > Added.
> > >
> > > > 82.2 FeatureLevel: The finalized feature in KIP-584 has the following
> > > > structure and needs to be reflected here.
> > > > {
> > > > "version": 0, // int32 -> Represents the version of the schema for the
> > > > data stored in the ZK node
> > > > "status": 1, // int32 -> Represents the status of the node
> > > > "features": {
> > > > "group_coordinator": { // string -> name of the feature
> > > > "min_version_level": 0, // int16 -> Represents the
> > > cluster-wide
> > > > finalized minimum version level (>=1) of this feature
> > > > "max_version_level": 3 // int16 -> Represents the
> > > cluster-wide
> > > > finalized maximum version level (>=1 and >= min_version_level) of this
> > > > feature
> > > > },
> > > > "consumer_offsets_topic_schema": {
> > > > "min_version_level": 0,
> > > > "max_version_level": 4
> > > > }
> > > > }
> > >
> > > Fair point. I will add both a minimum and maximum finalized feature level.
> > >
> > > >
> > > > 83. RPC
> > > > 83.1 BrokerHeartbeatRequest: Could you document when the controller
> > > assigns
> > > > a new broker epoch?
> > >
> > > The controller assigns a broker epoch if the heartbeat request was sent
> > > without an epoch (that is, it is in the INITIAL state), and the controller
> > > wants to add the new broker to the cluster. I added a note to the state
> > > machine.
> > >
> > > > 83.2 BrokerHeartbeatResponse: Do we need a new error code to indicate a
> > > > broker is fenced?
> > >
> > > I don't think we need a new error code, since targetState will just
> > > indicate that the broker should be fenced.
> > >
> > > > 83.3 What error/state indicates an unsuccessful controlled shutdown in
> > > > BrokerHeartbeatResponse? Currently, the controlled shutdown responses
> > > also
> > > > include a list of remaining partitions. That's mostly for debugging
> > > > purposes. Should we include it in BrokerHeartbeatResponse too?
> > >
> > > Once the controller wants the broker to shut down, it will set targetState
> > > to SHUTDOWN.
> > >
> > > We don't need to send a list of remaining partitions, since the broker is
> > > following the metadata log, and will get all its instructions from there.
> > >
> > > > 83.4 Could we document when Listeners, Features and Rack are expected to
> > > be
> > > > set in BrokerHeartbeatRequest?
> > >
> > > I added a line specifying that: The listeners, features, and rack fields
> > > only need to be set when the broker is in INITIAL state, and is requesting
> > > a new broker epoch.
> > >
> > > > 83.5 Currently, the metadata response only returns a list of brokers.
> > > > Should we have a separate field for the controller info or change the
> > > field
> > > > name to sth like nodes? Also, is the rack field relevant to the
> > > controller?
> > >
> > > As per KIP-590, we don't want the clients to communicate directly with the
> > > controller. So we weren't planning on including the controller's
> > > information in the MetadataResponse.
> > >
> > > We don't currently have rack-awareness for controllers, but I suppose we
> > > could in the future. I think we can leave that for later.
> > >
> > > >
> > > > 84. There are a few references to __kafka_metadata. However, KIP-595
> > > > uses __cluster_metadata.
> > >
> > > We agreed earlier to decide on the name of the metadata topic in this KIP,
> > > and to treat the KIP-595 name as provisional. Looking at it again, I think
> > > we should go with @metadata, in order to avoid colliding with topic names
> > > that already exist.
> > >
> > > >
> > > > 85. "When a broker is fenced, it cannot process any client requests. "
> > > > Should we send a new error code in the response to indicate that the
> > > broker
> > > > is fenced?
> > >
> > > I was hoping that we could avoid responding to requests when the broker
> > > was fenced. I suppose a new error code could work too, but then we also
> > > need a compatibility story for older clients.
> > >
> > > >
> > > > 86. "The controller can generate a new broker epoch by using the latest
> > > log
> > > > offset." Perhaps it's more accurate to say the new broker epoch is the
> > > > offset for the corresponding BrokerRecord in the metadata log.
> > >
> > > The idea here is slightly different. We want to use the current latest
> > > offset (prior to creating the BrokerRecord) as the new epoch. We can't use
> > > the offset of the BrokerRecord itself since we won't know what that is
> > > until we give the BrokerRecord to the Raft layer.
> > >
> > > >
> > > > 87. meta.properties:
> > > > 87.1 Could you document the format change to this file? Also, should we
> > > > bump up the version of the file?
> > >
> > > Hmm. The file is just a list of key=value pairs, one per line. So I'm
> > > not sure there is any format change per se, at least technically.
> > >
> > > Maybe we should bump up the version anyway, in order to prevent older
> > > pre-KIP-500 software from launching if the new stuff is present?
> > >
> > > > 87.2 How do we know a node is configured in KIP-500 mode?
> > > >
> > >
> > > A cluster configured in KIP-500 mode will contain the following line in
> > > the meta.properties file in each directory: kip.500.mode=enabled
> > >
> > > > 88. CurMetadataOffset: Should we use nextMetadataOffset since it's more
> > > > consistent with fetchOffset and HWM?
> > >
> > > I'm not sure. Drawing an analogy with HWM might be confusing, since
> > > CurMetadataOffset is not HWM. It's totally possible for the broker to have
> > > fetched up to offset 1000 but only applied up to offset 500.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, Oct 5, 2020 at 9:03 AM Colin McCabe <cm...@apache.org> wrote:
> > > >
> > > > > On Mon, Sep 28, 2020, at 11:41, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the reply. A few more comments below.
> > > > > >
> > > > > > 62.
> > > > > > 62.1 controller.listener.names: So, is this used for the controller
> > > or
> > > > > the
> > > > > > broker trying to connect to the controller?
> > > > > >
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > It's used by both. The broker tries to connect to controllers by using
> > > > > the first listener in this list. The controller uses the list to
> > > determine
> > > > > which listeners it should bind to.
> > > > >
> > > > > >
> > > > > > 62.2 If we want to take the approach to share the configs that are
> > > common
> > > > > > between the broker and the controller, should we share the id too?
> > > > > >
> > > > >
> > > > > On nodes where we are running both the controller and broker roles, we
> > > > > need two IDs: one for the controller, and one for the broker. So we
> > > can't
> > > > > share the same configuration.
> > > > >
> > > > > >
> > > > > > 62.3 We added some configs in KIP-595 prefixed with "quorum" and we
> > > plan
> > > > > to
> > > > > > add some controller specific configs prefixed with "controller".
> > > KIP-630
> > > > > > plans to add some other controller specific configs with no prefix.
> > > > > Should
> > > > > > we standardize all controller specific configs with the same prefix?
> > > > > >
> > > > >
> > > > > That's a good question. As Jose said elsewhere in the thread, let's
> > > use
> > > > > "metadata" as the prefix.
> > > > >
> > > > > >
> > > > > > 70. Could you explain the impact of process.roles a bit more? For
> > > > > example,
> > > > > > if process.roles=controller, does the node still maintain metadata
> > > cache
> > > > > as
> > > > > > described in KIP-630? Do we still return the host/port for those
> > > nodes in
> > > > > > the metadata response?
> > > > > >
> > > > >
> > > > > No, the node does not need any broker data structures if it is not a
> > > > > broker. Controllers also don't handle MetadataRequests. Clients will
> > > > > still go to the brokers for those.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Mon, Sep 28, 2020 at 9:24 AM Colin McCabe <cm...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > On Fri, Sep 25, 2020, at 17:35, Jun Rao wrote:
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the reply.
> > > > > > > >
> > > > > > > > 62. Thinking about this more, I am wondering what's our overall
> > > > > strategy
> > > > > > > > for configs shared between the broker and the controller. For
> > > > > example,
> > > > > > > both
> > > > > > > > the broker and the controller have to define listeners. So both
> > > need
> > > > > > > > configs like listeners/advertised.listeners. Both the new
> > > controller
> > > > > and
> > > > > > > > the broker replicate data. So both need to define some
> > > replication
> > > > > > > related
> > > > > > > > configurations (replica.fetch.min.bytes,
> > > replica.fetch.wait.max.ms,
> > > > > > > etc).
> > > > > > > > Should we let the new controller share those configs with
> > > brokers or
> > > > > > > should
> > > > > > > > we define new configs just for the controller? It seems that we
> > > want
> > > > > to
> > > > > > > > apply the same strategy consistently for all shared configs?
> > > > > > > >
> > > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > That's a good question. I think that we should share as many
> > > > > > > configurations as possible. There will be a few cases where this
> > > isn't
> > > > > > > possible, and we need to create a new configuration key that is
> > > > > > > controller-only, but I think that will be relatively rare.
> > > > > > >
> > > > > > > >
> > > > > > > > 63. If a node has process.roles set to controller, does one still
> > > > > need to
> > > > > > > > set broker.id on this node?
> > > > > > > >
> > > > > > >
> > > > > > > No, broker.id does not need to be set in that case.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Sep 25, 2020 at 2:17 PM Colin McCabe <cmccabe@apache.org
> > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Fri, Sep 25, 2020, at 10:17, Jun Rao wrote:
> > > > > > > > > > Hi, Colin,
> > > > > > > > > >
> > > > > > > > > > Thanks for the reply.
> > > > > > > > > >
> > > > > > > > > > 60. Yes, I think you are right. We probably need the
> > > controller
> > > > > id
> > > > > > > when a
> > > > > > > > > > broker starts up. A broker only stores the Raft leader id in
> > > the
> > > > > > > metadata
> > > > > > > > > > file. To do the initial fetch to the Raft leader, it needs to
> > > > > know
> > > > > > > the
> > > > > > > > > > host/port associated with the leader id.
> > > > > > > > > >
> > > > > > > > > > 62. It seems there are 2 parts to this : (1) which listener a
> > > > > client
> > > > > > > > > should
> > > > > > > > > > use to initiate a connection to the controller and (2) which
> > > > > listener
> > > > > > > > > > should a controller use to accept client requests. For (1),
> > > at
> > > > > any
> > > > > > > point
> > > > > > > > > of
> > > > > > > > > > time, a client only needs to use one listener. I think
> > > > > > > > > > controller.listener.name is meant for the client.
> > > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > controller.listener.names is also used by the controllers. In
> > > the
> > > > > case
> > > > > > > > > where we have a broker and a controller in the same JVM, we
> > > have a
> > > > > > > single
> > > > > > > > > config file. Then we need to know which listeners belong to
> > > the
> > > > > > > controller
> > > > > > > > > and which belong to the broker component. That's why it's a
> > > list.
> > > > > > > > >
> > > > > > > > > > So, a single value seems
> > > > > > > > > > to make more sense. Currently, we don't have a configuration
> > > for
> > > > > > > (2). We
> > > > > > > > > > could add a new one for that and support a list. I am
> > > wondering
> > > > > how
> > > > > > > > > useful
> > > > > > > > > > it will be. One example that I can think of is that we can
> > > reject
> > > > > > > > > > non-controller related requests if accepted on
> > > controller-only
> > > > > > > listeners.
> > > > > > > > > > However, we already support separate authentication for the
> > > > > > > controller
> > > > > > > > > > listener. So, not sure how useful it is.
> > > > > > > > >
> > > > > > > > > The controller always has a separate listener and does not
> > > share
> > > > > > > listeners
> > > > > > > > > with the broker. The main reason for this is to avoid giving
> > > > > clients
> > > > > > > the
> > > > > > > > > ability to launch a denial-of-service attack on the controller
> > > by
> > > > > > > flooding
> > > > > > > > > a broker port. A lot of times, these attacks are made
> > > > > unintentionally
> > > > > > > by
> > > > > > > > > poorly coded or configured clients. Additionally, broker
> > > ports can
> > > > > > > also be
> > > > > > > > > very busy with large fetch requests, and so on. It's just a
> > > bad
> > > > > > > > > configuration in general to try to overload the same port for
> > > the
> > > > > > > > > controller and broker, and we don't want to allow it. It
> > > would be
> > > > > a
> > > > > > > > > regression to go from the current system where control
> > > requests are
> > > > > > > safely
> > > > > > > > > isolated on a separate port, to one where they're not. It also
> > > > > makes
> > > > > > > the
> > > > > > > > > code and configuration a lot messier.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 63. (a) I think most users won't know controller.id
> > > defaults to
> > > > > > > > > broker.id +
> > > > > > > > > > 3000. So, it can be confusing for them to set up
> > > > > controller.connect.
> > > > > > > If
> > > > > > > > > > this is truly needed, it seems that it's less confusing to
> > > make
> > > > > > > > > > controller.id required.
> > > > > > > > > > (b) I am still trying to understand if we truly need to
> > > expose a
> > > > > > > > > > controller.id. What if we only expose broker.id and let
> > > > > > > > > controller.connect
> > > > > > > > > > just use broker.id? What will be missing?
> > > > > > > > >
> > > > > > > > > The controller has a separate ID from the broker, so knowing
> > > > > broker.id
> > > > > > > is
> > > > > > > > > not helpful here.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 24, 2020 at 10:55 PM Colin McCabe <
> > > > > cmccabe@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 24, 2020, at 16:24, Jun Rao wrote:
> > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the reply and the updated KIP. A few more
> > > comments
> > > > > > > below.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Jun,
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 53. It seems that you already incorporated the changes in
> > > > > > > KIP-516.
> > > > > > > > > With
> > > > > > > > > > > > topic ids, we don't need to wait for the topic's data to
> > > be
> > > > > > > deleted
> > > > > > > > > > > before
> > > > > > > > > > > > removing the topic metadata. If the topic is recreated,
> > > we
> > > > > can
> > > > > > > still
> > > > > > > > > > > delete
> > > > > > > > > > > > the data properly based on the topic id. So, it seems
> > > that
> > > > > we can
> > > > > > > > > remove
> > > > > > > > > > > > TopicRecord.Deleting.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the reply. What I was thinking of doing here
> > > was
> > > > > using
> > > > > > > > > topic
> > > > > > > > > > > IDs internally, but still using names externally. So the
> > > topic
> > > > > > > UUIDs
> > > > > > > > > are
> > > > > > > > > > > only for the purpose of associating topics with partitions
> > > --
> > > > > from
> > > > > > > the
> > > > > > > > > > > user's point of view, topics are still identified by names.
> > > > > > > > > > >
> > > > > > > > > > > You're right that KIP-516 will simplify things, but I'm not
> > > > > sure
> > > > > > > when
> > > > > > > > > that
> > > > > > > > > > > will land, so I wanted to avoid blocking the initial
> > > > > > > implementation of
> > > > > > > > > this
> > > > > > > > > > > KIP on it.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 55. It seems to me that the current behavior where we
> > > favor
> > > > > the
> > > > > > > > > current
> > > > > > > > > > > > broker registration is better. This is because
> > > uncontrolled
> > > > > > > broker
> > > > > > > > > > > shutdown
> > > > > > > > > > > > is rare and its impact is less severe since one just
> > > needs to
> > > > > > > wait
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > session timeout before restarting the broker. If a
> > > > > mis-configured
> > > > > > > > > broker
> > > > > > > > > > > > replaces an existing broker, the consequence is more
> > > severe
> > > > > > > since it
> > > > > > > > > can
> > > > > > > > > > > > cause the leader to be unavailable, a replica to be out
> > > of
> > > > > ISR,
> > > > > > > and
> > > > > > > > > add
> > > > > > > > > > > > more load on the leaders etc.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hmm, that's a good point. Let me check this a bit more
> > > before
> > > > > I
> > > > > > > change
> > > > > > > > > > > it, though.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 60. controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com
> > > :
> > > > > Do we
> > > > > > > > > need to
> > > > > > > > > > > > include the controller id before @? It seems that the
> > > > > host/port
> > > > > > > is
> > > > > > > > > enough
> > > > > > > > > > > > for establishing the connection. It would also be useful
> > > to
> > > > > make
> > > > > > > it
> > > > > > > > > clear
> > > > > > > > > > > > that controller.connect replaces quorum.voters in
> > > KIP-595.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I discussed this with Jason earlier, and he felt that the
> > > > > > > controller
> > > > > > > > > IDs
> > > > > > > > > > > were needed in this configuration key. It is certainly
> > > needed
> > > > > when
> > > > > > > > > > > configuring the controllers themselves, since they need to
> > > know
> > > > > > > each
> > > > > > > > > > > others' IDs.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 61. I am not sure that we need both controller.listeners
> > > and
> > > > > > > > > > > > controller.connect.security.protocol since the former
> > > > > implies the
> > > > > > > > > > > security
> > > > > > > > > > > > protocol. The reason that we have both
> > > > > > > inter.broker.listener.name
> > > > > > > > > and
> > > > > > > > > > > > inter.broker.security.protocol is because we had the
> > > latter
> > > > > > > first and
> > > > > > > > > > > later
> > > > > > > > > > > > realized that the former is more general.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > That's a good point. I've removed this from the KIP.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 62. I am still not sure that you need
> > > controller.listeners
> > > > > to be
> > > > > > > a
> > > > > > > > > list.
> > > > > > > > > > > > All listeners are already defined in listeners. To
> > > migrate
> > > > > from
> > > > > > > > > plaintext
> > > > > > > > > > > > to SSL, one can configure listeners to have both
> > > plaintext
> > > > > and
> > > > > > > SSL.
> > > > > > > > > After
> > > > > > > > > > > > that, one can just change controller.listeners from
> > > > > plaintext to
> > > > > > > SSL.
> > > > > > > > > > > This
> > > > > > > > > > > > is similar to how to change the listener for inter broker
> > > > > > > > > connections.
> > > > > > > > > > > > Also, controller.listener.name may be a more accurate
> > > name?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The issue that I see here is that if you are running with
> > > the
> > > > > > > > > controller
> > > > > > > > > > > and broker in the same JVM, if you define a few listeners
> > > in
> > > > > > > > > "listeners"
> > > > > > > > > > > they will get used as regular broker listeners, unless you
> > > put
> > > > > > > them in
> > > > > > > > > > > controller.listeners. Therefore, controller.listeners
> > > needs
> > > > > to be
> > > > > > > a
> > > > > > > > > list.
> > > > > > > > > > >
> > > > > > > > > > > controller.listener.names does sound like a better name,
> > > > > though...
> > > > > > > I've
> > > > > > > > > > > updated it to that.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 63. Regarding controller.id, I am trying to understand
> > > > > whether
> > > > > > > it's
> > > > > > > > > a
> > > > > > > > > > > > required configuration or an optional one. From the KIP,
> > > it
> > > > > > > sounds
> > > > > > > > > like
> > > > > > > > > > > > controller.id is optional. Then, if this is unset, it's
> > > not
> > > > > > > clear
> > > > > > > > > how
> > > > > > > > > > > the
> > > > > > > > > > > > user will obtain the controller id for setting
> > > > > > > controller.connect.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > If you specify broker.id but not controller.id, then
> > > > > controller.id
> > > > > > > > > will
> > > > > > > > > > > just be set to broker.id + 3000. This was intended to
> > > make
> > > > > some
> > > > > > > > > > > configurations a little bit more concise to write. You
> > > still
> > > > > do
> > > > > > > have
> > > > > > > > > to
> > > > > > > > > > > know the controller IDs when configuring the brokers,
> > > though.
> > > > > If
> > > > > > > this
> > > > > > > > > > > seems confusing then I can just make it mandatory.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 64. KIP-516 adds a flag in LeaderAndIsrRequest to
> > > indicate
> > > > > > > whether it
> > > > > > > > > > > > includes all partitions or not. This will be used to
> > > clean up
> > > > > > > strayed
> > > > > > > > > > > logs.
> > > > > > > > > > > > I was thinking how we will do the same thing once the
> > > > > controller
> > > > > > > > > moves to
> > > > > > > > > > > > Raft based. One way to do that is on broker startup, it
> > > gets
> > > > > the
> > > > > > > HWM
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > metadata log from the Raft leader and waits until its
> > > > > metadata
> > > > > > > > > catches up
> > > > > > > > > > > > to HWM. At that point, any local log not seen in the
> > > metadata
> > > > > > > can be
> > > > > > > > > > > > removed. Since the Fetch response returns the HWM, there
> > > > > seems
> > > > > > > to be
> > > > > > > > > > > enough
> > > > > > > > > > > > APIs to achieve this.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > That's a very good point. I added a note about this under
> > > > > Broker
> > > > > > > > > Startup.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Jun
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 24, 2020 at 1:07 PM Colin McCabe <
> > > > > colin@cmccabe.xyz>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Sep 21, 2020, at 18:13, Jun Rao wrote:
> > > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Sorry for the late reply. A few more comments below.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for taking another look.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 50. Configurations
> > > > > > > > > > > > > > 50.1 controller.listeners: It seems that a controller
> > > > > just
> > > > > > > needs
> > > > > > > > > one
> > > > > > > > > > > > > > listener. Why do we need to have a list here? Also,
> > > > > could you
> > > > > > > > > > > provide an
> > > > > > > > > > > > > > example of how this is set and what's its
> > > relationship
> > > > > with
> > > > > > > > > existing
> > > > > > > > > > > > > > configs such as "security.inter.broker.protocol" and
> > > "
> > > > > > > > > > > > > > inter.broker.listener.name"?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I agree that most administrators will want to run with
> > > > > only one
> > > > > > > > > > > controller
> > > > > > > > > > > > > listener. However, just as with brokers, it is nice to
> > > > > have
> > > > > > > the
> > > > > > > > > > > option to
> > > > > > > > > > > > > expose multiple ports.
> > > > > > > > > > > > >
> > > > > > > > > > > > > One reason why you might want multiple ports is if you
> > > were
> > > > > > > doing a
> > > > > > > > > > > > > migration from plaintext to SSL. You could add new SSL
> > > > > > > listeners
> > > > > > > > > but
> > > > > > > > > > > > > continue to expose the PLAINTEXT listeners. Then you
> > > could
> > > > > > > add the
> > > > > > > > > > > new SSL
> > > > > > > > > > > > > controller config to each broker and roll each broker.
> > > > > Then
> > > > > > > you
> > > > > > > > > could
> > > > > > > > > > > > > migrate from PLAINTEXT to SSL with no downtime.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Here's an example configuration for the controller:
> > > > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > > 1@controller1.example.com:9093,
> > > 2@controller2.example.com
> > > > > > > > > > > > > controller.listeners=CONTROLLER
> > > > > > > > > > > > > listeners=CONTROLLER://controller0.example.com:9093
> > > > > > > > > > > > > listener.security.protocol.map=CONTROLLER:SSL
> > > > > > > > > > > > >
> > > > > > > > > > > > > Here's an example configuration for the broker:
> > > > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > > 1@controller1.example.com:9093,
> > > 2@controller2.example.com
> > > > > > > > > > > > > controller.connect.security.protocol=SSL
> > > > > > > > > > > > >
> > > > > > > > > > > > > security.inter.broker.protocol or
> > > > > inter.broker.listener.name
> > > > > > > do
> > > > > > > > > not
> > > > > > > > > > > > > affect how the broker communicates with the controller.
> > > > > Those
> > > > > > > > > > > > > configurations relate to how brokers communicate with
> > > each
> > > > > > > other,
> > > > > > > > > but
> > > > > > > > > > > the
> > > > > > > > > > > > > controller is not a broker.
> > > > > > > > > > > > >
> > > > > > > > > > > > > (Note that I just added
> > > > > controller.connect.security.protocol
> > > > > > > to the
> > > > > > > > > > > KIP --
> > > > > > > > > > > > > I had forgotten to put it in earlier)
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 50.2 registration.heartbeat.interval.ms and
> > > > > > > > > > > > > registration.lease.timeout.ms.
> > > > > > > > > > > > > > Should we match their default value with the
> > > > > corresponding
> > > > > > > > > default
> > > > > > > > > > > for
> > > > > > > > > > > > > ZK?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Fair enough. I'll set them to the values of the
> > > > > > > > > > > zookeeper.sync.time.ms
> > > > > > > > > > > > > and zookeeper.connection.timeout.ms configurations.
> > > I do
> > > > > > > think we
> > > > > > > > > > > should
> > > > > > > > > > > > > experiment later on to see what works well here, but
> > > the ZK
> > > > > > > values
> > > > > > > > > are
> > > > > > > > > > > at
> > > > > > > > > > > > > least a starting point.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 50.3 controller.connect: Could you provide an
> > > example? I
> > > > > am
> > > > > > > > > > > wondering how
> > > > > > > > > > > > > > it differs from quorum.voters=1@kafka-1:9092,
> > > 2@kafka-2
> > > > > > > :9092,
> > > > > > > > > > > 3@kafka-3
> > > > > > > > > > > > > > :9092.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > controller.connect is intended to be the new name for
> > > > > > > > > quorum.voters.
> > > > > > > > > > > > > During the vote for KIP-595, we sort of agreed to
> > > defer the
> > > > > > > > > discussion
> > > > > > > > > > > > > about what this configuration should be called. I
> > > proposed
> > > > > > > this
> > > > > > > > > new
> > > > > > > > > > > name
> > > > > > > > > > > > > because it makes it clear what the configuration is for
> > > > > (how to
> > > > > > > > > > > connect to
> > > > > > > > > > > > > the controllers).
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 50.4 controller.id: I am still not sure how this is
> > > > > being
> > > > > > > used.
> > > > > > > > > > > Could
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > explain this in more detail?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > The controller ID needs to be set on each controller.
> > > It
> > > > > also
> > > > > > > > > appears
> > > > > > > > > > > in
> > > > > > > > > > > > > controller.connect as the thing before the "at" sign.
> > > Its
> > > > > > > > > function is
> > > > > > > > > > > > > pretty similar to broker.id.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Broker IDs and controller IDs exist in the same ID
> > > space,
> > > > > so
> > > > > > > you
> > > > > > > > > can't
> > > > > > > > > > > > > have a broker and a controller that use the same ID.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 51. BrokerHeartbeat: It seems a bit wasteful to
> > > include
> > > > > > > > > Listeners in
> > > > > > > > > > > > > every
> > > > > > > > > > > > > > heartbeat request since it typically doesn't change.
> > > > > Could we
> > > > > > > > > make
> > > > > > > > > > > that
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > optional field?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ok.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 52. KIP-584 adds a new ZK node /features. Should we
> > > add a
> > > > > > > > > > > corresponding
> > > > > > > > > > > > > > metadata record?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Good point. I added FeatureLevelRecord.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 53. TopicRecord and DeleteTopic: Both DeleteTopic and
> > > > > > > > > > > > > TopicRecord.Deleting
> > > > > > > > > > > > > > indicate topic deletion. Could we outline the flow
> > > when
> > > > > each
> > > > > > > > > will be
> > > > > > > > > > > set?
> > > > > > > > > > > > > > In particular, which one indicates the intention to
> > > > > delete
> > > > > > > and
> > > > > > > > > which
> > > > > > > > > > > one
> > > > > > > > > > > > > > indicates the completion of the deletion.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > TopicRecord.Deleting is set when we intend to delete
> > > the
> > > > > topic.
> > > > > > > > > > > > >
> > > > > > > > > > > > > DeleteTopic removes the topic completely. I will
> > > rename
> > > > > > > > > DeleteTopic to
> > > > > > > > > > > > > RemoveTopic to make this clearer.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 54. "The controller can generate a new broker epoch
> > > by
> > > > > using
> > > > > > > the
> > > > > > > > > > > latest
> > > > > > > > > > > > > log
> > > > > > > > > > > > > > offset." Which offset is that? Is it the offset of
> > > the
> > > > > > > metadata
> > > > > > > > > > > topic for
> > > > > > > > > > > > > > the corresponding BrokerRecord? Is it guaranteed to
> > > be
> > > > > > > different
> > > > > > > > > on
> > > > > > > > > > > each
> > > > > > > > > > > > > > broker restart?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes.... a new broker epoch implies that there has been
> > > a
> > > > > new
> > > > > > > record
> > > > > > > > > > > > > created in the metadata log. Therefore the last
> > > committed
> > > > > > > offset
> > > > > > > > > must
> > > > > > > > > > > be
> > > > > > > > > > > > > different.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 55. "Thereafter, it may lose subsequent conflicts if
> > > its
> > > > > > > broker
> > > > > > > > > > > epoch is
> > > > > > > > > > > > > > stale. (See KIP-380 for some background on broker
> > > > > epoch.)
> > > > > > > The
> > > > > > > > > > > reason
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > favoring new processes is to accommodate the common
> > > case
> > > > > > > where a
> > > > > > > > > > > process
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > killed with kill -9 and then restarted. " Are you
> > > saying
> > > > > > > that if
> > > > > > > > > > > there is
> > > > > > > > > > > > > > an active broker registered in the controller, a new
> > > > > broker
> > > > > > > > > heartbeat
> > > > > > > > > > > > > > request with the INITIAL state will fence the current
> > > > > broker
> > > > > > > > > > > session? Not
> > > > > > > > > > > > > > sure about this. For example, this will allow a
> > > broker
> > > > > with
> > > > > > > > > > > incorrectly
> > > > > > > > > > > > > > configured broker id to kill an existing valid
> > > broker.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, a new broker with an incorrectly configured
> > > broker id
> > > > > > > could
> > > > > > > > > fence
> > > > > > > > > > > an
> > > > > > > > > > > > > existing valid broker.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The goal of the conflict resolution process described
> > > here
> > > > > is
> > > > > > > to
> > > > > > > > > avoid
> > > > > > > > > > > > > having two brokers go back and forth in claiming the
> > > > > broker id.
> > > > > > > > > > > Choosing
> > > > > > > > > > > > > the newest was just an easy way to do that.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Choosing the oldest is another possibility, and more
> > > inline
> > > > > > > with
> > > > > > > > > what
> > > > > > > > > > > > > ZooKeeper does today. The problem with that is that it
> > > > > would
> > > > > > > not
> > > > > > > > > be
> > > > > > > > > > > > > possible to re-create the broker if the old instance
> > > died
> > > > > > > suddenly,
> > > > > > > > > > > except
> > > > > > > > > > > > > by waiting for the full lease timeout.
> > > > > > > > > > > > >
> > > > > > > > > > > > > It might be possible to do a little better here by
> > > > > breaking the
> > > > > > > > > lease
> > > > > > > > > > > if
> > > > > > > > > > > > > the TCP connection from the broker drops. But that
> > > > > requires
> > > > > > > > > crossing
> > > > > > > > > > > some
> > > > > > > > > > > > > layers of abstraction in the Kafka networking stack.
> > > > > There's
> > > > > > > also
> > > > > > > > > the
> > > > > > > > > > > > > possibility of false negatives or positives. For
> > > example,
> > > > > TCP
> > > > > > > > > > > connections
> > > > > > > > > > > > > sometimes just drop even if the node is still there.
> > > Or
> > > > > > > sometimes
> > > > > > > > > the
> > > > > > > > > > > > > networking stack keeps a TCP connection alive for a
> > > while
> > > > > when
> > > > > > > the
> > > > > > > > > > > other
> > > > > > > > > > > > > end is gone.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 56. kafka-storage.sh:
> > > > > > > > > > > > > > 56.1 In the info mode, what other information does it
> > > > > show in
> > > > > > > > > > > addition to
> > > > > > > > > > > > > > "kip.500.mode=enabled"?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think it should show a list of all the storage
> > > > > directories,
> > > > > > > and
> > > > > > > > > > > whether
> > > > > > > > > > > > > each one is formatted. In addition, it should show
> > > whether
> > > > > > > > > > > kip.500.mode is
> > > > > > > > > > > > > enabled, and what the cluster id is.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 56.2 Should the format mode take the config file as
> > > the
> > > > > > > input too
> > > > > > > > > > > like
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > info mode?
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, that's a good idea.
> > > > > > > > > > > > >
> > > > > > > > > > > > > best,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 17, 2020 at 4:12 PM Colin McCabe <
> > > > > > > cmccabe@apache.org
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > That's a fair point. I have moved the lease
> > > duration
> > > > > to
> > > > > > > the
> > > > > > > > > broker
> > > > > > > > > > > > > > > heartbeat response. That way lease durations can
> > > be
> > > > > > > changed
> > > > > > > > > just
> > > > > > > > > > > be
> > > > > > > > > > > > > > > reconfiguring the controllers.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Wed, Sep 16, 2020, at 07:40, Unmesh Joshi wrote:
> > > > > > > > > > > > > > > > Thanks Colin, the changes look good to me. One
> > > small
> > > > > > > thing.
> > > > > > > > > > > > > > > > registration.lease.timeout.ms is the
> > > configuration
> > > > > on
> > > > > > > the
> > > > > > > > > > > controller
> > > > > > > > > > > > > > > side.
> > > > > > > > > > > > > > > > It will be good to comment on how brokers know
> > > about
> > > > > it,
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > able
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > send LeaseDurationMs
> > > > > > > > > > > > > > > > in the heartbeat request,
> > > > > > > > > > > > > > > > or else it can be added in the heartbeat
> > > response for
> > > > > > > > > brokers to
> > > > > > > > > > > know
> > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Sep 11, 2020 at 10:32 PM Colin McCabe <
> > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think you're right that we should use a
> > > duration
> > > > > here
> > > > > > > > > rather
> > > > > > > > > > > > > than a
> > > > > > > > > > > > > > > > > time. As you said, the clock on the controller
> > > > > will
> > > > > > > > > probably
> > > > > > > > > > > not
> > > > > > > > > > > > > > > match the
> > > > > > > > > > > > > > > > > one on the broker. I have updated the KIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It's important to keep in mind that
> > > messages
> > > > > may be
> > > > > > > > > > > delayed in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > > > happens, we
> > > > > > > > > > > will
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > > time specified in the request to determine
> > > if
> > > > > the
> > > > > > > > > request
> > > > > > > > > > > is
> > > > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > > > I am assuming that there will be a single TCP
> > > > > > > connection
> > > > > > > > > > > > > maintained
> > > > > > > > > > > > > > > > > between
> > > > > > > > > > > > > > > > > > broker and active controller. So, there
> > > won't be
> > > > > any
> > > > > > > out
> > > > > > > > > of
> > > > > > > > > > > order
> > > > > > > > > > > > > > > > > requests?
> > > > > > > > > > > > > > > > > > There will be a scenario of broker GC pause,
> > > > > which
> > > > > > > might
> > > > > > > > > > > cause
> > > > > > > > > > > > > > > connection
> > > > > > > > > > > > > > > > > > timeout and broker might need to reestablish
> > > the
> > > > > > > > > connection.
> > > > > > > > > > > If
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > pause
> > > > > > > > > > > > > > > > > > is too long, lease will expire and the
> > > heartbeat
> > > > > sent
> > > > > > > > > after
> > > > > > > > > > > the
> > > > > > > > > > > > > pause
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > be treated as a new registration (similar to
> > > > > restart
> > > > > > > > > case),
> > > > > > > > > > > and
> > > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > > epoch
> > > > > > > > > > > > > > > > > > number will be assigned to the broker.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I agree with the end of this paragraph, but not
> > > > > with
> > > > > > > the
> > > > > > > > > start
> > > > > > > > > > > :)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There can be out-of-order requests, since the
> > > > > broker
> > > > > > > will
> > > > > > > > > > > simply
> > > > > > > > > > > > > use a
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > TCP connection if the old one has problems.
> > > This
> > > > > can
> > > > > > > > > happen
> > > > > > > > > > > for a
> > > > > > > > > > > > > > > variety
> > > > > > > > > > > > > > > > > of reasons. I don't think GC pauses are the
> > > most
> > > > > > > common
> > > > > > > > > > > reason for
> > > > > > > > > > > > > > > this to
> > > > > > > > > > > > > > > > > happen. It's more common to see issues issues
> > > in
> > > > > the
> > > > > > > > > network
> > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > result connections getting dropped from time to
> > > > > time.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > So we have to assume that messages may arrive
> > > out
> > > > > of
> > > > > > > > > order, and
> > > > > > > > > > > > > > > possibly
> > > > > > > > > > > > > > > > > be delayed. I added a note that heartbeat
> > > requests
> > > > > > > should
> > > > > > > > > be
> > > > > > > > > > > > > ignored
> > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > the metadata log offset they contain is smaller
> > > > > than a
> > > > > > > > > previous
> > > > > > > > > > > > > > > heartbeat.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > When the active controller fails, the new
> > > active
> > > > > > > > > controller
> > > > > > > > > > > > > needs to
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > sure that it considers all the known brokers
> > > as
> > > > > alive
> > > > > > > > > till
> > > > > > > > > > > the
> > > > > > > > > > > > > lease
> > > > > > > > > > > > > > > > > > expiration interval. Because
> > > > > > > > > registration.lease.timeout.ms,
> > > > > > > > > > > is
> > > > > > > > > > > > > > > > > configured
> > > > > > > > > > > > > > > > > > on the controller, the new active controller
> > > will
> > > > > > > extend
> > > > > > > > > all
> > > > > > > > > > > the
> > > > > > > > > > > > > > > leases
> > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > registration.lease.timeout.ms. I see that it
> > > > > won't
> > > > > > > need
> > > > > > > > > last
> > > > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Agreed.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Sat, Sep 5, 2020 at 1:28 AM Colin McCabe <
> > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Colin wrote:
> > > > > > > > > > > > > > > > > > > > > The reason for including
> > > LeaseStartTimeMs
> > > > > in
> > > > > > > the
> > > > > > > > > > > request
> > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > > > > that the time required to communicate
> > > with
> > > > > the
> > > > > > > > > > > controller
> > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > included in
> > > > > > > > > > > > > > > > > > > > > the lease time. Since requests can
> > > > > > > potentially be
> > > > > > > > > > > delayed
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > > > > for a long time, this is important.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020, at 05:58, Unmesh
> > > Joshi
> > > > > wrote:
> > > > > > > > > > > > > > > > > > > > The network time will be added anyway,
> > > > > because
> > > > > > > the
> > > > > > > > > lease
> > > > > > > > > > > > > timer
> > > > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > > > active controller will start only after
> > > the
> > > > > > > heartbeat
> > > > > > > > > > > request
> > > > > > > > > > > > > > > > > reaches the
> > > > > > > > > > > > > > > > > > > > server.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > If the start time is not specified in the
> > > > > request,
> > > > > > > > > then the
> > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > > > is excluded from the heartbeat time.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Here's an example:
> > > > > > > > > > > > > > > > > > > Let's say broker A sends a heartbeat at
> > > time
> > > > > 100,
> > > > > > > and
> > > > > > > > > it
> > > > > > > > > > > > > arrives
> > > > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > > controller at time 200, and the lease
> > > duration
> > > > > is
> > > > > > > 1000.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The controller looks at the start time in
> > > the
> > > > > > > request,
> > > > > > > > > > > which is
> > > > > > > > > > > > > > > 100,
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > adds 1000 to it, getting 1100. On the
> > > other
> > > > > hand,
> > > > > > > if
> > > > > > > > > start
> > > > > > > > > > > > > time
> > > > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > > > specified in the request, then the
> > > expiration
> > > > > will
> > > > > > > be
> > > > > > > > > at
> > > > > > > > > > > time
> > > > > > > > > > > > > 1200.
> > > > > > > > > > > > > > > > > > > That is what I mean by "the network time is
> > > > > > > included
> > > > > > > > > in the
> > > > > > > > > > > > > > > expiration
> > > > > > > > > > > > > > > > > > > time."
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > And I think, some assumption about
> > > network
> > > > > round
> > > > > > > trip
> > > > > > > > > > > time is
> > > > > > > > > > > > > > > > > > > > needed anyway to decide on the frequency
> > > of
> > > > > the
> > > > > > > > > > > heartbeat (
> > > > > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms), and
> > > > > lease
> > > > > > > > > timeout (
> > > > > > > > > > > > > > > > > > > > registration.lease.timeout.ms). So I
> > > think
> > > > > just
> > > > > > > > > having a
> > > > > > > > > > > > > > > leaseTTL
> > > > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > > > request is easier to understand and
> > > > > implement.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It's important to keep in mind that
> > > messages
> > > > > may be
> > > > > > > > > > > delayed in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > > > happens, we
> > > > > > > > > > > will
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > > time specified in the request to determine
> > > if
> > > > > the
> > > > > > > > > request
> > > > > > > > > > > is
> > > > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> > > the
> > > > > > > > > controller
> > > > > > > > > > > side
> > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > reset in the case of controller
> > > failover.
> > > > > The
> > > > > > > > > > > alternative
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > be to
> > > > > > > > > > > > > > > > > > > > > track the lease as hard state rather
> > > than
> > > > > soft
> > > > > > > > > state,
> > > > > > > > > > > but I
> > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > is not really needed, and would result
> > > in
> > > > > more
> > > > > > > log
> > > > > > > > > > > entries.
> > > > > > > > > > > > > > > > > > > > My interpretation of the mention of
> > > > > BrokerRecord
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > KIP
> > > > > > > > > > > > > was
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > record exists in the Raft log.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > BrokerRecord does exist in the Raft log,
> > > but
> > > > > does
> > > > > > > not
> > > > > > > > > > > include
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > > heartbeat time.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > By soft state, do you mean the broker
> > > > > > > > > > > > > > > > > > > > records exist only on the active leader
> > > and
> > > > > will
> > > > > > > not
> > > > > > > > > be
> > > > > > > > > > > > > > > replicated
> > > > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > > > raft log? If the live brokers list is
> > > > > maintained
> > > > > > > > > only on
> > > > > > > > > > > the
> > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > controller (raft leader), then, in case
> > > of
> > > > > leader
> > > > > > > > > > > failure,
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > will be
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > window where the new leader does not know
> > > > > about
> > > > > > > the
> > > > > > > > > live
> > > > > > > > > > > > > brokers,
> > > > > > > > > > > > > > > > > till
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > brokers establish the leases again.
> > > > > > > > > > > > > > > > > > > > I think it will be safer to have leases
> > > as a
> > > > > hard
> > > > > > > > > state
> > > > > > > > > > > > > managed
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > standard
> > > > > > > > > > > > > > > > > > > > Raft replication.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Leases are short, so the need to
> > > re-establish
> > > > > them
> > > > > > > > > after a
> > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > failover doesn't seem like a big problem.
> > > But
> > > > > > > this is
> > > > > > > > > > > > > something
> > > > > > > > > > > > > > > we can
> > > > > > > > > > > > > > > > > > > tweak if it becomes an issue. One option
> > > > > would be
> > > > > > > to
> > > > > > > > > have
> > > > > > > > > > > a
> > > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > > which is only used by the controller nodes
> > > for
> > > > > this
> > > > > > > > > (since,
> > > > > > > > > > > > > after
> > > > > > > > > > > > > > > all,
> > > > > > > > > > > > > > > > > > > brokers don't care about registration
> > > > > renewals).
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Or am I misunderstanding something? (I
> > > assume
> > > > > > > that
> > > > > > > > > with
> > > > > > > > > > > soft
> > > > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > mean something like zookeeper local
> > > sessions
> > > > > > > > > > > > > > > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1147
> > > > > > > > > .)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > > > think
> > > > > > > it
> > > > > > > > > makes
> > > > > > > > > > > > > sense
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> > > becomes
> > > > > > > very
> > > > > > > > > messy.
> > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > > > understand your question about
> > > duplicate
> > > > > > > broker ID
> > > > > > > > > > > > > detection,
> > > > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > > There's a section in the KIP about this
> > > -- is
> > > > > > > there a
> > > > > > > > > > > detail
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This is an implementation detail that
> > > doesn't
> > > > > need
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > in the
> > > > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I assumed broker leases are implemented
> > > as a
> > > > > hard
> > > > > > > > > state.
> > > > > > > > > > > In
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > case, to
> > > > > > > > > > > > > > > > > > > > check for broker id conflict, we need to
> > > > > check
> > > > > > > the
> > > > > > > > > broker
> > > > > > > > > > > > > ids at
> > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > places
> > > > > > > > > > > > > > > > > > > > 1. Pending broker registrations (which
> > > are
> > > > > yet
> > > > > > > to be
> > > > > > > > > > > > > committed)
> > > > > > > > > > > > > > > 2.
> > > > > > > > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > > > > > > committed broker registrations.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020 at 5:42 PM Colin
> > > McCabe
> > > > > <
> > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020, at 01:12, Unmesh
> > > > > Joshi
> > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >>>Can you repeat your questions
> > > about
> > > > > broker
> > > > > > > > > leases?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >>>>The LeaseStartTimeMs is expected
> > > to
> > > > > be
> > > > > > > the
> > > > > > > > > > > broker's
> > > > > > > > > > > > > > > > > > > > > > 'System.currentTimeMillis()' at the
> > > > > point of
> > > > > > > the
> > > > > > > > > > > > > request. The
> > > > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > > > controller will add its lease period
> > > to
> > > > > this
> > > > > > > in
> > > > > > > > > order
> > > > > > > > > > > > > >>>>to
> > > > > > > > > > > > > > > > > compute
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > LeaseEndTimeMs.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think the use of LeaseStartTimeMs
> > > and
> > > > > > > > > > > LeaseEndTimeMs
> > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > > confusing. Monotonic Clock
> > > > > > > (System.nanoTime) on
> > > > > > > > > the
> > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > should be used to track leases.
> > > > > > > > > > > > > > > > > > > > > > (For example,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-1616https://github.com/etcd-io/etcd/pull/6888/commits/e7f4010ccaf28b6ce64fe514d25a4b2fa459d114
> > > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Then we will not need
> > > LeaseStartTimeMs?
> > > > > > > > > > > > > > > > > > > > > > Instead of LeaseStartTimeMs, can we
> > > call
> > > > > it
> > > > > > > > > > > LeaseTTL? The
> > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > can then calculate LeaseEndTime =
> > > > > > > > > System.nanoTime() +
> > > > > > > > > > > > > > > LeaseTTL.
> > > > > > > > > > > > > > > > > > > > > > In this case we might just drop
> > > > > > > LeaseEndTimeMs
> > > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > response, as
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > broker already knows about the TTL
> > > and
> > > > > can
> > > > > > > send
> > > > > > > > > > > > > heartbeats at
> > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > fraction
> > > > > > > > > > > > > > > > > > > > > > of TTL, say every TTL/4
> > > > > milliseconds.(elapsed
> > > > > > > > > time
> > > > > > > > > > > on the
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > measured
> > > > > > > > > > > > > > > > > > > > > > by System.nanoTime)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I agree that the monotonic clock is
> > > > > probably a
> > > > > > > > > better
> > > > > > > > > > > idea
> > > > > > > > > > > > > > > here.
> > > > > > > > > > > > > > > > > It is
> > > > > > > > > > > > > > > > > > > > > good to be robust against wall clock
> > > > > changes,
> > > > > > > > > although
> > > > > > > > > > > I
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > > > > > which had them might suffer other
> > > issues.
> > > > > I
> > > > > > > will
> > > > > > > > > > > change
> > > > > > > > > > > > > it to
> > > > > > > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > monotonic clock.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The reason for including
> > > LeaseStartTimeMs
> > > > > in
> > > > > > > the
> > > > > > > > > > > request
> > > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > the time required to communicate with
> > > the
> > > > > > > > > controller
> > > > > > > > > > > gets
> > > > > > > > > > > > > > > included
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > lease time. Since requests can
> > > > > potentially be
> > > > > > > > > delayed
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > > > > > > long time, this is important.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I have a prototype built to
> > > demonstrate
> > > > > this
> > > > > > > as
> > > > > > > > > > > > > following:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The Kip631Controller itself depends
> > > on a
> > > > > > > > > Consensus
> > > > > > > > > > > > > module, to
> > > > > > > > > > > > > > > > > > > demonstrate
> > > > > > > > > > > > > > > > > > > > > > how possible interactions with the
> > > > > consensus
> > > > > > > > > module
> > > > > > > > > > > will
> > > > > > > > > > > > > look
> > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > > (The Consensus can be pluggable
> > > really,
> > > > > > > with an
> > > > > > > > > API
> > > > > > > > > > > to
> > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > > > replicated log upto HighWaterMark)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > It has an implementation of
> > > LeaseTracker
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/LeaderLeaseTracker.scala
> > > > > > > > > > > > > > > > > > > > > > to demonstrate LeaseTracker's
> > > interaction
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > consensus
> > > > > > > > > > > > > > > > > module.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The implementation has the following
> > > > > aspects:
> > > > > > > > > > > > > > > > > > > > > > 1. The lease tracking happens only
> > > on the
> > > > > > > active
> > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > (raft
> > > > > > > > > > > > > > > > > > > > > > leader)
> > > > > > > > > > > > > > > > > > > > > > 2. Once the lease expires, it needs
> > > to
> > > > > > > propose
> > > > > > > > > and
> > > > > > > > > > > > > commit a
> > > > > > > > > > > > > > > > > > > FenceBroker
> > > > > > > > > > > > > > > > > > > > > > record for that lease.
> > > > > > > > > > > > > > > > > > > > > > 3. In case of active controller
> > > failure,
> > > > > the
> > > > > > > > > lease
> > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > tracked by
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > newly raft leader. The new raft
> > > leader
> > > > > > > starts the
> > > > > > > > > > > lease
> > > > > > > > > > > > > timer
> > > > > > > > > > > > > > > > > again,
> > > > > > > > > > > > > > > > > > > (as
> > > > > > > > > > > > > > > > > > > > > > implemented in onBecomingLeader
> > > method of
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > > > in effect extending the lease by the
> > > time
> > > > > > > spent
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > election
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > whatever time was elapsed on the old
> > > > > leader.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> > > the
> > > > > > > > > controller
> > > > > > > > > > > side
> > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > reset
> > > > > > > > > > > > > > > > > > > > > in the case of controller failover.
> > > The
> > > > > > > > > alternative
> > > > > > > > > > > would
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > track
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > lease as hard state rather than soft
> > > state,
> > > > > > > but I
> > > > > > > > > think
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > > > > > > needed, and would result in more log
> > > > > entries.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > There are working tests for this
> > > > > > > implementation
> > > > > > > > > here.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > > > > and an end to end test here
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/ProducerConsumerKIP500Test.scala
> > > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >>'m not sure what you mean by
> > > > > > > "de-duplication
> > > > > > > > > of the
> > > > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > > > Apologies for using the confusing
> > > term
> > > > > > > > > > > deduplication. I
> > > > > > > > > > > > > meant
> > > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > > id
> > > > > > > > > > > > > > > > > > > > > > conflict.
> > > > > > > > > > > > > > > > > > > > > > As you can see in the prototype
> > > > > > > handleRequest of
> > > > > > > > > > > > > > > KIP631Controller
> > > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > > > the duplicate broker id needs to be
> > > > > detected
> > > > > > > > > before
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > BrokerRecord
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > submitted to the raft module.
> > > > > > > > > > > > > > > > > > > > > > Also as implemented in the
> > > prototype, the
> > > > > > > > > > > > > KIP631Controller is
> > > > > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > > > > > > threaded, handling requests one at a
> > > > > time.
> > > > > > > (an
> > > > > > > > > > > example of
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://martinfowler.com/articles/patterns-of-distributed-systems/singular-update-queue.html
> > > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > > > think
> > > > > > > it
> > > > > > > > > makes
> > > > > > > > > > > > > sense
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> > > becomes
> > > > > > > very
> > > > > > > > > messy.
> > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > > > understand your question about
> > > duplicate
> > > > > > > broker ID
> > > > > > > > > > > > > detection,
> > > > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > > > There's a section in the KIP about
> > > this --
> > > > > is
> > > > > > > > > there a
> > > > > > > > > > > > > detail we
> > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 10:49 AM
> > > Colin
> > > > > > > McCabe <
> > > > > > > > > > > > > > > colin@cmccabe.xyz
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2020, at 19:36,
> > > Unmesh
> > > > > > > Joshi
> > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > There were a few of questions I
> > > had..
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for the response.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1. Were my comments on the broker
> > > > > lease
> > > > > > > > > > > > > implementation
> > > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > > prototype) appropriate and do we
> > > > > need to
> > > > > > > > > change
> > > > > > > > > > > the
> > > > > > > > > > > > > KIP
> > > > > > > > > > > > > > > > > > > > > > > > description accordingly?.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Can you repeat your questions about
> > > > > broker
> > > > > > > > > leases?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2. How will broker epochs be
> > > > > generated?
> > > > > > > I am
> > > > > > > > > > > > > assuming it
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > committed log offset (like zxid?)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > There isn't any need to use a log
> > > > > offset.
> > > > > > > We
> > > > > > > > > can
> > > > > > > > > > > just
> > > > > > > > > > > > > > > look at
> > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > in-memory hash table and see what
> > > the
> > > > > > > latest
> > > > > > > > > > > number is,
> > > > > > > > > > > > > > > and add
> > > > > > > > > > > > > > > > > > > one, to
> > > > > > > > > > > > > > > > > > > > > > > generate a new broker epoch.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3. How will producer registration
> > > > > > > happen? I
> > > > > > > > > am
> > > > > > > > > > > > > assuming
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > should be
> > > > > > > > > > > > > > > > > > > > > > > > similar to broker registration,
> > > with
> > > > > a
> > > > > > > > > similar
> > > > > > > > > > > way to
> > > > > > > > > > > > > > > > > generate
> > > > > > > > > > > > > > > > > > > > > producer
> > > > > > > > > > > > > > > > > > > > > > > id.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > For the EOS stuff, we will need a
> > > few
> > > > > new
> > > > > > > RPCs
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > controller.
> > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > > we should do that in a follow-on
> > > KIP,
> > > > > > > though,
> > > > > > > > > since
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > one is
> > > > > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > > > > > pretty big.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 4. Because we expose Raft log to
> > > all
> > > > > the
> > > > > > > > > > > brokers, any
> > > > > > > > > > > > > > > > > > > de-duplication
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > broker needs to happen before the
> > > > > > > requests
> > > > > > > > > are
> > > > > > > > > > > > > proposed
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > Raft.
> > > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > the controller needs to be single
> > > > > > > threaded,
> > > > > > > > > and
> > > > > > > > > > > > > should do
> > > > > > > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > > > > > against the in-process or pending
> > > > > > > requests
> > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > > > final
> > > > > > > > > > > > > > > > > state. I
> > > > > > > > > > > > > > > > > > > > > read a
> > > > > > > > > > > > > > > > > > > > > > > > mention of this, in the
> > > responses in
> > > > > this
> > > > > > > > > > > > > thread.Will it
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > useful to
> > > > > > > > > > > > > > > > > > > > > > > > mention this in the KIP?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I'm not sure what you mean by
> > > > > > > "de-duplication
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 4:50 AM
> > > Colin
> > > > > > > McCabe
> > > > > > > > > <
> > > > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I'm thinking of calling a vote
> > > on
> > > > > > > KIP-631
> > > > > > > > > on
> > > > > > > > > > > > > Monday.
> > > > > > > > > > > > > > > Let
> > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > know
> > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > > there's any more comments I
> > > should
> > > > > > > address
> > > > > > > > > > > before I
> > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > vote.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020, at 05:39,
> > > > > Unmesh
> > > > > > > > > Joshi
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > >>Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > >>Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks. I will be adding
> > > more to
> > > > > the
> > > > > > > > > > > prototype
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > > > > > and collaborate.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020 at
> > > 12:28 AM
> > > > > > > Colin
> > > > > > > > > > > McCabe <
> > > > > > > > > > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Jose,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > That'a s good point that I
> > > > > hadn't
> > > > > > > > > > > considered.
> > > > > > > > > > > > > It's
> > > > > > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > > > a separate leader change
> > > > > message,
> > > > > > > as
> > > > > > > > > you
> > > > > > > > > > > > > mentioned.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2020, at
> > > 11:56,
> > > > > Jose
> > > > > > > > > Garcia
> > > > > > > > > > > > > Sancio
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Very cool prototype!
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > The KIP proposes a record
> > > > > called
> > > > > > > > > > > IsrChange
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > includes the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > partition, topic, isr,
> > > > > leader and
> > > > > > > > > leader
> > > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > > During
> > > > > > > > > > > > > > > > > > > normal
> > > > > > > > > > > > > > > > > > > > > > > > > > > > operation ISR changes do
> > > not
> > > > > > > result
> > > > > > > > > in
> > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > > > > Similarly,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > leader changes do not
> > > > > necessarily
> > > > > > > > > > > involve ISR
> > > > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > > > implementation that uses
> > > ZK
> > > > > > > modeled
> > > > > > > > > them
> > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. All of this
> > > information is
> > > > > > > stored
> > > > > > > > > in
> > > > > > > > > > > one
> > > > > > > > > > > > > > > znode.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. ZK's optimistic lock
> > > > > requires
> > > > > > > > > that you
> > > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > > the new
> > > > > > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. The change to that
> > > znode
> > > > > was
> > > > > > > being
> > > > > > > > > > > > > performed
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > > > and the leader.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > None of these reasons are
> > > > > true in
> > > > > > > > > > > KIP-500.
> > > > > > > > > > > > > Have
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > considered
> > > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > > > > two different records?
> > > For
> > > > > > > example
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. IsrChange record which
> > > > > > > includes
> > > > > > > > > topic,
> > > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > > isr
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. LeaderChange record
> > > which
> > > > > > > includes
> > > > > > > > > > > topic,
> > > > > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > leader epoch.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > I suspect that making
> > > this
> > > > > change
> > > > > > > > > will
> > > > > > > > > > > also
> > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > message AlterIsrRequest
> > > > > > > introduced in
> > > > > > > > > > > > > KIP-497:
> > > > > > > > > > > > > > > Add
> > > > > > > > > > > > > > > > > > > > > inter-broker
> > > > > > > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > alter ISR.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > > > > > > -Jose
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Tue, Oct 13, 2020, at 18:30, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the reply. A few more comments below.
>
> 80.1 controller.listener.names only defines the name of the listener. The
> actual listener including host/port/security_protocol is typically defined
> in advertised_listners. Does that mean advertised_listners is a required
> config now?
>
Hi Jun,
Thanks for the re-review.
The controller listeners are not advertised to clients. So I think they should go in listeners, rather than advertised.listeners.
I agree that this makes listeners a required configuration. At very least, it is required to have the controller listener in there.
>
> 83.1 broker state machine: It seems that we should transition from FENCED
> => INITIAL since only INITIAL generates new broker epoch?
>
I changed the broker state machine a bit-- take a look. In the new state machine, the FENCED state can re-register.
>
> 83.5. It's true that the controller node doesn't serve metadata requests.
> However, there are admin requests such as topic creation/deletion are sent
> to the controller directly. So, it seems that the client needs to know
> the controller host/port?
>
The client sends admin requests to a random broker, which forwards them to the controller. This is described in KIP-590.
>
> 85. "I was hoping that we could avoid responding to requests when the
> broker was fenced." This issue is that if we don't send a response, the
> client won't know the reason and can't act properly.
>
Sorry, when I said "does not respond" I meant it would return an error.
>
> 88. CurMetadataOffset: I was thinking that we may want to
> use CurMetadataOffset to compute the MetadataLag. Since HWM is exclusive,
> it's more convenient if CurMetadataOffset is also exclusive.
>
OK, I will change it to NextMetadataOffset and make it exclusive.
> 90. It would be useful to add a rejected section on why separate controller
> and broker id is preferred over just broker id. For example, the following
> are some potential reasons. (a) We can guard duplicated brokerID, but it's
> hard to guard against duplicated controllerId. (b) brokerID can be auto
> assigned in the future, but controllerId is hard to be generated
> automatically.
>
OK. I added a rejected alternatives section about sharing IDs between multiple nodes.
Colin
> Thanks,
>
> Jun
>
> On Mon, Oct 12, 2020 at 11:14 AM Colin McCabe <cm...@apache.org> wrote:
>
> > On Tue, Oct 6, 2020, at 16:09, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply. Made another pass of the KIP. A few more comments
> > > below.
> > >
> >
> > Hi Jun,
> >
> > Thanks for the review.
> >
> > > 55. We discussed earlier why the current behavior where we favor the
> > > current broker registration is better. Have you given this more thought?
> > >
> >
> > Yes, I think we should favor the current broker registration, as you
> > suggested earlier.
> >
> > > 80. Config related.
> > > 80.1 Currently, each broker only has the following 3 required configs. It
> > > will be useful to document the required configs post KIP-500 (in both the
> > > dedicated and shared controller mode).
> > > broker.id
> > > log.dirs
> > > zookeeper.connect
> >
> > For the broker, these configs will be required:
> >
> > broker.id
> > log.dirs
> > process.roles
> > controller.listener.names
> > controller.connect
> >
> > For the controller, these configs will be required:
> >
> > controller.id
> > log.dirs
> > process.roles
> > controller.listener.names
> > controller.connect
> >
> > For broker+controller, it will be the union of these two, which
> > essentially means we need both broker.id and controller.id, but all
> > others are the same as standalone.
> >
> > > 80.2 It would be useful to document all deprecated configs post KIP-500.
> > > For example, all zookeeper.* are obviously deprecated. But there could be
> > > others. For example, since we don't plan to support auto broker id
> > > generation, it seems broker.id.generation.enable is deprecated too.
> > > 80.3 Could we make it clear that controller.connect replaces
> > quorum.voters
> > > in KIP-595?
> >
> > OK. I added a comment about this in the table.
> >
> > > 80.4 Could we document that broker.id is now optional?
> >
> > OK. I have added a line for broker.id.
> >
> > > 80.5 The KIP suggests that controller.id is optional on the controller
> > > node. I am concerned that this can cause a bit of confusion in 2 aspects.
> > > First, in the dedicated controller mode, controller.id is not optional
> > > (since broker.id is now optional). Second, in the shared controller
> > mode,
> > > it may not be easy for the user to figure out the default value of
> > > controller.id to set controller.connect properly.
> >
> > I got rid of the default value of controller.id. I agree it was a bit
> > confusing to explain.
> >
> > > 80.6 Regarding the consistency of config names, our metrics already
> > include
> > > controller. So, prefixing all controller related configs with
> > "controller"
> > > may be more consistent. If we choose to do that, could we rename all new
> > > configs here and in KIP-595 consistently?
> > >
> >
> > I added the new names for the KIP-595 configurations to the configuration
> > table. I think prefixing these configurations with "controller.quorum" is
> > the best option since it makes it clear that they are related to the
> > controller quorum.
> >
> > > 81. Metrics
> > > 81.1 kafka.controller:type=KafkaController,name=MetadataSnapshotLag: This
> > > is now redundant since KIP-630 already has
> > > kafka.controller:type=KafkaController,name=SnapshotLag.
> >
> > Thanks for bringing up the metric name issue. I think it would be good to
> > clarify that this is the lag for the metadata snapshot specifically. So I
> > prefer the name MetadataSnapshotLag. I'll add a note to the table.
> >
> > > 81.2 Do we need both kafka.controller:type=KafkaServer,name=MetadataLag
> > and
> > > kafka.controller:type=KafkaController,name=MetadataLag since in the
> > shared
> > > controller mode, the metadata log is shared?
> >
> > Yes, both are needed. This metric is about what metadata logs we have
> > applied, not about the Raft log's high water mark. If the broker or
> > controller takes a while to apply the log messages (for example, because of
> > lock contention or some other issue), they could fall significantly behind
> > the high watermark.
> >
> > >
> > > 82. Metadata records
> > > 82.1 BrokerRecord: It needs to include supported features based on
> > KIP-584.
> >
> > Added.
> >
> > > 82.2 FeatureLevel: The finalized feature in KIP-584 has the following
> > > structure and needs to be reflected here.
> > > {
> > > "version": 0, // int32 -> Represents the version of the schema for the
> > > data stored in the ZK node
> > > "status": 1, // int32 -> Represents the status of the node
> > > "features": {
> > > "group_coordinator": { // string -> name of the feature
> > > "min_version_level": 0, // int16 -> Represents the
> > cluster-wide
> > > finalized minimum version level (>=1) of this feature
> > > "max_version_level": 3 // int16 -> Represents the
> > cluster-wide
> > > finalized maximum version level (>=1 and >= min_version_level) of this
> > > feature
> > > },
> > > "consumer_offsets_topic_schema": {
> > > "min_version_level": 0,
> > > "max_version_level": 4
> > > }
> > > }
> >
> > Fair point. I will add both a minimum and maximum finalized feature level.
> >
> > >
> > > 83. RPC
> > > 83.1 BrokerHeartbeatRequest: Could you document when the controller
> > assigns
> > > a new broker epoch?
> >
> > The controller assigns a broker epoch if the heartbeat request was sent
> > without an epoch (that is, it is in the INITIAL state), and the controller
> > wants to add the new broker to the cluster. I added a note to the state
> > machine.
> >
> > > 83.2 BrokerHeartbeatResponse: Do we need a new error code to indicate a
> > > broker is fenced?
> >
> > I don't think we need a new error code, since targetState will just
> > indicate that the broker should be fenced.
> >
> > > 83.3 What error/state indicates an unsuccessful controlled shutdown in
> > > BrokerHeartbeatResponse? Currently, the controlled shutdown responses
> > also
> > > include a list of remaining partitions. That's mostly for debugging
> > > purposes. Should we include it in BrokerHeartbeatResponse too?
> >
> > Once the controller wants the broker to shut down, it will set targetState
> > to SHUTDOWN.
> >
> > We don't need to send a list of remaining partitions, since the broker is
> > following the metadata log, and will get all its instructions from there.
> >
> > > 83.4 Could we document when Listeners, Features and Rack are expected to
> > be
> > > set in BrokerHeartbeatRequest?
> >
> > I added a line specifying that: The listeners, features, and rack fields
> > only need to be set when the broker is in INITIAL state, and is requesting
> > a new broker epoch.
> >
> > > 83.5 Currently, the metadata response only returns a list of brokers.
> > > Should we have a separate field for the controller info or change the
> > field
> > > name to sth like nodes? Also, is the rack field relevant to the
> > controller?
> >
> > As per KIP-590, we don't want the clients to communicate directly with the
> > controller. So we weren't planning on including the controller's
> > information in the MetadataResponse.
> >
> > We don't currently have rack-awareness for controllers, but I suppose we
> > could in the future. I think we can leave that for later.
> >
> > >
> > > 84. There are a few references to __kafka_metadata. However, KIP-595
> > > uses __cluster_metadata.
> >
> > We agreed earlier to decide on the name of the metadata topic in this KIP,
> > and to treat the KIP-595 name as provisional. Looking at it again, I think
> > we should go with @metadata, in order to avoid colliding with topic names
> > that already exist.
> >
> > >
> > > 85. "When a broker is fenced, it cannot process any client requests. "
> > > Should we send a new error code in the response to indicate that the
> > broker
> > > is fenced?
> >
> > I was hoping that we could avoid responding to requests when the broker
> > was fenced. I suppose a new error code could work too, but then we also
> > need a compatibility story for older clients.
> >
> > >
> > > 86. "The controller can generate a new broker epoch by using the latest
> > log
> > > offset." Perhaps it's more accurate to say the new broker epoch is the
> > > offset for the corresponding BrokerRecord in the metadata log.
> >
> > The idea here is slightly different. We want to use the current latest
> > offset (prior to creating the BrokerRecord) as the new epoch. We can't use
> > the offset of the BrokerRecord itself since we won't know what that is
> > until we give the BrokerRecord to the Raft layer.
> >
> > >
> > > 87. meta.properties:
> > > 87.1 Could you document the format change to this file? Also, should we
> > > bump up the version of the file?
> >
> > Hmm. The file is just a list of key=value pairs, one per line. So I'm
> > not sure there is any format change per se, at least technically.
> >
> > Maybe we should bump up the version anyway, in order to prevent older
> > pre-KIP-500 software from launching if the new stuff is present?
> >
> > > 87.2 How do we know a node is configured in KIP-500 mode?
> > >
> >
> > A cluster configured in KIP-500 mode will contain the following line in
> > the meta.properties file in each directory: kip.500.mode=enabled
> >
> > > 88. CurMetadataOffset: Should we use nextMetadataOffset since it's more
> > > consistent with fetchOffset and HWM?
> >
> > I'm not sure. Drawing an analogy with HWM might be confusing, since
> > CurMetadataOffset is not HWM. It's totally possible for the broker to have
> > fetched up to offset 1000 but only applied up to offset 500.
> >
> > best,
> > Colin
> >
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Oct 5, 2020 at 9:03 AM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Mon, Sep 28, 2020, at 11:41, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the reply. A few more comments below.
> > > > >
> > > > > 62.
> > > > > 62.1 controller.listener.names: So, is this used for the controller
> > or
> > > > the
> > > > > broker trying to connect to the controller?
> > > > >
> > > >
> > > > Hi Jun,
> > > >
> > > > It's used by both. The broker tries to connect to controllers by using
> > > > the first listener in this list. The controller uses the list to
> > determine
> > > > which listeners it should bind to.
> > > >
> > > > >
> > > > > 62.2 If we want to take the approach to share the configs that are
> > common
> > > > > between the broker and the controller, should we share the id too?
> > > > >
> > > >
> > > > On nodes where we are running both the controller and broker roles, we
> > > > need two IDs: one for the controller, and one for the broker. So we
> > can't
> > > > share the same configuration.
> > > >
> > > > >
> > > > > 62.3 We added some configs in KIP-595 prefixed with "quorum" and we
> > plan
> > > > to
> > > > > add some controller specific configs prefixed with "controller".
> > KIP-630
> > > > > plans to add some other controller specific configs with no prefix.
> > > > Should
> > > > > we standardize all controller specific configs with the same prefix?
> > > > >
> > > >
> > > > That's a good question. As Jose said elsewhere in the thread, let's
> > use
> > > > "metadata" as the prefix.
> > > >
> > > > >
> > > > > 70. Could you explain the impact of process.roles a bit more? For
> > > > example,
> > > > > if process.roles=controller, does the node still maintain metadata
> > cache
> > > > as
> > > > > described in KIP-630? Do we still return the host/port for those
> > nodes in
> > > > > the metadata response?
> > > > >
> > > >
> > > > No, the node does not need any broker data structures if it is not a
> > > > broker. Controllers also don't handle MetadataRequests. Clients will
> > > > still go to the brokers for those.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Sep 28, 2020 at 9:24 AM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > On Fri, Sep 25, 2020, at 17:35, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the reply.
> > > > > > >
> > > > > > > 62. Thinking about this more, I am wondering what's our overall
> > > > strategy
> > > > > > > for configs shared between the broker and the controller. For
> > > > example,
> > > > > > both
> > > > > > > the broker and the controller have to define listeners. So both
> > need
> > > > > > > configs like listeners/advertised.listeners. Both the new
> > controller
> > > > and
> > > > > > > the broker replicate data. So both need to define some
> > replication
> > > > > > related
> > > > > > > configurations (replica.fetch.min.bytes,
> > replica.fetch.wait.max.ms,
> > > > > > etc).
> > > > > > > Should we let the new controller share those configs with
> > brokers or
> > > > > > should
> > > > > > > we define new configs just for the controller? It seems that we
> > want
> > > > to
> > > > > > > apply the same strategy consistently for all shared configs?
> > > > > > >
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > That's a good question. I think that we should share as many
> > > > > > configurations as possible. There will be a few cases where this
> > isn't
> > > > > > possible, and we need to create a new configuration key that is
> > > > > > controller-only, but I think that will be relatively rare.
> > > > > >
> > > > > > >
> > > > > > > 63. If a node has process.roles set to controller, does one still
> > > > need to
> > > > > > > set broker.id on this node?
> > > > > > >
> > > > > >
> > > > > > No, broker.id does not need to be set in that case.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Sep 25, 2020 at 2:17 PM Colin McCabe <cmccabe@apache.org
> > >
> > > > wrote:
> > > > > > >
> > > > > > > > On Fri, Sep 25, 2020, at 10:17, Jun Rao wrote:
> > > > > > > > > Hi, Colin,
> > > > > > > > >
> > > > > > > > > Thanks for the reply.
> > > > > > > > >
> > > > > > > > > 60. Yes, I think you are right. We probably need the
> > controller
> > > > id
> > > > > > when a
> > > > > > > > > broker starts up. A broker only stores the Raft leader id in
> > the
> > > > > > metadata
> > > > > > > > > file. To do the initial fetch to the Raft leader, it needs to
> > > > know
> > > > > > the
> > > > > > > > > host/port associated with the leader id.
> > > > > > > > >
> > > > > > > > > 62. It seems there are 2 parts to this : (1) which listener a
> > > > client
> > > > > > > > should
> > > > > > > > > use to initiate a connection to the controller and (2) which
> > > > listener
> > > > > > > > > should a controller use to accept client requests. For (1),
> > at
> > > > any
> > > > > > point
> > > > > > > > of
> > > > > > > > > time, a client only needs to use one listener. I think
> > > > > > > > > controller.listener.name is meant for the client.
> > > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > controller.listener.names is also used by the controllers. In
> > the
> > > > case
> > > > > > > > where we have a broker and a controller in the same JVM, we
> > have a
> > > > > > single
> > > > > > > > config file. Then we need to know which listeners belong to
> > the
> > > > > > controller
> > > > > > > > and which belong to the broker component. That's why it's a
> > list.
> > > > > > > >
> > > > > > > > > So, a single value seems
> > > > > > > > > to make more sense. Currently, we don't have a configuration
> > for
> > > > > > (2). We
> > > > > > > > > could add a new one for that and support a list. I am
> > wondering
> > > > how
> > > > > > > > useful
> > > > > > > > > it will be. One example that I can think of is that we can
> > reject
> > > > > > > > > non-controller related requests if accepted on
> > controller-only
> > > > > > listeners.
> > > > > > > > > However, we already support separate authentication for the
> > > > > > controller
> > > > > > > > > listener. So, not sure how useful it is.
> > > > > > > >
> > > > > > > > The controller always has a separate listener and does not
> > share
> > > > > > listeners
> > > > > > > > with the broker. The main reason for this is to avoid giving
> > > > clients
> > > > > > the
> > > > > > > > ability to launch a denial-of-service attack on the controller
> > by
> > > > > > flooding
> > > > > > > > a broker port. A lot of times, these attacks are made
> > > > unintentionally
> > > > > > by
> > > > > > > > poorly coded or configured clients. Additionally, broker
> > ports can
> > > > > > also be
> > > > > > > > very busy with large fetch requests, and so on. It's just a
> > bad
> > > > > > > > configuration in general to try to overload the same port for
> > the
> > > > > > > > controller and broker, and we don't want to allow it. It
> > would be
> > > > a
> > > > > > > > regression to go from the current system where control
> > requests are
> > > > > > safely
> > > > > > > > isolated on a separate port, to one where they're not. It also
> > > > makes
> > > > > > the
> > > > > > > > code and configuration a lot messier.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 63. (a) I think most users won't know controller.id
> > defaults to
> > > > > > > > broker.id +
> > > > > > > > > 3000. So, it can be confusing for them to set up
> > > > controller.connect.
> > > > > > If
> > > > > > > > > this is truly needed, it seems that it's less confusing to
> > make
> > > > > > > > > controller.id required.
> > > > > > > > > (b) I am still trying to understand if we truly need to
> > expose a
> > > > > > > > > controller.id. What if we only expose broker.id and let
> > > > > > > > controller.connect
> > > > > > > > > just use broker.id? What will be missing?
> > > > > > > >
> > > > > > > > The controller has a separate ID from the broker, so knowing
> > > > broker.id
> > > > > > is
> > > > > > > > not helpful here.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Thu, Sep 24, 2020 at 10:55 PM Colin McCabe <
> > > > cmccabe@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Thu, Sep 24, 2020, at 16:24, Jun Rao wrote:
> > > > > > > > > > > Hi, Colin,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the reply and the updated KIP. A few more
> > comments
> > > > > > below.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi Jun,
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 53. It seems that you already incorporated the changes in
> > > > > > KIP-516.
> > > > > > > > With
> > > > > > > > > > > topic ids, we don't need to wait for the topic's data to
> > be
> > > > > > deleted
> > > > > > > > > > before
> > > > > > > > > > > removing the topic metadata. If the topic is recreated,
> > we
> > > > can
> > > > > > still
> > > > > > > > > > delete
> > > > > > > > > > > the data properly based on the topic id. So, it seems
> > that
> > > > we can
> > > > > > > > remove
> > > > > > > > > > > TopicRecord.Deleting.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks for the reply. What I was thinking of doing here
> > was
> > > > using
> > > > > > > > topic
> > > > > > > > > > IDs internally, but still using names externally. So the
> > topic
> > > > > > UUIDs
> > > > > > > > are
> > > > > > > > > > only for the purpose of associating topics with partitions
> > --
> > > > from
> > > > > > the
> > > > > > > > > > user's point of view, topics are still identified by names.
> > > > > > > > > >
> > > > > > > > > > You're right that KIP-516 will simplify things, but I'm not
> > > > sure
> > > > > > when
> > > > > > > > that
> > > > > > > > > > will land, so I wanted to avoid blocking the initial
> > > > > > implementation of
> > > > > > > > this
> > > > > > > > > > KIP on it.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 55. It seems to me that the current behavior where we
> > favor
> > > > the
> > > > > > > > current
> > > > > > > > > > > broker registration is better. This is because
> > uncontrolled
> > > > > > broker
> > > > > > > > > > shutdown
> > > > > > > > > > > is rare and its impact is less severe since one just
> > needs to
> > > > > > wait
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > session timeout before restarting the broker. If a
> > > > mis-configured
> > > > > > > > broker
> > > > > > > > > > > replaces an existing broker, the consequence is more
> > severe
> > > > > > since it
> > > > > > > > can
> > > > > > > > > > > cause the leader to be unavailable, a replica to be out
> > of
> > > > ISR,
> > > > > > and
> > > > > > > > add
> > > > > > > > > > > more load on the leaders etc.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hmm, that's a good point. Let me check this a bit more
> > before
> > > > I
> > > > > > change
> > > > > > > > > > it, though.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 60. controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com
> > :
> > > > Do we
> > > > > > > > need to
> > > > > > > > > > > include the controller id before @? It seems that the
> > > > host/port
> > > > > > is
> > > > > > > > enough
> > > > > > > > > > > for establishing the connection. It would also be useful
> > to
> > > > make
> > > > > > it
> > > > > > > > clear
> > > > > > > > > > > that controller.connect replaces quorum.voters in
> > KIP-595.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I discussed this with Jason earlier, and he felt that the
> > > > > > controller
> > > > > > > > IDs
> > > > > > > > > > were needed in this configuration key. It is certainly
> > needed
> > > > when
> > > > > > > > > > configuring the controllers themselves, since they need to
> > know
> > > > > > each
> > > > > > > > > > others' IDs.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 61. I am not sure that we need both controller.listeners
> > and
> > > > > > > > > > > controller.connect.security.protocol since the former
> > > > implies the
> > > > > > > > > > security
> > > > > > > > > > > protocol. The reason that we have both
> > > > > > inter.broker.listener.name
> > > > > > > > and
> > > > > > > > > > > inter.broker.security.protocol is because we had the
> > latter
> > > > > > first and
> > > > > > > > > > later
> > > > > > > > > > > realized that the former is more general.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > That's a good point. I've removed this from the KIP.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 62. I am still not sure that you need
> > controller.listeners
> > > > to be
> > > > > > a
> > > > > > > > list.
> > > > > > > > > > > All listeners are already defined in listeners. To
> > migrate
> > > > from
> > > > > > > > plaintext
> > > > > > > > > > > to SSL, one can configure listeners to have both
> > plaintext
> > > > and
> > > > > > SSL.
> > > > > > > > After
> > > > > > > > > > > that, one can just change controller.listeners from
> > > > plaintext to
> > > > > > SSL.
> > > > > > > > > > This
> > > > > > > > > > > is similar to how to change the listener for inter broker
> > > > > > > > connections.
> > > > > > > > > > > Also, controller.listener.name may be a more accurate
> > name?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The issue that I see here is that if you are running with
> > the
> > > > > > > > controller
> > > > > > > > > > and broker in the same JVM, if you define a few listeners
> > in
> > > > > > > > "listeners"
> > > > > > > > > > they will get used as regular broker listeners, unless you
> > put
> > > > > > them in
> > > > > > > > > > controller.listeners. Therefore, controller.listeners
> > needs
> > > > to be
> > > > > > a
> > > > > > > > list.
> > > > > > > > > >
> > > > > > > > > > controller.listener.names does sound like a better name,
> > > > though...
> > > > > > I've
> > > > > > > > > > updated it to that.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 63. Regarding controller.id, I am trying to understand
> > > > whether
> > > > > > it's
> > > > > > > > a
> > > > > > > > > > > required configuration or an optional one. From the KIP,
> > it
> > > > > > sounds
> > > > > > > > like
> > > > > > > > > > > controller.id is optional. Then, if this is unset, it's
> > not
> > > > > > clear
> > > > > > > > how
> > > > > > > > > > the
> > > > > > > > > > > user will obtain the controller id for setting
> > > > > > controller.connect.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > If you specify broker.id but not controller.id, then
> > > > controller.id
> > > > > > > > will
> > > > > > > > > > just be set to broker.id + 3000. This was intended to
> > make
> > > > some
> > > > > > > > > > configurations a little bit more concise to write. You
> > still
> > > > do
> > > > > > have
> > > > > > > > to
> > > > > > > > > > know the controller IDs when configuring the brokers,
> > though.
> > > > If
> > > > > > this
> > > > > > > > > > seems confusing then I can just make it mandatory.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 64. KIP-516 adds a flag in LeaderAndIsrRequest to
> > indicate
> > > > > > whether it
> > > > > > > > > > > includes all partitions or not. This will be used to
> > clean up
> > > > > > strayed
> > > > > > > > > > logs.
> > > > > > > > > > > I was thinking how we will do the same thing once the
> > > > controller
> > > > > > > > moves to
> > > > > > > > > > > Raft based. One way to do that is on broker startup, it
> > gets
> > > > the
> > > > > > HWM
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > metadata log from the Raft leader and waits until its
> > > > metadata
> > > > > > > > catches up
> > > > > > > > > > > to HWM. At that point, any local log not seen in the
> > metadata
> > > > > > can be
> > > > > > > > > > > removed. Since the Fetch response returns the HWM, there
> > > > seems
> > > > > > to be
> > > > > > > > > > enough
> > > > > > > > > > > APIs to achieve this.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > That's a very good point. I added a note about this under
> > > > Broker
> > > > > > > > Startup.
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 24, 2020 at 1:07 PM Colin McCabe <
> > > > colin@cmccabe.xyz>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > On Mon, Sep 21, 2020, at 18:13, Jun Rao wrote:
> > > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Sorry for the late reply. A few more comments below.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Jun,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for taking another look.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 50. Configurations
> > > > > > > > > > > > > 50.1 controller.listeners: It seems that a controller
> > > > just
> > > > > > needs
> > > > > > > > one
> > > > > > > > > > > > > listener. Why do we need to have a list here? Also,
> > > > could you
> > > > > > > > > > provide an
> > > > > > > > > > > > > example of how this is set and what's its
> > relationship
> > > > with
> > > > > > > > existing
> > > > > > > > > > > > > configs such as "security.inter.broker.protocol" and
> > "
> > > > > > > > > > > > > inter.broker.listener.name"?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I agree that most administrators will want to run with
> > > > only one
> > > > > > > > > > controller
> > > > > > > > > > > > listener. However, just as with brokers, it is nice to
> > > > have
> > > > > > the
> > > > > > > > > > option to
> > > > > > > > > > > > expose multiple ports.
> > > > > > > > > > > >
> > > > > > > > > > > > One reason why you might want multiple ports is if you
> > were
> > > > > > doing a
> > > > > > > > > > > > migration from plaintext to SSL. You could add new SSL
> > > > > > listeners
> > > > > > > > but
> > > > > > > > > > > > continue to expose the PLAINTEXT listeners. Then you
> > could
> > > > > > add the
> > > > > > > > > > new SSL
> > > > > > > > > > > > controller config to each broker and roll each broker.
> > > > Then
> > > > > > you
> > > > > > > > could
> > > > > > > > > > > > migrate from PLAINTEXT to SSL with no downtime.
> > > > > > > > > > > >
> > > > > > > > > > > > Here's an example configuration for the controller:
> > > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > 1@controller1.example.com:9093,
> > 2@controller2.example.com
> > > > > > > > > > > > controller.listeners=CONTROLLER
> > > > > > > > > > > > listeners=CONTROLLER://controller0.example.com:9093
> > > > > > > > > > > > listener.security.protocol.map=CONTROLLER:SSL
> > > > > > > > > > > >
> > > > > > > > > > > > Here's an example configuration for the broker:
> > > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > > 1@controller1.example.com:9093,
> > 2@controller2.example.com
> > > > > > > > > > > > controller.connect.security.protocol=SSL
> > > > > > > > > > > >
> > > > > > > > > > > > security.inter.broker.protocol or
> > > > inter.broker.listener.name
> > > > > > do
> > > > > > > > not
> > > > > > > > > > > > affect how the broker communicates with the controller.
> > > > Those
> > > > > > > > > > > > configurations relate to how brokers communicate with
> > each
> > > > > > other,
> > > > > > > > but
> > > > > > > > > > the
> > > > > > > > > > > > controller is not a broker.
> > > > > > > > > > > >
> > > > > > > > > > > > (Note that I just added
> > > > controller.connect.security.protocol
> > > > > > to the
> > > > > > > > > > KIP --
> > > > > > > > > > > > I had forgotten to put it in earlier)
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 50.2 registration.heartbeat.interval.ms and
> > > > > > > > > > > > registration.lease.timeout.ms.
> > > > > > > > > > > > > Should we match their default value with the
> > > > corresponding
> > > > > > > > default
> > > > > > > > > > for
> > > > > > > > > > > > ZK?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Fair enough. I'll set them to the values of the
> > > > > > > > > > zookeeper.sync.time.ms
> > > > > > > > > > > > and zookeeper.connection.timeout.ms configurations.
> > I do
> > > > > > think we
> > > > > > > > > > should
> > > > > > > > > > > > experiment later on to see what works well here, but
> > the ZK
> > > > > > values
> > > > > > > > are
> > > > > > > > > > at
> > > > > > > > > > > > least a starting point.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 50.3 controller.connect: Could you provide an
> > example? I
> > > > am
> > > > > > > > > > wondering how
> > > > > > > > > > > > > it differs from quorum.voters=1@kafka-1:9092,
> > 2@kafka-2
> > > > > > :9092,
> > > > > > > > > > 3@kafka-3
> > > > > > > > > > > > > :9092.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > controller.connect is intended to be the new name for
> > > > > > > > quorum.voters.
> > > > > > > > > > > > During the vote for KIP-595, we sort of agreed to
> > defer the
> > > > > > > > discussion
> > > > > > > > > > > > about what this configuration should be called. I
> > proposed
> > > > > > this
> > > > > > > > new
> > > > > > > > > > name
> > > > > > > > > > > > because it makes it clear what the configuration is for
> > > > (how to
> > > > > > > > > > connect to
> > > > > > > > > > > > the controllers).
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 50.4 controller.id: I am still not sure how this is
> > > > being
> > > > > > used.
> > > > > > > > > > Could
> > > > > > > > > > > > you
> > > > > > > > > > > > > explain this in more detail?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > The controller ID needs to be set on each controller.
> > It
> > > > also
> > > > > > > > appears
> > > > > > > > > > in
> > > > > > > > > > > > controller.connect as the thing before the "at" sign.
> > Its
> > > > > > > > function is
> > > > > > > > > > > > pretty similar to broker.id.
> > > > > > > > > > > >
> > > > > > > > > > > > Broker IDs and controller IDs exist in the same ID
> > space,
> > > > so
> > > > > > you
> > > > > > > > can't
> > > > > > > > > > > > have a broker and a controller that use the same ID.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 51. BrokerHeartbeat: It seems a bit wasteful to
> > include
> > > > > > > > Listeners in
> > > > > > > > > > > > every
> > > > > > > > > > > > > heartbeat request since it typically doesn't change.
> > > > Could we
> > > > > > > > make
> > > > > > > > > > that
> > > > > > > > > > > > an
> > > > > > > > > > > > > optional field?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Ok.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 52. KIP-584 adds a new ZK node /features. Should we
> > add a
> > > > > > > > > > corresponding
> > > > > > > > > > > > > metadata record?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Good point. I added FeatureLevelRecord.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 53. TopicRecord and DeleteTopic: Both DeleteTopic and
> > > > > > > > > > > > TopicRecord.Deleting
> > > > > > > > > > > > > indicate topic deletion. Could we outline the flow
> > when
> > > > each
> > > > > > > > will be
> > > > > > > > > > set?
> > > > > > > > > > > > > In particular, which one indicates the intention to
> > > > delete
> > > > > > and
> > > > > > > > which
> > > > > > > > > > one
> > > > > > > > > > > > > indicates the completion of the deletion.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > TopicRecord.Deleting is set when we intend to delete
> > the
> > > > topic.
> > > > > > > > > > > >
> > > > > > > > > > > > DeleteTopic removes the topic completely. I will
> > rename
> > > > > > > > DeleteTopic to
> > > > > > > > > > > > RemoveTopic to make this clearer.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 54. "The controller can generate a new broker epoch
> > by
> > > > using
> > > > > > the
> > > > > > > > > > latest
> > > > > > > > > > > > log
> > > > > > > > > > > > > offset." Which offset is that? Is it the offset of
> > the
> > > > > > metadata
> > > > > > > > > > topic for
> > > > > > > > > > > > > the corresponding BrokerRecord? Is it guaranteed to
> > be
> > > > > > different
> > > > > > > > on
> > > > > > > > > > each
> > > > > > > > > > > > > broker restart?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Yes.... a new broker epoch implies that there has been
> > a
> > > > new
> > > > > > record
> > > > > > > > > > > > created in the metadata log. Therefore the last
> > committed
> > > > > > offset
> > > > > > > > must
> > > > > > > > > > be
> > > > > > > > > > > > different.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 55. "Thereafter, it may lose subsequent conflicts if
> > its
> > > > > > broker
> > > > > > > > > > epoch is
> > > > > > > > > > > > > stale. (See KIP-380 for some background on broker
> > > > epoch.)
> > > > > > The
> > > > > > > > > > reason
> > > > > > > > > > > > for
> > > > > > > > > > > > > favoring new processes is to accommodate the common
> > case
> > > > > > where a
> > > > > > > > > > process
> > > > > > > > > > > > is
> > > > > > > > > > > > > killed with kill -9 and then restarted. " Are you
> > saying
> > > > > > that if
> > > > > > > > > > there is
> > > > > > > > > > > > > an active broker registered in the controller, a new
> > > > broker
> > > > > > > > heartbeat
> > > > > > > > > > > > > request with the INITIAL state will fence the current
> > > > broker
> > > > > > > > > > session? Not
> > > > > > > > > > > > > sure about this. For example, this will allow a
> > broker
> > > > with
> > > > > > > > > > incorrectly
> > > > > > > > > > > > > configured broker id to kill an existing valid
> > broker.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, a new broker with an incorrectly configured
> > broker id
> > > > > > could
> > > > > > > > fence
> > > > > > > > > > an
> > > > > > > > > > > > existing valid broker.
> > > > > > > > > > > >
> > > > > > > > > > > > The goal of the conflict resolution process described
> > here
> > > > is
> > > > > > to
> > > > > > > > avoid
> > > > > > > > > > > > having two brokers go back and forth in claiming the
> > > > broker id.
> > > > > > > > > > Choosing
> > > > > > > > > > > > the newest was just an easy way to do that.
> > > > > > > > > > > >
> > > > > > > > > > > > Choosing the oldest is another possibility, and more
> > inline
> > > > > > with
> > > > > > > > what
> > > > > > > > > > > > ZooKeeper does today. The problem with that is that it
> > > > would
> > > > > > not
> > > > > > > > be
> > > > > > > > > > > > possible to re-create the broker if the old instance
> > died
> > > > > > suddenly,
> > > > > > > > > > except
> > > > > > > > > > > > by waiting for the full lease timeout.
> > > > > > > > > > > >
> > > > > > > > > > > > It might be possible to do a little better here by
> > > > breaking the
> > > > > > > > lease
> > > > > > > > > > if
> > > > > > > > > > > > the TCP connection from the broker drops. But that
> > > > requires
> > > > > > > > crossing
> > > > > > > > > > some
> > > > > > > > > > > > layers of abstraction in the Kafka networking stack.
> > > > There's
> > > > > > also
> > > > > > > > the
> > > > > > > > > > > > possibility of false negatives or positives. For
> > example,
> > > > TCP
> > > > > > > > > > connections
> > > > > > > > > > > > sometimes just drop even if the node is still there.
> > Or
> > > > > > sometimes
> > > > > > > > the
> > > > > > > > > > > > networking stack keeps a TCP connection alive for a
> > while
> > > > when
> > > > > > the
> > > > > > > > > > other
> > > > > > > > > > > > end is gone.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 56. kafka-storage.sh:
> > > > > > > > > > > > > 56.1 In the info mode, what other information does it
> > > > show in
> > > > > > > > > > addition to
> > > > > > > > > > > > > "kip.500.mode=enabled"?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I think it should show a list of all the storage
> > > > directories,
> > > > > > and
> > > > > > > > > > whether
> > > > > > > > > > > > each one is formatted. In addition, it should show
> > whether
> > > > > > > > > > kip.500.mode is
> > > > > > > > > > > > enabled, and what the cluster id is.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 56.2 Should the format mode take the config file as
> > the
> > > > > > input too
> > > > > > > > > > like
> > > > > > > > > > > > the
> > > > > > > > > > > > > info mode?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, that's a good idea.
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jun
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 17, 2020 at 4:12 PM Colin McCabe <
> > > > > > cmccabe@apache.org
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > That's a fair point. I have moved the lease
> > duration
> > > > to
> > > > > > the
> > > > > > > > broker
> > > > > > > > > > > > > > heartbeat response. That way lease durations can
> > be
> > > > > > changed
> > > > > > > > just
> > > > > > > > > > be
> > > > > > > > > > > > > > reconfiguring the controllers.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, Sep 16, 2020, at 07:40, Unmesh Joshi wrote:
> > > > > > > > > > > > > > > Thanks Colin, the changes look good to me. One
> > small
> > > > > > thing.
> > > > > > > > > > > > > > > registration.lease.timeout.ms is the
> > configuration
> > > > on
> > > > > > the
> > > > > > > > > > controller
> > > > > > > > > > > > > > side.
> > > > > > > > > > > > > > > It will be good to comment on how brokers know
> > about
> > > > it,
> > > > > > to
> > > > > > > > be
> > > > > > > > > > able
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > send LeaseDurationMs
> > > > > > > > > > > > > > > in the heartbeat request,
> > > > > > > > > > > > > > > or else it can be added in the heartbeat
> > response for
> > > > > > > > brokers to
> > > > > > > > > > know
> > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Sep 11, 2020 at 10:32 PM Colin McCabe <
> > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think you're right that we should use a
> > duration
> > > > here
> > > > > > > > rather
> > > > > > > > > > > > than a
> > > > > > > > > > > > > > > > time. As you said, the clock on the controller
> > > > will
> > > > > > > > probably
> > > > > > > > > > not
> > > > > > > > > > > > > > match the
> > > > > > > > > > > > > > > > one on the broker. I have updated the KIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It's important to keep in mind that
> > messages
> > > > may be
> > > > > > > > > > delayed in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > > happens, we
> > > > > > > > > > will
> > > > > > > > > > > > use
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > time specified in the request to determine
> > if
> > > > the
> > > > > > > > request
> > > > > > > > > > is
> > > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > > I am assuming that there will be a single TCP
> > > > > > connection
> > > > > > > > > > > > maintained
> > > > > > > > > > > > > > > > between
> > > > > > > > > > > > > > > > > broker and active controller. So, there
> > won't be
> > > > any
> > > > > > out
> > > > > > > > of
> > > > > > > > > > order
> > > > > > > > > > > > > > > > requests?
> > > > > > > > > > > > > > > > > There will be a scenario of broker GC pause,
> > > > which
> > > > > > might
> > > > > > > > > > cause
> > > > > > > > > > > > > > connection
> > > > > > > > > > > > > > > > > timeout and broker might need to reestablish
> > the
> > > > > > > > connection.
> > > > > > > > > > If
> > > > > > > > > > > > the
> > > > > > > > > > > > > > pause
> > > > > > > > > > > > > > > > > is too long, lease will expire and the
> > heartbeat
> > > > sent
> > > > > > > > after
> > > > > > > > > > the
> > > > > > > > > > > > pause
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > be treated as a new registration (similar to
> > > > restart
> > > > > > > > case),
> > > > > > > > > > and
> > > > > > > > > > > > a new
> > > > > > > > > > > > > > > > epoch
> > > > > > > > > > > > > > > > > number will be assigned to the broker.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I agree with the end of this paragraph, but not
> > > > with
> > > > > > the
> > > > > > > > start
> > > > > > > > > > :)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There can be out-of-order requests, since the
> > > > broker
> > > > > > will
> > > > > > > > > > simply
> > > > > > > > > > > > use a
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > TCP connection if the old one has problems.
> > This
> > > > can
> > > > > > > > happen
> > > > > > > > > > for a
> > > > > > > > > > > > > > variety
> > > > > > > > > > > > > > > > of reasons. I don't think GC pauses are the
> > most
> > > > > > common
> > > > > > > > > > reason for
> > > > > > > > > > > > > > this to
> > > > > > > > > > > > > > > > happen. It's more common to see issues issues
> > in
> > > > the
> > > > > > > > network
> > > > > > > > > > > > itself
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > result connections getting dropped from time to
> > > > time.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > So we have to assume that messages may arrive
> > out
> > > > of
> > > > > > > > order, and
> > > > > > > > > > > > > > possibly
> > > > > > > > > > > > > > > > be delayed. I added a note that heartbeat
> > requests
> > > > > > should
> > > > > > > > be
> > > > > > > > > > > > ignored
> > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > the metadata log offset they contain is smaller
> > > > than a
> > > > > > > > previous
> > > > > > > > > > > > > > heartbeat.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > When the active controller fails, the new
> > active
> > > > > > > > controller
> > > > > > > > > > > > needs to
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > sure that it considers all the known brokers
> > as
> > > > alive
> > > > > > > > till
> > > > > > > > > > the
> > > > > > > > > > > > lease
> > > > > > > > > > > > > > > > > expiration interval. Because
> > > > > > > > registration.lease.timeout.ms,
> > > > > > > > > > is
> > > > > > > > > > > > > > > > configured
> > > > > > > > > > > > > > > > > on the controller, the new active controller
> > will
> > > > > > extend
> > > > > > > > all
> > > > > > > > > > the
> > > > > > > > > > > > > > leases
> > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > registration.lease.timeout.ms. I see that it
> > > > won't
> > > > > > need
> > > > > > > > last
> > > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Agreed.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Sat, Sep 5, 2020 at 1:28 AM Colin McCabe <
> > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Colin wrote:
> > > > > > > > > > > > > > > > > > > > The reason for including
> > LeaseStartTimeMs
> > > > in
> > > > > > the
> > > > > > > > > > request
> > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > > > that the time required to communicate
> > with
> > > > the
> > > > > > > > > > controller
> > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > included in
> > > > > > > > > > > > > > > > > > > > the lease time. Since requests can
> > > > > > potentially be
> > > > > > > > > > delayed
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > > > for a long time, this is important.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020, at 05:58, Unmesh
> > Joshi
> > > > wrote:
> > > > > > > > > > > > > > > > > > > The network time will be added anyway,
> > > > because
> > > > > > the
> > > > > > > > lease
> > > > > > > > > > > > timer
> > > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > > active controller will start only after
> > the
> > > > > > heartbeat
> > > > > > > > > > request
> > > > > > > > > > > > > > > > reaches the
> > > > > > > > > > > > > > > > > > > server.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > If the start time is not specified in the
> > > > request,
> > > > > > > > then the
> > > > > > > > > > > > network
> > > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > > is excluded from the heartbeat time.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Here's an example:
> > > > > > > > > > > > > > > > > > Let's say broker A sends a heartbeat at
> > time
> > > > 100,
> > > > > > and
> > > > > > > > it
> > > > > > > > > > > > arrives
> > > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > controller at time 200, and the lease
> > duration
> > > > is
> > > > > > 1000.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The controller looks at the start time in
> > the
> > > > > > request,
> > > > > > > > > > which is
> > > > > > > > > > > > > > 100,
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > adds 1000 to it, getting 1100. On the
> > other
> > > > hand,
> > > > > > if
> > > > > > > > start
> > > > > > > > > > > > time
> > > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > > specified in the request, then the
> > expiration
> > > > will
> > > > > > be
> > > > > > > > at
> > > > > > > > > > time
> > > > > > > > > > > > 1200.
> > > > > > > > > > > > > > > > > > That is what I mean by "the network time is
> > > > > > included
> > > > > > > > in the
> > > > > > > > > > > > > > expiration
> > > > > > > > > > > > > > > > > > time."
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > And I think, some assumption about
> > network
> > > > round
> > > > > > trip
> > > > > > > > > > time is
> > > > > > > > > > > > > > > > > > > needed anyway to decide on the frequency
> > of
> > > > the
> > > > > > > > > > heartbeat (
> > > > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms), and
> > > > lease
> > > > > > > > timeout (
> > > > > > > > > > > > > > > > > > > registration.lease.timeout.ms). So I
> > think
> > > > just
> > > > > > > > having a
> > > > > > > > > > > > > > leaseTTL
> > > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > > request is easier to understand and
> > > > implement.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It's important to keep in mind that
> > messages
> > > > may be
> > > > > > > > > > delayed in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > > happens, we
> > > > > > > > > > will
> > > > > > > > > > > > use
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > > time specified in the request to determine
> > if
> > > > the
> > > > > > > > request
> > > > > > > > > > is
> > > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> > the
> > > > > > > > controller
> > > > > > > > > > side
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > reset in the case of controller
> > failover.
> > > > The
> > > > > > > > > > alternative
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > be to
> > > > > > > > > > > > > > > > > > > > track the lease as hard state rather
> > than
> > > > soft
> > > > > > > > state,
> > > > > > > > > > but I
> > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > is not really needed, and would result
> > in
> > > > more
> > > > > > log
> > > > > > > > > > entries.
> > > > > > > > > > > > > > > > > > > My interpretation of the mention of
> > > > BrokerRecord
> > > > > > in
> > > > > > > > the
> > > > > > > > > > KIP
> > > > > > > > > > > > was
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > record exists in the Raft log.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > BrokerRecord does exist in the Raft log,
> > but
> > > > does
> > > > > > not
> > > > > > > > > > include
> > > > > > > > > > > > the
> > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > heartbeat time.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > By soft state, do you mean the broker
> > > > > > > > > > > > > > > > > > > records exist only on the active leader
> > and
> > > > will
> > > > > > not
> > > > > > > > be
> > > > > > > > > > > > > > replicated
> > > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > > raft log? If the live brokers list is
> > > > maintained
> > > > > > > > only on
> > > > > > > > > > the
> > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > controller (raft leader), then, in case
> > of
> > > > leader
> > > > > > > > > > failure,
> > > > > > > > > > > > there
> > > > > > > > > > > > > > > > will be
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > window where the new leader does not know
> > > > about
> > > > > > the
> > > > > > > > live
> > > > > > > > > > > > brokers,
> > > > > > > > > > > > > > > > till
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > brokers establish the leases again.
> > > > > > > > > > > > > > > > > > > I think it will be safer to have leases
> > as a
> > > > hard
> > > > > > > > state
> > > > > > > > > > > > managed
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > standard
> > > > > > > > > > > > > > > > > > > Raft replication.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Leases are short, so the need to
> > re-establish
> > > > them
> > > > > > > > after a
> > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > failover doesn't seem like a big problem.
> > But
> > > > > > this is
> > > > > > > > > > > > something
> > > > > > > > > > > > > > we can
> > > > > > > > > > > > > > > > > > tweak if it becomes an issue. One option
> > > > would be
> > > > > > to
> > > > > > > > have
> > > > > > > > > > a
> > > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > which is only used by the controller nodes
> > for
> > > > this
> > > > > > > > (since,
> > > > > > > > > > > > after
> > > > > > > > > > > > > > all,
> > > > > > > > > > > > > > > > > > brokers don't care about registration
> > > > renewals).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Or am I misunderstanding something? (I
> > assume
> > > > > > that
> > > > > > > > with
> > > > > > > > > > soft
> > > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > mean something like zookeeper local
> > sessions
> > > > > > > > > > > > > > > > > > >
> > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1147
> > > > > > > > .)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > > think
> > > > > > it
> > > > > > > > makes
> > > > > > > > > > > > sense
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> > becomes
> > > > > > very
> > > > > > > > messy.
> > > > > > > > > > > > I'm
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > > understand your question about
> > duplicate
> > > > > > broker ID
> > > > > > > > > > > > detection,
> > > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > There's a section in the KIP about this
> > -- is
> > > > > > there a
> > > > > > > > > > detail
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This is an implementation detail that
> > doesn't
> > > > need
> > > > > > to
> > > > > > > > be
> > > > > > > > > > in the
> > > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I assumed broker leases are implemented
> > as a
> > > > hard
> > > > > > > > state.
> > > > > > > > > > In
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > case, to
> > > > > > > > > > > > > > > > > > > check for broker id conflict, we need to
> > > > check
> > > > > > the
> > > > > > > > broker
> > > > > > > > > > > > ids at
> > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > places
> > > > > > > > > > > > > > > > > > > 1. Pending broker registrations (which
> > are
> > > > yet
> > > > > > to be
> > > > > > > > > > > > committed)
> > > > > > > > > > > > > > 2.
> > > > > > > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > > > > > committed broker registrations.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020 at 5:42 PM Colin
> > McCabe
> > > > <
> > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020, at 01:12, Unmesh
> > > > Joshi
> > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >>>Can you repeat your questions
> > about
> > > > broker
> > > > > > > > leases?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >>>>The LeaseStartTimeMs is expected
> > to
> > > > be
> > > > > > the
> > > > > > > > > > broker's
> > > > > > > > > > > > > > > > > > > > > 'System.currentTimeMillis()' at the
> > > > point of
> > > > > > the
> > > > > > > > > > > > request. The
> > > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > > controller will add its lease period
> > to
> > > > this
> > > > > > in
> > > > > > > > order
> > > > > > > > > > > > >>>>to
> > > > > > > > > > > > > > > > compute
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > LeaseEndTimeMs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think the use of LeaseStartTimeMs
> > and
> > > > > > > > > > LeaseEndTimeMs
> > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > confusing. Monotonic Clock
> > > > > > (System.nanoTime) on
> > > > > > > > the
> > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > should be used to track leases.
> > > > > > > > > > > > > > > > > > > > > (For example,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1616https://github.com/etcd-io/etcd/pull/6888/commits/e7f4010ccaf28b6ce64fe514d25a4b2fa459d114
> > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Then we will not need
> > LeaseStartTimeMs?
> > > > > > > > > > > > > > > > > > > > > Instead of LeaseStartTimeMs, can we
> > call
> > > > it
> > > > > > > > > > LeaseTTL? The
> > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > can then calculate LeaseEndTime =
> > > > > > > > System.nanoTime() +
> > > > > > > > > > > > > > LeaseTTL.
> > > > > > > > > > > > > > > > > > > > > In this case we might just drop
> > > > > > LeaseEndTimeMs
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > > > > > response, as
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > broker already knows about the TTL
> > and
> > > > can
> > > > > > send
> > > > > > > > > > > > heartbeats at
> > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > fraction
> > > > > > > > > > > > > > > > > > > > > of TTL, say every TTL/4
> > > > milliseconds.(elapsed
> > > > > > > > time
> > > > > > > > > > on the
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > measured
> > > > > > > > > > > > > > > > > > > > > by System.nanoTime)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I agree that the monotonic clock is
> > > > probably a
> > > > > > > > better
> > > > > > > > > > idea
> > > > > > > > > > > > > > here.
> > > > > > > > > > > > > > > > It is
> > > > > > > > > > > > > > > > > > > > good to be robust against wall clock
> > > > changes,
> > > > > > > > although
> > > > > > > > > > I
> > > > > > > > > > > > think
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > > > > which had them might suffer other
> > issues.
> > > > I
> > > > > > will
> > > > > > > > > > change
> > > > > > > > > > > > it to
> > > > > > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > monotonic clock.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The reason for including
> > LeaseStartTimeMs
> > > > in
> > > > > > the
> > > > > > > > > > request
> > > > > > > > > > > > is to
> > > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > the time required to communicate with
> > the
> > > > > > > > controller
> > > > > > > > > > gets
> > > > > > > > > > > > > > included
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > lease time. Since requests can
> > > > potentially be
> > > > > > > > delayed
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > > > > > long time, this is important.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I have a prototype built to
> > demonstrate
> > > > this
> > > > > > as
> > > > > > > > > > > > following:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The Kip631Controller itself depends
> > on a
> > > > > > > > Consensus
> > > > > > > > > > > > module, to
> > > > > > > > > > > > > > > > > > demonstrate
> > > > > > > > > > > > > > > > > > > > > how possible interactions with the
> > > > consensus
> > > > > > > > module
> > > > > > > > > > will
> > > > > > > > > > > > look
> > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > (The Consensus can be pluggable
> > really,
> > > > > > with an
> > > > > > > > API
> > > > > > > > > > to
> > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > > replicated log upto HighWaterMark)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > It has an implementation of
> > LeaseTracker
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/LeaderLeaseTracker.scala
> > > > > > > > > > > > > > > > > > > > > to demonstrate LeaseTracker's
> > interaction
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > > consensus
> > > > > > > > > > > > > > > > module.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The implementation has the following
> > > > aspects:
> > > > > > > > > > > > > > > > > > > > > 1. The lease tracking happens only
> > on the
> > > > > > active
> > > > > > > > > > > > controller
> > > > > > > > > > > > > > (raft
> > > > > > > > > > > > > > > > > > > > > leader)
> > > > > > > > > > > > > > > > > > > > > 2. Once the lease expires, it needs
> > to
> > > > > > propose
> > > > > > > > and
> > > > > > > > > > > > commit a
> > > > > > > > > > > > > > > > > > FenceBroker
> > > > > > > > > > > > > > > > > > > > > record for that lease.
> > > > > > > > > > > > > > > > > > > > > 3. In case of active controller
> > failure,
> > > > the
> > > > > > > > lease
> > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > tracked by
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > newly raft leader. The new raft
> > leader
> > > > > > starts the
> > > > > > > > > > lease
> > > > > > > > > > > > timer
> > > > > > > > > > > > > > > > again,
> > > > > > > > > > > > > > > > > > (as
> > > > > > > > > > > > > > > > > > > > > implemented in onBecomingLeader
> > method of
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > > in effect extending the lease by the
> > time
> > > > > > spent
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > election
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > whatever time was elapsed on the old
> > > > leader.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> > the
> > > > > > > > controller
> > > > > > > > > > side
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > reset
> > > > > > > > > > > > > > > > > > > > in the case of controller failover.
> > The
> > > > > > > > alternative
> > > > > > > > > > would
> > > > > > > > > > > > be
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > track
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > lease as hard state rather than soft
> > state,
> > > > > > but I
> > > > > > > > think
> > > > > > > > > > > > that
> > > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > > > > > needed, and would result in more log
> > > > entries.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > There are working tests for this
> > > > > > implementation
> > > > > > > > here.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > > > and an end to end test here
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/ProducerConsumerKIP500Test.scala
> > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >>'m not sure what you mean by
> > > > > > "de-duplication
> > > > > > > > of the
> > > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > > Apologies for using the confusing
> > term
> > > > > > > > > > deduplication. I
> > > > > > > > > > > > meant
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > id
> > > > > > > > > > > > > > > > > > > > > conflict.
> > > > > > > > > > > > > > > > > > > > > As you can see in the prototype
> > > > > > handleRequest of
> > > > > > > > > > > > > > KIP631Controller
> > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > > the duplicate broker id needs to be
> > > > detected
> > > > > > > > before
> > > > > > > > > > the
> > > > > > > > > > > > > > > > BrokerRecord
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > submitted to the raft module.
> > > > > > > > > > > > > > > > > > > > > Also as implemented in the
> > prototype, the
> > > > > > > > > > > > KIP631Controller is
> > > > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > > > > > threaded, handling requests one at a
> > > > time.
> > > > > > (an
> > > > > > > > > > example of
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://martinfowler.com/articles/patterns-of-distributed-systems/singular-update-queue.html
> > > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > > think
> > > > > > it
> > > > > > > > makes
> > > > > > > > > > > > sense
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> > becomes
> > > > > > very
> > > > > > > > messy.
> > > > > > > > > > > > I'm
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > > understand your question about
> > duplicate
> > > > > > broker ID
> > > > > > > > > > > > detection,
> > > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > > There's a section in the KIP about
> > this --
> > > > is
> > > > > > > > there a
> > > > > > > > > > > > detail we
> > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 10:49 AM
> > Colin
> > > > > > McCabe <
> > > > > > > > > > > > > > colin@cmccabe.xyz
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2020, at 19:36,
> > Unmesh
> > > > > > Joshi
> > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > There were a few of questions I
> > had..
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for the response.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. Were my comments on the broker
> > > > lease
> > > > > > > > > > > > implementation
> > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > prototype) appropriate and do we
> > > > need to
> > > > > > > > change
> > > > > > > > > > the
> > > > > > > > > > > > KIP
> > > > > > > > > > > > > > > > > > > > > > > description accordingly?.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Can you repeat your questions about
> > > > broker
> > > > > > > > leases?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. How will broker epochs be
> > > > generated?
> > > > > > I am
> > > > > > > > > > > > assuming it
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > committed log offset (like zxid?)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > There isn't any need to use a log
> > > > offset.
> > > > > > We
> > > > > > > > can
> > > > > > > > > > just
> > > > > > > > > > > > > > look at
> > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > in-memory hash table and see what
> > the
> > > > > > latest
> > > > > > > > > > number is,
> > > > > > > > > > > > > > and add
> > > > > > > > > > > > > > > > > > one, to
> > > > > > > > > > > > > > > > > > > > > > generate a new broker epoch.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3. How will producer registration
> > > > > > happen? I
> > > > > > > > am
> > > > > > > > > > > > assuming
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > should be
> > > > > > > > > > > > > > > > > > > > > > > similar to broker registration,
> > with
> > > > a
> > > > > > > > similar
> > > > > > > > > > way to
> > > > > > > > > > > > > > > > generate
> > > > > > > > > > > > > > > > > > > > producer
> > > > > > > > > > > > > > > > > > > > > > id.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > For the EOS stuff, we will need a
> > few
> > > > new
> > > > > > RPCs
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > > > controller.
> > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > we should do that in a follow-on
> > KIP,
> > > > > > though,
> > > > > > > > since
> > > > > > > > > > > > this
> > > > > > > > > > > > > > one is
> > > > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > > > > pretty big.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 4. Because we expose Raft log to
> > all
> > > > the
> > > > > > > > > > brokers, any
> > > > > > > > > > > > > > > > > > de-duplication
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > broker needs to happen before the
> > > > > > requests
> > > > > > > > are
> > > > > > > > > > > > proposed
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > Raft.
> > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > the controller needs to be single
> > > > > > threaded,
> > > > > > > > and
> > > > > > > > > > > > should do
> > > > > > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > > > > against the in-process or pending
> > > > > > requests
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > > final
> > > > > > > > > > > > > > > > state. I
> > > > > > > > > > > > > > > > > > > > read a
> > > > > > > > > > > > > > > > > > > > > > > mention of this, in the
> > responses in
> > > > this
> > > > > > > > > > > > thread.Will it
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > useful to
> > > > > > > > > > > > > > > > > > > > > > > mention this in the KIP?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I'm not sure what you mean by
> > > > > > "de-duplication
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 4:50 AM
> > Colin
> > > > > > McCabe
> > > > > > > > <
> > > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I'm thinking of calling a vote
> > on
> > > > > > KIP-631
> > > > > > > > on
> > > > > > > > > > > > Monday.
> > > > > > > > > > > > > > Let
> > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > know
> > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > there's any more comments I
> > should
> > > > > > address
> > > > > > > > > > before I
> > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > vote.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020, at 05:39,
> > > > Unmesh
> > > > > > > > Joshi
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >>Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > >>Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks. I will be adding
> > more to
> > > > the
> > > > > > > > > > prototype
> > > > > > > > > > > > and
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > > > > and collaborate.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020 at
> > 12:28 AM
> > > > > > Colin
> > > > > > > > > > McCabe <
> > > > > > > > > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Jose,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > That'a s good point that I
> > > > hadn't
> > > > > > > > > > considered.
> > > > > > > > > > > > It's
> > > > > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > > a separate leader change
> > > > message,
> > > > > > as
> > > > > > > > you
> > > > > > > > > > > > mentioned.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2020, at
> > 11:56,
> > > > Jose
> > > > > > > > Garcia
> > > > > > > > > > > > Sancio
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Very cool prototype!
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > The KIP proposes a record
> > > > called
> > > > > > > > > > IsrChange
> > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > includes the
> > > > > > > > > > > > > > > > > > > > > > > > > > > partition, topic, isr,
> > > > leader and
> > > > > > > > leader
> > > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > > During
> > > > > > > > > > > > > > > > > > normal
> > > > > > > > > > > > > > > > > > > > > > > > > > > operation ISR changes do
> > not
> > > > > > result
> > > > > > > > in
> > > > > > > > > > leader
> > > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > > > Similarly,
> > > > > > > > > > > > > > > > > > > > > > > > > > > leader changes do not
> > > > necessarily
> > > > > > > > > > involve ISR
> > > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > > implementation that uses
> > ZK
> > > > > > modeled
> > > > > > > > them
> > > > > > > > > > > > together
> > > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. All of this
> > information is
> > > > > > stored
> > > > > > > > in
> > > > > > > > > > one
> > > > > > > > > > > > > > znode.
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. ZK's optimistic lock
> > > > requires
> > > > > > > > that you
> > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > the new
> > > > > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > > > > > > > > 3. The change to that
> > znode
> > > > was
> > > > > > being
> > > > > > > > > > > > performed
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > > and the leader.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > None of these reasons are
> > > > true in
> > > > > > > > > > KIP-500.
> > > > > > > > > > > > Have
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > considered
> > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > > > two different records?
> > For
> > > > > > example
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. IsrChange record which
> > > > > > includes
> > > > > > > > topic,
> > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > isr
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. LeaderChange record
> > which
> > > > > > includes
> > > > > > > > > > topic,
> > > > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > leader epoch.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I suspect that making
> > this
> > > > change
> > > > > > > > will
> > > > > > > > > > also
> > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > message AlterIsrRequest
> > > > > > introduced in
> > > > > > > > > > > > KIP-497:
> > > > > > > > > > > > > > Add
> > > > > > > > > > > > > > > > > > > > inter-broker
> > > > > > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > alter ISR.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > > > > > -Jose
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the reply. A few more comments below.
80.1 controller.listener.names only defines the name of the listener. The
actual listener including host/port/security_protocol is typically defined
in advertised_listners. Does that mean advertised_listners is a required
config now?
83.1 broker state machine: It seems that we should transition from FENCED
=> INITIAL since only INITIAL generates new broker epoch?
83.5. It's true that the controller node doesn't serve metadata requests.
However, there are admin requests such as topic creation/deletion are sent
to the controller directly. So, it seems that the client needs to know
the controller host/port?
85. "I was hoping that we could avoid responding to requests when the
broker was fenced." This issue is that if we don't send a response, the
client won't know the reason and can't act properly.
88. CurMetadataOffset: I was thinking that we may want to
use CurMetadataOffset to compute the MetadataLag. Since HWM is exclusive,
it's more convenient if CurMetadataOffset is also exclusive.
90. It would be useful to add a rejected section on why separate controller
and broker id is preferred over just broker id. For example, the following
are some potential reasons. (a) We can guard duplicated brokerID, but it's
hard to guard against duplicated controllerId. (b) brokerID can be auto
assigned in the future, but controllerId is hard to be generated
automatically.
Thanks,
Jun
On Mon, Oct 12, 2020 at 11:14 AM Colin McCabe <cm...@apache.org> wrote:
> On Tue, Oct 6, 2020, at 16:09, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the reply. Made another pass of the KIP. A few more comments
> > below.
> >
>
> Hi Jun,
>
> Thanks for the review.
>
> > 55. We discussed earlier why the current behavior where we favor the
> > current broker registration is better. Have you given this more thought?
> >
>
> Yes, I think we should favor the current broker registration, as you
> suggested earlier.
>
> > 80. Config related.
> > 80.1 Currently, each broker only has the following 3 required configs. It
> > will be useful to document the required configs post KIP-500 (in both the
> > dedicated and shared controller mode).
> > broker.id
> > log.dirs
> > zookeeper.connect
>
> For the broker, these configs will be required:
>
> broker.id
> log.dirs
> process.roles
> controller.listener.names
> controller.connect
>
> For the controller, these configs will be required:
>
> controller.id
> log.dirs
> process.roles
> controller.listener.names
> controller.connect
>
> For broker+controller, it will be the union of these two, which
> essentially means we need both broker.id and controller.id, but all
> others are the same as standalone.
>
> > 80.2 It would be useful to document all deprecated configs post KIP-500.
> > For example, all zookeeper.* are obviously deprecated. But there could be
> > others. For example, since we don't plan to support auto broker id
> > generation, it seems broker.id.generation.enable is deprecated too.
> > 80.3 Could we make it clear that controller.connect replaces
> quorum.voters
> > in KIP-595?
>
> OK. I added a comment about this in the table.
>
> > 80.4 Could we document that broker.id is now optional?
>
> OK. I have added a line for broker.id.
>
> > 80.5 The KIP suggests that controller.id is optional on the controller
> > node. I am concerned that this can cause a bit of confusion in 2 aspects.
> > First, in the dedicated controller mode, controller.id is not optional
> > (since broker.id is now optional). Second, in the shared controller
> mode,
> > it may not be easy for the user to figure out the default value of
> > controller.id to set controller.connect properly.
>
> I got rid of the default value of controller.id. I agree it was a bit
> confusing to explain.
>
> > 80.6 Regarding the consistency of config names, our metrics already
> include
> > controller. So, prefixing all controller related configs with
> "controller"
> > may be more consistent. If we choose to do that, could we rename all new
> > configs here and in KIP-595 consistently?
> >
>
> I added the new names for the KIP-595 configurations to the configuration
> table. I think prefixing these configurations with "controller.quorum" is
> the best option since it makes it clear that they are related to the
> controller quorum.
>
> > 81. Metrics
> > 81.1 kafka.controller:type=KafkaController,name=MetadataSnapshotLag: This
> > is now redundant since KIP-630 already has
> > kafka.controller:type=KafkaController,name=SnapshotLag.
>
> Thanks for bringing up the metric name issue. I think it would be good to
> clarify that this is the lag for the metadata snapshot specifically. So I
> prefer the name MetadataSnapshotLag. I'll add a note to the table.
>
> > 81.2 Do we need both kafka.controller:type=KafkaServer,name=MetadataLag
> and
> > kafka.controller:type=KafkaController,name=MetadataLag since in the
> shared
> > controller mode, the metadata log is shared?
>
> Yes, both are needed. This metric is about what metadata logs we have
> applied, not about the Raft log's high water mark. If the broker or
> controller takes a while to apply the log messages (for example, because of
> lock contention or some other issue), they could fall significantly behind
> the high watermark.
>
> >
> > 82. Metadata records
> > 82.1 BrokerRecord: It needs to include supported features based on
> KIP-584.
>
> Added.
>
> > 82.2 FeatureLevel: The finalized feature in KIP-584 has the following
> > structure and needs to be reflected here.
> > {
> > "version": 0, // int32 -> Represents the version of the schema for the
> > data stored in the ZK node
> > "status": 1, // int32 -> Represents the status of the node
> > "features": {
> > "group_coordinator": { // string -> name of the feature
> > "min_version_level": 0, // int16 -> Represents the
> cluster-wide
> > finalized minimum version level (>=1) of this feature
> > "max_version_level": 3 // int16 -> Represents the
> cluster-wide
> > finalized maximum version level (>=1 and >= min_version_level) of this
> > feature
> > },
> > "consumer_offsets_topic_schema": {
> > "min_version_level": 0,
> > "max_version_level": 4
> > }
> > }
>
> Fair point. I will add both a minimum and maximum finalized feature level.
>
> >
> > 83. RPC
> > 83.1 BrokerHeartbeatRequest: Could you document when the controller
> assigns
> > a new broker epoch?
>
> The controller assigns a broker epoch if the heartbeat request was sent
> without an epoch (that is, it is in the INITIAL state), and the controller
> wants to add the new broker to the cluster. I added a note to the state
> machine.
>
> > 83.2 BrokerHeartbeatResponse: Do we need a new error code to indicate a
> > broker is fenced?
>
> I don't think we need a new error code, since targetState will just
> indicate that the broker should be fenced.
>
> > 83.3 What error/state indicates an unsuccessful controlled shutdown in
> > BrokerHeartbeatResponse? Currently, the controlled shutdown responses
> also
> > include a list of remaining partitions. That's mostly for debugging
> > purposes. Should we include it in BrokerHeartbeatResponse too?
>
> Once the controller wants the broker to shut down, it will set targetState
> to SHUTDOWN.
>
> We don't need to send a list of remaining partitions, since the broker is
> following the metadata log, and will get all its instructions from there.
>
> > 83.4 Could we document when Listeners, Features and Rack are expected to
> be
> > set in BrokerHeartbeatRequest?
>
> I added a line specifying that: The listeners, features, and rack fields
> only need to be set when the broker is in INITIAL state, and is requesting
> a new broker epoch.
>
> > 83.5 Currently, the metadata response only returns a list of brokers.
> > Should we have a separate field for the controller info or change the
> field
> > name to sth like nodes? Also, is the rack field relevant to the
> controller?
>
> As per KIP-590, we don't want the clients to communicate directly with the
> controller. So we weren't planning on including the controller's
> information in the MetadataResponse.
>
> We don't currently have rack-awareness for controllers, but I suppose we
> could in the future. I think we can leave that for later.
>
> >
> > 84. There are a few references to __kafka_metadata. However, KIP-595
> > uses __cluster_metadata.
>
> We agreed earlier to decide on the name of the metadata topic in this KIP,
> and to treat the KIP-595 name as provisional. Looking at it again, I think
> we should go with @metadata, in order to avoid colliding with topic names
> that already exist.
>
> >
> > 85. "When a broker is fenced, it cannot process any client requests. "
> > Should we send a new error code in the response to indicate that the
> broker
> > is fenced?
>
> I was hoping that we could avoid responding to requests when the broker
> was fenced. I suppose a new error code could work too, but then we also
> need a compatibility story for older clients.
>
> >
> > 86. "The controller can generate a new broker epoch by using the latest
> log
> > offset." Perhaps it's more accurate to say the new broker epoch is the
> > offset for the corresponding BrokerRecord in the metadata log.
>
> The idea here is slightly different. We want to use the current latest
> offset (prior to creating the BrokerRecord) as the new epoch. We can't use
> the offset of the BrokerRecord itself since we won't know what that is
> until we give the BrokerRecord to the Raft layer.
>
> >
> > 87. meta.properties:
> > 87.1 Could you document the format change to this file? Also, should we
> > bump up the version of the file?
>
> Hmm. The file is just a list of key=value pairs, one per line. So I'm
> not sure there is any format change per se, at least technically.
>
> Maybe we should bump up the version anyway, in order to prevent older
> pre-KIP-500 software from launching if the new stuff is present?
>
> > 87.2 How do we know a node is configured in KIP-500 mode?
> >
>
> A cluster configured in KIP-500 mode will contain the following line in
> the meta.properties file in each directory: kip.500.mode=enabled
>
> > 88. CurMetadataOffset: Should we use nextMetadataOffset since it's more
> > consistent with fetchOffset and HWM?
>
> I'm not sure. Drawing an analogy with HWM might be confusing, since
> CurMetadataOffset is not HWM. It's totally possible for the broker to have
> fetched up to offset 1000 but only applied up to offset 500.
>
> best,
> Colin
>
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Oct 5, 2020 at 9:03 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Mon, Sep 28, 2020, at 11:41, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply. A few more comments below.
> > > >
> > > > 62.
> > > > 62.1 controller.listener.names: So, is this used for the controller
> or
> > > the
> > > > broker trying to connect to the controller?
> > > >
> > >
> > > Hi Jun,
> > >
> > > It's used by both. The broker tries to connect to controllers by using
> > > the first listener in this list. The controller uses the list to
> determine
> > > which listeners it should bind to.
> > >
> > > >
> > > > 62.2 If we want to take the approach to share the configs that are
> common
> > > > between the broker and the controller, should we share the id too?
> > > >
> > >
> > > On nodes where we are running both the controller and broker roles, we
> > > need two IDs: one for the controller, and one for the broker. So we
> can't
> > > share the same configuration.
> > >
> > > >
> > > > 62.3 We added some configs in KIP-595 prefixed with "quorum" and we
> plan
> > > to
> > > > add some controller specific configs prefixed with "controller".
> KIP-630
> > > > plans to add some other controller specific configs with no prefix.
> > > Should
> > > > we standardize all controller specific configs with the same prefix?
> > > >
> > >
> > > That's a good question. As Jose said elsewhere in the thread, let's
> use
> > > "metadata" as the prefix.
> > >
> > > >
> > > > 70. Could you explain the impact of process.roles a bit more? For
> > > example,
> > > > if process.roles=controller, does the node still maintain metadata
> cache
> > > as
> > > > described in KIP-630? Do we still return the host/port for those
> nodes in
> > > > the metadata response?
> > > >
> > >
> > > No, the node does not need any broker data structures if it is not a
> > > broker. Controllers also don't handle MetadataRequests. Clients will
> > > still go to the brokers for those.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Sep 28, 2020 at 9:24 AM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > On Fri, Sep 25, 2020, at 17:35, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the reply.
> > > > > >
> > > > > > 62. Thinking about this more, I am wondering what's our overall
> > > strategy
> > > > > > for configs shared between the broker and the controller. For
> > > example,
> > > > > both
> > > > > > the broker and the controller have to define listeners. So both
> need
> > > > > > configs like listeners/advertised.listeners. Both the new
> controller
> > > and
> > > > > > the broker replicate data. So both need to define some
> replication
> > > > > related
> > > > > > configurations (replica.fetch.min.bytes,
> replica.fetch.wait.max.ms,
> > > > > etc).
> > > > > > Should we let the new controller share those configs with
> brokers or
> > > > > should
> > > > > > we define new configs just for the controller? It seems that we
> want
> > > to
> > > > > > apply the same strategy consistently for all shared configs?
> > > > > >
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > That's a good question. I think that we should share as many
> > > > > configurations as possible. There will be a few cases where this
> isn't
> > > > > possible, and we need to create a new configuration key that is
> > > > > controller-only, but I think that will be relatively rare.
> > > > >
> > > > > >
> > > > > > 63. If a node has process.roles set to controller, does one still
> > > need to
> > > > > > set broker.id on this node?
> > > > > >
> > > > >
> > > > > No, broker.id does not need to be set in that case.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 25, 2020 at 2:17 PM Colin McCabe <cmccabe@apache.org
> >
> > > wrote:
> > > > > >
> > > > > > > On Fri, Sep 25, 2020, at 10:17, Jun Rao wrote:
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the reply.
> > > > > > > >
> > > > > > > > 60. Yes, I think you are right. We probably need the
> controller
> > > id
> > > > > when a
> > > > > > > > broker starts up. A broker only stores the Raft leader id in
> the
> > > > > metadata
> > > > > > > > file. To do the initial fetch to the Raft leader, it needs to
> > > know
> > > > > the
> > > > > > > > host/port associated with the leader id.
> > > > > > > >
> > > > > > > > 62. It seems there are 2 parts to this : (1) which listener a
> > > client
> > > > > > > should
> > > > > > > > use to initiate a connection to the controller and (2) which
> > > listener
> > > > > > > > should a controller use to accept client requests. For (1),
> at
> > > any
> > > > > point
> > > > > > > of
> > > > > > > > time, a client only needs to use one listener. I think
> > > > > > > > controller.listener.name is meant for the client.
> > > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > controller.listener.names is also used by the controllers. In
> the
> > > case
> > > > > > > where we have a broker and a controller in the same JVM, we
> have a
> > > > > single
> > > > > > > config file. Then we need to know which listeners belong to
> the
> > > > > controller
> > > > > > > and which belong to the broker component. That's why it's a
> list.
> > > > > > >
> > > > > > > > So, a single value seems
> > > > > > > > to make more sense. Currently, we don't have a configuration
> for
> > > > > (2). We
> > > > > > > > could add a new one for that and support a list. I am
> wondering
> > > how
> > > > > > > useful
> > > > > > > > it will be. One example that I can think of is that we can
> reject
> > > > > > > > non-controller related requests if accepted on
> controller-only
> > > > > listeners.
> > > > > > > > However, we already support separate authentication for the
> > > > > controller
> > > > > > > > listener. So, not sure how useful it is.
> > > > > > >
> > > > > > > The controller always has a separate listener and does not
> share
> > > > > listeners
> > > > > > > with the broker. The main reason for this is to avoid giving
> > > clients
> > > > > the
> > > > > > > ability to launch a denial-of-service attack on the controller
> by
> > > > > flooding
> > > > > > > a broker port. A lot of times, these attacks are made
> > > unintentionally
> > > > > by
> > > > > > > poorly coded or configured clients. Additionally, broker
> ports can
> > > > > also be
> > > > > > > very busy with large fetch requests, and so on. It's just a
> bad
> > > > > > > configuration in general to try to overload the same port for
> the
> > > > > > > controller and broker, and we don't want to allow it. It
> would be
> > > a
> > > > > > > regression to go from the current system where control
> requests are
> > > > > safely
> > > > > > > isolated on a separate port, to one where they're not. It also
> > > makes
> > > > > the
> > > > > > > code and configuration a lot messier.
> > > > > > >
> > > > > > > >
> > > > > > > > 63. (a) I think most users won't know controller.id
> defaults to
> > > > > > > broker.id +
> > > > > > > > 3000. So, it can be confusing for them to set up
> > > controller.connect.
> > > > > If
> > > > > > > > this is truly needed, it seems that it's less confusing to
> make
> > > > > > > > controller.id required.
> > > > > > > > (b) I am still trying to understand if we truly need to
> expose a
> > > > > > > > controller.id. What if we only expose broker.id and let
> > > > > > > controller.connect
> > > > > > > > just use broker.id? What will be missing?
> > > > > > >
> > > > > > > The controller has a separate ID from the broker, so knowing
> > > broker.id
> > > > > is
> > > > > > > not helpful here.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Thu, Sep 24, 2020 at 10:55 PM Colin McCabe <
> > > cmccabe@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > On Thu, Sep 24, 2020, at 16:24, Jun Rao wrote:
> > > > > > > > > > Hi, Colin,
> > > > > > > > > >
> > > > > > > > > > Thanks for the reply and the updated KIP. A few more
> comments
> > > > > below.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 53. It seems that you already incorporated the changes in
> > > > > KIP-516.
> > > > > > > With
> > > > > > > > > > topic ids, we don't need to wait for the topic's data to
> be
> > > > > deleted
> > > > > > > > > before
> > > > > > > > > > removing the topic metadata. If the topic is recreated,
> we
> > > can
> > > > > still
> > > > > > > > > delete
> > > > > > > > > > the data properly based on the topic id. So, it seems
> that
> > > we can
> > > > > > > remove
> > > > > > > > > > TopicRecord.Deleting.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks for the reply. What I was thinking of doing here
> was
> > > using
> > > > > > > topic
> > > > > > > > > IDs internally, but still using names externally. So the
> topic
> > > > > UUIDs
> > > > > > > are
> > > > > > > > > only for the purpose of associating topics with partitions
> --
> > > from
> > > > > the
> > > > > > > > > user's point of view, topics are still identified by names.
> > > > > > > > >
> > > > > > > > > You're right that KIP-516 will simplify things, but I'm not
> > > sure
> > > > > when
> > > > > > > that
> > > > > > > > > will land, so I wanted to avoid blocking the initial
> > > > > implementation of
> > > > > > > this
> > > > > > > > > KIP on it.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 55. It seems to me that the current behavior where we
> favor
> > > the
> > > > > > > current
> > > > > > > > > > broker registration is better. This is because
> uncontrolled
> > > > > broker
> > > > > > > > > shutdown
> > > > > > > > > > is rare and its impact is less severe since one just
> needs to
> > > > > wait
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > session timeout before restarting the broker. If a
> > > mis-configured
> > > > > > > broker
> > > > > > > > > > replaces an existing broker, the consequence is more
> severe
> > > > > since it
> > > > > > > can
> > > > > > > > > > cause the leader to be unavailable, a replica to be out
> of
> > > ISR,
> > > > > and
> > > > > > > add
> > > > > > > > > > more load on the leaders etc.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hmm, that's a good point. Let me check this a bit more
> before
> > > I
> > > > > change
> > > > > > > > > it, though.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 60. controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com
> :
> > > Do we
> > > > > > > need to
> > > > > > > > > > include the controller id before @? It seems that the
> > > host/port
> > > > > is
> > > > > > > enough
> > > > > > > > > > for establishing the connection. It would also be useful
> to
> > > make
> > > > > it
> > > > > > > clear
> > > > > > > > > > that controller.connect replaces quorum.voters in
> KIP-595.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I discussed this with Jason earlier, and he felt that the
> > > > > controller
> > > > > > > IDs
> > > > > > > > > were needed in this configuration key. It is certainly
> needed
> > > when
> > > > > > > > > configuring the controllers themselves, since they need to
> know
> > > > > each
> > > > > > > > > others' IDs.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 61. I am not sure that we need both controller.listeners
> and
> > > > > > > > > > controller.connect.security.protocol since the former
> > > implies the
> > > > > > > > > security
> > > > > > > > > > protocol. The reason that we have both
> > > > > inter.broker.listener.name
> > > > > > > and
> > > > > > > > > > inter.broker.security.protocol is because we had the
> latter
> > > > > first and
> > > > > > > > > later
> > > > > > > > > > realized that the former is more general.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > That's a good point. I've removed this from the KIP.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 62. I am still not sure that you need
> controller.listeners
> > > to be
> > > > > a
> > > > > > > list.
> > > > > > > > > > All listeners are already defined in listeners. To
> migrate
> > > from
> > > > > > > plaintext
> > > > > > > > > > to SSL, one can configure listeners to have both
> plaintext
> > > and
> > > > > SSL.
> > > > > > > After
> > > > > > > > > > that, one can just change controller.listeners from
> > > plaintext to
> > > > > SSL.
> > > > > > > > > This
> > > > > > > > > > is similar to how to change the listener for inter broker
> > > > > > > connections.
> > > > > > > > > > Also, controller.listener.name may be a more accurate
> name?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > The issue that I see here is that if you are running with
> the
> > > > > > > controller
> > > > > > > > > and broker in the same JVM, if you define a few listeners
> in
> > > > > > > "listeners"
> > > > > > > > > they will get used as regular broker listeners, unless you
> put
> > > > > them in
> > > > > > > > > controller.listeners. Therefore, controller.listeners
> needs
> > > to be
> > > > > a
> > > > > > > list.
> > > > > > > > >
> > > > > > > > > controller.listener.names does sound like a better name,
> > > though...
> > > > > I've
> > > > > > > > > updated it to that.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 63. Regarding controller.id, I am trying to understand
> > > whether
> > > > > it's
> > > > > > > a
> > > > > > > > > > required configuration or an optional one. From the KIP,
> it
> > > > > sounds
> > > > > > > like
> > > > > > > > > > controller.id is optional. Then, if this is unset, it's
> not
> > > > > clear
> > > > > > > how
> > > > > > > > > the
> > > > > > > > > > user will obtain the controller id for setting
> > > > > controller.connect.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > If you specify broker.id but not controller.id, then
> > > controller.id
> > > > > > > will
> > > > > > > > > just be set to broker.id + 3000. This was intended to
> make
> > > some
> > > > > > > > > configurations a little bit more concise to write. You
> still
> > > do
> > > > > have
> > > > > > > to
> > > > > > > > > know the controller IDs when configuring the brokers,
> though.
> > > If
> > > > > this
> > > > > > > > > seems confusing then I can just make it mandatory.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 64. KIP-516 adds a flag in LeaderAndIsrRequest to
> indicate
> > > > > whether it
> > > > > > > > > > includes all partitions or not. This will be used to
> clean up
> > > > > strayed
> > > > > > > > > logs.
> > > > > > > > > > I was thinking how we will do the same thing once the
> > > controller
> > > > > > > moves to
> > > > > > > > > > Raft based. One way to do that is on broker startup, it
> gets
> > > the
> > > > > HWM
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > metadata log from the Raft leader and waits until its
> > > metadata
> > > > > > > catches up
> > > > > > > > > > to HWM. At that point, any local log not seen in the
> metadata
> > > > > can be
> > > > > > > > > > removed. Since the Fetch response returns the HWM, there
> > > seems
> > > > > to be
> > > > > > > > > enough
> > > > > > > > > > APIs to achieve this.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > That's a very good point. I added a note about this under
> > > Broker
> > > > > > > Startup.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 24, 2020 at 1:07 PM Colin McCabe <
> > > colin@cmccabe.xyz>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > On Mon, Sep 21, 2020, at 18:13, Jun Rao wrote:
> > > > > > > > > > > > Hi, Colin,
> > > > > > > > > > > >
> > > > > > > > > > > > Sorry for the late reply. A few more comments below.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Jun,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for taking another look.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 50. Configurations
> > > > > > > > > > > > 50.1 controller.listeners: It seems that a controller
> > > just
> > > > > needs
> > > > > > > one
> > > > > > > > > > > > listener. Why do we need to have a list here? Also,
> > > could you
> > > > > > > > > provide an
> > > > > > > > > > > > example of how this is set and what's its
> relationship
> > > with
> > > > > > > existing
> > > > > > > > > > > > configs such as "security.inter.broker.protocol" and
> "
> > > > > > > > > > > > inter.broker.listener.name"?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I agree that most administrators will want to run with
> > > only one
> > > > > > > > > controller
> > > > > > > > > > > listener. However, just as with brokers, it is nice to
> > > have
> > > > > the
> > > > > > > > > option to
> > > > > > > > > > > expose multiple ports.
> > > > > > > > > > >
> > > > > > > > > > > One reason why you might want multiple ports is if you
> were
> > > > > doing a
> > > > > > > > > > > migration from plaintext to SSL. You could add new SSL
> > > > > listeners
> > > > > > > but
> > > > > > > > > > > continue to expose the PLAINTEXT listeners. Then you
> could
> > > > > add the
> > > > > > > > > new SSL
> > > > > > > > > > > controller config to each broker and roll each broker.
> > > Then
> > > > > you
> > > > > > > could
> > > > > > > > > > > migrate from PLAINTEXT to SSL with no downtime.
> > > > > > > > > > >
> > > > > > > > > > > Here's an example configuration for the controller:
> > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > 1@controller1.example.com:9093,
> 2@controller2.example.com
> > > > > > > > > > > controller.listeners=CONTROLLER
> > > > > > > > > > > listeners=CONTROLLER://controller0.example.com:9093
> > > > > > > > > > > listener.security.protocol.map=CONTROLLER:SSL
> > > > > > > > > > >
> > > > > > > > > > > Here's an example configuration for the broker:
> > > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > > 1@controller1.example.com:9093,
> 2@controller2.example.com
> > > > > > > > > > > controller.connect.security.protocol=SSL
> > > > > > > > > > >
> > > > > > > > > > > security.inter.broker.protocol or
> > > inter.broker.listener.name
> > > > > do
> > > > > > > not
> > > > > > > > > > > affect how the broker communicates with the controller.
> > > Those
> > > > > > > > > > > configurations relate to how brokers communicate with
> each
> > > > > other,
> > > > > > > but
> > > > > > > > > the
> > > > > > > > > > > controller is not a broker.
> > > > > > > > > > >
> > > > > > > > > > > (Note that I just added
> > > controller.connect.security.protocol
> > > > > to the
> > > > > > > > > KIP --
> > > > > > > > > > > I had forgotten to put it in earlier)
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 50.2 registration.heartbeat.interval.ms and
> > > > > > > > > > > registration.lease.timeout.ms.
> > > > > > > > > > > > Should we match their default value with the
> > > corresponding
> > > > > > > default
> > > > > > > > > for
> > > > > > > > > > > ZK?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Fair enough. I'll set them to the values of the
> > > > > > > > > zookeeper.sync.time.ms
> > > > > > > > > > > and zookeeper.connection.timeout.ms configurations.
> I do
> > > > > think we
> > > > > > > > > should
> > > > > > > > > > > experiment later on to see what works well here, but
> the ZK
> > > > > values
> > > > > > > are
> > > > > > > > > at
> > > > > > > > > > > least a starting point.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 50.3 controller.connect: Could you provide an
> example? I
> > > am
> > > > > > > > > wondering how
> > > > > > > > > > > > it differs from quorum.voters=1@kafka-1:9092,
> 2@kafka-2
> > > > > :9092,
> > > > > > > > > 3@kafka-3
> > > > > > > > > > > > :9092.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > controller.connect is intended to be the new name for
> > > > > > > quorum.voters.
> > > > > > > > > > > During the vote for KIP-595, we sort of agreed to
> defer the
> > > > > > > discussion
> > > > > > > > > > > about what this configuration should be called. I
> proposed
> > > > > this
> > > > > > > new
> > > > > > > > > name
> > > > > > > > > > > because it makes it clear what the configuration is for
> > > (how to
> > > > > > > > > connect to
> > > > > > > > > > > the controllers).
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 50.4 controller.id: I am still not sure how this is
> > > being
> > > > > used.
> > > > > > > > > Could
> > > > > > > > > > > you
> > > > > > > > > > > > explain this in more detail?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The controller ID needs to be set on each controller.
> It
> > > also
> > > > > > > appears
> > > > > > > > > in
> > > > > > > > > > > controller.connect as the thing before the "at" sign.
> Its
> > > > > > > function is
> > > > > > > > > > > pretty similar to broker.id.
> > > > > > > > > > >
> > > > > > > > > > > Broker IDs and controller IDs exist in the same ID
> space,
> > > so
> > > > > you
> > > > > > > can't
> > > > > > > > > > > have a broker and a controller that use the same ID.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 51. BrokerHeartbeat: It seems a bit wasteful to
> include
> > > > > > > Listeners in
> > > > > > > > > > > every
> > > > > > > > > > > > heartbeat request since it typically doesn't change.
> > > Could we
> > > > > > > make
> > > > > > > > > that
> > > > > > > > > > > an
> > > > > > > > > > > > optional field?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Ok.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 52. KIP-584 adds a new ZK node /features. Should we
> add a
> > > > > > > > > corresponding
> > > > > > > > > > > > metadata record?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Good point. I added FeatureLevelRecord.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 53. TopicRecord and DeleteTopic: Both DeleteTopic and
> > > > > > > > > > > TopicRecord.Deleting
> > > > > > > > > > > > indicate topic deletion. Could we outline the flow
> when
> > > each
> > > > > > > will be
> > > > > > > > > set?
> > > > > > > > > > > > In particular, which one indicates the intention to
> > > delete
> > > > > and
> > > > > > > which
> > > > > > > > > one
> > > > > > > > > > > > indicates the completion of the deletion.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > TopicRecord.Deleting is set when we intend to delete
> the
> > > topic.
> > > > > > > > > > >
> > > > > > > > > > > DeleteTopic removes the topic completely. I will
> rename
> > > > > > > DeleteTopic to
> > > > > > > > > > > RemoveTopic to make this clearer.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 54. "The controller can generate a new broker epoch
> by
> > > using
> > > > > the
> > > > > > > > > latest
> > > > > > > > > > > log
> > > > > > > > > > > > offset." Which offset is that? Is it the offset of
> the
> > > > > metadata
> > > > > > > > > topic for
> > > > > > > > > > > > the corresponding BrokerRecord? Is it guaranteed to
> be
> > > > > different
> > > > > > > on
> > > > > > > > > each
> > > > > > > > > > > > broker restart?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Yes.... a new broker epoch implies that there has been
> a
> > > new
> > > > > record
> > > > > > > > > > > created in the metadata log. Therefore the last
> committed
> > > > > offset
> > > > > > > must
> > > > > > > > > be
> > > > > > > > > > > different.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 55. "Thereafter, it may lose subsequent conflicts if
> its
> > > > > broker
> > > > > > > > > epoch is
> > > > > > > > > > > > stale. (See KIP-380 for some background on broker
> > > epoch.)
> > > > > The
> > > > > > > > > reason
> > > > > > > > > > > for
> > > > > > > > > > > > favoring new processes is to accommodate the common
> case
> > > > > where a
> > > > > > > > > process
> > > > > > > > > > > is
> > > > > > > > > > > > killed with kill -9 and then restarted. " Are you
> saying
> > > > > that if
> > > > > > > > > there is
> > > > > > > > > > > > an active broker registered in the controller, a new
> > > broker
> > > > > > > heartbeat
> > > > > > > > > > > > request with the INITIAL state will fence the current
> > > broker
> > > > > > > > > session? Not
> > > > > > > > > > > > sure about this. For example, this will allow a
> broker
> > > with
> > > > > > > > > incorrectly
> > > > > > > > > > > > configured broker id to kill an existing valid
> broker.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Yes, a new broker with an incorrectly configured
> broker id
> > > > > could
> > > > > > > fence
> > > > > > > > > an
> > > > > > > > > > > existing valid broker.
> > > > > > > > > > >
> > > > > > > > > > > The goal of the conflict resolution process described
> here
> > > is
> > > > > to
> > > > > > > avoid
> > > > > > > > > > > having two brokers go back and forth in claiming the
> > > broker id.
> > > > > > > > > Choosing
> > > > > > > > > > > the newest was just an easy way to do that.
> > > > > > > > > > >
> > > > > > > > > > > Choosing the oldest is another possibility, and more
> inline
> > > > > with
> > > > > > > what
> > > > > > > > > > > ZooKeeper does today. The problem with that is that it
> > > would
> > > > > not
> > > > > > > be
> > > > > > > > > > > possible to re-create the broker if the old instance
> died
> > > > > suddenly,
> > > > > > > > > except
> > > > > > > > > > > by waiting for the full lease timeout.
> > > > > > > > > > >
> > > > > > > > > > > It might be possible to do a little better here by
> > > breaking the
> > > > > > > lease
> > > > > > > > > if
> > > > > > > > > > > the TCP connection from the broker drops. But that
> > > requires
> > > > > > > crossing
> > > > > > > > > some
> > > > > > > > > > > layers of abstraction in the Kafka networking stack.
> > > There's
> > > > > also
> > > > > > > the
> > > > > > > > > > > possibility of false negatives or positives. For
> example,
> > > TCP
> > > > > > > > > connections
> > > > > > > > > > > sometimes just drop even if the node is still there.
> Or
> > > > > sometimes
> > > > > > > the
> > > > > > > > > > > networking stack keeps a TCP connection alive for a
> while
> > > when
> > > > > the
> > > > > > > > > other
> > > > > > > > > > > end is gone.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 56. kafka-storage.sh:
> > > > > > > > > > > > 56.1 In the info mode, what other information does it
> > > show in
> > > > > > > > > addition to
> > > > > > > > > > > > "kip.500.mode=enabled"?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I think it should show a list of all the storage
> > > directories,
> > > > > and
> > > > > > > > > whether
> > > > > > > > > > > each one is formatted. In addition, it should show
> whether
> > > > > > > > > kip.500.mode is
> > > > > > > > > > > enabled, and what the cluster id is.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 56.2 Should the format mode take the config file as
> the
> > > > > input too
> > > > > > > > > like
> > > > > > > > > > > the
> > > > > > > > > > > > info mode?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Yes, that's a good idea.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jun
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 17, 2020 at 4:12 PM Colin McCabe <
> > > > > cmccabe@apache.org
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > >
> > > > > > > > > > > > > That's a fair point. I have moved the lease
> duration
> > > to
> > > > > the
> > > > > > > broker
> > > > > > > > > > > > > heartbeat response. That way lease durations can
> be
> > > > > changed
> > > > > > > just
> > > > > > > > > be
> > > > > > > > > > > > > reconfiguring the controllers.
> > > > > > > > > > > > >
> > > > > > > > > > > > > best,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, Sep 16, 2020, at 07:40, Unmesh Joshi wrote:
> > > > > > > > > > > > > > Thanks Colin, the changes look good to me. One
> small
> > > > > thing.
> > > > > > > > > > > > > > registration.lease.timeout.ms is the
> configuration
> > > on
> > > > > the
> > > > > > > > > controller
> > > > > > > > > > > > > side.
> > > > > > > > > > > > > > It will be good to comment on how brokers know
> about
> > > it,
> > > > > to
> > > > > > > be
> > > > > > > > > able
> > > > > > > > > > > to
> > > > > > > > > > > > > > send LeaseDurationMs
> > > > > > > > > > > > > > in the heartbeat request,
> > > > > > > > > > > > > > or else it can be added in the heartbeat
> response for
> > > > > > > brokers to
> > > > > > > > > know
> > > > > > > > > > > > > about
> > > > > > > > > > > > > > it.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Sep 11, 2020 at 10:32 PM Colin McCabe <
> > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think you're right that we should use a
> duration
> > > here
> > > > > > > rather
> > > > > > > > > > > than a
> > > > > > > > > > > > > > > time. As you said, the clock on the controller
> > > will
> > > > > > > probably
> > > > > > > > > not
> > > > > > > > > > > > > match the
> > > > > > > > > > > > > > > one on the broker. I have updated the KIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It's important to keep in mind that
> messages
> > > may be
> > > > > > > > > delayed in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > happens, we
> > > > > > > > > will
> > > > > > > > > > > use
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > time specified in the request to determine
> if
> > > the
> > > > > > > request
> > > > > > > > > is
> > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > I am assuming that there will be a single TCP
> > > > > connection
> > > > > > > > > > > maintained
> > > > > > > > > > > > > > > between
> > > > > > > > > > > > > > > > broker and active controller. So, there
> won't be
> > > any
> > > > > out
> > > > > > > of
> > > > > > > > > order
> > > > > > > > > > > > > > > requests?
> > > > > > > > > > > > > > > > There will be a scenario of broker GC pause,
> > > which
> > > > > might
> > > > > > > > > cause
> > > > > > > > > > > > > connection
> > > > > > > > > > > > > > > > timeout and broker might need to reestablish
> the
> > > > > > > connection.
> > > > > > > > > If
> > > > > > > > > > > the
> > > > > > > > > > > > > pause
> > > > > > > > > > > > > > > > is too long, lease will expire and the
> heartbeat
> > > sent
> > > > > > > after
> > > > > > > > > the
> > > > > > > > > > > pause
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > be treated as a new registration (similar to
> > > restart
> > > > > > > case),
> > > > > > > > > and
> > > > > > > > > > > a new
> > > > > > > > > > > > > > > epoch
> > > > > > > > > > > > > > > > number will be assigned to the broker.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I agree with the end of this paragraph, but not
> > > with
> > > > > the
> > > > > > > start
> > > > > > > > > :)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There can be out-of-order requests, since the
> > > broker
> > > > > will
> > > > > > > > > simply
> > > > > > > > > > > use a
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > TCP connection if the old one has problems.
> This
> > > can
> > > > > > > happen
> > > > > > > > > for a
> > > > > > > > > > > > > variety
> > > > > > > > > > > > > > > of reasons. I don't think GC pauses are the
> most
> > > > > common
> > > > > > > > > reason for
> > > > > > > > > > > > > this to
> > > > > > > > > > > > > > > happen. It's more common to see issues issues
> in
> > > the
> > > > > > > network
> > > > > > > > > > > itself
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > result connections getting dropped from time to
> > > time.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > So we have to assume that messages may arrive
> out
> > > of
> > > > > > > order, and
> > > > > > > > > > > > > possibly
> > > > > > > > > > > > > > > be delayed. I added a note that heartbeat
> requests
> > > > > should
> > > > > > > be
> > > > > > > > > > > ignored
> > > > > > > > > > > > > if
> > > > > > > > > > > > > > > the metadata log offset they contain is smaller
> > > than a
> > > > > > > previous
> > > > > > > > > > > > > heartbeat.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > When the active controller fails, the new
> active
> > > > > > > controller
> > > > > > > > > > > needs to
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > sure that it considers all the known brokers
> as
> > > alive
> > > > > > > till
> > > > > > > > > the
> > > > > > > > > > > lease
> > > > > > > > > > > > > > > > expiration interval. Because
> > > > > > > registration.lease.timeout.ms,
> > > > > > > > > is
> > > > > > > > > > > > > > > configured
> > > > > > > > > > > > > > > > on the controller, the new active controller
> will
> > > > > extend
> > > > > > > all
> > > > > > > > > the
> > > > > > > > > > > > > leases
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > registration.lease.timeout.ms. I see that it
> > > won't
> > > > > need
> > > > > > > last
> > > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Agreed.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Sat, Sep 5, 2020 at 1:28 AM Colin McCabe <
> > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Colin wrote:
> > > > > > > > > > > > > > > > > > > The reason for including
> LeaseStartTimeMs
> > > in
> > > > > the
> > > > > > > > > request
> > > > > > > > > > > is to
> > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > > that the time required to communicate
> with
> > > the
> > > > > > > > > controller
> > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > included in
> > > > > > > > > > > > > > > > > > > the lease time. Since requests can
> > > > > potentially be
> > > > > > > > > delayed
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > > for a long time, this is important.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020, at 05:58, Unmesh
> Joshi
> > > wrote:
> > > > > > > > > > > > > > > > > > The network time will be added anyway,
> > > because
> > > > > the
> > > > > > > lease
> > > > > > > > > > > timer
> > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > active controller will start only after
> the
> > > > > heartbeat
> > > > > > > > > request
> > > > > > > > > > > > > > > reaches the
> > > > > > > > > > > > > > > > > > server.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > If the start time is not specified in the
> > > request,
> > > > > > > then the
> > > > > > > > > > > network
> > > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > > is excluded from the heartbeat time.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Here's an example:
> > > > > > > > > > > > > > > > > Let's say broker A sends a heartbeat at
> time
> > > 100,
> > > > > and
> > > > > > > it
> > > > > > > > > > > arrives
> > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > controller at time 200, and the lease
> duration
> > > is
> > > > > 1000.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The controller looks at the start time in
> the
> > > > > request,
> > > > > > > > > which is
> > > > > > > > > > > > > 100,
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > adds 1000 to it, getting 1100. On the
> other
> > > hand,
> > > > > if
> > > > > > > start
> > > > > > > > > > > time
> > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > specified in the request, then the
> expiration
> > > will
> > > > > be
> > > > > > > at
> > > > > > > > > time
> > > > > > > > > > > 1200.
> > > > > > > > > > > > > > > > > That is what I mean by "the network time is
> > > > > included
> > > > > > > in the
> > > > > > > > > > > > > expiration
> > > > > > > > > > > > > > > > > time."
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > And I think, some assumption about
> network
> > > round
> > > > > trip
> > > > > > > > > time is
> > > > > > > > > > > > > > > > > > needed anyway to decide on the frequency
> of
> > > the
> > > > > > > > > heartbeat (
> > > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms), and
> > > lease
> > > > > > > timeout (
> > > > > > > > > > > > > > > > > > registration.lease.timeout.ms). So I
> think
> > > just
> > > > > > > having a
> > > > > > > > > > > > > leaseTTL
> > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > request is easier to understand and
> > > implement.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It's important to keep in mind that
> messages
> > > may be
> > > > > > > > > delayed in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > > happens, we
> > > > > > > > > will
> > > > > > > > > > > use
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > > time specified in the request to determine
> if
> > > the
> > > > > > > request
> > > > > > > > > is
> > > > > > > > > > > stale.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> the
> > > > > > > controller
> > > > > > > > > side
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > reset in the case of controller
> failover.
> > > The
> > > > > > > > > alternative
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > be to
> > > > > > > > > > > > > > > > > > > track the lease as hard state rather
> than
> > > soft
> > > > > > > state,
> > > > > > > > > but I
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > is not really needed, and would result
> in
> > > more
> > > > > log
> > > > > > > > > entries.
> > > > > > > > > > > > > > > > > > My interpretation of the mention of
> > > BrokerRecord
> > > > > in
> > > > > > > the
> > > > > > > > > KIP
> > > > > > > > > > > was
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > record exists in the Raft log.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > BrokerRecord does exist in the Raft log,
> but
> > > does
> > > > > not
> > > > > > > > > include
> > > > > > > > > > > the
> > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > heartbeat time.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > By soft state, do you mean the broker
> > > > > > > > > > > > > > > > > > records exist only on the active leader
> and
> > > will
> > > > > not
> > > > > > > be
> > > > > > > > > > > > > replicated
> > > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > > raft log? If the live brokers list is
> > > maintained
> > > > > > > only on
> > > > > > > > > the
> > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > controller (raft leader), then, in case
> of
> > > leader
> > > > > > > > > failure,
> > > > > > > > > > > there
> > > > > > > > > > > > > > > will be
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > window where the new leader does not know
> > > about
> > > > > the
> > > > > > > live
> > > > > > > > > > > brokers,
> > > > > > > > > > > > > > > till
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > brokers establish the leases again.
> > > > > > > > > > > > > > > > > > I think it will be safer to have leases
> as a
> > > hard
> > > > > > > state
> > > > > > > > > > > managed
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > standard
> > > > > > > > > > > > > > > > > > Raft replication.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Leases are short, so the need to
> re-establish
> > > them
> > > > > > > after a
> > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > failover doesn't seem like a big problem.
> But
> > > > > this is
> > > > > > > > > > > something
> > > > > > > > > > > > > we can
> > > > > > > > > > > > > > > > > tweak if it becomes an issue. One option
> > > would be
> > > > > to
> > > > > > > have
> > > > > > > > > a
> > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > which is only used by the controller nodes
> for
> > > this
> > > > > > > (since,
> > > > > > > > > > > after
> > > > > > > > > > > > > all,
> > > > > > > > > > > > > > > > > brokers don't care about registration
> > > renewals).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Or am I misunderstanding something? (I
> assume
> > > > > that
> > > > > > > with
> > > > > > > > > soft
> > > > > > > > > > > > > state,
> > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > mean something like zookeeper local
> sessions
> > > > > > > > > > > > > > > > > >
> > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1147
> > > > > > > .)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > think
> > > > > it
> > > > > > > makes
> > > > > > > > > > > sense
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> becomes
> > > > > very
> > > > > > > messy.
> > > > > > > > > > > I'm
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > understand your question about
> duplicate
> > > > > broker ID
> > > > > > > > > > > detection,
> > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > There's a section in the KIP about this
> -- is
> > > > > there a
> > > > > > > > > detail
> > > > > > > > > > > we
> > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This is an implementation detail that
> doesn't
> > > need
> > > > > to
> > > > > > > be
> > > > > > > > > in the
> > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I assumed broker leases are implemented
> as a
> > > hard
> > > > > > > state.
> > > > > > > > > In
> > > > > > > > > > > that
> > > > > > > > > > > > > > > case, to
> > > > > > > > > > > > > > > > > > check for broker id conflict, we need to
> > > check
> > > > > the
> > > > > > > broker
> > > > > > > > > > > ids at
> > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > places
> > > > > > > > > > > > > > > > > > 1. Pending broker registrations (which
> are
> > > yet
> > > > > to be
> > > > > > > > > > > committed)
> > > > > > > > > > > > > 2.
> > > > > > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > > > > committed broker registrations.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020 at 5:42 PM Colin
> McCabe
> > > <
> > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020, at 01:12, Unmesh
> > > Joshi
> > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >>>Can you repeat your questions
> about
> > > broker
> > > > > > > leases?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >>>>The LeaseStartTimeMs is expected
> to
> > > be
> > > > > the
> > > > > > > > > broker's
> > > > > > > > > > > > > > > > > > > > 'System.currentTimeMillis()' at the
> > > point of
> > > > > the
> > > > > > > > > > > request. The
> > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > controller will add its lease period
> to
> > > this
> > > > > in
> > > > > > > order
> > > > > > > > > > > >>>>to
> > > > > > > > > > > > > > > compute
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > LeaseEndTimeMs.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think the use of LeaseStartTimeMs
> and
> > > > > > > > > LeaseEndTimeMs
> > > > > > > > > > > in the
> > > > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > confusing. Monotonic Clock
> > > > > (System.nanoTime) on
> > > > > > > the
> > > > > > > > > > > active
> > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > should be used to track leases.
> > > > > > > > > > > > > > > > > > > > (For example,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://issues.apache.org/jira/browse/ZOOKEEPER-1616https://github.com/etcd-io/etcd/pull/6888/commits/e7f4010ccaf28b6ce64fe514d25a4b2fa459d114
> > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Then we will not need
> LeaseStartTimeMs?
> > > > > > > > > > > > > > > > > > > > Instead of LeaseStartTimeMs, can we
> call
> > > it
> > > > > > > > > LeaseTTL? The
> > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > can then calculate LeaseEndTime =
> > > > > > > System.nanoTime() +
> > > > > > > > > > > > > LeaseTTL.
> > > > > > > > > > > > > > > > > > > > In this case we might just drop
> > > > > LeaseEndTimeMs
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > > > > > response, as
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > broker already knows about the TTL
> and
> > > can
> > > > > send
> > > > > > > > > > > heartbeats at
> > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > fraction
> > > > > > > > > > > > > > > > > > > > of TTL, say every TTL/4
> > > milliseconds.(elapsed
> > > > > > > time
> > > > > > > > > on the
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > measured
> > > > > > > > > > > > > > > > > > > > by System.nanoTime)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I agree that the monotonic clock is
> > > probably a
> > > > > > > better
> > > > > > > > > idea
> > > > > > > > > > > > > here.
> > > > > > > > > > > > > > > It is
> > > > > > > > > > > > > > > > > > > good to be robust against wall clock
> > > changes,
> > > > > > > although
> > > > > > > > > I
> > > > > > > > > > > think
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > > > which had them might suffer other
> issues.
> > > I
> > > > > will
> > > > > > > > > change
> > > > > > > > > > > it to
> > > > > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > monotonic clock.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The reason for including
> LeaseStartTimeMs
> > > in
> > > > > the
> > > > > > > > > request
> > > > > > > > > > > is to
> > > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > the time required to communicate with
> the
> > > > > > > controller
> > > > > > > > > gets
> > > > > > > > > > > > > included
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > lease time. Since requests can
> > > potentially be
> > > > > > > delayed
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > > > > long time, this is important.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I have a prototype built to
> demonstrate
> > > this
> > > > > as
> > > > > > > > > > > following:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The Kip631Controller itself depends
> on a
> > > > > > > Consensus
> > > > > > > > > > > module, to
> > > > > > > > > > > > > > > > > demonstrate
> > > > > > > > > > > > > > > > > > > > how possible interactions with the
> > > consensus
> > > > > > > module
> > > > > > > > > will
> > > > > > > > > > > look
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > (The Consensus can be pluggable
> really,
> > > > > with an
> > > > > > > API
> > > > > > > > > to
> > > > > > > > > > > allow
> > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > replicated log upto HighWaterMark)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > It has an implementation of
> LeaseTracker
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/LeaderLeaseTracker.scala
> > > > > > > > > > > > > > > > > > > > to demonstrate LeaseTracker's
> interaction
> > > > > with
> > > > > > > the
> > > > > > > > > > > consensus
> > > > > > > > > > > > > > > module.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The implementation has the following
> > > aspects:
> > > > > > > > > > > > > > > > > > > > 1. The lease tracking happens only
> on the
> > > > > active
> > > > > > > > > > > controller
> > > > > > > > > > > > > (raft
> > > > > > > > > > > > > > > > > > > > leader)
> > > > > > > > > > > > > > > > > > > > 2. Once the lease expires, it needs
> to
> > > > > propose
> > > > > > > and
> > > > > > > > > > > commit a
> > > > > > > > > > > > > > > > > FenceBroker
> > > > > > > > > > > > > > > > > > > > record for that lease.
> > > > > > > > > > > > > > > > > > > > 3. In case of active controller
> failure,
> > > the
> > > > > > > lease
> > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > > > tracked by
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > newly raft leader. The new raft
> leader
> > > > > starts the
> > > > > > > > > lease
> > > > > > > > > > > timer
> > > > > > > > > > > > > > > again,
> > > > > > > > > > > > > > > > > (as
> > > > > > > > > > > > > > > > > > > > implemented in onBecomingLeader
> method of
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > > in effect extending the lease by the
> time
> > > > > spent
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > election
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > whatever time was elapsed on the old
> > > leader.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on
> the
> > > > > > > controller
> > > > > > > > > side
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > reset
> > > > > > > > > > > > > > > > > > > in the case of controller failover.
> The
> > > > > > > alternative
> > > > > > > > > would
> > > > > > > > > > > be
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > track
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > lease as hard state rather than soft
> state,
> > > > > but I
> > > > > > > think
> > > > > > > > > > > that
> > > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > > > > needed, and would result in more log
> > > entries.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > There are working tests for this
> > > > > implementation
> > > > > > > here.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > > and an end to end test here
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/ProducerConsumerKIP500Test.scala
> > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >>'m not sure what you mean by
> > > > > "de-duplication
> > > > > > > of the
> > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > Apologies for using the confusing
> term
> > > > > > > > > deduplication. I
> > > > > > > > > > > meant
> > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > id
> > > > > > > > > > > > > > > > > > > > conflict.
> > > > > > > > > > > > > > > > > > > > As you can see in the prototype
> > > > > handleRequest of
> > > > > > > > > > > > > KIP631Controller
> > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > > the duplicate broker id needs to be
> > > detected
> > > > > > > before
> > > > > > > > > the
> > > > > > > > > > > > > > > BrokerRecord
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > submitted to the raft module.
> > > > > > > > > > > > > > > > > > > > Also as implemented in the
> prototype, the
> > > > > > > > > > > KIP631Controller is
> > > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > > > > threaded, handling requests one at a
> > > time.
> > > > > (an
> > > > > > > > > example of
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://martinfowler.com/articles/patterns-of-distributed-systems/singular-update-queue.html
> > > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > > think
> > > > > it
> > > > > > > makes
> > > > > > > > > > > sense
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > controller, since otherwise locking
> becomes
> > > > > very
> > > > > > > messy.
> > > > > > > > > > > I'm
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > > understand your question about
> duplicate
> > > > > broker ID
> > > > > > > > > > > detection,
> > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > There's a section in the KIP about
> this --
> > > is
> > > > > > > there a
> > > > > > > > > > > detail we
> > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 10:49 AM
> Colin
> > > > > McCabe <
> > > > > > > > > > > > > colin@cmccabe.xyz
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2020, at 19:36,
> Unmesh
> > > > > Joshi
> > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > There were a few of questions I
> had..
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for the response.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. Were my comments on the broker
> > > lease
> > > > > > > > > > > implementation
> > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > prototype) appropriate and do we
> > > need to
> > > > > > > change
> > > > > > > > > the
> > > > > > > > > > > KIP
> > > > > > > > > > > > > > > > > > > > > > description accordingly?.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Can you repeat your questions about
> > > broker
> > > > > > > leases?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2. How will broker epochs be
> > > generated?
> > > > > I am
> > > > > > > > > > > assuming it
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > committed log offset (like zxid?)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > There isn't any need to use a log
> > > offset.
> > > > > We
> > > > > > > can
> > > > > > > > > just
> > > > > > > > > > > > > look at
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > in-memory hash table and see what
> the
> > > > > latest
> > > > > > > > > number is,
> > > > > > > > > > > > > and add
> > > > > > > > > > > > > > > > > one, to
> > > > > > > > > > > > > > > > > > > > > generate a new broker epoch.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3. How will producer registration
> > > > > happen? I
> > > > > > > am
> > > > > > > > > > > assuming
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > should be
> > > > > > > > > > > > > > > > > > > > > > similar to broker registration,
> with
> > > a
> > > > > > > similar
> > > > > > > > > way to
> > > > > > > > > > > > > > > generate
> > > > > > > > > > > > > > > > > > > producer
> > > > > > > > > > > > > > > > > > > > > id.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > For the EOS stuff, we will need a
> few
> > > new
> > > > > RPCs
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > > > controller.
> > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > we should do that in a follow-on
> KIP,
> > > > > though,
> > > > > > > since
> > > > > > > > > > > this
> > > > > > > > > > > > > one is
> > > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > > > pretty big.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 4. Because we expose Raft log to
> all
> > > the
> > > > > > > > > brokers, any
> > > > > > > > > > > > > > > > > de-duplication
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > broker needs to happen before the
> > > > > requests
> > > > > > > are
> > > > > > > > > > > proposed
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > Raft.
> > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > the controller needs to be single
> > > > > threaded,
> > > > > > > and
> > > > > > > > > > > should do
> > > > > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > > > against the in-process or pending
> > > > > requests
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > > final
> > > > > > > > > > > > > > > state. I
> > > > > > > > > > > > > > > > > > > read a
> > > > > > > > > > > > > > > > > > > > > > mention of this, in the
> responses in
> > > this
> > > > > > > > > > > thread.Will it
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > useful to
> > > > > > > > > > > > > > > > > > > > > > mention this in the KIP?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I'm not sure what you mean by
> > > > > "de-duplication
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > broker."
> > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 4:50 AM
> Colin
> > > > > McCabe
> > > > > > > <
> > > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I'm thinking of calling a vote
> on
> > > > > KIP-631
> > > > > > > on
> > > > > > > > > > > Monday.
> > > > > > > > > > > > > Let
> > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > know
> > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > there's any more comments I
> should
> > > > > address
> > > > > > > > > before I
> > > > > > > > > > > > > start
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > vote.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020, at 05:39,
> > > Unmesh
> > > > > > > Joshi
> > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >>Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > >>Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > Thanks. I will be adding
> more to
> > > the
> > > > > > > > > prototype
> > > > > > > > > > > and
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > > > and collaborate.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020 at
> 12:28 AM
> > > > > Colin
> > > > > > > > > McCabe <
> > > > > > > > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Jose,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > That'a s good point that I
> > > hadn't
> > > > > > > > > considered.
> > > > > > > > > > > It's
> > > > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > a separate leader change
> > > message,
> > > > > as
> > > > > > > you
> > > > > > > > > > > mentioned.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2020, at
> 11:56,
> > > Jose
> > > > > > > Garcia
> > > > > > > > > > > Sancio
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Very cool prototype!
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > The KIP proposes a record
> > > called
> > > > > > > > > IsrChange
> > > > > > > > > > > which
> > > > > > > > > > > > > > > > > includes the
> > > > > > > > > > > > > > > > > > > > > > > > > > partition, topic, isr,
> > > leader and
> > > > > > > leader
> > > > > > > > > > > epoch.
> > > > > > > > > > > > > > > During
> > > > > > > > > > > > > > > > > normal
> > > > > > > > > > > > > > > > > > > > > > > > > > operation ISR changes do
> not
> > > > > result
> > > > > > > in
> > > > > > > > > leader
> > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > > Similarly,
> > > > > > > > > > > > > > > > > > > > > > > > > > leader changes do not
> > > necessarily
> > > > > > > > > involve ISR
> > > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > implementation that uses
> ZK
> > > > > modeled
> > > > > > > them
> > > > > > > > > > > together
> > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. All of this
> information is
> > > > > stored
> > > > > > > in
> > > > > > > > > one
> > > > > > > > > > > > > znode.
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. ZK's optimistic lock
> > > requires
> > > > > > > that you
> > > > > > > > > > > specify
> > > > > > > > > > > > > > > the new
> > > > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > > > > > > > 3. The change to that
> znode
> > > was
> > > > > being
> > > > > > > > > > > performed
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > > and the leader.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > None of these reasons are
> > > true in
> > > > > > > > > KIP-500.
> > > > > > > > > > > Have
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > considered
> > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > > two different records?
> For
> > > > > example
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. IsrChange record which
> > > > > includes
> > > > > > > topic,
> > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > isr
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. LeaderChange record
> which
> > > > > includes
> > > > > > > > > topic,
> > > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > leader epoch.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I suspect that making
> this
> > > change
> > > > > > > will
> > > > > > > > > also
> > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > message AlterIsrRequest
> > > > > introduced in
> > > > > > > > > > > KIP-497:
> > > > > > > > > > > > > Add
> > > > > > > > > > > > > > > > > > > inter-broker
> > > > > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > alter ISR.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > > > > -Jose
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Colin McCabe <cm...@apache.org>.
On Tue, Oct 6, 2020, at 16:09, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the reply. Made another pass of the KIP. A few more comments
> below.
>
Hi Jun,
Thanks for the review.
> 55. We discussed earlier why the current behavior where we favor the
> current broker registration is better. Have you given this more thought?
>
Yes, I think we should favor the current broker registration, as you suggested earlier.
> 80. Config related.
> 80.1 Currently, each broker only has the following 3 required configs. It
> will be useful to document the required configs post KIP-500 (in both the
> dedicated and shared controller mode).
> broker.id
> log.dirs
> zookeeper.connect
For the broker, these configs will be required:
broker.id
log.dirs
process.roles
controller.listener.names
controller.connect
For the controller, these configs will be required:
controller.id
log.dirs
process.roles
controller.listener.names
controller.connect
For broker+controller, it will be the union of these two, which essentially means we need both broker.id and controller.id, but all others are the same as standalone.
> 80.2 It would be useful to document all deprecated configs post KIP-500.
> For example, all zookeeper.* are obviously deprecated. But there could be
> others. For example, since we don't plan to support auto broker id
> generation, it seems broker.id.generation.enable is deprecated too.
> 80.3 Could we make it clear that controller.connect replaces quorum.voters
> in KIP-595?
OK. I added a comment about this in the table.
> 80.4 Could we document that broker.id is now optional?
OK. I have added a line for broker.id.
> 80.5 The KIP suggests that controller.id is optional on the controller
> node. I am concerned that this can cause a bit of confusion in 2 aspects.
> First, in the dedicated controller mode, controller.id is not optional
> (since broker.id is now optional). Second, in the shared controller mode,
> it may not be easy for the user to figure out the default value of
> controller.id to set controller.connect properly.
I got rid of the default value of controller.id. I agree it was a bit confusing to explain.
> 80.6 Regarding the consistency of config names, our metrics already include
> controller. So, prefixing all controller related configs with "controller"
> may be more consistent. If we choose to do that, could we rename all new
> configs here and in KIP-595 consistently?
>
I added the new names for the KIP-595 configurations to the configuration table. I think prefixing these configurations with "controller.quorum" is the best option since it makes it clear that they are related to the controller quorum.
> 81. Metrics
> 81.1 kafka.controller:type=KafkaController,name=MetadataSnapshotLag: This
> is now redundant since KIP-630 already has
> kafka.controller:type=KafkaController,name=SnapshotLag.
Thanks for bringing up the metric name issue. I think it would be good to clarify that this is the lag for the metadata snapshot specifically. So I prefer the name MetadataSnapshotLag. I'll add a note to the table.
> 81.2 Do we need both kafka.controller:type=KafkaServer,name=MetadataLag and
> kafka.controller:type=KafkaController,name=MetadataLag since in the shared
> controller mode, the metadata log is shared?
Yes, both are needed. This metric is about what metadata logs we have applied, not about the Raft log's high water mark. If the broker or controller takes a while to apply the log messages (for example, because of lock contention or some other issue), they could fall significantly behind the high watermark.
>
> 82. Metadata records
> 82.1 BrokerRecord: It needs to include supported features based on KIP-584.
Added.
> 82.2 FeatureLevel: The finalized feature in KIP-584 has the following
> structure and needs to be reflected here.
> {
> "version": 0, // int32 -> Represents the version of the schema for the
> data stored in the ZK node
> "status": 1, // int32 -> Represents the status of the node
> "features": {
> "group_coordinator": { // string -> name of the feature
> "min_version_level": 0, // int16 -> Represents the cluster-wide
> finalized minimum version level (>=1) of this feature
> "max_version_level": 3 // int16 -> Represents the cluster-wide
> finalized maximum version level (>=1 and >= min_version_level) of this
> feature
> },
> "consumer_offsets_topic_schema": {
> "min_version_level": 0,
> "max_version_level": 4
> }
> }
Fair point. I will add both a minimum and maximum finalized feature level.
>
> 83. RPC
> 83.1 BrokerHeartbeatRequest: Could you document when the controller assigns
> a new broker epoch?
The controller assigns a broker epoch if the heartbeat request was sent without an epoch (that is, it is in the INITIAL state), and the controller wants to add the new broker to the cluster. I added a note to the state machine.
> 83.2 BrokerHeartbeatResponse: Do we need a new error code to indicate a
> broker is fenced?
I don't think we need a new error code, since targetState will just indicate that the broker should be fenced.
> 83.3 What error/state indicates an unsuccessful controlled shutdown in
> BrokerHeartbeatResponse? Currently, the controlled shutdown responses also
> include a list of remaining partitions. That's mostly for debugging
> purposes. Should we include it in BrokerHeartbeatResponse too?
Once the controller wants the broker to shut down, it will set targetState to SHUTDOWN.
We don't need to send a list of remaining partitions, since the broker is following the metadata log, and will get all its instructions from there.
> 83.4 Could we document when Listeners, Features and Rack are expected to be
> set in BrokerHeartbeatRequest?
I added a line specifying that: The listeners, features, and rack fields only need to be set when the broker is in INITIAL state, and is requesting a new broker epoch.
> 83.5 Currently, the metadata response only returns a list of brokers.
> Should we have a separate field for the controller info or change the field
> name to sth like nodes? Also, is the rack field relevant to the controller?
As per KIP-590, we don't want the clients to communicate directly with the controller. So we weren't planning on including the controller's information in the MetadataResponse.
We don't currently have rack-awareness for controllers, but I suppose we could in the future. I think we can leave that for later.
>
> 84. There are a few references to __kafka_metadata. However, KIP-595
> uses __cluster_metadata.
We agreed earlier to decide on the name of the metadata topic in this KIP, and to treat the KIP-595 name as provisional. Looking at it again, I think we should go with @metadata, in order to avoid colliding with topic names that already exist.
>
> 85. "When a broker is fenced, it cannot process any client requests. "
> Should we send a new error code in the response to indicate that the broker
> is fenced?
I was hoping that we could avoid responding to requests when the broker was fenced. I suppose a new error code could work too, but then we also need a compatibility story for older clients.
>
> 86. "The controller can generate a new broker epoch by using the latest log
> offset." Perhaps it's more accurate to say the new broker epoch is the
> offset for the corresponding BrokerRecord in the metadata log.
The idea here is slightly different. We want to use the current latest offset (prior to creating the BrokerRecord) as the new epoch. We can't use the offset of the BrokerRecord itself since we won't know what that is until we give the BrokerRecord to the Raft layer.
>
> 87. meta.properties:
> 87.1 Could you document the format change to this file? Also, should we
> bump up the version of the file?
Hmm. The file is just a list of key=value pairs, one per line. So I'm not sure there is any format change per se, at least technically.
Maybe we should bump up the version anyway, in order to prevent older pre-KIP-500 software from launching if the new stuff is present?
> 87.2 How do we know a node is configured in KIP-500 mode?
>
A cluster configured in KIP-500 mode will contain the following line in the meta.properties file in each directory: kip.500.mode=enabled
> 88. CurMetadataOffset: Should we use nextMetadataOffset since it's more
> consistent with fetchOffset and HWM?
I'm not sure. Drawing an analogy with HWM might be confusing, since CurMetadataOffset is not HWM. It's totally possible for the broker to have fetched up to offset 1000 but only applied up to offset 500.
best,
Colin
>
> Thanks,
>
> Jun
>
>
> On Mon, Oct 5, 2020 at 9:03 AM Colin McCabe <cm...@apache.org> wrote:
>
> > On Mon, Sep 28, 2020, at 11:41, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply. A few more comments below.
> > >
> > > 62.
> > > 62.1 controller.listener.names: So, is this used for the controller or
> > the
> > > broker trying to connect to the controller?
> > >
> >
> > Hi Jun,
> >
> > It's used by both. The broker tries to connect to controllers by using
> > the first listener in this list. The controller uses the list to determine
> > which listeners it should bind to.
> >
> > >
> > > 62.2 If we want to take the approach to share the configs that are common
> > > between the broker and the controller, should we share the id too?
> > >
> >
> > On nodes where we are running both the controller and broker roles, we
> > need two IDs: one for the controller, and one for the broker. So we can't
> > share the same configuration.
> >
> > >
> > > 62.3 We added some configs in KIP-595 prefixed with "quorum" and we plan
> > to
> > > add some controller specific configs prefixed with "controller". KIP-630
> > > plans to add some other controller specific configs with no prefix.
> > Should
> > > we standardize all controller specific configs with the same prefix?
> > >
> >
> > That's a good question. As Jose said elsewhere in the thread, let's use
> > "metadata" as the prefix.
> >
> > >
> > > 70. Could you explain the impact of process.roles a bit more? For
> > example,
> > > if process.roles=controller, does the node still maintain metadata cache
> > as
> > > described in KIP-630? Do we still return the host/port for those nodes in
> > > the metadata response?
> > >
> >
> > No, the node does not need any broker data structures if it is not a
> > broker. Controllers also don't handle MetadataRequests. Clients will
> > still go to the brokers for those.
> >
> > best,
> > Colin
> >
> >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Sep 28, 2020 at 9:24 AM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Fri, Sep 25, 2020, at 17:35, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > 62. Thinking about this more, I am wondering what's our overall
> > strategy
> > > > > for configs shared between the broker and the controller. For
> > example,
> > > > both
> > > > > the broker and the controller have to define listeners. So both need
> > > > > configs like listeners/advertised.listeners. Both the new controller
> > and
> > > > > the broker replicate data. So both need to define some replication
> > > > related
> > > > > configurations (replica.fetch.min.bytes, replica.fetch.wait.max.ms,
> > > > etc).
> > > > > Should we let the new controller share those configs with brokers or
> > > > should
> > > > > we define new configs just for the controller? It seems that we want
> > to
> > > > > apply the same strategy consistently for all shared configs?
> > > > >
> > > >
> > > > Hi Jun,
> > > >
> > > > That's a good question. I think that we should share as many
> > > > configurations as possible. There will be a few cases where this isn't
> > > > possible, and we need to create a new configuration key that is
> > > > controller-only, but I think that will be relatively rare.
> > > >
> > > > >
> > > > > 63. If a node has process.roles set to controller, does one still
> > need to
> > > > > set broker.id on this node?
> > > > >
> > > >
> > > > No, broker.id does not need to be set in that case.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Sep 25, 2020 at 2:17 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > On Fri, Sep 25, 2020, at 10:17, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the reply.
> > > > > > >
> > > > > > > 60. Yes, I think you are right. We probably need the controller
> > id
> > > > when a
> > > > > > > broker starts up. A broker only stores the Raft leader id in the
> > > > metadata
> > > > > > > file. To do the initial fetch to the Raft leader, it needs to
> > know
> > > > the
> > > > > > > host/port associated with the leader id.
> > > > > > >
> > > > > > > 62. It seems there are 2 parts to this : (1) which listener a
> > client
> > > > > > should
> > > > > > > use to initiate a connection to the controller and (2) which
> > listener
> > > > > > > should a controller use to accept client requests. For (1), at
> > any
> > > > point
> > > > > > of
> > > > > > > time, a client only needs to use one listener. I think
> > > > > > > controller.listener.name is meant for the client.
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > controller.listener.names is also used by the controllers. In the
> > case
> > > > > > where we have a broker and a controller in the same JVM, we have a
> > > > single
> > > > > > config file. Then we need to know which listeners belong to the
> > > > controller
> > > > > > and which belong to the broker component. That's why it's a list.
> > > > > >
> > > > > > > So, a single value seems
> > > > > > > to make more sense. Currently, we don't have a configuration for
> > > > (2). We
> > > > > > > could add a new one for that and support a list. I am wondering
> > how
> > > > > > useful
> > > > > > > it will be. One example that I can think of is that we can reject
> > > > > > > non-controller related requests if accepted on controller-only
> > > > listeners.
> > > > > > > However, we already support separate authentication for the
> > > > controller
> > > > > > > listener. So, not sure how useful it is.
> > > > > >
> > > > > > The controller always has a separate listener and does not share
> > > > listeners
> > > > > > with the broker. The main reason for this is to avoid giving
> > clients
> > > > the
> > > > > > ability to launch a denial-of-service attack on the controller by
> > > > flooding
> > > > > > a broker port. A lot of times, these attacks are made
> > unintentionally
> > > > by
> > > > > > poorly coded or configured clients. Additionally, broker ports can
> > > > also be
> > > > > > very busy with large fetch requests, and so on. It's just a bad
> > > > > > configuration in general to try to overload the same port for the
> > > > > > controller and broker, and we don't want to allow it. It would be
> > a
> > > > > > regression to go from the current system where control requests are
> > > > safely
> > > > > > isolated on a separate port, to one where they're not. It also
> > makes
> > > > the
> > > > > > code and configuration a lot messier.
> > > > > >
> > > > > > >
> > > > > > > 63. (a) I think most users won't know controller.id defaults to
> > > > > > broker.id +
> > > > > > > 3000. So, it can be confusing for them to set up
> > controller.connect.
> > > > If
> > > > > > > this is truly needed, it seems that it's less confusing to make
> > > > > > > controller.id required.
> > > > > > > (b) I am still trying to understand if we truly need to expose a
> > > > > > > controller.id. What if we only expose broker.id and let
> > > > > > controller.connect
> > > > > > > just use broker.id? What will be missing?
> > > > > >
> > > > > > The controller has a separate ID from the broker, so knowing
> > broker.id
> > > > is
> > > > > > not helpful here.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Thu, Sep 24, 2020 at 10:55 PM Colin McCabe <
> > cmccabe@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > On Thu, Sep 24, 2020, at 16:24, Jun Rao wrote:
> > > > > > > > > Hi, Colin,
> > > > > > > > >
> > > > > > > > > Thanks for the reply and the updated KIP. A few more comments
> > > > below.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 53. It seems that you already incorporated the changes in
> > > > KIP-516.
> > > > > > With
> > > > > > > > > topic ids, we don't need to wait for the topic's data to be
> > > > deleted
> > > > > > > > before
> > > > > > > > > removing the topic metadata. If the topic is recreated, we
> > can
> > > > still
> > > > > > > > delete
> > > > > > > > > the data properly based on the topic id. So, it seems that
> > we can
> > > > > > remove
> > > > > > > > > TopicRecord.Deleting.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Thanks for the reply. What I was thinking of doing here was
> > using
> > > > > > topic
> > > > > > > > IDs internally, but still using names externally. So the topic
> > > > UUIDs
> > > > > > are
> > > > > > > > only for the purpose of associating topics with partitions --
> > from
> > > > the
> > > > > > > > user's point of view, topics are still identified by names.
> > > > > > > >
> > > > > > > > You're right that KIP-516 will simplify things, but I'm not
> > sure
> > > > when
> > > > > > that
> > > > > > > > will land, so I wanted to avoid blocking the initial
> > > > implementation of
> > > > > > this
> > > > > > > > KIP on it.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 55. It seems to me that the current behavior where we favor
> > the
> > > > > > current
> > > > > > > > > broker registration is better. This is because uncontrolled
> > > > broker
> > > > > > > > shutdown
> > > > > > > > > is rare and its impact is less severe since one just needs to
> > > > wait
> > > > > > for
> > > > > > > > the
> > > > > > > > > session timeout before restarting the broker. If a
> > mis-configured
> > > > > > broker
> > > > > > > > > replaces an existing broker, the consequence is more severe
> > > > since it
> > > > > > can
> > > > > > > > > cause the leader to be unavailable, a replica to be out of
> > ISR,
> > > > and
> > > > > > add
> > > > > > > > > more load on the leaders etc.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hmm, that's a good point. Let me check this a bit more before
> > I
> > > > change
> > > > > > > > it, though.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 60. controller.connect=0@controller0.example.com:9093,
> > > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com :
> > Do we
> > > > > > need to
> > > > > > > > > include the controller id before @? It seems that the
> > host/port
> > > > is
> > > > > > enough
> > > > > > > > > for establishing the connection. It would also be useful to
> > make
> > > > it
> > > > > > clear
> > > > > > > > > that controller.connect replaces quorum.voters in KIP-595.
> > > > > > > > >
> > > > > > > >
> > > > > > > > I discussed this with Jason earlier, and he felt that the
> > > > controller
> > > > > > IDs
> > > > > > > > were needed in this configuration key. It is certainly needed
> > when
> > > > > > > > configuring the controllers themselves, since they need to know
> > > > each
> > > > > > > > others' IDs.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 61. I am not sure that we need both controller.listeners and
> > > > > > > > > controller.connect.security.protocol since the former
> > implies the
> > > > > > > > security
> > > > > > > > > protocol. The reason that we have both
> > > > inter.broker.listener.name
> > > > > > and
> > > > > > > > > inter.broker.security.protocol is because we had the latter
> > > > first and
> > > > > > > > later
> > > > > > > > > realized that the former is more general.
> > > > > > > > >
> > > > > > > >
> > > > > > > > That's a good point. I've removed this from the KIP.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 62. I am still not sure that you need controller.listeners
> > to be
> > > > a
> > > > > > list.
> > > > > > > > > All listeners are already defined in listeners. To migrate
> > from
> > > > > > plaintext
> > > > > > > > > to SSL, one can configure listeners to have both plaintext
> > and
> > > > SSL.
> > > > > > After
> > > > > > > > > that, one can just change controller.listeners from
> > plaintext to
> > > > SSL.
> > > > > > > > This
> > > > > > > > > is similar to how to change the listener for inter broker
> > > > > > connections.
> > > > > > > > > Also, controller.listener.name may be a more accurate name?
> > > > > > > > >
> > > > > > > >
> > > > > > > > The issue that I see here is that if you are running with the
> > > > > > controller
> > > > > > > > and broker in the same JVM, if you define a few listeners in
> > > > > > "listeners"
> > > > > > > > they will get used as regular broker listeners, unless you put
> > > > them in
> > > > > > > > controller.listeners. Therefore, controller.listeners needs
> > to be
> > > > a
> > > > > > list.
> > > > > > > >
> > > > > > > > controller.listener.names does sound like a better name,
> > though...
> > > > I've
> > > > > > > > updated it to that.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 63. Regarding controller.id, I am trying to understand
> > whether
> > > > it's
> > > > > > a
> > > > > > > > > required configuration or an optional one. From the KIP, it
> > > > sounds
> > > > > > like
> > > > > > > > > controller.id is optional. Then, if this is unset, it's not
> > > > clear
> > > > > > how
> > > > > > > > the
> > > > > > > > > user will obtain the controller id for setting
> > > > controller.connect.
> > > > > > > > >
> > > > > > > >
> > > > > > > > If you specify broker.id but not controller.id, then
> > controller.id
> > > > > > will
> > > > > > > > just be set to broker.id + 3000. This was intended to make
> > some
> > > > > > > > configurations a little bit more concise to write. You still
> > do
> > > > have
> > > > > > to
> > > > > > > > know the controller IDs when configuring the brokers, though.
> > If
> > > > this
> > > > > > > > seems confusing then I can just make it mandatory.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 64. KIP-516 adds a flag in LeaderAndIsrRequest to indicate
> > > > whether it
> > > > > > > > > includes all partitions or not. This will be used to clean up
> > > > strayed
> > > > > > > > logs.
> > > > > > > > > I was thinking how we will do the same thing once the
> > controller
> > > > > > moves to
> > > > > > > > > Raft based. One way to do that is on broker startup, it gets
> > the
> > > > HWM
> > > > > > in
> > > > > > > > the
> > > > > > > > > metadata log from the Raft leader and waits until its
> > metadata
> > > > > > catches up
> > > > > > > > > to HWM. At that point, any local log not seen in the metadata
> > > > can be
> > > > > > > > > removed. Since the Fetch response returns the HWM, there
> > seems
> > > > to be
> > > > > > > > enough
> > > > > > > > > APIs to achieve this.
> > > > > > > > >
> > > > > > > >
> > > > > > > > That's a very good point. I added a note about this under
> > Broker
> > > > > > Startup.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Sep 24, 2020 at 1:07 PM Colin McCabe <
> > colin@cmccabe.xyz>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Mon, Sep 21, 2020, at 18:13, Jun Rao wrote:
> > > > > > > > > > > Hi, Colin,
> > > > > > > > > > >
> > > > > > > > > > > Sorry for the late reply. A few more comments below.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi Jun,
> > > > > > > > > >
> > > > > > > > > > Thanks for taking another look.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 50. Configurations
> > > > > > > > > > > 50.1 controller.listeners: It seems that a controller
> > just
> > > > needs
> > > > > > one
> > > > > > > > > > > listener. Why do we need to have a list here? Also,
> > could you
> > > > > > > > provide an
> > > > > > > > > > > example of how this is set and what's its relationship
> > with
> > > > > > existing
> > > > > > > > > > > configs such as "security.inter.broker.protocol" and "
> > > > > > > > > > > inter.broker.listener.name"?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I agree that most administrators will want to run with
> > only one
> > > > > > > > controller
> > > > > > > > > > listener. However, just as with brokers, it is nice to
> > have
> > > > the
> > > > > > > > option to
> > > > > > > > > > expose multiple ports.
> > > > > > > > > >
> > > > > > > > > > One reason why you might want multiple ports is if you were
> > > > doing a
> > > > > > > > > > migration from plaintext to SSL. You could add new SSL
> > > > listeners
> > > > > > but
> > > > > > > > > > continue to expose the PLAINTEXT listeners. Then you could
> > > > add the
> > > > > > > > new SSL
> > > > > > > > > > controller config to each broker and roll each broker.
> > Then
> > > > you
> > > > > > could
> > > > > > > > > > migrate from PLAINTEXT to SSL with no downtime.
> > > > > > > > > >
> > > > > > > > > > Here's an example configuration for the controller:
> > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com
> > > > > > > > > > controller.listeners=CONTROLLER
> > > > > > > > > > listeners=CONTROLLER://controller0.example.com:9093
> > > > > > > > > > listener.security.protocol.map=CONTROLLER:SSL
> > > > > > > > > >
> > > > > > > > > > Here's an example configuration for the broker:
> > > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com
> > > > > > > > > > controller.connect.security.protocol=SSL
> > > > > > > > > >
> > > > > > > > > > security.inter.broker.protocol or
> > inter.broker.listener.name
> > > > do
> > > > > > not
> > > > > > > > > > affect how the broker communicates with the controller.
> > Those
> > > > > > > > > > configurations relate to how brokers communicate with each
> > > > other,
> > > > > > but
> > > > > > > > the
> > > > > > > > > > controller is not a broker.
> > > > > > > > > >
> > > > > > > > > > (Note that I just added
> > controller.connect.security.protocol
> > > > to the
> > > > > > > > KIP --
> > > > > > > > > > I had forgotten to put it in earlier)
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 50.2 registration.heartbeat.interval.ms and
> > > > > > > > > > registration.lease.timeout.ms.
> > > > > > > > > > > Should we match their default value with the
> > corresponding
> > > > > > default
> > > > > > > > for
> > > > > > > > > > ZK?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Fair enough. I'll set them to the values of the
> > > > > > > > zookeeper.sync.time.ms
> > > > > > > > > > and zookeeper.connection.timeout.ms configurations. I do
> > > > think we
> > > > > > > > should
> > > > > > > > > > experiment later on to see what works well here, but the ZK
> > > > values
> > > > > > are
> > > > > > > > at
> > > > > > > > > > least a starting point.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 50.3 controller.connect: Could you provide an example? I
> > am
> > > > > > > > wondering how
> > > > > > > > > > > it differs from quorum.voters=1@kafka-1:9092, 2@kafka-2
> > > > :9092,
> > > > > > > > 3@kafka-3
> > > > > > > > > > > :9092.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > controller.connect is intended to be the new name for
> > > > > > quorum.voters.
> > > > > > > > > > During the vote for KIP-595, we sort of agreed to defer the
> > > > > > discussion
> > > > > > > > > > about what this configuration should be called. I proposed
> > > > this
> > > > > > new
> > > > > > > > name
> > > > > > > > > > because it makes it clear what the configuration is for
> > (how to
> > > > > > > > connect to
> > > > > > > > > > the controllers).
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 50.4 controller.id: I am still not sure how this is
> > being
> > > > used.
> > > > > > > > Could
> > > > > > > > > > you
> > > > > > > > > > > explain this in more detail?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The controller ID needs to be set on each controller. It
> > also
> > > > > > appears
> > > > > > > > in
> > > > > > > > > > controller.connect as the thing before the "at" sign. Its
> > > > > > function is
> > > > > > > > > > pretty similar to broker.id.
> > > > > > > > > >
> > > > > > > > > > Broker IDs and controller IDs exist in the same ID space,
> > so
> > > > you
> > > > > > can't
> > > > > > > > > > have a broker and a controller that use the same ID.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 51. BrokerHeartbeat: It seems a bit wasteful to include
> > > > > > Listeners in
> > > > > > > > > > every
> > > > > > > > > > > heartbeat request since it typically doesn't change.
> > Could we
> > > > > > make
> > > > > > > > that
> > > > > > > > > > an
> > > > > > > > > > > optional field?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Ok.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 52. KIP-584 adds a new ZK node /features. Should we add a
> > > > > > > > corresponding
> > > > > > > > > > > metadata record?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Good point. I added FeatureLevelRecord.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 53. TopicRecord and DeleteTopic: Both DeleteTopic and
> > > > > > > > > > TopicRecord.Deleting
> > > > > > > > > > > indicate topic deletion. Could we outline the flow when
> > each
> > > > > > will be
> > > > > > > > set?
> > > > > > > > > > > In particular, which one indicates the intention to
> > delete
> > > > and
> > > > > > which
> > > > > > > > one
> > > > > > > > > > > indicates the completion of the deletion.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > TopicRecord.Deleting is set when we intend to delete the
> > topic.
> > > > > > > > > >
> > > > > > > > > > DeleteTopic removes the topic completely. I will rename
> > > > > > DeleteTopic to
> > > > > > > > > > RemoveTopic to make this clearer.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 54. "The controller can generate a new broker epoch by
> > using
> > > > the
> > > > > > > > latest
> > > > > > > > > > log
> > > > > > > > > > > offset." Which offset is that? Is it the offset of the
> > > > metadata
> > > > > > > > topic for
> > > > > > > > > > > the corresponding BrokerRecord? Is it guaranteed to be
> > > > different
> > > > > > on
> > > > > > > > each
> > > > > > > > > > > broker restart?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Yes.... a new broker epoch implies that there has been a
> > new
> > > > record
> > > > > > > > > > created in the metadata log. Therefore the last committed
> > > > offset
> > > > > > must
> > > > > > > > be
> > > > > > > > > > different.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 55. "Thereafter, it may lose subsequent conflicts if its
> > > > broker
> > > > > > > > epoch is
> > > > > > > > > > > stale. (See KIP-380 for some background on broker
> > epoch.)
> > > > The
> > > > > > > > reason
> > > > > > > > > > for
> > > > > > > > > > > favoring new processes is to accommodate the common case
> > > > where a
> > > > > > > > process
> > > > > > > > > > is
> > > > > > > > > > > killed with kill -9 and then restarted. " Are you saying
> > > > that if
> > > > > > > > there is
> > > > > > > > > > > an active broker registered in the controller, a new
> > broker
> > > > > > heartbeat
> > > > > > > > > > > request with the INITIAL state will fence the current
> > broker
> > > > > > > > session? Not
> > > > > > > > > > > sure about this. For example, this will allow a broker
> > with
> > > > > > > > incorrectly
> > > > > > > > > > > configured broker id to kill an existing valid broker.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Yes, a new broker with an incorrectly configured broker id
> > > > could
> > > > > > fence
> > > > > > > > an
> > > > > > > > > > existing valid broker.
> > > > > > > > > >
> > > > > > > > > > The goal of the conflict resolution process described here
> > is
> > > > to
> > > > > > avoid
> > > > > > > > > > having two brokers go back and forth in claiming the
> > broker id.
> > > > > > > > Choosing
> > > > > > > > > > the newest was just an easy way to do that.
> > > > > > > > > >
> > > > > > > > > > Choosing the oldest is another possibility, and more inline
> > > > with
> > > > > > what
> > > > > > > > > > ZooKeeper does today. The problem with that is that it
> > would
> > > > not
> > > > > > be
> > > > > > > > > > possible to re-create the broker if the old instance died
> > > > suddenly,
> > > > > > > > except
> > > > > > > > > > by waiting for the full lease timeout.
> > > > > > > > > >
> > > > > > > > > > It might be possible to do a little better here by
> > breaking the
> > > > > > lease
> > > > > > > > if
> > > > > > > > > > the TCP connection from the broker drops. But that
> > requires
> > > > > > crossing
> > > > > > > > some
> > > > > > > > > > layers of abstraction in the Kafka networking stack.
> > There's
> > > > also
> > > > > > the
> > > > > > > > > > possibility of false negatives or positives. For example,
> > TCP
> > > > > > > > connections
> > > > > > > > > > sometimes just drop even if the node is still there. Or
> > > > sometimes
> > > > > > the
> > > > > > > > > > networking stack keeps a TCP connection alive for a while
> > when
> > > > the
> > > > > > > > other
> > > > > > > > > > end is gone.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 56. kafka-storage.sh:
> > > > > > > > > > > 56.1 In the info mode, what other information does it
> > show in
> > > > > > > > addition to
> > > > > > > > > > > "kip.500.mode=enabled"?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I think it should show a list of all the storage
> > directories,
> > > > and
> > > > > > > > whether
> > > > > > > > > > each one is formatted. In addition, it should show whether
> > > > > > > > kip.500.mode is
> > > > > > > > > > enabled, and what the cluster id is.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 56.2 Should the format mode take the config file as the
> > > > input too
> > > > > > > > like
> > > > > > > > > > the
> > > > > > > > > > > info mode?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Yes, that's a good idea.
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 17, 2020 at 4:12 PM Colin McCabe <
> > > > cmccabe@apache.org
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > >
> > > > > > > > > > > > That's a fair point. I have moved the lease duration
> > to
> > > > the
> > > > > > broker
> > > > > > > > > > > > heartbeat response. That way lease durations can be
> > > > changed
> > > > > > just
> > > > > > > > be
> > > > > > > > > > > > reconfiguring the controllers.
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Sep 16, 2020, at 07:40, Unmesh Joshi wrote:
> > > > > > > > > > > > > Thanks Colin, the changes look good to me. One small
> > > > thing.
> > > > > > > > > > > > > registration.lease.timeout.ms is the configuration
> > on
> > > > the
> > > > > > > > controller
> > > > > > > > > > > > side.
> > > > > > > > > > > > > It will be good to comment on how brokers know about
> > it,
> > > > to
> > > > > > be
> > > > > > > > able
> > > > > > > > > > to
> > > > > > > > > > > > > send LeaseDurationMs
> > > > > > > > > > > > > in the heartbeat request,
> > > > > > > > > > > > > or else it can be added in the heartbeat response for
> > > > > > brokers to
> > > > > > > > know
> > > > > > > > > > > > about
> > > > > > > > > > > > > it.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Sep 11, 2020 at 10:32 PM Colin McCabe <
> > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think you're right that we should use a duration
> > here
> > > > > > rather
> > > > > > > > > > than a
> > > > > > > > > > > > > > time. As you said, the clock on the controller
> > will
> > > > > > probably
> > > > > > > > not
> > > > > > > > > > > > match the
> > > > > > > > > > > > > > one on the broker. I have updated the KIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It's important to keep in mind that messages
> > may be
> > > > > > > > delayed in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > happens, we
> > > > > > > > will
> > > > > > > > > > use
> > > > > > > > > > > > the
> > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > time specified in the request to determine if
> > the
> > > > > > request
> > > > > > > > is
> > > > > > > > > > stale.
> > > > > > > > > > > > > > > I am assuming that there will be a single TCP
> > > > connection
> > > > > > > > > > maintained
> > > > > > > > > > > > > > between
> > > > > > > > > > > > > > > broker and active controller. So, there won't be
> > any
> > > > out
> > > > > > of
> > > > > > > > order
> > > > > > > > > > > > > > requests?
> > > > > > > > > > > > > > > There will be a scenario of broker GC pause,
> > which
> > > > might
> > > > > > > > cause
> > > > > > > > > > > > connection
> > > > > > > > > > > > > > > timeout and broker might need to reestablish the
> > > > > > connection.
> > > > > > > > If
> > > > > > > > > > the
> > > > > > > > > > > > pause
> > > > > > > > > > > > > > > is too long, lease will expire and the heartbeat
> > sent
> > > > > > after
> > > > > > > > the
> > > > > > > > > > pause
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > be treated as a new registration (similar to
> > restart
> > > > > > case),
> > > > > > > > and
> > > > > > > > > > a new
> > > > > > > > > > > > > > epoch
> > > > > > > > > > > > > > > number will be assigned to the broker.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I agree with the end of this paragraph, but not
> > with
> > > > the
> > > > > > start
> > > > > > > > :)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There can be out-of-order requests, since the
> > broker
> > > > will
> > > > > > > > simply
> > > > > > > > > > use a
> > > > > > > > > > > > new
> > > > > > > > > > > > > > TCP connection if the old one has problems. This
> > can
> > > > > > happen
> > > > > > > > for a
> > > > > > > > > > > > variety
> > > > > > > > > > > > > > of reasons. I don't think GC pauses are the most
> > > > common
> > > > > > > > reason for
> > > > > > > > > > > > this to
> > > > > > > > > > > > > > happen. It's more common to see issues issues in
> > the
> > > > > > network
> > > > > > > > > > itself
> > > > > > > > > > > > that
> > > > > > > > > > > > > > result connections getting dropped from time to
> > time.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So we have to assume that messages may arrive out
> > of
> > > > > > order, and
> > > > > > > > > > > > possibly
> > > > > > > > > > > > > > be delayed. I added a note that heartbeat requests
> > > > should
> > > > > > be
> > > > > > > > > > ignored
> > > > > > > > > > > > if
> > > > > > > > > > > > > > the metadata log offset they contain is smaller
> > than a
> > > > > > previous
> > > > > > > > > > > > heartbeat.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > When the active controller fails, the new active
> > > > > > controller
> > > > > > > > > > needs to
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > sure that it considers all the known brokers as
> > alive
> > > > > > till
> > > > > > > > the
> > > > > > > > > > lease
> > > > > > > > > > > > > > > expiration interval. Because
> > > > > > registration.lease.timeout.ms,
> > > > > > > > is
> > > > > > > > > > > > > > configured
> > > > > > > > > > > > > > > on the controller, the new active controller will
> > > > extend
> > > > > > all
> > > > > > > > the
> > > > > > > > > > > > leases
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > registration.lease.timeout.ms. I see that it
> > won't
> > > > need
> > > > > > last
> > > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Agreed.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sat, Sep 5, 2020 at 1:28 AM Colin McCabe <
> > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Colin wrote:
> > > > > > > > > > > > > > > > > > The reason for including LeaseStartTimeMs
> > in
> > > > the
> > > > > > > > request
> > > > > > > > > > is to
> > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > > that the time required to communicate with
> > the
> > > > > > > > controller
> > > > > > > > > > gets
> > > > > > > > > > > > > > > > included in
> > > > > > > > > > > > > > > > > > the lease time. Since requests can
> > > > potentially be
> > > > > > > > delayed
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > > for a long time, this is important.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Aug 31, 2020, at 05:58, Unmesh Joshi
> > wrote:
> > > > > > > > > > > > > > > > > The network time will be added anyway,
> > because
> > > > the
> > > > > > lease
> > > > > > > > > > timer
> > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > active controller will start only after the
> > > > heartbeat
> > > > > > > > request
> > > > > > > > > > > > > > reaches the
> > > > > > > > > > > > > > > > > server.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > If the start time is not specified in the
> > request,
> > > > > > then the
> > > > > > > > > > network
> > > > > > > > > > > > > > time
> > > > > > > > > > > > > > > > is excluded from the heartbeat time.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Here's an example:
> > > > > > > > > > > > > > > > Let's say broker A sends a heartbeat at time
> > 100,
> > > > and
> > > > > > it
> > > > > > > > > > arrives
> > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > controller at time 200, and the lease duration
> > is
> > > > 1000.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The controller looks at the start time in the
> > > > request,
> > > > > > > > which is
> > > > > > > > > > > > 100,
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > adds 1000 to it, getting 1100. On the other
> > hand,
> > > > if
> > > > > > start
> > > > > > > > > > time
> > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > specified in the request, then the expiration
> > will
> > > > be
> > > > > > at
> > > > > > > > time
> > > > > > > > > > 1200.
> > > > > > > > > > > > > > > > That is what I mean by "the network time is
> > > > included
> > > > > > in the
> > > > > > > > > > > > expiration
> > > > > > > > > > > > > > > > time."
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > And I think, some assumption about network
> > round
> > > > trip
> > > > > > > > time is
> > > > > > > > > > > > > > > > > needed anyway to decide on the frequency of
> > the
> > > > > > > > heartbeat (
> > > > > > > > > > > > > > > > > registration.heartbeat.interval.ms), and
> > lease
> > > > > > timeout (
> > > > > > > > > > > > > > > > > registration.lease.timeout.ms). So I think
> > just
> > > > > > having a
> > > > > > > > > > > > leaseTTL
> > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > request is easier to understand and
> > implement.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It's important to keep in mind that messages
> > may be
> > > > > > > > delayed in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > > happens, we
> > > > > > > > will
> > > > > > > > > > use
> > > > > > > > > > > > the
> > > > > > > > > > > > > > start
> > > > > > > > > > > > > > > > time specified in the request to determine if
> > the
> > > > > > request
> > > > > > > > is
> > > > > > > > > > stale.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on the
> > > > > > controller
> > > > > > > > side
> > > > > > > > > > > > should
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > reset in the case of controller failover.
> > The
> > > > > > > > alternative
> > > > > > > > > > > > would
> > > > > > > > > > > > > > be to
> > > > > > > > > > > > > > > > > > track the lease as hard state rather than
> > soft
> > > > > > state,
> > > > > > > > but I
> > > > > > > > > > > > think
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > is not really needed, and would result in
> > more
> > > > log
> > > > > > > > entries.
> > > > > > > > > > > > > > > > > My interpretation of the mention of
> > BrokerRecord
> > > > in
> > > > > > the
> > > > > > > > KIP
> > > > > > > > > > was
> > > > > > > > > > > > that
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > record exists in the Raft log.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > BrokerRecord does exist in the Raft log, but
> > does
> > > > not
> > > > > > > > include
> > > > > > > > > > the
> > > > > > > > > > > > last
> > > > > > > > > > > > > > > > heartbeat time.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > By soft state, do you mean the broker
> > > > > > > > > > > > > > > > > records exist only on the active leader and
> > will
> > > > not
> > > > > > be
> > > > > > > > > > > > replicated
> > > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > > raft log? If the live brokers list is
> > maintained
> > > > > > only on
> > > > > > > > the
> > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > controller (raft leader), then, in case of
> > leader
> > > > > > > > failure,
> > > > > > > > > > there
> > > > > > > > > > > > > > will be
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > window where the new leader does not know
> > about
> > > > the
> > > > > > live
> > > > > > > > > > brokers,
> > > > > > > > > > > > > > till
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > brokers establish the leases again.
> > > > > > > > > > > > > > > > > I think it will be safer to have leases as a
> > hard
> > > > > > state
> > > > > > > > > > managed
> > > > > > > > > > > > by
> > > > > > > > > > > > > > > > standard
> > > > > > > > > > > > > > > > > Raft replication.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Leases are short, so the need to re-establish
> > them
> > > > > > after a
> > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > failover doesn't seem like a big problem. But
> > > > this is
> > > > > > > > > > something
> > > > > > > > > > > > we can
> > > > > > > > > > > > > > > > tweak if it becomes an issue. One option
> > would be
> > > > to
> > > > > > have
> > > > > > > > a
> > > > > > > > > > > > separate
> > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > which is only used by the controller nodes for
> > this
> > > > > > (since,
> > > > > > > > > > after
> > > > > > > > > > > > all,
> > > > > > > > > > > > > > > > brokers don't care about registration
> > renewals).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Or am I misunderstanding something? (I assume
> > > > that
> > > > > > with
> > > > > > > > soft
> > > > > > > > > > > > state,
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > mean something like zookeeper local sessions
> > > > > > > > > > > > > > > > >
> > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1147
> > > > > > .)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > think
> > > > it
> > > > > > makes
> > > > > > > > > > sense
> > > > > > > > > > > > for
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > controller, since otherwise locking becomes
> > > > very
> > > > > > messy.
> > > > > > > > > > I'm
> > > > > > > > > > > > not
> > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > understand your question about duplicate
> > > > broker ID
> > > > > > > > > > detection,
> > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > There's a section in the KIP about this -- is
> > > > there a
> > > > > > > > detail
> > > > > > > > > > we
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This is an implementation detail that doesn't
> > need
> > > > to
> > > > > > be
> > > > > > > > in the
> > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I assumed broker leases are implemented as a
> > hard
> > > > > > state.
> > > > > > > > In
> > > > > > > > > > that
> > > > > > > > > > > > > > case, to
> > > > > > > > > > > > > > > > > check for broker id conflict, we need to
> > check
> > > > the
> > > > > > broker
> > > > > > > > > > ids at
> > > > > > > > > > > > two
> > > > > > > > > > > > > > > > places
> > > > > > > > > > > > > > > > > 1. Pending broker registrations (which are
> > yet
> > > > to be
> > > > > > > > > > committed)
> > > > > > > > > > > > 2.
> > > > > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > > > committed broker registrations.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, Aug 31, 2020 at 5:42 PM Colin McCabe
> > <
> > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020, at 01:12, Unmesh
> > Joshi
> > > > wrote:
> > > > > > > > > > > > > > > > > > > >>>Can you repeat your questions about
> > broker
> > > > > > leases?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >>>>The LeaseStartTimeMs is expected to
> > be
> > > > the
> > > > > > > > broker's
> > > > > > > > > > > > > > > > > > > 'System.currentTimeMillis()' at the
> > point of
> > > > the
> > > > > > > > > > request. The
> > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > controller will add its lease period to
> > this
> > > > in
> > > > > > order
> > > > > > > > > > >>>>to
> > > > > > > > > > > > > > compute
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > LeaseEndTimeMs.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think the use of LeaseStartTimeMs and
> > > > > > > > LeaseEndTimeMs
> > > > > > > > > > in the
> > > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > confusing. Monotonic Clock
> > > > (System.nanoTime) on
> > > > > > the
> > > > > > > > > > active
> > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > should be used to track leases.
> > > > > > > > > > > > > > > > > > > (For example,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1616https://github.com/etcd-io/etcd/pull/6888/commits/e7f4010ccaf28b6ce64fe514d25a4b2fa459d114
> > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Then we will not need LeaseStartTimeMs?
> > > > > > > > > > > > > > > > > > > Instead of LeaseStartTimeMs, can we call
> > it
> > > > > > > > LeaseTTL? The
> > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > can then calculate LeaseEndTime =
> > > > > > System.nanoTime() +
> > > > > > > > > > > > LeaseTTL.
> > > > > > > > > > > > > > > > > > > In this case we might just drop
> > > > LeaseEndTimeMs
> > > > > > from
> > > > > > > > the
> > > > > > > > > > > > > > response, as
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > broker already knows about the TTL and
> > can
> > > > send
> > > > > > > > > > heartbeats at
> > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > fraction
> > > > > > > > > > > > > > > > > > > of TTL, say every TTL/4
> > milliseconds.(elapsed
> > > > > > time
> > > > > > > > on the
> > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > measured
> > > > > > > > > > > > > > > > > > > by System.nanoTime)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I agree that the monotonic clock is
> > probably a
> > > > > > better
> > > > > > > > idea
> > > > > > > > > > > > here.
> > > > > > > > > > > > > > It is
> > > > > > > > > > > > > > > > > > good to be robust against wall clock
> > changes,
> > > > > > although
> > > > > > > > I
> > > > > > > > > > think
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > > which had them might suffer other issues.
> > I
> > > > will
> > > > > > > > change
> > > > > > > > > > it to
> > > > > > > > > > > > > > specify
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > monotonic clock.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The reason for including LeaseStartTimeMs
> > in
> > > > the
> > > > > > > > request
> > > > > > > > > > is to
> > > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > the time required to communicate with the
> > > > > > controller
> > > > > > > > gets
> > > > > > > > > > > > included
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > lease time. Since requests can
> > potentially be
> > > > > > delayed
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > > > long time, this is important.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I have a prototype built to demonstrate
> > this
> > > > as
> > > > > > > > > > following:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The Kip631Controller itself depends on a
> > > > > > Consensus
> > > > > > > > > > module, to
> > > > > > > > > > > > > > > > demonstrate
> > > > > > > > > > > > > > > > > > > how possible interactions with the
> > consensus
> > > > > > module
> > > > > > > > will
> > > > > > > > > > look
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > (The Consensus can be pluggable really,
> > > > with an
> > > > > > API
> > > > > > > > to
> > > > > > > > > > allow
> > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > replicated log upto HighWaterMark)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It has an implementation of LeaseTracker
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/LeaderLeaseTracker.scala
> > > > > > > > > > > > > > > > > > > to demonstrate LeaseTracker's interaction
> > > > with
> > > > > > the
> > > > > > > > > > consensus
> > > > > > > > > > > > > > module.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The implementation has the following
> > aspects:
> > > > > > > > > > > > > > > > > > > 1. The lease tracking happens only on the
> > > > active
> > > > > > > > > > controller
> > > > > > > > > > > > (raft
> > > > > > > > > > > > > > > > > > > leader)
> > > > > > > > > > > > > > > > > > > 2. Once the lease expires, it needs to
> > > > propose
> > > > > > and
> > > > > > > > > > commit a
> > > > > > > > > > > > > > > > FenceBroker
> > > > > > > > > > > > > > > > > > > record for that lease.
> > > > > > > > > > > > > > > > > > > 3. In case of active controller failure,
> > the
> > > > > > lease
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > > > tracked by
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > newly raft leader. The new raft leader
> > > > starts the
> > > > > > > > lease
> > > > > > > > > > timer
> > > > > > > > > > > > > > again,
> > > > > > > > > > > > > > > > (as
> > > > > > > > > > > > > > > > > > > implemented in onBecomingLeader method of
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > > in effect extending the lease by the time
> > > > spent
> > > > > > in
> > > > > > > > the
> > > > > > > > > > leader
> > > > > > > > > > > > > > > > election
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > whatever time was elapsed on the old
> > leader.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on the
> > > > > > controller
> > > > > > > > side
> > > > > > > > > > > > should
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > reset
> > > > > > > > > > > > > > > > > > in the case of controller failover. The
> > > > > > alternative
> > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > > to
> > > > > > > > > > > > > > track
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > lease as hard state rather than soft state,
> > > > but I
> > > > > > think
> > > > > > > > > > that
> > > > > > > > > > > > is not
> > > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > > > needed, and would result in more log
> > entries.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There are working tests for this
> > > > implementation
> > > > > > here.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > > and an end to end test here
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/ProducerConsumerKIP500Test.scala
> > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >>'m not sure what you mean by
> > > > "de-duplication
> > > > > > of the
> > > > > > > > > > > > broker."
> > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > Apologies for using the confusing term
> > > > > > > > deduplication. I
> > > > > > > > > > meant
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > id
> > > > > > > > > > > > > > > > > > > conflict.
> > > > > > > > > > > > > > > > > > > As you can see in the prototype
> > > > handleRequest of
> > > > > > > > > > > > KIP631Controller
> > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > > the duplicate broker id needs to be
> > detected
> > > > > > before
> > > > > > > > the
> > > > > > > > > > > > > > BrokerRecord
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > submitted to the raft module.
> > > > > > > > > > > > > > > > > > > Also as implemented in the prototype, the
> > > > > > > > > > KIP631Controller is
> > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > > > threaded, handling requests one at a
> > time.
> > > > (an
> > > > > > > > example of
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://martinfowler.com/articles/patterns-of-distributed-systems/singular-update-queue.html
> > > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> > think
> > > > it
> > > > > > makes
> > > > > > > > > > sense
> > > > > > > > > > > > for
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > controller, since otherwise locking becomes
> > > > very
> > > > > > messy.
> > > > > > > > > > I'm
> > > > > > > > > > > > not
> > > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > > understand your question about duplicate
> > > > broker ID
> > > > > > > > > > detection,
> > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > There's a section in the KIP about this --
> > is
> > > > > > there a
> > > > > > > > > > detail we
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 10:49 AM Colin
> > > > McCabe <
> > > > > > > > > > > > colin@cmccabe.xyz
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2020, at 19:36, Unmesh
> > > > Joshi
> > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > There were a few of questions I had..
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for the response.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. Were my comments on the broker
> > lease
> > > > > > > > > > implementation
> > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > prototype) appropriate and do we
> > need to
> > > > > > change
> > > > > > > > the
> > > > > > > > > > KIP
> > > > > > > > > > > > > > > > > > > > > description accordingly?.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Can you repeat your questions about
> > broker
> > > > > > leases?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. How will broker epochs be
> > generated?
> > > > I am
> > > > > > > > > > assuming it
> > > > > > > > > > > > can
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > committed log offset (like zxid?)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > There isn't any need to use a log
> > offset.
> > > > We
> > > > > > can
> > > > > > > > just
> > > > > > > > > > > > look at
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > in-memory hash table and see what the
> > > > latest
> > > > > > > > number is,
> > > > > > > > > > > > and add
> > > > > > > > > > > > > > > > one, to
> > > > > > > > > > > > > > > > > > > > generate a new broker epoch.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3. How will producer registration
> > > > happen? I
> > > > > > am
> > > > > > > > > > assuming
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > > should be
> > > > > > > > > > > > > > > > > > > > > similar to broker registration, with
> > a
> > > > > > similar
> > > > > > > > way to
> > > > > > > > > > > > > > generate
> > > > > > > > > > > > > > > > > > producer
> > > > > > > > > > > > > > > > > > > > id.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > For the EOS stuff, we will need a few
> > new
> > > > RPCs
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > > > controller.
> > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > we should do that in a follow-on KIP,
> > > > though,
> > > > > > since
> > > > > > > > > > this
> > > > > > > > > > > > one is
> > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > > pretty big.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 4. Because we expose Raft log to all
> > the
> > > > > > > > brokers, any
> > > > > > > > > > > > > > > > de-duplication
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > broker needs to happen before the
> > > > requests
> > > > > > are
> > > > > > > > > > proposed
> > > > > > > > > > > > to
> > > > > > > > > > > > > > Raft.
> > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > the controller needs to be single
> > > > threaded,
> > > > > > and
> > > > > > > > > > should do
> > > > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > > against the in-process or pending
> > > > requests
> > > > > > and
> > > > > > > > the
> > > > > > > > > > final
> > > > > > > > > > > > > > state. I
> > > > > > > > > > > > > > > > > > read a
> > > > > > > > > > > > > > > > > > > > > mention of this, in the responses in
> > this
> > > > > > > > > > thread.Will it
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > useful to
> > > > > > > > > > > > > > > > > > > > > mention this in the KIP?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I'm not sure what you mean by
> > > > "de-duplication
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > broker."
> > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 4:50 AM Colin
> > > > McCabe
> > > > > > <
> > > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I'm thinking of calling a vote on
> > > > KIP-631
> > > > > > on
> > > > > > > > > > Monday.
> > > > > > > > > > > > Let
> > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > know
> > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > there's any more comments I should
> > > > address
> > > > > > > > before I
> > > > > > > > > > > > start
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > vote.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020, at 05:39,
> > Unmesh
> > > > > > Joshi
> > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >>Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > >>Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > Thanks. I will be adding more to
> > the
> > > > > > > > prototype
> > > > > > > > > > and
> > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > > and collaborate.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020 at 12:28 AM
> > > > Colin
> > > > > > > > McCabe <
> > > > > > > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Jose,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > That'a s good point that I
> > hadn't
> > > > > > > > considered.
> > > > > > > > > > It's
> > > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > a separate leader change
> > message,
> > > > as
> > > > > > you
> > > > > > > > > > mentioned.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2020, at 11:56,
> > Jose
> > > > > > Garcia
> > > > > > > > > > Sancio
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Very cool prototype!
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The KIP proposes a record
> > called
> > > > > > > > IsrChange
> > > > > > > > > > which
> > > > > > > > > > > > > > > > includes the
> > > > > > > > > > > > > > > > > > > > > > > > > partition, topic, isr,
> > leader and
> > > > > > leader
> > > > > > > > > > epoch.
> > > > > > > > > > > > > > During
> > > > > > > > > > > > > > > > normal
> > > > > > > > > > > > > > > > > > > > > > > > > operation ISR changes do not
> > > > result
> > > > > > in
> > > > > > > > leader
> > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > > Similarly,
> > > > > > > > > > > > > > > > > > > > > > > > > leader changes do not
> > necessarily
> > > > > > > > involve ISR
> > > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > implementation that uses ZK
> > > > modeled
> > > > > > them
> > > > > > > > > > together
> > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > > > > > > > 1. All of this information is
> > > > stored
> > > > > > in
> > > > > > > > one
> > > > > > > > > > > > znode.
> > > > > > > > > > > > > > > > > > > > > > > > > 2. ZK's optimistic lock
> > requires
> > > > > > that you
> > > > > > > > > > specify
> > > > > > > > > > > > > > the new
> > > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > > > > > > 3. The change to that znode
> > was
> > > > being
> > > > > > > > > > performed
> > > > > > > > > > > > by
> > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > > and the leader.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > None of these reasons are
> > true in
> > > > > > > > KIP-500.
> > > > > > > > > > Have
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > > considered
> > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > > two different records? For
> > > > example
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. IsrChange record which
> > > > includes
> > > > > > topic,
> > > > > > > > > > > > partition,
> > > > > > > > > > > > > > isr
> > > > > > > > > > > > > > > > > > > > > > > > > 2. LeaderChange record which
> > > > includes
> > > > > > > > topic,
> > > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > leader epoch.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I suspect that making this
> > change
> > > > > > will
> > > > > > > > also
> > > > > > > > > > > > require
> > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > message AlterIsrRequest
> > > > introduced in
> > > > > > > > > > KIP-497:
> > > > > > > > > > > > Add
> > > > > > > > > > > > > > > > > > inter-broker
> > > > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > alter ISR.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > > > -Jose
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the reply. Made another pass of the KIP. A few more comments
below.
55. We discussed earlier why the current behavior where we favor the
current broker registration is better. Have you given this more thought?
80. Config related.
80.1 Currently, each broker only has the following 3 required configs. It
will be useful to document the required configs post KIP-500 (in both the
dedicated and shared controller mode).
broker.id
log.dirs
zookeeper.connect
80.2 It would be useful to document all deprecated configs post KIP-500.
For example, all zookeeper.* are obviously deprecated. But there could be
others. For example, since we don't plan to support auto broker id
generation, it seems broker.id.generation.enable is deprecated too.
80.3 Could we make it clear that controller.connect replaces quorum.voters
in KIP-595?
80.4 Could we document that broker.id is now optional?
80.5 The KIP suggests that controller.id is optional on the controller
node. I am concerned that this can cause a bit of confusion in 2 aspects.
First, in the dedicated controller mode, controller.id is not optional
(since broker.id is now optional). Second, in the shared controller mode,
it may not be easy for the user to figure out the default value of
controller.id to set controller.connect properly.
80.6 Regarding the consistency of config names, our metrics already include
controller. So, prefixing all controller related configs with "controller"
may be more consistent. If we choose to do that, could we rename all new
configs here and in KIP-595 consistently?
81. Metrics
81.1 kafka.controller:type=KafkaController,name=MetadataSnapshotLag: This
is now redundant since KIP-630 already has
kafka.controller:type=KafkaController,name=SnapshotLag.
81.2 Do we need both kafka.controller:type=KafkaServer,name=MetadataLag and
kafka.controller:type=KafkaController,name=MetadataLag since in the shared
controller mode, the metadata log is shared?
82. Metadata records
82.1 BrokerRecord: It needs to include supported features based on KIP-584.
82.2 FeatureLevel: The finalized feature in KIP-584 has the following
structure and needs to be reflected here.
{
"version": 0, // int32 -> Represents the version of the schema for the
data stored in the ZK node
"status": 1, // int32 -> Represents the status of the node
"features": {
"group_coordinator": { // string -> name of the feature
"min_version_level": 0, // int16 -> Represents the cluster-wide
finalized minimum version level (>=1) of this feature
"max_version_level": 3 // int16 -> Represents the cluster-wide
finalized maximum version level (>=1 and >= min_version_level) of this
feature
},
"consumer_offsets_topic_schema": {
"min_version_level": 0,
"max_version_level": 4
}
}
83. RPC
83.1 BrokerHeartbeatRequest: Could you document when the controller assigns
a new broker epoch?
83.2 BrokerHeartbeatResponse: Do we need a new error code to indicate a
broker is fenced?
83.3 What error/state indicates an unsuccessful controlled shutdown in
BrokerHeartbeatResponse? Currently, the controlled shutdown responses also
include a list of remaining partitions. That's mostly for debugging
purposes. Should we include it in BrokerHeartbeatResponse too?
83.4 Could we document when Listeners, Features and Rack are expected to be
set in BrokerHeartbeatRequest?
83.5 Currently, the metadata response only returns a list of brokers.
Should we have a separate field for the controller info or change the field
name to sth like nodes? Also, is the rack field relevant to the controller?
84. There are a few references to __kafka_metadata. However, KIP-595
uses __cluster_metadata.
85. "When a broker is fenced, it cannot process any client requests. "
Should we send a new error code in the response to indicate that the broker
is fenced?
86. "The controller can generate a new broker epoch by using the latest log
offset." Perhaps it's more accurate to say the new broker epoch is the
offset for the corresponding BrokerRecord in the metadata log.
87. meta.properties:
87.1 Could you document the format change to this file? Also, should we
bump up the version of the file?
87.2 How do we know a node is configured in KIP-500 mode?
88. CurMetadataOffset: Should we use nextMetadataOffset since it's more
consistent with fetchOffset and HWM?
Thanks,
Jun
On Mon, Oct 5, 2020 at 9:03 AM Colin McCabe <cm...@apache.org> wrote:
> On Mon, Sep 28, 2020, at 11:41, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the reply. A few more comments below.
> >
> > 62.
> > 62.1 controller.listener.names: So, is this used for the controller or
> the
> > broker trying to connect to the controller?
> >
>
> Hi Jun,
>
> It's used by both. The broker tries to connect to controllers by using
> the first listener in this list. The controller uses the list to determine
> which listeners it should bind to.
>
> >
> > 62.2 If we want to take the approach to share the configs that are common
> > between the broker and the controller, should we share the id too?
> >
>
> On nodes where we are running both the controller and broker roles, we
> need two IDs: one for the controller, and one for the broker. So we can't
> share the same configuration.
>
> >
> > 62.3 We added some configs in KIP-595 prefixed with "quorum" and we plan
> to
> > add some controller specific configs prefixed with "controller". KIP-630
> > plans to add some other controller specific configs with no prefix.
> Should
> > we standardize all controller specific configs with the same prefix?
> >
>
> That's a good question. As Jose said elsewhere in the thread, let's use
> "metadata" as the prefix.
>
> >
> > 70. Could you explain the impact of process.roles a bit more? For
> example,
> > if process.roles=controller, does the node still maintain metadata cache
> as
> > described in KIP-630? Do we still return the host/port for those nodes in
> > the metadata response?
> >
>
> No, the node does not need any broker data structures if it is not a
> broker. Controllers also don't handle MetadataRequests. Clients will
> still go to the brokers for those.
>
> best,
> Colin
>
>
> > Thanks,
> >
> > Jun
> >
> > On Mon, Sep 28, 2020 at 9:24 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Fri, Sep 25, 2020, at 17:35, Jun Rao wrote:
> > > > Hi, Colin,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > 62. Thinking about this more, I am wondering what's our overall
> strategy
> > > > for configs shared between the broker and the controller. For
> example,
> > > both
> > > > the broker and the controller have to define listeners. So both need
> > > > configs like listeners/advertised.listeners. Both the new controller
> and
> > > > the broker replicate data. So both need to define some replication
> > > related
> > > > configurations (replica.fetch.min.bytes, replica.fetch.wait.max.ms,
> > > etc).
> > > > Should we let the new controller share those configs with brokers or
> > > should
> > > > we define new configs just for the controller? It seems that we want
> to
> > > > apply the same strategy consistently for all shared configs?
> > > >
> > >
> > > Hi Jun,
> > >
> > > That's a good question. I think that we should share as many
> > > configurations as possible. There will be a few cases where this isn't
> > > possible, and we need to create a new configuration key that is
> > > controller-only, but I think that will be relatively rare.
> > >
> > > >
> > > > 63. If a node has process.roles set to controller, does one still
> need to
> > > > set broker.id on this node?
> > > >
> > >
> > > No, broker.id does not need to be set in that case.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > >
> > > > On Fri, Sep 25, 2020 at 2:17 PM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > On Fri, Sep 25, 2020, at 10:17, Jun Rao wrote:
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the reply.
> > > > > >
> > > > > > 60. Yes, I think you are right. We probably need the controller
> id
> > > when a
> > > > > > broker starts up. A broker only stores the Raft leader id in the
> > > metadata
> > > > > > file. To do the initial fetch to the Raft leader, it needs to
> know
> > > the
> > > > > > host/port associated with the leader id.
> > > > > >
> > > > > > 62. It seems there are 2 parts to this : (1) which listener a
> client
> > > > > should
> > > > > > use to initiate a connection to the controller and (2) which
> listener
> > > > > > should a controller use to accept client requests. For (1), at
> any
> > > point
> > > > > of
> > > > > > time, a client only needs to use one listener. I think
> > > > > > controller.listener.name is meant for the client.
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > controller.listener.names is also used by the controllers. In the
> case
> > > > > where we have a broker and a controller in the same JVM, we have a
> > > single
> > > > > config file. Then we need to know which listeners belong to the
> > > controller
> > > > > and which belong to the broker component. That's why it's a list.
> > > > >
> > > > > > So, a single value seems
> > > > > > to make more sense. Currently, we don't have a configuration for
> > > (2). We
> > > > > > could add a new one for that and support a list. I am wondering
> how
> > > > > useful
> > > > > > it will be. One example that I can think of is that we can reject
> > > > > > non-controller related requests if accepted on controller-only
> > > listeners.
> > > > > > However, we already support separate authentication for the
> > > controller
> > > > > > listener. So, not sure how useful it is.
> > > > >
> > > > > The controller always has a separate listener and does not share
> > > listeners
> > > > > with the broker. The main reason for this is to avoid giving
> clients
> > > the
> > > > > ability to launch a denial-of-service attack on the controller by
> > > flooding
> > > > > a broker port. A lot of times, these attacks are made
> unintentionally
> > > by
> > > > > poorly coded or configured clients. Additionally, broker ports can
> > > also be
> > > > > very busy with large fetch requests, and so on. It's just a bad
> > > > > configuration in general to try to overload the same port for the
> > > > > controller and broker, and we don't want to allow it. It would be
> a
> > > > > regression to go from the current system where control requests are
> > > safely
> > > > > isolated on a separate port, to one where they're not. It also
> makes
> > > the
> > > > > code and configuration a lot messier.
> > > > >
> > > > > >
> > > > > > 63. (a) I think most users won't know controller.id defaults to
> > > > > broker.id +
> > > > > > 3000. So, it can be confusing for them to set up
> controller.connect.
> > > If
> > > > > > this is truly needed, it seems that it's less confusing to make
> > > > > > controller.id required.
> > > > > > (b) I am still trying to understand if we truly need to expose a
> > > > > > controller.id. What if we only expose broker.id and let
> > > > > controller.connect
> > > > > > just use broker.id? What will be missing?
> > > > >
> > > > > The controller has a separate ID from the broker, so knowing
> broker.id
> > > is
> > > > > not helpful here.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Thu, Sep 24, 2020 at 10:55 PM Colin McCabe <
> cmccabe@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > On Thu, Sep 24, 2020, at 16:24, Jun Rao wrote:
> > > > > > > > Hi, Colin,
> > > > > > > >
> > > > > > > > Thanks for the reply and the updated KIP. A few more comments
> > > below.
> > > > > > > >
> > > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > >
> > > > > > > > 53. It seems that you already incorporated the changes in
> > > KIP-516.
> > > > > With
> > > > > > > > topic ids, we don't need to wait for the topic's data to be
> > > deleted
> > > > > > > before
> > > > > > > > removing the topic metadata. If the topic is recreated, we
> can
> > > still
> > > > > > > delete
> > > > > > > > the data properly based on the topic id. So, it seems that
> we can
> > > > > remove
> > > > > > > > TopicRecord.Deleting.
> > > > > > > >
> > > > > > >
> > > > > > > Thanks for the reply. What I was thinking of doing here was
> using
> > > > > topic
> > > > > > > IDs internally, but still using names externally. So the topic
> > > UUIDs
> > > > > are
> > > > > > > only for the purpose of associating topics with partitions --
> from
> > > the
> > > > > > > user's point of view, topics are still identified by names.
> > > > > > >
> > > > > > > You're right that KIP-516 will simplify things, but I'm not
> sure
> > > when
> > > > > that
> > > > > > > will land, so I wanted to avoid blocking the initial
> > > implementation of
> > > > > this
> > > > > > > KIP on it.
> > > > > > >
> > > > > > > >
> > > > > > > > 55. It seems to me that the current behavior where we favor
> the
> > > > > current
> > > > > > > > broker registration is better. This is because uncontrolled
> > > broker
> > > > > > > shutdown
> > > > > > > > is rare and its impact is less severe since one just needs to
> > > wait
> > > > > for
> > > > > > > the
> > > > > > > > session timeout before restarting the broker. If a
> mis-configured
> > > > > broker
> > > > > > > > replaces an existing broker, the consequence is more severe
> > > since it
> > > > > can
> > > > > > > > cause the leader to be unavailable, a replica to be out of
> ISR,
> > > and
> > > > > add
> > > > > > > > more load on the leaders etc.
> > > > > > > >
> > > > > > >
> > > > > > > Hmm, that's a good point. Let me check this a bit more before
> I
> > > change
> > > > > > > it, though.
> > > > > > >
> > > > > > > >
> > > > > > > > 60. controller.connect=0@controller0.example.com:9093,
> > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com :
> Do we
> > > > > need to
> > > > > > > > include the controller id before @? It seems that the
> host/port
> > > is
> > > > > enough
> > > > > > > > for establishing the connection. It would also be useful to
> make
> > > it
> > > > > clear
> > > > > > > > that controller.connect replaces quorum.voters in KIP-595.
> > > > > > > >
> > > > > > >
> > > > > > > I discussed this with Jason earlier, and he felt that the
> > > controller
> > > > > IDs
> > > > > > > were needed in this configuration key. It is certainly needed
> when
> > > > > > > configuring the controllers themselves, since they need to know
> > > each
> > > > > > > others' IDs.
> > > > > > >
> > > > > > > >
> > > > > > > > 61. I am not sure that we need both controller.listeners and
> > > > > > > > controller.connect.security.protocol since the former
> implies the
> > > > > > > security
> > > > > > > > protocol. The reason that we have both
> > > inter.broker.listener.name
> > > > > and
> > > > > > > > inter.broker.security.protocol is because we had the latter
> > > first and
> > > > > > > later
> > > > > > > > realized that the former is more general.
> > > > > > > >
> > > > > > >
> > > > > > > That's a good point. I've removed this from the KIP.
> > > > > > >
> > > > > > > >
> > > > > > > > 62. I am still not sure that you need controller.listeners
> to be
> > > a
> > > > > list.
> > > > > > > > All listeners are already defined in listeners. To migrate
> from
> > > > > plaintext
> > > > > > > > to SSL, one can configure listeners to have both plaintext
> and
> > > SSL.
> > > > > After
> > > > > > > > that, one can just change controller.listeners from
> plaintext to
> > > SSL.
> > > > > > > This
> > > > > > > > is similar to how to change the listener for inter broker
> > > > > connections.
> > > > > > > > Also, controller.listener.name may be a more accurate name?
> > > > > > > >
> > > > > > >
> > > > > > > The issue that I see here is that if you are running with the
> > > > > controller
> > > > > > > and broker in the same JVM, if you define a few listeners in
> > > > > "listeners"
> > > > > > > they will get used as regular broker listeners, unless you put
> > > them in
> > > > > > > controller.listeners. Therefore, controller.listeners needs
> to be
> > > a
> > > > > list.
> > > > > > >
> > > > > > > controller.listener.names does sound like a better name,
> though...
> > > I've
> > > > > > > updated it to that.
> > > > > > >
> > > > > > > >
> > > > > > > > 63. Regarding controller.id, I am trying to understand
> whether
> > > it's
> > > > > a
> > > > > > > > required configuration or an optional one. From the KIP, it
> > > sounds
> > > > > like
> > > > > > > > controller.id is optional. Then, if this is unset, it's not
> > > clear
> > > > > how
> > > > > > > the
> > > > > > > > user will obtain the controller id for setting
> > > controller.connect.
> > > > > > > >
> > > > > > >
> > > > > > > If you specify broker.id but not controller.id, then
> controller.id
> > > > > will
> > > > > > > just be set to broker.id + 3000. This was intended to make
> some
> > > > > > > configurations a little bit more concise to write. You still
> do
> > > have
> > > > > to
> > > > > > > know the controller IDs when configuring the brokers, though.
> If
> > > this
> > > > > > > seems confusing then I can just make it mandatory.
> > > > > > >
> > > > > > > >
> > > > > > > > 64. KIP-516 adds a flag in LeaderAndIsrRequest to indicate
> > > whether it
> > > > > > > > includes all partitions or not. This will be used to clean up
> > > strayed
> > > > > > > logs.
> > > > > > > > I was thinking how we will do the same thing once the
> controller
> > > > > moves to
> > > > > > > > Raft based. One way to do that is on broker startup, it gets
> the
> > > HWM
> > > > > in
> > > > > > > the
> > > > > > > > metadata log from the Raft leader and waits until its
> metadata
> > > > > catches up
> > > > > > > > to HWM. At that point, any local log not seen in the metadata
> > > can be
> > > > > > > > removed. Since the Fetch response returns the HWM, there
> seems
> > > to be
> > > > > > > enough
> > > > > > > > APIs to achieve this.
> > > > > > > >
> > > > > > >
> > > > > > > That's a very good point. I added a note about this under
> Broker
> > > > > Startup.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Sep 24, 2020 at 1:07 PM Colin McCabe <
> colin@cmccabe.xyz>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Mon, Sep 21, 2020, at 18:13, Jun Rao wrote:
> > > > > > > > > > Hi, Colin,
> > > > > > > > > >
> > > > > > > > > > Sorry for the late reply. A few more comments below.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > Thanks for taking another look.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 50. Configurations
> > > > > > > > > > 50.1 controller.listeners: It seems that a controller
> just
> > > needs
> > > > > one
> > > > > > > > > > listener. Why do we need to have a list here? Also,
> could you
> > > > > > > provide an
> > > > > > > > > > example of how this is set and what's its relationship
> with
> > > > > existing
> > > > > > > > > > configs such as "security.inter.broker.protocol" and "
> > > > > > > > > > inter.broker.listener.name"?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I agree that most administrators will want to run with
> only one
> > > > > > > controller
> > > > > > > > > listener. However, just as with brokers, it is nice to
> have
> > > the
> > > > > > > option to
> > > > > > > > > expose multiple ports.
> > > > > > > > >
> > > > > > > > > One reason why you might want multiple ports is if you were
> > > doing a
> > > > > > > > > migration from plaintext to SSL. You could add new SSL
> > > listeners
> > > > > but
> > > > > > > > > continue to expose the PLAINTEXT listeners. Then you could
> > > add the
> > > > > > > new SSL
> > > > > > > > > controller config to each broker and roll each broker.
> Then
> > > you
> > > > > could
> > > > > > > > > migrate from PLAINTEXT to SSL with no downtime.
> > > > > > > > >
> > > > > > > > > Here's an example configuration for the controller:
> > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com
> > > > > > > > > controller.listeners=CONTROLLER
> > > > > > > > > listeners=CONTROLLER://controller0.example.com:9093
> > > > > > > > > listener.security.protocol.map=CONTROLLER:SSL
> > > > > > > > >
> > > > > > > > > Here's an example configuration for the broker:
> > > > > > > > > controller.connect=0@controller0.example.com:9093,
> > > > > > > > > 1@controller1.example.com:9093,2@controller2.example.com
> > > > > > > > > controller.connect.security.protocol=SSL
> > > > > > > > >
> > > > > > > > > security.inter.broker.protocol or
> inter.broker.listener.name
> > > do
> > > > > not
> > > > > > > > > affect how the broker communicates with the controller.
> Those
> > > > > > > > > configurations relate to how brokers communicate with each
> > > other,
> > > > > but
> > > > > > > the
> > > > > > > > > controller is not a broker.
> > > > > > > > >
> > > > > > > > > (Note that I just added
> controller.connect.security.protocol
> > > to the
> > > > > > > KIP --
> > > > > > > > > I had forgotten to put it in earlier)
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 50.2 registration.heartbeat.interval.ms and
> > > > > > > > > registration.lease.timeout.ms.
> > > > > > > > > > Should we match their default value with the
> corresponding
> > > > > default
> > > > > > > for
> > > > > > > > > ZK?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Fair enough. I'll set them to the values of the
> > > > > > > zookeeper.sync.time.ms
> > > > > > > > > and zookeeper.connection.timeout.ms configurations. I do
> > > think we
> > > > > > > should
> > > > > > > > > experiment later on to see what works well here, but the ZK
> > > values
> > > > > are
> > > > > > > at
> > > > > > > > > least a starting point.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 50.3 controller.connect: Could you provide an example? I
> am
> > > > > > > wondering how
> > > > > > > > > > it differs from quorum.voters=1@kafka-1:9092, 2@kafka-2
> > > :9092,
> > > > > > > 3@kafka-3
> > > > > > > > > > :9092.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > controller.connect is intended to be the new name for
> > > > > quorum.voters.
> > > > > > > > > During the vote for KIP-595, we sort of agreed to defer the
> > > > > discussion
> > > > > > > > > about what this configuration should be called. I proposed
> > > this
> > > > > new
> > > > > > > name
> > > > > > > > > because it makes it clear what the configuration is for
> (how to
> > > > > > > connect to
> > > > > > > > > the controllers).
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 50.4 controller.id: I am still not sure how this is
> being
> > > used.
> > > > > > > Could
> > > > > > > > > you
> > > > > > > > > > explain this in more detail?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > The controller ID needs to be set on each controller. It
> also
> > > > > appears
> > > > > > > in
> > > > > > > > > controller.connect as the thing before the "at" sign. Its
> > > > > function is
> > > > > > > > > pretty similar to broker.id.
> > > > > > > > >
> > > > > > > > > Broker IDs and controller IDs exist in the same ID space,
> so
> > > you
> > > > > can't
> > > > > > > > > have a broker and a controller that use the same ID.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 51. BrokerHeartbeat: It seems a bit wasteful to include
> > > > > Listeners in
> > > > > > > > > every
> > > > > > > > > > heartbeat request since it typically doesn't change.
> Could we
> > > > > make
> > > > > > > that
> > > > > > > > > an
> > > > > > > > > > optional field?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Ok.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 52. KIP-584 adds a new ZK node /features. Should we add a
> > > > > > > corresponding
> > > > > > > > > > metadata record?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Good point. I added FeatureLevelRecord.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 53. TopicRecord and DeleteTopic: Both DeleteTopic and
> > > > > > > > > TopicRecord.Deleting
> > > > > > > > > > indicate topic deletion. Could we outline the flow when
> each
> > > > > will be
> > > > > > > set?
> > > > > > > > > > In particular, which one indicates the intention to
> delete
> > > and
> > > > > which
> > > > > > > one
> > > > > > > > > > indicates the completion of the deletion.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > TopicRecord.Deleting is set when we intend to delete the
> topic.
> > > > > > > > >
> > > > > > > > > DeleteTopic removes the topic completely. I will rename
> > > > > DeleteTopic to
> > > > > > > > > RemoveTopic to make this clearer.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 54. "The controller can generate a new broker epoch by
> using
> > > the
> > > > > > > latest
> > > > > > > > > log
> > > > > > > > > > offset." Which offset is that? Is it the offset of the
> > > metadata
> > > > > > > topic for
> > > > > > > > > > the corresponding BrokerRecord? Is it guaranteed to be
> > > different
> > > > > on
> > > > > > > each
> > > > > > > > > > broker restart?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes.... a new broker epoch implies that there has been a
> new
> > > record
> > > > > > > > > created in the metadata log. Therefore the last committed
> > > offset
> > > > > must
> > > > > > > be
> > > > > > > > > different.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 55. "Thereafter, it may lose subsequent conflicts if its
> > > broker
> > > > > > > epoch is
> > > > > > > > > > stale. (See KIP-380 for some background on broker
> epoch.)
> > > The
> > > > > > > reason
> > > > > > > > > for
> > > > > > > > > > favoring new processes is to accommodate the common case
> > > where a
> > > > > > > process
> > > > > > > > > is
> > > > > > > > > > killed with kill -9 and then restarted. " Are you saying
> > > that if
> > > > > > > there is
> > > > > > > > > > an active broker registered in the controller, a new
> broker
> > > > > heartbeat
> > > > > > > > > > request with the INITIAL state will fence the current
> broker
> > > > > > > session? Not
> > > > > > > > > > sure about this. For example, this will allow a broker
> with
> > > > > > > incorrectly
> > > > > > > > > > configured broker id to kill an existing valid broker.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes, a new broker with an incorrectly configured broker id
> > > could
> > > > > fence
> > > > > > > an
> > > > > > > > > existing valid broker.
> > > > > > > > >
> > > > > > > > > The goal of the conflict resolution process described here
> is
> > > to
> > > > > avoid
> > > > > > > > > having two brokers go back and forth in claiming the
> broker id.
> > > > > > > Choosing
> > > > > > > > > the newest was just an easy way to do that.
> > > > > > > > >
> > > > > > > > > Choosing the oldest is another possibility, and more inline
> > > with
> > > > > what
> > > > > > > > > ZooKeeper does today. The problem with that is that it
> would
> > > not
> > > > > be
> > > > > > > > > possible to re-create the broker if the old instance died
> > > suddenly,
> > > > > > > except
> > > > > > > > > by waiting for the full lease timeout.
> > > > > > > > >
> > > > > > > > > It might be possible to do a little better here by
> breaking the
> > > > > lease
> > > > > > > if
> > > > > > > > > the TCP connection from the broker drops. But that
> requires
> > > > > crossing
> > > > > > > some
> > > > > > > > > layers of abstraction in the Kafka networking stack.
> There's
> > > also
> > > > > the
> > > > > > > > > possibility of false negatives or positives. For example,
> TCP
> > > > > > > connections
> > > > > > > > > sometimes just drop even if the node is still there. Or
> > > sometimes
> > > > > the
> > > > > > > > > networking stack keeps a TCP connection alive for a while
> when
> > > the
> > > > > > > other
> > > > > > > > > end is gone.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 56. kafka-storage.sh:
> > > > > > > > > > 56.1 In the info mode, what other information does it
> show in
> > > > > > > addition to
> > > > > > > > > > "kip.500.mode=enabled"?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I think it should show a list of all the storage
> directories,
> > > and
> > > > > > > whether
> > > > > > > > > each one is formatted. In addition, it should show whether
> > > > > > > kip.500.mode is
> > > > > > > > > enabled, and what the cluster id is.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 56.2 Should the format mode take the config file as the
> > > input too
> > > > > > > like
> > > > > > > > > the
> > > > > > > > > > info mode?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes, that's a good idea.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 17, 2020 at 4:12 PM Colin McCabe <
> > > cmccabe@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > >
> > > > > > > > > > > That's a fair point. I have moved the lease duration
> to
> > > the
> > > > > broker
> > > > > > > > > > > heartbeat response. That way lease durations can be
> > > changed
> > > > > just
> > > > > > > be
> > > > > > > > > > > reconfiguring the controllers.
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Sep 16, 2020, at 07:40, Unmesh Joshi wrote:
> > > > > > > > > > > > Thanks Colin, the changes look good to me. One small
> > > thing.
> > > > > > > > > > > > registration.lease.timeout.ms is the configuration
> on
> > > the
> > > > > > > controller
> > > > > > > > > > > side.
> > > > > > > > > > > > It will be good to comment on how brokers know about
> it,
> > > to
> > > > > be
> > > > > > > able
> > > > > > > > > to
> > > > > > > > > > > > send LeaseDurationMs
> > > > > > > > > > > > in the heartbeat request,
> > > > > > > > > > > > or else it can be added in the heartbeat response for
> > > > > brokers to
> > > > > > > know
> > > > > > > > > > > about
> > > > > > > > > > > > it.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Unmesh
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Sep 11, 2020 at 10:32 PM Colin McCabe <
> > > > > > > cmccabe@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think you're right that we should use a duration
> here
> > > > > rather
> > > > > > > > > than a
> > > > > > > > > > > > > time. As you said, the clock on the controller
> will
> > > > > probably
> > > > > > > not
> > > > > > > > > > > match the
> > > > > > > > > > > > > one on the broker. I have updated the KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > It's important to keep in mind that messages
> may be
> > > > > > > delayed in
> > > > > > > > > the
> > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > happens, we
> > > > > > > will
> > > > > > > > > use
> > > > > > > > > > > the
> > > > > > > > > > > > > start
> > > > > > > > > > > > > > > time specified in the request to determine if
> the
> > > > > request
> > > > > > > is
> > > > > > > > > stale.
> > > > > > > > > > > > > > I am assuming that there will be a single TCP
> > > connection
> > > > > > > > > maintained
> > > > > > > > > > > > > between
> > > > > > > > > > > > > > broker and active controller. So, there won't be
> any
> > > out
> > > > > of
> > > > > > > order
> > > > > > > > > > > > > requests?
> > > > > > > > > > > > > > There will be a scenario of broker GC pause,
> which
> > > might
> > > > > > > cause
> > > > > > > > > > > connection
> > > > > > > > > > > > > > timeout and broker might need to reestablish the
> > > > > connection.
> > > > > > > If
> > > > > > > > > the
> > > > > > > > > > > pause
> > > > > > > > > > > > > > is too long, lease will expire and the heartbeat
> sent
> > > > > after
> > > > > > > the
> > > > > > > > > pause
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > be treated as a new registration (similar to
> restart
> > > > > case),
> > > > > > > and
> > > > > > > > > a new
> > > > > > > > > > > > > epoch
> > > > > > > > > > > > > > number will be assigned to the broker.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I agree with the end of this paragraph, but not
> with
> > > the
> > > > > start
> > > > > > > :)
> > > > > > > > > > > > >
> > > > > > > > > > > > > There can be out-of-order requests, since the
> broker
> > > will
> > > > > > > simply
> > > > > > > > > use a
> > > > > > > > > > > new
> > > > > > > > > > > > > TCP connection if the old one has problems. This
> can
> > > > > happen
> > > > > > > for a
> > > > > > > > > > > variety
> > > > > > > > > > > > > of reasons. I don't think GC pauses are the most
> > > common
> > > > > > > reason for
> > > > > > > > > > > this to
> > > > > > > > > > > > > happen. It's more common to see issues issues in
> the
> > > > > network
> > > > > > > > > itself
> > > > > > > > > > > that
> > > > > > > > > > > > > result connections getting dropped from time to
> time.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So we have to assume that messages may arrive out
> of
> > > > > order, and
> > > > > > > > > > > possibly
> > > > > > > > > > > > > be delayed. I added a note that heartbeat requests
> > > should
> > > > > be
> > > > > > > > > ignored
> > > > > > > > > > > if
> > > > > > > > > > > > > the metadata log offset they contain is smaller
> than a
> > > > > previous
> > > > > > > > > > > heartbeat.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > When the active controller fails, the new active
> > > > > controller
> > > > > > > > > needs to
> > > > > > > > > > > be
> > > > > > > > > > > > > > sure that it considers all the known brokers as
> alive
> > > > > till
> > > > > > > the
> > > > > > > > > lease
> > > > > > > > > > > > > > expiration interval. Because
> > > > > registration.lease.timeout.ms,
> > > > > > > is
> > > > > > > > > > > > > configured
> > > > > > > > > > > > > > on the controller, the new active controller will
> > > extend
> > > > > all
> > > > > > > the
> > > > > > > > > > > leases
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > registration.lease.timeout.ms. I see that it
> won't
> > > need
> > > > > last
> > > > > > > > > > > heartbeat
> > > > > > > > > > > > > > time.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Agreed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > best,
> > > > > > > > > > > > > Colin
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sat, Sep 5, 2020 at 1:28 AM Colin McCabe <
> > > > > > > cmccabe@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Colin wrote:
> > > > > > > > > > > > > > > > > The reason for including LeaseStartTimeMs
> in
> > > the
> > > > > > > request
> > > > > > > > > is to
> > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > > that the time required to communicate with
> the
> > > > > > > controller
> > > > > > > > > gets
> > > > > > > > > > > > > > > included in
> > > > > > > > > > > > > > > > > the lease time. Since requests can
> > > potentially be
> > > > > > > delayed
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > network
> > > > > > > > > > > > > > > > > for a long time, this is important.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Aug 31, 2020, at 05:58, Unmesh Joshi
> wrote:
> > > > > > > > > > > > > > > > The network time will be added anyway,
> because
> > > the
> > > > > lease
> > > > > > > > > timer
> > > > > > > > > > > on the
> > > > > > > > > > > > > > > > active controller will start only after the
> > > heartbeat
> > > > > > > request
> > > > > > > > > > > > > reaches the
> > > > > > > > > > > > > > > > server.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If the start time is not specified in the
> request,
> > > > > then the
> > > > > > > > > network
> > > > > > > > > > > > > time
> > > > > > > > > > > > > > > is excluded from the heartbeat time.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Here's an example:
> > > > > > > > > > > > > > > Let's say broker A sends a heartbeat at time
> 100,
> > > and
> > > > > it
> > > > > > > > > arrives
> > > > > > > > > > > on the
> > > > > > > > > > > > > > > controller at time 200, and the lease duration
> is
> > > 1000.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The controller looks at the start time in the
> > > request,
> > > > > > > which is
> > > > > > > > > > > 100,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > adds 1000 to it, getting 1100. On the other
> hand,
> > > if
> > > > > start
> > > > > > > > > time
> > > > > > > > > > > is not
> > > > > > > > > > > > > > > specified in the request, then the expiration
> will
> > > be
> > > > > at
> > > > > > > time
> > > > > > > > > 1200.
> > > > > > > > > > > > > > > That is what I mean by "the network time is
> > > included
> > > > > in the
> > > > > > > > > > > expiration
> > > > > > > > > > > > > > > time."
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > And I think, some assumption about network
> round
> > > trip
> > > > > > > time is
> > > > > > > > > > > > > > > > needed anyway to decide on the frequency of
> the
> > > > > > > heartbeat (
> > > > > > > > > > > > > > > > registration.heartbeat.interval.ms), and
> lease
> > > > > timeout (
> > > > > > > > > > > > > > > > registration.lease.timeout.ms). So I think
> just
> > > > > having a
> > > > > > > > > > > leaseTTL
> > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > request is easier to understand and
> implement.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It's important to keep in mind that messages
> may be
> > > > > > > delayed in
> > > > > > > > > the
> > > > > > > > > > > > > > > network, or arrive out of order. When this
> > > happens, we
> > > > > > > will
> > > > > > > > > use
> > > > > > > > > > > the
> > > > > > > > > > > > > start
> > > > > > > > > > > > > > > time specified in the request to determine if
> the
> > > > > request
> > > > > > > is
> > > > > > > > > stale.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on the
> > > > > controller
> > > > > > > side
> > > > > > > > > > > should
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > reset in the case of controller failover.
> The
> > > > > > > alternative
> > > > > > > > > > > would
> > > > > > > > > > > > > be to
> > > > > > > > > > > > > > > > > track the lease as hard state rather than
> soft
> > > > > state,
> > > > > > > but I
> > > > > > > > > > > think
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > is not really needed, and would result in
> more
> > > log
> > > > > > > entries.
> > > > > > > > > > > > > > > > My interpretation of the mention of
> BrokerRecord
> > > in
> > > > > the
> > > > > > > KIP
> > > > > > > > > was
> > > > > > > > > > > that
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > record exists in the Raft log.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > BrokerRecord does exist in the Raft log, but
> does
> > > not
> > > > > > > include
> > > > > > > > > the
> > > > > > > > > > > last
> > > > > > > > > > > > > > > heartbeat time.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > By soft state, do you mean the broker
> > > > > > > > > > > > > > > > records exist only on the active leader and
> will
> > > not
> > > > > be
> > > > > > > > > > > replicated
> > > > > > > > > > > > > in the
> > > > > > > > > > > > > > > > raft log? If the live brokers list is
> maintained
> > > > > only on
> > > > > > > the
> > > > > > > > > > > active
> > > > > > > > > > > > > > > > controller (raft leader), then, in case of
> leader
> > > > > > > failure,
> > > > > > > > > there
> > > > > > > > > > > > > will be
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > window where the new leader does not know
> about
> > > the
> > > > > live
> > > > > > > > > brokers,
> > > > > > > > > > > > > till
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > brokers establish the leases again.
> > > > > > > > > > > > > > > > I think it will be safer to have leases as a
> hard
> > > > > state
> > > > > > > > > managed
> > > > > > > > > > > by
> > > > > > > > > > > > > > > standard
> > > > > > > > > > > > > > > > Raft replication.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Leases are short, so the need to re-establish
> them
> > > > > after a
> > > > > > > > > > > controller
> > > > > > > > > > > > > > > failover doesn't seem like a big problem. But
> > > this is
> > > > > > > > > something
> > > > > > > > > > > we can
> > > > > > > > > > > > > > > tweak if it becomes an issue. One option
> would be
> > > to
> > > > > have
> > > > > > > a
> > > > > > > > > > > separate
> > > > > > > > > > > > > log
> > > > > > > > > > > > > > > which is only used by the controller nodes for
> this
> > > > > (since,
> > > > > > > > > after
> > > > > > > > > > > all,
> > > > > > > > > > > > > > > brokers don't care about registration
> renewals).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Or am I misunderstanding something? (I assume
> > > that
> > > > > with
> > > > > > > soft
> > > > > > > > > > > state,
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > mean something like zookeeper local sessions
> > > > > > > > > > > > > > > >
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-1147
> > > > > .)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> think
> > > it
> > > > > makes
> > > > > > > > > sense
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > controller, since otherwise locking becomes
> > > very
> > > > > messy.
> > > > > > > > > I'm
> > > > > > > > > > > not
> > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > understand your question about duplicate
> > > broker ID
> > > > > > > > > detection,
> > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > There's a section in the KIP about this -- is
> > > there a
> > > > > > > detail
> > > > > > > > > we
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This is an implementation detail that doesn't
> need
> > > to
> > > > > be
> > > > > > > in the
> > > > > > > > > > > KIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I assumed broker leases are implemented as a
> hard
> > > > > state.
> > > > > > > In
> > > > > > > > > that
> > > > > > > > > > > > > case, to
> > > > > > > > > > > > > > > > check for broker id conflict, we need to
> check
> > > the
> > > > > broker
> > > > > > > > > ids at
> > > > > > > > > > > two
> > > > > > > > > > > > > > > places
> > > > > > > > > > > > > > > > 1. Pending broker registrations (which are
> yet
> > > to be
> > > > > > > > > committed)
> > > > > > > > > > > 2.
> > > > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > > committed broker registrations.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Aug 31, 2020 at 5:42 PM Colin McCabe
> <
> > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020, at 01:12, Unmesh
> Joshi
> > > wrote:
> > > > > > > > > > > > > > > > > > >>>Can you repeat your questions about
> broker
> > > > > leases?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >>>>The LeaseStartTimeMs is expected to
> be
> > > the
> > > > > > > broker's
> > > > > > > > > > > > > > > > > > 'System.currentTimeMillis()' at the
> point of
> > > the
> > > > > > > > > request. The
> > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > controller will add its lease period to
> this
> > > in
> > > > > order
> > > > > > > > > >>>>to
> > > > > > > > > > > > > compute
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > LeaseEndTimeMs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think the use of LeaseStartTimeMs and
> > > > > > > LeaseEndTimeMs
> > > > > > > > > in the
> > > > > > > > > > > > > KIP is
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > confusing. Monotonic Clock
> > > (System.nanoTime) on
> > > > > the
> > > > > > > > > active
> > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > should be used to track leases.
> > > > > > > > > > > > > > > > > > (For example,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://issues.apache.org/jira/browse/ZOOKEEPER-1616https://github.com/etcd-io/etcd/pull/6888/commits/e7f4010ccaf28b6ce64fe514d25a4b2fa459d114
> > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Then we will not need LeaseStartTimeMs?
> > > > > > > > > > > > > > > > > > Instead of LeaseStartTimeMs, can we call
> it
> > > > > > > LeaseTTL? The
> > > > > > > > > > > active
> > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > can then calculate LeaseEndTime =
> > > > > System.nanoTime() +
> > > > > > > > > > > LeaseTTL.
> > > > > > > > > > > > > > > > > > In this case we might just drop
> > > LeaseEndTimeMs
> > > > > from
> > > > > > > the
> > > > > > > > > > > > > response, as
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > broker already knows about the TTL and
> can
> > > send
> > > > > > > > > heartbeats at
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > fraction
> > > > > > > > > > > > > > > > > > of TTL, say every TTL/4
> milliseconds.(elapsed
> > > > > time
> > > > > > > on the
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > measured
> > > > > > > > > > > > > > > > > > by System.nanoTime)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I agree that the monotonic clock is
> probably a
> > > > > better
> > > > > > > idea
> > > > > > > > > > > here.
> > > > > > > > > > > > > It is
> > > > > > > > > > > > > > > > > good to be robust against wall clock
> changes,
> > > > > although
> > > > > > > I
> > > > > > > > > think
> > > > > > > > > > > a
> > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > which had them might suffer other issues.
> I
> > > will
> > > > > > > change
> > > > > > > > > it to
> > > > > > > > > > > > > specify
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > monotonic clock.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The reason for including LeaseStartTimeMs
> in
> > > the
> > > > > > > request
> > > > > > > > > is to
> > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > the time required to communicate with the
> > > > > controller
> > > > > > > gets
> > > > > > > > > > > included
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > lease time. Since requests can
> potentially be
> > > > > delayed
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > network
> > > > > > > > > > > > > > > for a
> > > > > > > > > > > > > > > > > long time, this is important.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I have a prototype built to demonstrate
> this
> > > as
> > > > > > > > > following:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The Kip631Controller itself depends on a
> > > > > Consensus
> > > > > > > > > module, to
> > > > > > > > > > > > > > > demonstrate
> > > > > > > > > > > > > > > > > > how possible interactions with the
> consensus
> > > > > module
> > > > > > > will
> > > > > > > > > look
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > (The Consensus can be pluggable really,
> > > with an
> > > > > API
> > > > > > > to
> > > > > > > > > allow
> > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > replicated log upto HighWaterMark)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It has an implementation of LeaseTracker
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/LeaderLeaseTracker.scala
> > > > > > > > > > > > > > > > > > to demonstrate LeaseTracker's interaction
> > > with
> > > > > the
> > > > > > > > > consensus
> > > > > > > > > > > > > module.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The implementation has the following
> aspects:
> > > > > > > > > > > > > > > > > > 1. The lease tracking happens only on the
> > > active
> > > > > > > > > controller
> > > > > > > > > > > (raft
> > > > > > > > > > > > > > > > > > leader)
> > > > > > > > > > > > > > > > > > 2. Once the lease expires, it needs to
> > > propose
> > > > > and
> > > > > > > > > commit a
> > > > > > > > > > > > > > > FenceBroker
> > > > > > > > > > > > > > > > > > record for that lease.
> > > > > > > > > > > > > > > > > > 3. In case of active controller failure,
> the
> > > > > lease
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > > tracked by
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > newly raft leader. The new raft leader
> > > starts the
> > > > > > > lease
> > > > > > > > > timer
> > > > > > > > > > > > > again,
> > > > > > > > > > > > > > > (as
> > > > > > > > > > > > > > > > > > implemented in onBecomingLeader method of
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > in effect extending the lease by the time
> > > spent
> > > > > in
> > > > > > > the
> > > > > > > > > leader
> > > > > > > > > > > > > > > election
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > whatever time was elapsed on the old
> leader.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Yes, I agree that the lease timeout on the
> > > > > controller
> > > > > > > side
> > > > > > > > > > > should
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > reset
> > > > > > > > > > > > > > > > > in the case of controller failover. The
> > > > > alternative
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > > to
> > > > > > > > > > > > > track
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > lease as hard state rather than soft state,
> > > but I
> > > > > think
> > > > > > > > > that
> > > > > > > > > > > is not
> > > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > > needed, and would result in more log
> entries.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > There are working tests for this
> > > implementation
> > > > > here.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > > and an end to end test here
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/ProducerConsumerKIP500Test.scala
> > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >>'m not sure what you mean by
> > > "de-duplication
> > > > > of the
> > > > > > > > > > > broker."
> > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > Apologies for using the confusing term
> > > > > > > deduplication. I
> > > > > > > > > meant
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > id
> > > > > > > > > > > > > > > > > > conflict.
> > > > > > > > > > > > > > > > > > As you can see in the prototype
> > > handleRequest of
> > > > > > > > > > > KIP631Controller
> > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
> > > > > > > > > > > > > > > > > >,
> > > > > > > > > > > > > > > > > > the duplicate broker id needs to be
> detected
> > > > > before
> > > > > > > the
> > > > > > > > > > > > > BrokerRecord
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > submitted to the raft module.
> > > > > > > > > > > > > > > > > > Also as implemented in the prototype, the
> > > > > > > > > KIP631Controller is
> > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > > threaded, handling requests one at a
> time.
> > > (an
> > > > > > > example of
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://martinfowler.com/articles/patterns-of-distributed-systems/singular-update-queue.html
> > > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Our code is single threaded as well. I
> think
> > > it
> > > > > makes
> > > > > > > > > sense
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > controller, since otherwise locking becomes
> > > very
> > > > > messy.
> > > > > > > > > I'm
> > > > > > > > > > > not
> > > > > > > > > > > > > sure I
> > > > > > > > > > > > > > > > > understand your question about duplicate
> > > broker ID
> > > > > > > > > detection,
> > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > There's a section in the KIP about this --
> is
> > > > > there a
> > > > > > > > > detail we
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > there?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 10:49 AM Colin
> > > McCabe <
> > > > > > > > > > > colin@cmccabe.xyz
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2020, at 19:36, Unmesh
> > > Joshi
> > > > > wrote:
> > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > There were a few of questions I had..
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for the response.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. Were my comments on the broker
> lease
> > > > > > > > > implementation
> > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > prototype) appropriate and do we
> need to
> > > > > change
> > > > > > > the
> > > > > > > > > KIP
> > > > > > > > > > > > > > > > > > > > description accordingly?.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Can you repeat your questions about
> broker
> > > > > leases?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2. How will broker epochs be
> generated?
> > > I am
> > > > > > > > > assuming it
> > > > > > > > > > > can
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > committed log offset (like zxid?)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There isn't any need to use a log
> offset.
> > > We
> > > > > can
> > > > > > > just
> > > > > > > > > > > look at
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > in-memory hash table and see what the
> > > latest
> > > > > > > number is,
> > > > > > > > > > > and add
> > > > > > > > > > > > > > > one, to
> > > > > > > > > > > > > > > > > > > generate a new broker epoch.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3. How will producer registration
> > > happen? I
> > > > > am
> > > > > > > > > assuming
> > > > > > > > > > > it
> > > > > > > > > > > > > > > should be
> > > > > > > > > > > > > > > > > > > > similar to broker registration, with
> a
> > > > > similar
> > > > > > > way to
> > > > > > > > > > > > > generate
> > > > > > > > > > > > > > > > > producer
> > > > > > > > > > > > > > > > > > > id.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > For the EOS stuff, we will need a few
> new
> > > RPCs
> > > > > to
> > > > > > > the
> > > > > > > > > > > > > controller.
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > we should do that in a follow-on KIP,
> > > though,
> > > > > since
> > > > > > > > > this
> > > > > > > > > > > one is
> > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > pretty big.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 4. Because we expose Raft log to all
> the
> > > > > > > brokers, any
> > > > > > > > > > > > > > > de-duplication
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > broker needs to happen before the
> > > requests
> > > > > are
> > > > > > > > > proposed
> > > > > > > > > > > to
> > > > > > > > > > > > > Raft.
> > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > the controller needs to be single
> > > threaded,
> > > > > and
> > > > > > > > > should do
> > > > > > > > > > > > > > > validation
> > > > > > > > > > > > > > > > > > > > against the in-process or pending
> > > requests
> > > > > and
> > > > > > > the
> > > > > > > > > final
> > > > > > > > > > > > > state. I
> > > > > > > > > > > > > > > > > read a
> > > > > > > > > > > > > > > > > > > > mention of this, in the responses in
> this
> > > > > > > > > thread.Will it
> > > > > > > > > > > be
> > > > > > > > > > > > > > > useful to
> > > > > > > > > > > > > > > > > > > > mention this in the KIP?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I'm not sure what you mean by
> > > "de-duplication
> > > > > of
> > > > > > > the
> > > > > > > > > > > broker."
> > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > give a little more context?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Sat, Aug 29, 2020 at 4:50 AM Colin
> > > McCabe
> > > > > <
> > > > > > > > > > > > > cmccabe@apache.org
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I'm thinking of calling a vote on
> > > KIP-631
> > > > > on
> > > > > > > > > Monday.
> > > > > > > > > > > Let
> > > > > > > > > > > > > me
> > > > > > > > > > > > > > > know
> > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > there's any more comments I should
> > > address
> > > > > > > before I
> > > > > > > > > > > start
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > vote.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > cheers,
> > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020, at 05:39,
> Unmesh
> > > > > Joshi
> > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >>Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > >>Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > Thanks. I will be adding more to
> the
> > > > > > > prototype
> > > > > > > > > and
> > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > happy
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > and collaborate.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > Unmesh
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 11, 2020 at 12:28 AM
> > > Colin
> > > > > > > McCabe <
> > > > > > > > > > > > > > > > > cmccabe@apache.org>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Jose,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > That'a s good point that I
> hadn't
> > > > > > > considered.
> > > > > > > > > It's
> > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > a separate leader change
> message,
> > > as
> > > > > you
> > > > > > > > > mentioned.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks, I'll take a look.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > best,
> > > > > > > > > > > > > > > > > > > > > > > Colin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2020, at 11:56,
> Jose
> > > > > Garcia
> > > > > > > > > Sancio
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > Hi Unmesh,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Very cool prototype!
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Colin,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The KIP proposes a record
> called
> > > > > > > IsrChange
> > > > > > > > > which
> > > > > > > > > > > > > > > includes the
> > > > > > > > > > > > > > > > > > > > > > > > partition, topic, isr,
> leader and
> > > > > leader
> > > > > > > > > epoch.
> > > > > > > > > > > > > During
> > > > > > > > > > > > > > > normal
> > > > > > > > > > > > > > > > > > > > > > > > operation ISR changes do not
> > > result
> > > > > in
> > > > > > > leader
> > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > > > Similarly,
> > > > > > > > > > > > > > > > > > > > > > > > leader changes do not
> necessarily
> > > > > > > involve ISR
> > > > > > > > > > > > > changes.
> > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > implementation that uses ZK
> > > modeled
> > > > > them
> > > > > > > > > together
> > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > > > > > > 1. All of this information is
> > > stored
> > > > > in
> > > > > > > one
> > > > > > > > > > > znode.
> > > > > > > > > > > > > > > > > > > > > > > > 2. ZK's optimistic lock
> requires
> > > > > that you
> > > > > > > > > specify
> > > > > > > > > > > > > the new
> > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > > > > > 3. The change to that znode
> was
> > > being
> > > > > > > > > performed
> > > > > > > > > > > by
> > > > > > > > > > > > > both
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > controller
> > > > > > > > > > > > > > > > > > > > > > > > and the leader.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > None of these reasons are
> true in
> > > > > > > KIP-500.
> > > > > > > > > Have
> > > > > > > > > > > we
> > > > > > > > > > > > > > > considered
> > > > > > > > > > > > > > > > > > > having
> > > > > > > > > > > > > > > > > > > > > > > > two different records? For
> > > example
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1. IsrChange record which
> > > includes
> > > > > topic,
> > > > > > > > > > > partition,
> > > > > > > > > > > > > isr
> > > > > > > > > > > > > > > > > > > > > > > > 2. LeaderChange record which
> > > includes
> > > > > > > topic,
> > > > > > > > > > > > > partition,
> > > > > > > > > > > > > > > > > leader
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > leader epoch.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I suspect that making this
> change
> > > > > will
> > > > > > > also
> > > > > > > > > > > require
> > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > message AlterIsrRequest
> > > introduced in
> > > > > > > > > KIP-497:
> > > > > > > > > > > Add
> > > > > > > > > > > > > > > > > inter-broker
> > > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > alter ISR.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > > -Jose
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>