[DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Gwen Shapira <gs...@cloudera.com>]

Hi Kafka Devs,

While reviewing the patch for KAFKA-1809, we came across two questions
that we are interested in hearing the community out on.

1. This patch changes the Broker class and adds a new class
BrokerEndPoint that behaves like the previous broker.

While technically kafka.cluster.Broker is not part of the public API,
it is returned by javaapi, used with the SimpleConsumer.

Getting replicas from PartitionMetadata will now return BrokerEndPoint
instead of Broker. All method calls remain the same, but since we
return a new type, we break the API.

Note that this breakage does not prevent upgrades - existing
SimpleConsumers will continue working (because we are
wire-compatible).
The only thing that won't work is building SimpleConsumers with
dependency on Kafka versions higher than 0.8.2. Arguably, we don't
want anyone to do it anyway :)

So:
Do we state that the highest release on which SimpleConsumers can
depend is 0.8.2? Or shall we keep Broker as is and create an
UberBroker which will contain multiple brokers as its endpoints?

2.
The KIP suggests "use.new.wire.protocol" configuration to decide which
protocols the brokers will use to talk to each other. The problem is
that after the next upgrade, the wire protocol is no longer new, so
we'll have to reset it to false for the following upgrade, then change
to true again... and upgrading more than a single version will be
impossible.
Bad idea :)

As an alternative, we can have a property for each version and set one
of them to true. Or (simple, I think) have "wire.protocol.version"
property and accept version numbers (0.8.2, 0.8.3, 0.9) as values.

Please share your thoughts :)

Gwen

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Gwen Shapira <gs...@cloudera.com>]

Jun,

I'm not sure we should default wire.protocol.version to the previous
version. This will make fresh installs a bit weird :)
I think we should default to the new version and assume that when I'm
upgrading a broker, I'm re-using an existing configuration file.

This way, if I'm upgrading 0.8.3.0 to 0.9.0.0, the configuration file
already says "wire.protocol.version=0.8.3.0" and I need to bump it post
upgrade.
Fresh install will include 0.9.0.0, so I won't need to bump anything.

The only exception is with 0.8.2.0, where I'll need to add
"wire.protocol.version=0.8.2.0" before upgrading to 0.8.3.0.

Does that make sense?

Regarding the naming, I agree that this parameter only controls the
protocol between brokers (clients control the version of the protocol when
they are involved, on a per-message basis). However,
inter.broker.wire.protocol.version makes it sound like there may be other
types of wire.protocol.version in the future, and I'm pretty sure we want a
single parameter for controlling protocol versions from broker side.
Not a big deal for me either way.

On Wed, Feb 11, 2015 at 9:40 AM, Jun Rao <ju...@confluent.io> wrote:

> +1 for proposed changes in 1 and 2.
>
> 1. The impact is that if someone uses SimpleConsumer and references Broker
> explicitly, the application needs code change to compile with 0.8.3. Since
> SimpleConsumer is not widely used, breaking the API in SimpleConsumer but
> maintaining overall code cleanness seems to be a better tradeoff.
>
> 2. For clarification, the issue is the following. In 0.8.3, we will be
> evolving the wire protocol of UpdateMedataRequest (to send info about
> endpoints for different security protocols). Since this is used in
> intra-cluster communication, we need to do the upgrade in two steps. The
> idea is that in 0.8.3, we will default wire.protocol.version to 0.8.2. When
> upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3. After step
> 1, all brokers will be capable for processing the new protocol in 0.8.3,
> but without actually using it. In step 2, we
> configure wire.protocol.version to 0.8.3 in each broker and do another
> rolling restart. After step 2, all brokers will start using the new
> protocol in 0.8.3. Let's say that in the next release 0.9, we are changing
> the intra-cluster wire protocol again. We will do the similar thing:
> defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can upgrade
> from 0.8.3 to 0.9 in two steps. For people who want to upgrade from 0.8.2
> to 0.9 directly, they will have to configure wire.protocol.version to 0.8.2
> first and then do the two-step upgrade to 0.9.
>
> Gwen,
>
> In KIP2, there is still a reference to use.new.protocol. This needs to be
> removed. Also, would it be better to use
> intra.cluster.wire.protocol.version
> since this only applies to the wire protocol among brokers?
>
> Others,
>
> The patch in KAFKA-1809 is almost ready. It would be good to wrap up the
> discussion on KIP2 soon. So, if you haven't looked at this KIP, please take
> a look and send your comments.
>
> Thanks,
>
> Jun
>
>
> On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
> > Hi Kafka Devs,
> >
> > While reviewing the patch for KAFKA-1809, we came across two questions
> > that we are interested in hearing the community out on.
> >
> > 1. This patch changes the Broker class and adds a new class
> > BrokerEndPoint that behaves like the previous broker.
> >
> > While technically kafka.cluster.Broker is not part of the public API,
> > it is returned by javaapi, used with the SimpleConsumer.
> >
> > Getting replicas from PartitionMetadata will now return BrokerEndPoint
> > instead of Broker. All method calls remain the same, but since we
> > return a new type, we break the API.
> >
> > Note that this breakage does not prevent upgrades - existing
> > SimpleConsumers will continue working (because we are
> > wire-compatible).
> > The only thing that won't work is building SimpleConsumers with
> > dependency on Kafka versions higher than 0.8.2. Arguably, we don't
> > want anyone to do it anyway :)
> >
> > So:
> > Do we state that the highest release on which SimpleConsumers can
> > depend is 0.8.2? Or shall we keep Broker as is and create an
> > UberBroker which will contain multiple brokers as its endpoints?
> >
> > 2.
> > The KIP suggests "use.new.wire.protocol" configuration to decide which
> > protocols the brokers will use to talk to each other. The problem is
> > that after the next upgrade, the wire protocol is no longer new, so
> > we'll have to reset it to false for the following upgrade, then change
> > to true again... and upgrading more than a single version will be
> > impossible.
> > Bad idea :)
> >
> > As an alternative, we can have a property for each version and set one
> > of them to true. Or (simple, I think) have "wire.protocol.version"
> > property and accept version numbers (0.8.2, 0.8.3, 0.9) as values.
> >
> > Please share your thoughts :)
> >
> > Gwen
> >
>

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Jun Rao <ju...@confluent.io>]

For rolling downgrading of the intra-cluster protocol, we mostly just need
to reverse the steps. If none of the brokers have set wire.protocol.version
to the newer version, downgrading can be done by just downgrading the jar,
followed by a rolling restart. Otherwise, downgrading needs to be done in
two steps: (1) Reconfigure wire.protocol.version to an old version and do a
rolling restart of the brokers. After this step, all brokers will be using
the old protocol for intra-cluster communication. (2) Downgrade the jar and
do another round of rolling restart.

Gwen,

Could you document that in the WIP-2 wiki too?

Thanks,

Jun

On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino <tp...@gmail.com> wrote:

> Thanks, Gwen. This looks good to me as far as the wire protocol versioning
> goes. I agree with you on defaulting to the new wire protocol version for
> new installs. I think it will also need to be very clear (to general
> installer of Kafka, and not just developers) in documentation when the wire
> protocol version changes moving forwards, and what the risk/benefit of
> changing to the new version is.
>
> Since a rolling upgrade of the intra-cluster protocol is supported, will a
> rolling downgrade work as well? Should a flaw (bug, security, or otherwise)
> be discovered after upgrade, is it possible to change the
> wire.protocol.version
> back to 0.8.2 and do a rolling bounce?
>
> On the host/port/protocol specification, specifically the ZK config format,
> is it possible to have an un-advertised endpoint? I would see this as
> potentially useful if you wanted to have an endpoint that you are reserving
> for intra-cluster communication, and you would prefer to not have it
> advertised at all. Perhaps it is blocked by a firewall rule or other
> authentication method. This could also allow you to duplicate a security
> protocol type but segregate it on a different port or interface (if it is
> unadvertised, there is no ambiguity to the clients as to which endpoint
> should be selected). I believe I asked about that previously, and I didn't
> track what the final outcome was or even if it was discussed further.
>
>
> -Todd
>
>
> On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
> > Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun. I
> was
> > clearly struggling with this...) and removed the reference to
> > use.new.wire.protocol.
> >
> > On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy <jj...@gmail.com> wrote:
> >
> > > The description that Jun gave for (2) was the detail I was looking for
> > > - Gwen can you update the KIP with that for completeness/clarity?
> > >
> > > I'm +1 as well overall. However, I think it would be good if we also
> > > get an ack from someone who is more experienced on the operations side
> > > (say, Todd) to review especially the upgrade plan.
> > >
> > > On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote:
> > > > +1 for proposed changes in 1 and 2.
> > > >
> > > > 1. The impact is that if someone uses SimpleConsumer and references
> > > Broker
> > > > explicitly, the application needs code change to compile with 0.8.3.
> > > Since
> > > > SimpleConsumer is not widely used, breaking the API in SimpleConsumer
> > but
> > > > maintaining overall code cleanness seems to be a better tradeoff.
> > > >
> > > > 2. For clarification, the issue is the following. In 0.8.3, we will
> be
> > > > evolving the wire protocol of UpdateMedataRequest (to send info about
> > > > endpoints for different security protocols). Since this is used in
> > > > intra-cluster communication, we need to do the upgrade in two steps.
> > The
> > > > idea is that in 0.8.3, we will default wire.protocol.version to
> 0.8.2.
> > > When
> > > > upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3.
> After
> > > step
> > > > 1, all brokers will be capable for processing the new protocol in
> > 0.8.3,
> > > > but without actually using it. In step 2, we
> > > > configure wire.protocol.version to 0.8.3 in each broker and do
> another
> > > > rolling restart. After step 2, all brokers will start using the new
> > > > protocol in 0.8.3. Let's say that in the next release 0.9, we are
> > > changing
> > > > the intra-cluster wire protocol again. We will do the similar thing:
> > > > defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can
> > > upgrade
> > > > from 0.8.3 to 0.9 in two steps. For people who want to upgrade from
> > 0.8.2
> > > > to 0.9 directly, they will have to configure wire.protocol.version to
> > > 0.8.2
> > > > first and then do the two-step upgrade to 0.9.
> > > >
> > > > Gwen,
> > > >
> > > > In KIP2, there is still a reference to use.new.protocol. This needs
> to
> > be
> > > > removed. Also, would it be better to use
> > > intra.cluster.wire.protocol.version
> > > > since this only applies to the wire protocol among brokers?
> > > >
> > > > Others,
> > > >
> > > > The patch in KAFKA-1809 is almost ready. It would be good to wrap up
> > the
> > > > discussion on KIP2 soon. So, if you haven't looked at this KIP,
> please
> > > take
> > > > a look and send your comments.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <gshapira@cloudera.com
> >
> > > wrote:
> > > >
> > > > > Hi Kafka Devs,
> > > > >
> > > > > While reviewing the patch for KAFKA-1809, we came across two
> > questions
> > > > > that we are interested in hearing the community out on.
> > > > >
> > > > > 1. This patch changes the Broker class and adds a new class
> > > > > BrokerEndPoint that behaves like the previous broker.
> > > > >
> > > > > While technically kafka.cluster.Broker is not part of the public
> API,
> > > > > it is returned by javaapi, used with the SimpleConsumer.
> > > > >
> > > > > Getting replicas from PartitionMetadata will now return
> > BrokerEndPoint
> > > > > instead of Broker. All method calls remain the same, but since we
> > > > > return a new type, we break the API.
> > > > >
> > > > > Note that this breakage does not prevent upgrades - existing
> > > > > SimpleConsumers will continue working (because we are
> > > > > wire-compatible).
> > > > > The only thing that won't work is building SimpleConsumers with
> > > > > dependency on Kafka versions higher than 0.8.2. Arguably, we don't
> > > > > want anyone to do it anyway :)
> > > > >
> > > > > So:
> > > > > Do we state that the highest release on which SimpleConsumers can
> > > > > depend is 0.8.2? Or shall we keep Broker as is and create an
> > > > > UberBroker which will contain multiple brokers as its endpoints?
> > > > >
> > > > > 2.
> > > > > The KIP suggests "use.new.wire.protocol" configuration to decide
> > which
> > > > > protocols the brokers will use to talk to each other. The problem
> is
> > > > > that after the next upgrade, the wire protocol is no longer new, so
> > > > > we'll have to reset it to false for the following upgrade, then
> > change
> > > > > to true again... and upgrading more than a single version will be
> > > > > impossible.
> > > > > Bad idea :)
> > > > >
> > > > > As an alternative, we can have a property for each version and set
> > one
> > > > > of them to true. Or (simple, I think) have "wire.protocol.version"
> > > > > property and accept version numbers (0.8.2, 0.8.3, 0.9) as values.
> > > > >
> > > > > Please share your thoughts :)
> > > > >
> > > > > Gwen
> > > > >
> > >
> > >
> >
>

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Gwen Shapira <gs...@cloudera.com>]

Hi,

1. We have another issue with reserving a port for inter-broker
communication (at least by not advertising the endpoint).
Currently brokers are pretty much "normal" consumers when replicating. Not
exactly, but close. Which mean they also learn about other brokers from the
registration in ZK. If a broker fails to advertise an endpoint, no one will
know about it - neither clients nor other brokers.

If Todd has a good use-case for reserving ports for inter-broker messages
(firewalls?), we can support it as a follow-up JIRA.

2. I updated the "upgrade guide" in the KIP and created KAFKA-1949 to
follow up in the docs.

Gwen

On Thu, Feb 12, 2015 at 11:38 AM, Jun Rao <ju...@confluent.io> wrote:

> Todd,
>
> Could you elaborate on the benefit for having a separate endpoint for
> intra-cluster communication? Is it mainly for giving intra-cluster requests
> a high priority? At this moment, having a separate endpoint just means that
> the socket connection for the intra-cluster communication is handled by a
> separate acceptor thread. The processing of the requests from the network
> and the handling of the requests are still shared by a single thread pool.
> So, if any of the thread pool is exhausted, the intra-cluster requests will
> still be delayed. We can potentially change this model, but this requires
> more work.
>
> An alternative is to just rely on quotas. Intra-cluster requests will be
> exempt from any kind of throttling.
>
> Gwen,
>
> I agree that defaulting wire.protocol.version to the current version is
> probably better. It just means that we need to document the migration path
> for previous versions.
>
> Thanks,
>
> Jun
>
>
> On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino <tp...@gmail.com> wrote:
>
> > Thanks, Gwen. This looks good to me as far as the wire protocol
> versioning
> > goes. I agree with you on defaulting to the new wire protocol version for
> > new installs. I think it will also need to be very clear (to general
> > installer of Kafka, and not just developers) in documentation when the
> wire
> > protocol version changes moving forwards, and what the risk/benefit of
> > changing to the new version is.
> >
> > Since a rolling upgrade of the intra-cluster protocol is supported, will
> a
> > rolling downgrade work as well? Should a flaw (bug, security, or
> otherwise)
> > be discovered after upgrade, is it possible to change the
> > wire.protocol.version
> > back to 0.8.2 and do a rolling bounce?
> >
> > On the host/port/protocol specification, specifically the ZK config
> format,
> > is it possible to have an un-advertised endpoint? I would see this as
> > potentially useful if you wanted to have an endpoint that you are
> reserving
> > for intra-cluster communication, and you would prefer to not have it
> > advertised at all. Perhaps it is blocked by a firewall rule or other
> > authentication method. This could also allow you to duplicate a security
> > protocol type but segregate it on a different port or interface (if it is
> > unadvertised, there is no ambiguity to the clients as to which endpoint
> > should be selected). I believe I asked about that previously, and I
> didn't
> > track what the final outcome was or even if it was discussed further.
> >
> >
> > -Todd
> >
> >
> > On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira <gs...@cloudera.com>
> > wrote:
> >
> > > Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun. I
> > was
> > > clearly struggling with this...) and removed the reference to
> > > use.new.wire.protocol.
> > >
> > > On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy <jj...@gmail.com>
> wrote:
> > >
> > > > The description that Jun gave for (2) was the detail I was looking
> for
> > > > - Gwen can you update the KIP with that for completeness/clarity?
> > > >
> > > > I'm +1 as well overall. However, I think it would be good if we also
> > > > get an ack from someone who is more experienced on the operations
> side
> > > > (say, Todd) to review especially the upgrade plan.
> > > >
> > > > On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote:
> > > > > +1 for proposed changes in 1 and 2.
> > > > >
> > > > > 1. The impact is that if someone uses SimpleConsumer and references
> > > > Broker
> > > > > explicitly, the application needs code change to compile with
> 0.8.3.
> > > > Since
> > > > > SimpleConsumer is not widely used, breaking the API in
> SimpleConsumer
> > > but
> > > > > maintaining overall code cleanness seems to be a better tradeoff.
> > > > >
> > > > > 2. For clarification, the issue is the following. In 0.8.3, we will
> > be
> > > > > evolving the wire protocol of UpdateMedataRequest (to send info
> about
> > > > > endpoints for different security protocols). Since this is used in
> > > > > intra-cluster communication, we need to do the upgrade in two
> steps.
> > > The
> > > > > idea is that in 0.8.3, we will default wire.protocol.version to
> > 0.8.2.
> > > > When
> > > > > upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3.
> > After
> > > > step
> > > > > 1, all brokers will be capable for processing the new protocol in
> > > 0.8.3,
> > > > > but without actually using it. In step 2, we
> > > > > configure wire.protocol.version to 0.8.3 in each broker and do
> > another
> > > > > rolling restart. After step 2, all brokers will start using the new
> > > > > protocol in 0.8.3. Let's say that in the next release 0.9, we are
> > > > changing
> > > > > the intra-cluster wire protocol again. We will do the similar
> thing:
> > > > > defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can
> > > > upgrade
> > > > > from 0.8.3 to 0.9 in two steps. For people who want to upgrade from
> > > 0.8.2
> > > > > to 0.9 directly, they will have to configure wire.protocol.version
> to
> > > > 0.8.2
> > > > > first and then do the two-step upgrade to 0.9.
> > > > >
> > > > > Gwen,
> > > > >
> > > > > In KIP2, there is still a reference to use.new.protocol. This needs
> > to
> > > be
> > > > > removed. Also, would it be better to use
> > > > intra.cluster.wire.protocol.version
> > > > > since this only applies to the wire protocol among brokers?
> > > > >
> > > > > Others,
> > > > >
> > > > > The patch in KAFKA-1809 is almost ready. It would be good to wrap
> up
> > > the
> > > > > discussion on KIP2 soon. So, if you haven't looked at this KIP,
> > please
> > > > take
> > > > > a look and send your comments.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <
> gshapira@cloudera.com
> > >
> > > > wrote:
> > > > >
> > > > > > Hi Kafka Devs,
> > > > > >
> > > > > > While reviewing the patch for KAFKA-1809, we came across two
> > > questions
> > > > > > that we are interested in hearing the community out on.
> > > > > >
> > > > > > 1. This patch changes the Broker class and adds a new class
> > > > > > BrokerEndPoint that behaves like the previous broker.
> > > > > >
> > > > > > While technically kafka.cluster.Broker is not part of the public
> > API,
> > > > > > it is returned by javaapi, used with the SimpleConsumer.
> > > > > >
> > > > > > Getting replicas from PartitionMetadata will now return
> > > BrokerEndPoint
> > > > > > instead of Broker. All method calls remain the same, but since we
> > > > > > return a new type, we break the API.
> > > > > >
> > > > > > Note that this breakage does not prevent upgrades - existing
> > > > > > SimpleConsumers will continue working (because we are
> > > > > > wire-compatible).
> > > > > > The only thing that won't work is building SimpleConsumers with
> > > > > > dependency on Kafka versions higher than 0.8.2. Arguably, we
> don't
> > > > > > want anyone to do it anyway :)
> > > > > >
> > > > > > So:
> > > > > > Do we state that the highest release on which SimpleConsumers can
> > > > > > depend is 0.8.2? Or shall we keep Broker as is and create an
> > > > > > UberBroker which will contain multiple brokers as its endpoints?
> > > > > >
> > > > > > 2.
> > > > > > The KIP suggests "use.new.wire.protocol" configuration to decide
> > > which
> > > > > > protocols the brokers will use to talk to each other. The problem
> > is
> > > > > > that after the next upgrade, the wire protocol is no longer new,
> so
> > > > > > we'll have to reset it to false for the following upgrade, then
> > > change
> > > > > > to true again... and upgrading more than a single version will be
> > > > > > impossible.
> > > > > > Bad idea :)
> > > > > >
> > > > > > As an alternative, we can have a property for each version and
> set
> > > one
> > > > > > of them to true. Or (simple, I think) have
> "wire.protocol.version"
> > > > > > property and accept version numbers (0.8.2, 0.8.3, 0.9) as
> values.
> > > > > >
> > > > > > Please share your thoughts :)
> > > > > >
> > > > > > Gwen
> > > > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Gwen Shapira <gs...@cloudera.com>]

I'll be happy to give the initial design a go, but will probably only get
to it after Strata.

So either wait a bit (there are enough KIPs to review ;) or someone else
can get started.

Gwen

On Thu, Feb 12, 2015 at 6:55 PM, Joel Koshy <jj...@gmail.com> wrote:

> +1 on investigating it further as a separate feature that will improve
> ops significantly (especially since an expert on the operations side
> has described use cases from actual experience).
>
> On Thu, Feb 12, 2015 at 05:47:50PM -0800, Gwen Shapira wrote:
> > I REALLY like the idea of supporting separate network for inter-broker
> > communication (and probably Zookeeper too).
> > I think its actually a pretty typical configuration in clusters, so I'm
> > surprised we didn't think of it before :)
> > Servers arrive with multiple cards specifically for "admin nic" vs.
> > "clients nic" vs "storage nic".
> >
> > That said, I'd like to handle it in a separate patch. First because
> > KAFKA-1809 is big enough already, and second because this really deserve
> > its own requirement-gathering and design.
> >
> > Does that make sense?
> >
> > Gwen
> >
> >
> >
> > On Thu, Feb 12, 2015 at 12:34 PM, Todd Palino <tp...@gmail.com> wrote:
> >
> > > The idea is more about isolating the intra-cluster traffic from the
> normal
> > > clients as much as possible. There's a couple situations we've seen
> where
> > > this would be useful that I can think of immediately:
> > >
> > > 1) Normal operation - just having the intra-cluster traffic on a
> separate
> > > network interface would allow it to not get overwhelmed by something
> like a
> > > bootstrapping client who is saturating the network interface. We see
> this
> > > fairly often, where the replication falls behind because of heavy
> traffic
> > > from one application. We can always adjust the network threads, but
> > > segregating the traffic is the first step.
> > >
> > > 2) Isolation in case of an error - We have had situations, more than
> once,
> > > where we are needing to rebuild a cluster after a catastrophic problem
> and
> > > the clients are causing that process to take too long, or are causing
> > > additional failures. This has mostly come into play with file
> descriptor
> > > limits in the past, but it's certainly not the only situation.
> Constantly
> > > reconnecting clients continue to cause the brokers to fall over while
> we
> > > are trying to recover a down cluster. The only solution was to
> firewall off
> > > all the clients temporarily. This is a great deal more complicated if
> the
> > > brokers and the clients are all operating over the same port.
> > >
> > > Now, that said, quotas can be a partial solution to this. I don't want
> to
> > > jump the gun on that discussion (because it's going to come up
> separately
> > > and in more detail), but it is possible to structure quotas in a way
> that
> > > will allow the intra-cluster replication to continue to function in the
> > > case of high load. That would partially address case 1, but it does
> nothing
> > > for case 2. Additionally, I think it is also desirable to segregate the
> > > traffic even with quotas, so that regardless of the client load, the
> > > cluster itself is able to be healthy.
> > >
> > > -Todd
> > >
> > >
> > > On Thu, Feb 12, 2015 at 11:38 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Todd,
> > > >
> > > > Could you elaborate on the benefit for having a separate endpoint for
> > > > intra-cluster communication? Is it mainly for giving intra-cluster
> > > requests
> > > > a high priority? At this moment, having a separate endpoint just
> means
> > > that
> > > > the socket connection for the intra-cluster communication is handled
> by a
> > > > separate acceptor thread. The processing of the requests from the
> network
> > > > and the handling of the requests are still shared by a single thread
> > > pool.
> > > > So, if any of the thread pool is exhausted, the intra-cluster
> requests
> > > will
> > > > still be delayed. We can potentially change this model, but this
> requires
> > > > more work.
> > > >
> > > > An alternative is to just rely on quotas. Intra-cluster requests
> will be
> > > > exempt from any kind of throttling.
> > > >
> > > > Gwen,
> > > >
> > > > I agree that defaulting wire.protocol.version to the current version
> is
> > > > probably better. It just means that we need to document the migration
> > > path
> > > > for previous versions.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino <tp...@gmail.com>
> wrote:
> > > >
> > > > > Thanks, Gwen. This looks good to me as far as the wire protocol
> > > > versioning
> > > > > goes. I agree with you on defaulting to the new wire protocol
> version
> > > for
> > > > > new installs. I think it will also need to be very clear (to
> general
> > > > > installer of Kafka, and not just developers) in documentation when
> the
> > > > wire
> > > > > protocol version changes moving forwards, and what the
> risk/benefit of
> > > > > changing to the new version is.
> > > > >
> > > > > Since a rolling upgrade of the intra-cluster protocol is supported,
> > > will
> > > > a
> > > > > rolling downgrade work as well? Should a flaw (bug, security, or
> > > > otherwise)
> > > > > be discovered after upgrade, is it possible to change the
> > > > > wire.protocol.version
> > > > > back to 0.8.2 and do a rolling bounce?
> > > > >
> > > > > On the host/port/protocol specification, specifically the ZK config
> > > > format,
> > > > > is it possible to have an un-advertised endpoint? I would see this
> as
> > > > > potentially useful if you wanted to have an endpoint that you are
> > > > reserving
> > > > > for intra-cluster communication, and you would prefer to not have
> it
> > > > > advertised at all. Perhaps it is blocked by a firewall rule or
> other
> > > > > authentication method. This could also allow you to duplicate a
> > > security
> > > > > protocol type but segregate it on a different port or interface
> (if it
> > > is
> > > > > unadvertised, there is no ambiguity to the clients as to which
> endpoint
> > > > > should be selected). I believe I asked about that previously, and I
> > > > didn't
> > > > > track what the final outcome was or even if it was discussed
> further.
> > > > >
> > > > >
> > > > > -Todd
> > > > >
> > > > >
> > > > > On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira <
> gshapira@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > Added Jun's notes to the KIP (Thanks for explaining so clearly,
> Jun.
> > > I
> > > > > was
> > > > > > clearly struggling with this...) and removed the reference to
> > > > > > use.new.wire.protocol.
> > > > > >
> > > > > > On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy <jjkoshy.w@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > The description that Jun gave for (2) was the detail I was
> looking
> > > > for
> > > > > > > - Gwen can you update the KIP with that for
> completeness/clarity?
> > > > > > >
> > > > > > > I'm +1 as well overall. However, I think it would be good if we
> > > also
> > > > > > > get an ack from someone who is more experienced on the
> operations
> > > > side
> > > > > > > (say, Todd) to review especially the upgrade plan.
> > > > > > >
> > > > > > > On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote:
> > > > > > > > +1 for proposed changes in 1 and 2.
> > > > > > > >
> > > > > > > > 1. The impact is that if someone uses SimpleConsumer and
> > > references
> > > > > > > Broker
> > > > > > > > explicitly, the application needs code change to compile with
> > > > 0.8.3.
> > > > > > > Since
> > > > > > > > SimpleConsumer is not widely used, breaking the API in
> > > > SimpleConsumer
> > > > > > but
> > > > > > > > maintaining overall code cleanness seems to be a better
> tradeoff.
> > > > > > > >
> > > > > > > > 2. For clarification, the issue is the following. In 0.8.3,
> we
> > > will
> > > > > be
> > > > > > > > evolving the wire protocol of UpdateMedataRequest (to send
> info
> > > > about
> > > > > > > > endpoints for different security protocols). Since this is
> used
> > > in
> > > > > > > > intra-cluster communication, we need to do the upgrade in two
> > > > steps.
> > > > > > The
> > > > > > > > idea is that in 0.8.3, we will default wire.protocol.version
> to
> > > > > 0.8.2.
> > > > > > > When
> > > > > > > > upgrading to 0.8.3, in step 1, we do a rolling upgrade to
> 0.8.3.
> > > > > After
> > > > > > > step
> > > > > > > > 1, all brokers will be capable for processing the new
> protocol in
> > > > > > 0.8.3,
> > > > > > > > but without actually using it. In step 2, we
> > > > > > > > configure wire.protocol.version to 0.8.3 in each broker and
> do
> > > > > another
> > > > > > > > rolling restart. After step 2, all brokers will start using
> the
> > > new
> > > > > > > > protocol in 0.8.3. Let's say that in the next release 0.9,
> we are
> > > > > > > changing
> > > > > > > > the intra-cluster wire protocol again. We will do the similar
> > > > thing:
> > > > > > > > defaulting wire.protocol.version to 0.8.3 in 0.9 so that
> people
> > > can
> > > > > > > upgrade
> > > > > > > > from 0.8.3 to 0.9 in two steps. For people who want to
> upgrade
> > > from
> > > > > > 0.8.2
> > > > > > > > to 0.9 directly, they will have to configure
> > > wire.protocol.version
> > > > to
> > > > > > > 0.8.2
> > > > > > > > first and then do the two-step upgrade to 0.9.
> > > > > > > >
> > > > > > > > Gwen,
> > > > > > > >
> > > > > > > > In KIP2, there is still a reference to use.new.protocol. This
> > > needs
> > > > > to
> > > > > > be
> > > > > > > > removed. Also, would it be better to use
> > > > > > > intra.cluster.wire.protocol.version
> > > > > > > > since this only applies to the wire protocol among brokers?
> > > > > > > >
> > > > > > > > Others,
> > > > > > > >
> > > > > > > > The patch in KAFKA-1809 is almost ready. It would be good to
> wrap
> > > > up
> > > > > > the
> > > > > > > > discussion on KIP2 soon. So, if you haven't looked at this
> KIP,
> > > > > please
> > > > > > > take
> > > > > > > > a look and send your comments.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <
> > > > gshapira@cloudera.com
> > > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Kafka Devs,
> > > > > > > > >
> > > > > > > > > While reviewing the patch for KAFKA-1809, we came across
> two
> > > > > > questions
> > > > > > > > > that we are interested in hearing the community out on.
> > > > > > > > >
> > > > > > > > > 1. This patch changes the Broker class and adds a new class
> > > > > > > > > BrokerEndPoint that behaves like the previous broker.
> > > > > > > > >
> > > > > > > > > While technically kafka.cluster.Broker is not part of the
> > > public
> > > > > API,
> > > > > > > > > it is returned by javaapi, used with the SimpleConsumer.
> > > > > > > > >
> > > > > > > > > Getting replicas from PartitionMetadata will now return
> > > > > > BrokerEndPoint
> > > > > > > > > instead of Broker. All method calls remain the same, but
> since
> > > we
> > > > > > > > > return a new type, we break the API.
> > > > > > > > >
> > > > > > > > > Note that this breakage does not prevent upgrades -
> existing
> > > > > > > > > SimpleConsumers will continue working (because we are
> > > > > > > > > wire-compatible).
> > > > > > > > > The only thing that won't work is building SimpleConsumers
> with
> > > > > > > > > dependency on Kafka versions higher than 0.8.2. Arguably,
> we
> > > > don't
> > > > > > > > > want anyone to do it anyway :)
> > > > > > > > >
> > > > > > > > > So:
> > > > > > > > > Do we state that the highest release on which
> SimpleConsumers
> > > can
> > > > > > > > > depend is 0.8.2? Or shall we keep Broker as is and create
> an
> > > > > > > > > UberBroker which will contain multiple brokers as its
> > > endpoints?
> > > > > > > > >
> > > > > > > > > 2.
> > > > > > > > > The KIP suggests "use.new.wire.protocol" configuration to
> > > decide
> > > > > > which
> > > > > > > > > protocols the brokers will use to talk to each other. The
> > > problem
> > > > > is
> > > > > > > > > that after the next upgrade, the wire protocol is no longer
> > > new,
> > > > so
> > > > > > > > > we'll have to reset it to false for the following upgrade,
> then
> > > > > > change
> > > > > > > > > to true again... and upgrading more than a single version
> will
> > > be
> > > > > > > > > impossible.
> > > > > > > > > Bad idea :)
> > > > > > > > >
> > > > > > > > > As an alternative, we can have a property for each version
> and
> > > > set
> > > > > > one
> > > > > > > > > of them to true. Or (simple, I think) have
> > > > "wire.protocol.version"
> > > > > > > > > property and accept version numbers (0.8.2, 0.8.3, 0.9) as
> > > > values.
> > > > > > > > >
> > > > > > > > > Please share your thoughts :)
> > > > > > > > >
> > > > > > > > > Gwen
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
>

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Joel Koshy <jj...@gmail.com>]

+1 on investigating it further as a separate feature that will improve
ops significantly (especially since an expert on the operations side
has described use cases from actual experience).

On Thu, Feb 12, 2015 at 05:47:50PM -0800, Gwen Shapira wrote:
> I REALLY like the idea of supporting separate network for inter-broker
> communication (and probably Zookeeper too).
> I think its actually a pretty typical configuration in clusters, so I'm
> surprised we didn't think of it before :)
> Servers arrive with multiple cards specifically for "admin nic" vs.
> "clients nic" vs "storage nic".
> 
> That said, I'd like to handle it in a separate patch. First because
> KAFKA-1809 is big enough already, and second because this really deserve
> its own requirement-gathering and design.
> 
> Does that make sense?
> 
> Gwen
> 
> 
> 
> On Thu, Feb 12, 2015 at 12:34 PM, Todd Palino <tp...@gmail.com> wrote:
> 
> > The idea is more about isolating the intra-cluster traffic from the normal
> > clients as much as possible. There's a couple situations we've seen where
> > this would be useful that I can think of immediately:
> >
> > 1) Normal operation - just having the intra-cluster traffic on a separate
> > network interface would allow it to not get overwhelmed by something like a
> > bootstrapping client who is saturating the network interface. We see this
> > fairly often, where the replication falls behind because of heavy traffic
> > from one application. We can always adjust the network threads, but
> > segregating the traffic is the first step.
> >
> > 2) Isolation in case of an error - We have had situations, more than once,
> > where we are needing to rebuild a cluster after a catastrophic problem and
> > the clients are causing that process to take too long, or are causing
> > additional failures. This has mostly come into play with file descriptor
> > limits in the past, but it's certainly not the only situation. Constantly
> > reconnecting clients continue to cause the brokers to fall over while we
> > are trying to recover a down cluster. The only solution was to firewall off
> > all the clients temporarily. This is a great deal more complicated if the
> > brokers and the clients are all operating over the same port.
> >
> > Now, that said, quotas can be a partial solution to this. I don't want to
> > jump the gun on that discussion (because it's going to come up separately
> > and in more detail), but it is possible to structure quotas in a way that
> > will allow the intra-cluster replication to continue to function in the
> > case of high load. That would partially address case 1, but it does nothing
> > for case 2. Additionally, I think it is also desirable to segregate the
> > traffic even with quotas, so that regardless of the client load, the
> > cluster itself is able to be healthy.
> >
> > -Todd
> >
> >
> > On Thu, Feb 12, 2015 at 11:38 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Todd,
> > >
> > > Could you elaborate on the benefit for having a separate endpoint for
> > > intra-cluster communication? Is it mainly for giving intra-cluster
> > requests
> > > a high priority? At this moment, having a separate endpoint just means
> > that
> > > the socket connection for the intra-cluster communication is handled by a
> > > separate acceptor thread. The processing of the requests from the network
> > > and the handling of the requests are still shared by a single thread
> > pool.
> > > So, if any of the thread pool is exhausted, the intra-cluster requests
> > will
> > > still be delayed. We can potentially change this model, but this requires
> > > more work.
> > >
> > > An alternative is to just rely on quotas. Intra-cluster requests will be
> > > exempt from any kind of throttling.
> > >
> > > Gwen,
> > >
> > > I agree that defaulting wire.protocol.version to the current version is
> > > probably better. It just means that we need to document the migration
> > path
> > > for previous versions.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino <tp...@gmail.com> wrote:
> > >
> > > > Thanks, Gwen. This looks good to me as far as the wire protocol
> > > versioning
> > > > goes. I agree with you on defaulting to the new wire protocol version
> > for
> > > > new installs. I think it will also need to be very clear (to general
> > > > installer of Kafka, and not just developers) in documentation when the
> > > wire
> > > > protocol version changes moving forwards, and what the risk/benefit of
> > > > changing to the new version is.
> > > >
> > > > Since a rolling upgrade of the intra-cluster protocol is supported,
> > will
> > > a
> > > > rolling downgrade work as well? Should a flaw (bug, security, or
> > > otherwise)
> > > > be discovered after upgrade, is it possible to change the
> > > > wire.protocol.version
> > > > back to 0.8.2 and do a rolling bounce?
> > > >
> > > > On the host/port/protocol specification, specifically the ZK config
> > > format,
> > > > is it possible to have an un-advertised endpoint? I would see this as
> > > > potentially useful if you wanted to have an endpoint that you are
> > > reserving
> > > > for intra-cluster communication, and you would prefer to not have it
> > > > advertised at all. Perhaps it is blocked by a firewall rule or other
> > > > authentication method. This could also allow you to duplicate a
> > security
> > > > protocol type but segregate it on a different port or interface (if it
> > is
> > > > unadvertised, there is no ambiguity to the clients as to which endpoint
> > > > should be selected). I believe I asked about that previously, and I
> > > didn't
> > > > track what the final outcome was or even if it was discussed further.
> > > >
> > > >
> > > > -Todd
> > > >
> > > >
> > > > On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira <gs...@cloudera.com>
> > > > wrote:
> > > >
> > > > > Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun.
> > I
> > > > was
> > > > > clearly struggling with this...) and removed the reference to
> > > > > use.new.wire.protocol.
> > > > >
> > > > > On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy <jj...@gmail.com>
> > > wrote:
> > > > >
> > > > > > The description that Jun gave for (2) was the detail I was looking
> > > for
> > > > > > - Gwen can you update the KIP with that for completeness/clarity?
> > > > > >
> > > > > > I'm +1 as well overall. However, I think it would be good if we
> > also
> > > > > > get an ack from someone who is more experienced on the operations
> > > side
> > > > > > (say, Todd) to review especially the upgrade plan.
> > > > > >
> > > > > > On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote:
> > > > > > > +1 for proposed changes in 1 and 2.
> > > > > > >
> > > > > > > 1. The impact is that if someone uses SimpleConsumer and
> > references
> > > > > > Broker
> > > > > > > explicitly, the application needs code change to compile with
> > > 0.8.3.
> > > > > > Since
> > > > > > > SimpleConsumer is not widely used, breaking the API in
> > > SimpleConsumer
> > > > > but
> > > > > > > maintaining overall code cleanness seems to be a better tradeoff.
> > > > > > >
> > > > > > > 2. For clarification, the issue is the following. In 0.8.3, we
> > will
> > > > be
> > > > > > > evolving the wire protocol of UpdateMedataRequest (to send info
> > > about
> > > > > > > endpoints for different security protocols). Since this is used
> > in
> > > > > > > intra-cluster communication, we need to do the upgrade in two
> > > steps.
> > > > > The
> > > > > > > idea is that in 0.8.3, we will default wire.protocol.version to
> > > > 0.8.2.
> > > > > > When
> > > > > > > upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3.
> > > > After
> > > > > > step
> > > > > > > 1, all brokers will be capable for processing the new protocol in
> > > > > 0.8.3,
> > > > > > > but without actually using it. In step 2, we
> > > > > > > configure wire.protocol.version to 0.8.3 in each broker and do
> > > > another
> > > > > > > rolling restart. After step 2, all brokers will start using the
> > new
> > > > > > > protocol in 0.8.3. Let's say that in the next release 0.9, we are
> > > > > > changing
> > > > > > > the intra-cluster wire protocol again. We will do the similar
> > > thing:
> > > > > > > defaulting wire.protocol.version to 0.8.3 in 0.9 so that people
> > can
> > > > > > upgrade
> > > > > > > from 0.8.3 to 0.9 in two steps. For people who want to upgrade
> > from
> > > > > 0.8.2
> > > > > > > to 0.9 directly, they will have to configure
> > wire.protocol.version
> > > to
> > > > > > 0.8.2
> > > > > > > first and then do the two-step upgrade to 0.9.
> > > > > > >
> > > > > > > Gwen,
> > > > > > >
> > > > > > > In KIP2, there is still a reference to use.new.protocol. This
> > needs
> > > > to
> > > > > be
> > > > > > > removed. Also, would it be better to use
> > > > > > intra.cluster.wire.protocol.version
> > > > > > > since this only applies to the wire protocol among brokers?
> > > > > > >
> > > > > > > Others,
> > > > > > >
> > > > > > > The patch in KAFKA-1809 is almost ready. It would be good to wrap
> > > up
> > > > > the
> > > > > > > discussion on KIP2 soon. So, if you haven't looked at this KIP,
> > > > please
> > > > > > take
> > > > > > > a look and send your comments.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <
> > > gshapira@cloudera.com
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Kafka Devs,
> > > > > > > >
> > > > > > > > While reviewing the patch for KAFKA-1809, we came across two
> > > > > questions
> > > > > > > > that we are interested in hearing the community out on.
> > > > > > > >
> > > > > > > > 1. This patch changes the Broker class and adds a new class
> > > > > > > > BrokerEndPoint that behaves like the previous broker.
> > > > > > > >
> > > > > > > > While technically kafka.cluster.Broker is not part of the
> > public
> > > > API,
> > > > > > > > it is returned by javaapi, used with the SimpleConsumer.
> > > > > > > >
> > > > > > > > Getting replicas from PartitionMetadata will now return
> > > > > BrokerEndPoint
> > > > > > > > instead of Broker. All method calls remain the same, but since
> > we
> > > > > > > > return a new type, we break the API.
> > > > > > > >
> > > > > > > > Note that this breakage does not prevent upgrades - existing
> > > > > > > > SimpleConsumers will continue working (because we are
> > > > > > > > wire-compatible).
> > > > > > > > The only thing that won't work is building SimpleConsumers with
> > > > > > > > dependency on Kafka versions higher than 0.8.2. Arguably, we
> > > don't
> > > > > > > > want anyone to do it anyway :)
> > > > > > > >
> > > > > > > > So:
> > > > > > > > Do we state that the highest release on which SimpleConsumers
> > can
> > > > > > > > depend is 0.8.2? Or shall we keep Broker as is and create an
> > > > > > > > UberBroker which will contain multiple brokers as its
> > endpoints?
> > > > > > > >
> > > > > > > > 2.
> > > > > > > > The KIP suggests "use.new.wire.protocol" configuration to
> > decide
> > > > > which
> > > > > > > > protocols the brokers will use to talk to each other. The
> > problem
> > > > is
> > > > > > > > that after the next upgrade, the wire protocol is no longer
> > new,
> > > so
> > > > > > > > we'll have to reset it to false for the following upgrade, then
> > > > > change
> > > > > > > > to true again... and upgrading more than a single version will
> > be
> > > > > > > > impossible.
> > > > > > > > Bad idea :)
> > > > > > > >
> > > > > > > > As an alternative, we can have a property for each version and
> > > set
> > > > > one
> > > > > > > > of them to true. Or (simple, I think) have
> > > "wire.protocol.version"
> > > > > > > > property and accept version numbers (0.8.2, 0.8.3, 0.9) as
> > > values.
> > > > > > > >
> > > > > > > > Please share your thoughts :)
> > > > > > > >
> > > > > > > > Gwen
> > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Todd Palino <tp...@gmail.com>]

Thanks, Gwen. Perfectly reasonable to split it out into a separate ticket,
and I definitely want to hear some other views on this and make sure we
cover the use cases (not just the ones I can think of right now)
appropriately.

-Todd

On Thu, Feb 12, 2015 at 5:47 PM, Gwen Shapira <gs...@cloudera.com> wrote:

> I REALLY like the idea of supporting separate network for inter-broker
> communication (and probably Zookeeper too).
> I think its actually a pretty typical configuration in clusters, so I'm
> surprised we didn't think of it before :)
> Servers arrive with multiple cards specifically for "admin nic" vs.
> "clients nic" vs "storage nic".
>
> That said, I'd like to handle it in a separate patch. First because
> KAFKA-1809 is big enough already, and second because this really deserve
> its own requirement-gathering and design.
>
> Does that make sense?
>
> Gwen
>
>
>
> On Thu, Feb 12, 2015 at 12:34 PM, Todd Palino <tp...@gmail.com> wrote:
>
> > The idea is more about isolating the intra-cluster traffic from the
> normal
> > clients as much as possible. There's a couple situations we've seen where
> > this would be useful that I can think of immediately:
> >
> > 1) Normal operation - just having the intra-cluster traffic on a separate
> > network interface would allow it to not get overwhelmed by something
> like a
> > bootstrapping client who is saturating the network interface. We see this
> > fairly often, where the replication falls behind because of heavy traffic
> > from one application. We can always adjust the network threads, but
> > segregating the traffic is the first step.
> >
> > 2) Isolation in case of an error - We have had situations, more than
> once,
> > where we are needing to rebuild a cluster after a catastrophic problem
> and
> > the clients are causing that process to take too long, or are causing
> > additional failures. This has mostly come into play with file descriptor
> > limits in the past, but it's certainly not the only situation. Constantly
> > reconnecting clients continue to cause the brokers to fall over while we
> > are trying to recover a down cluster. The only solution was to firewall
> off
> > all the clients temporarily. This is a great deal more complicated if the
> > brokers and the clients are all operating over the same port.
> >
> > Now, that said, quotas can be a partial solution to this. I don't want to
> > jump the gun on that discussion (because it's going to come up separately
> > and in more detail), but it is possible to structure quotas in a way that
> > will allow the intra-cluster replication to continue to function in the
> > case of high load. That would partially address case 1, but it does
> nothing
> > for case 2. Additionally, I think it is also desirable to segregate the
> > traffic even with quotas, so that regardless of the client load, the
> > cluster itself is able to be healthy.
> >
> > -Todd
> >
> >
> > On Thu, Feb 12, 2015 at 11:38 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Todd,
> > >
> > > Could you elaborate on the benefit for having a separate endpoint for
> > > intra-cluster communication? Is it mainly for giving intra-cluster
> > requests
> > > a high priority? At this moment, having a separate endpoint just means
> > that
> > > the socket connection for the intra-cluster communication is handled
> by a
> > > separate acceptor thread. The processing of the requests from the
> network
> > > and the handling of the requests are still shared by a single thread
> > pool.
> > > So, if any of the thread pool is exhausted, the intra-cluster requests
> > will
> > > still be delayed. We can potentially change this model, but this
> requires
> > > more work.
> > >
> > > An alternative is to just rely on quotas. Intra-cluster requests will
> be
> > > exempt from any kind of throttling.
> > >
> > > Gwen,
> > >
> > > I agree that defaulting wire.protocol.version to the current version is
> > > probably better. It just means that we need to document the migration
> > path
> > > for previous versions.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino <tp...@gmail.com>
> wrote:
> > >
> > > > Thanks, Gwen. This looks good to me as far as the wire protocol
> > > versioning
> > > > goes. I agree with you on defaulting to the new wire protocol version
> > for
> > > > new installs. I think it will also need to be very clear (to general
> > > > installer of Kafka, and not just developers) in documentation when
> the
> > > wire
> > > > protocol version changes moving forwards, and what the risk/benefit
> of
> > > > changing to the new version is.
> > > >
> > > > Since a rolling upgrade of the intra-cluster protocol is supported,
> > will
> > > a
> > > > rolling downgrade work as well? Should a flaw (bug, security, or
> > > otherwise)
> > > > be discovered after upgrade, is it possible to change the
> > > > wire.protocol.version
> > > > back to 0.8.2 and do a rolling bounce?
> > > >
> > > > On the host/port/protocol specification, specifically the ZK config
> > > format,
> > > > is it possible to have an un-advertised endpoint? I would see this as
> > > > potentially useful if you wanted to have an endpoint that you are
> > > reserving
> > > > for intra-cluster communication, and you would prefer to not have it
> > > > advertised at all. Perhaps it is blocked by a firewall rule or other
> > > > authentication method. This could also allow you to duplicate a
> > security
> > > > protocol type but segregate it on a different port or interface (if
> it
> > is
> > > > unadvertised, there is no ambiguity to the clients as to which
> endpoint
> > > > should be selected). I believe I asked about that previously, and I
> > > didn't
> > > > track what the final outcome was or even if it was discussed further.
> > > >
> > > >
> > > > -Todd
> > > >
> > > >
> > > > On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira <gshapira@cloudera.com
> >
> > > > wrote:
> > > >
> > > > > Added Jun's notes to the KIP (Thanks for explaining so clearly,
> Jun.
> > I
> > > > was
> > > > > clearly struggling with this...) and removed the reference to
> > > > > use.new.wire.protocol.
> > > > >
> > > > > On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy <jj...@gmail.com>
> > > wrote:
> > > > >
> > > > > > The description that Jun gave for (2) was the detail I was
> looking
> > > for
> > > > > > - Gwen can you update the KIP with that for completeness/clarity?
> > > > > >
> > > > > > I'm +1 as well overall. However, I think it would be good if we
> > also
> > > > > > get an ack from someone who is more experienced on the operations
> > > side
> > > > > > (say, Todd) to review especially the upgrade plan.
> > > > > >
> > > > > > On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote:
> > > > > > > +1 for proposed changes in 1 and 2.
> > > > > > >
> > > > > > > 1. The impact is that if someone uses SimpleConsumer and
> > references
> > > > > > Broker
> > > > > > > explicitly, the application needs code change to compile with
> > > 0.8.3.
> > > > > > Since
> > > > > > > SimpleConsumer is not widely used, breaking the API in
> > > SimpleConsumer
> > > > > but
> > > > > > > maintaining overall code cleanness seems to be a better
> tradeoff.
> > > > > > >
> > > > > > > 2. For clarification, the issue is the following. In 0.8.3, we
> > will
> > > > be
> > > > > > > evolving the wire protocol of UpdateMedataRequest (to send info
> > > about
> > > > > > > endpoints for different security protocols). Since this is used
> > in
> > > > > > > intra-cluster communication, we need to do the upgrade in two
> > > steps.
> > > > > The
> > > > > > > idea is that in 0.8.3, we will default wire.protocol.version to
> > > > 0.8.2.
> > > > > > When
> > > > > > > upgrading to 0.8.3, in step 1, we do a rolling upgrade to
> 0.8.3.
> > > > After
> > > > > > step
> > > > > > > 1, all brokers will be capable for processing the new protocol
> in
> > > > > 0.8.3,
> > > > > > > but without actually using it. In step 2, we
> > > > > > > configure wire.protocol.version to 0.8.3 in each broker and do
> > > > another
> > > > > > > rolling restart. After step 2, all brokers will start using the
> > new
> > > > > > > protocol in 0.8.3. Let's say that in the next release 0.9, we
> are
> > > > > > changing
> > > > > > > the intra-cluster wire protocol again. We will do the similar
> > > thing:
> > > > > > > defaulting wire.protocol.version to 0.8.3 in 0.9 so that people
> > can
> > > > > > upgrade
> > > > > > > from 0.8.3 to 0.9 in two steps. For people who want to upgrade
> > from
> > > > > 0.8.2
> > > > > > > to 0.9 directly, they will have to configure
> > wire.protocol.version
> > > to
> > > > > > 0.8.2
> > > > > > > first and then do the two-step upgrade to 0.9.
> > > > > > >
> > > > > > > Gwen,
> > > > > > >
> > > > > > > In KIP2, there is still a reference to use.new.protocol. This
> > needs
> > > > to
> > > > > be
> > > > > > > removed. Also, would it be better to use
> > > > > > intra.cluster.wire.protocol.version
> > > > > > > since this only applies to the wire protocol among brokers?
> > > > > > >
> > > > > > > Others,
> > > > > > >
> > > > > > > The patch in KAFKA-1809 is almost ready. It would be good to
> wrap
> > > up
> > > > > the
> > > > > > > discussion on KIP2 soon. So, if you haven't looked at this KIP,
> > > > please
> > > > > > take
> > > > > > > a look and send your comments.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <
> > > gshapira@cloudera.com
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Kafka Devs,
> > > > > > > >
> > > > > > > > While reviewing the patch for KAFKA-1809, we came across two
> > > > > questions
> > > > > > > > that we are interested in hearing the community out on.
> > > > > > > >
> > > > > > > > 1. This patch changes the Broker class and adds a new class
> > > > > > > > BrokerEndPoint that behaves like the previous broker.
> > > > > > > >
> > > > > > > > While technically kafka.cluster.Broker is not part of the
> > public
> > > > API,
> > > > > > > > it is returned by javaapi, used with the SimpleConsumer.
> > > > > > > >
> > > > > > > > Getting replicas from PartitionMetadata will now return
> > > > > BrokerEndPoint
> > > > > > > > instead of Broker. All method calls remain the same, but
> since
> > we
> > > > > > > > return a new type, we break the API.
> > > > > > > >
> > > > > > > > Note that this breakage does not prevent upgrades - existing
> > > > > > > > SimpleConsumers will continue working (because we are
> > > > > > > > wire-compatible).
> > > > > > > > The only thing that won't work is building SimpleConsumers
> with
> > > > > > > > dependency on Kafka versions higher than 0.8.2. Arguably, we
> > > don't
> > > > > > > > want anyone to do it anyway :)
> > > > > > > >
> > > > > > > > So:
> > > > > > > > Do we state that the highest release on which SimpleConsumers
> > can
> > > > > > > > depend is 0.8.2? Or shall we keep Broker as is and create an
> > > > > > > > UberBroker which will contain multiple brokers as its
> > endpoints?
> > > > > > > >
> > > > > > > > 2.
> > > > > > > > The KIP suggests "use.new.wire.protocol" configuration to
> > decide
> > > > > which
> > > > > > > > protocols the brokers will use to talk to each other. The
> > problem
> > > > is
> > > > > > > > that after the next upgrade, the wire protocol is no longer
> > new,
> > > so
> > > > > > > > we'll have to reset it to false for the following upgrade,
> then
> > > > > change
> > > > > > > > to true again... and upgrading more than a single version
> will
> > be
> > > > > > > > impossible.
> > > > > > > > Bad idea :)
> > > > > > > >
> > > > > > > > As an alternative, we can have a property for each version
> and
> > > set
> > > > > one
> > > > > > > > of them to true. Or (simple, I think) have
> > > "wire.protocol.version"
> > > > > > > > property and accept version numbers (0.8.2, 0.8.3, 0.9) as
> > > values.
> > > > > > > >
> > > > > > > > Please share your thoughts :)
> > > > > > > >
> > > > > > > > Gwen
> > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

+1 on separating the end points for different purposes.

On 2/12/15, 5:47 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:

>I REALLY like the idea of supporting separate network for inter-broker
>communication (and probably Zookeeper too).
>I think its actually a pretty typical configuration in clusters, so I'm
>surprised we didn't think of it before :)
>Servers arrive with multiple cards specifically for "admin nic" vs.
>"clients nic" vs "storage nic".
>
>That said, I'd like to handle it in a separate patch. First because
>KAFKA-1809 is big enough already, and second because this really deserve
>its own requirement-gathering and design.
>
>Does that make sense?
>
>Gwen
>
>
>
>On Thu, Feb 12, 2015 at 12:34 PM, Todd Palino <tp...@gmail.com> wrote:
>
>> The idea is more about isolating the intra-cluster traffic from the
>>normal
>> clients as much as possible. There's a couple situations we've seen
>>where
>> this would be useful that I can think of immediately:
>>
>> 1) Normal operation - just having the intra-cluster traffic on a
>>separate
>> network interface would allow it to not get overwhelmed by something
>>like a
>> bootstrapping client who is saturating the network interface. We see
>>this
>> fairly often, where the replication falls behind because of heavy
>>traffic
>> from one application. We can always adjust the network threads, but
>> segregating the traffic is the first step.
>>
>> 2) Isolation in case of an error - We have had situations, more than
>>once,
>> where we are needing to rebuild a cluster after a catastrophic problem
>>and
>> the clients are causing that process to take too long, or are causing
>> additional failures. This has mostly come into play with file descriptor
>> limits in the past, but it's certainly not the only situation.
>>Constantly
>> reconnecting clients continue to cause the brokers to fall over while we
>> are trying to recover a down cluster. The only solution was to firewall
>>off
>> all the clients temporarily. This is a great deal more complicated if
>>the
>> brokers and the clients are all operating over the same port.
>>
>> Now, that said, quotas can be a partial solution to this. I don't want
>>to
>> jump the gun on that discussion (because it's going to come up
>>separately
>> and in more detail), but it is possible to structure quotas in a way
>>that
>> will allow the intra-cluster replication to continue to function in the
>> case of high load. That would partially address case 1, but it does
>>nothing
>> for case 2. Additionally, I think it is also desirable to segregate the
>> traffic even with quotas, so that regardless of the client load, the
>> cluster itself is able to be healthy.
>>
>> -Todd
>>
>>
>> On Thu, Feb 12, 2015 at 11:38 AM, Jun Rao <ju...@confluent.io> wrote:
>>
>> > Todd,
>> >
>> > Could you elaborate on the benefit for having a separate endpoint for
>> > intra-cluster communication? Is it mainly for giving intra-cluster
>> requests
>> > a high priority? At this moment, having a separate endpoint just means
>> that
>> > the socket connection for the intra-cluster communication is handled
>>by a
>> > separate acceptor thread. The processing of the requests from the
>>network
>> > and the handling of the requests are still shared by a single thread
>> pool.
>> > So, if any of the thread pool is exhausted, the intra-cluster requests
>> will
>> > still be delayed. We can potentially change this model, but this
>>requires
>> > more work.
>> >
>> > An alternative is to just rely on quotas. Intra-cluster requests will
>>be
>> > exempt from any kind of throttling.
>> >
>> > Gwen,
>> >
>> > I agree that defaulting wire.protocol.version to the current version
>>is
>> > probably better. It just means that we need to document the migration
>> path
>> > for previous versions.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> >
>> > On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino <tp...@gmail.com>
>>wrote:
>> >
>> > > Thanks, Gwen. This looks good to me as far as the wire protocol
>> > versioning
>> > > goes. I agree with you on defaulting to the new wire protocol
>>version
>> for
>> > > new installs. I think it will also need to be very clear (to general
>> > > installer of Kafka, and not just developers) in documentation when
>>the
>> > wire
>> > > protocol version changes moving forwards, and what the risk/benefit
>>of
>> > > changing to the new version is.
>> > >
>> > > Since a rolling upgrade of the intra-cluster protocol is supported,
>> will
>> > a
>> > > rolling downgrade work as well? Should a flaw (bug, security, or
>> > otherwise)
>> > > be discovered after upgrade, is it possible to change the
>> > > wire.protocol.version
>> > > back to 0.8.2 and do a rolling bounce?
>> > >
>> > > On the host/port/protocol specification, specifically the ZK config
>> > format,
>> > > is it possible to have an un-advertised endpoint? I would see this
>>as
>> > > potentially useful if you wanted to have an endpoint that you are
>> > reserving
>> > > for intra-cluster communication, and you would prefer to not have it
>> > > advertised at all. Perhaps it is blocked by a firewall rule or other
>> > > authentication method. This could also allow you to duplicate a
>> security
>> > > protocol type but segregate it on a different port or interface (if
>>it
>> is
>> > > unadvertised, there is no ambiguity to the clients as to which
>>endpoint
>> > > should be selected). I believe I asked about that previously, and I
>> > didn't
>> > > track what the final outcome was or even if it was discussed
>>further.
>> > >
>> > >
>> > > -Todd
>> > >
>> > >
>> > > On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira
>><gs...@cloudera.com>
>> > > wrote:
>> > >
>> > > > Added Jun's notes to the KIP (Thanks for explaining so clearly,
>>Jun.
>> I
>> > > was
>> > > > clearly struggling with this...) and removed the reference to
>> > > > use.new.wire.protocol.
>> > > >
>> > > > On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy <jj...@gmail.com>
>> > wrote:
>> > > >
>> > > > > The description that Jun gave for (2) was the detail I was
>>looking
>> > for
>> > > > > - Gwen can you update the KIP with that for
>>completeness/clarity?
>> > > > >
>> > > > > I'm +1 as well overall. However, I think it would be good if we
>> also
>> > > > > get an ack from someone who is more experienced on the
>>operations
>> > side
>> > > > > (say, Todd) to review especially the upgrade plan.
>> > > > >
>> > > > > On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote:
>> > > > > > +1 for proposed changes in 1 and 2.
>> > > > > >
>> > > > > > 1. The impact is that if someone uses SimpleConsumer and
>> references
>> > > > > Broker
>> > > > > > explicitly, the application needs code change to compile with
>> > 0.8.3.
>> > > > > Since
>> > > > > > SimpleConsumer is not widely used, breaking the API in
>> > SimpleConsumer
>> > > > but
>> > > > > > maintaining overall code cleanness seems to be a better
>>tradeoff.
>> > > > > >
>> > > > > > 2. For clarification, the issue is the following. In 0.8.3, we
>> will
>> > > be
>> > > > > > evolving the wire protocol of UpdateMedataRequest (to send
>>info
>> > about
>> > > > > > endpoints for different security protocols). Since this is
>>used
>> in
>> > > > > > intra-cluster communication, we need to do the upgrade in two
>> > steps.
>> > > > The
>> > > > > > idea is that in 0.8.3, we will default wire.protocol.version
>>to
>> > > 0.8.2.
>> > > > > When
>> > > > > > upgrading to 0.8.3, in step 1, we do a rolling upgrade to
>>0.8.3.
>> > > After
>> > > > > step
>> > > > > > 1, all brokers will be capable for processing the new
>>protocol in
>> > > > 0.8.3,
>> > > > > > but without actually using it. In step 2, we
>> > > > > > configure wire.protocol.version to 0.8.3 in each broker and do
>> > > another
>> > > > > > rolling restart. After step 2, all brokers will start using
>>the
>> new
>> > > > > > protocol in 0.8.3. Let's say that in the next release 0.9, we
>>are
>> > > > > changing
>> > > > > > the intra-cluster wire protocol again. We will do the similar
>> > thing:
>> > > > > > defaulting wire.protocol.version to 0.8.3 in 0.9 so that
>>people
>> can
>> > > > > upgrade
>> > > > > > from 0.8.3 to 0.9 in two steps. For people who want to upgrade
>> from
>> > > > 0.8.2
>> > > > > > to 0.9 directly, they will have to configure
>> wire.protocol.version
>> > to
>> > > > > 0.8.2
>> > > > > > first and then do the two-step upgrade to 0.9.
>> > > > > >
>> > > > > > Gwen,
>> > > > > >
>> > > > > > In KIP2, there is still a reference to use.new.protocol. This
>> needs
>> > > to
>> > > > be
>> > > > > > removed. Also, would it be better to use
>> > > > > intra.cluster.wire.protocol.version
>> > > > > > since this only applies to the wire protocol among brokers?
>> > > > > >
>> > > > > > Others,
>> > > > > >
>> > > > > > The patch in KAFKA-1809 is almost ready. It would be good to
>>wrap
>> > up
>> > > > the
>> > > > > > discussion on KIP2 soon. So, if you haven't looked at this
>>KIP,
>> > > please
>> > > > > take
>> > > > > > a look and send your comments.
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > Jun
>> > > > > >
>> > > > > >
>> > > > > > On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <
>> > gshapira@cloudera.com
>> > > >
>> > > > > wrote:
>> > > > > >
>> > > > > > > Hi Kafka Devs,
>> > > > > > >
>> > > > > > > While reviewing the patch for KAFKA-1809, we came across two
>> > > > questions
>> > > > > > > that we are interested in hearing the community out on.
>> > > > > > >
>> > > > > > > 1. This patch changes the Broker class and adds a new class
>> > > > > > > BrokerEndPoint that behaves like the previous broker.
>> > > > > > >
>> > > > > > > While technically kafka.cluster.Broker is not part of the
>> public
>> > > API,
>> > > > > > > it is returned by javaapi, used with the SimpleConsumer.
>> > > > > > >
>> > > > > > > Getting replicas from PartitionMetadata will now return
>> > > > BrokerEndPoint
>> > > > > > > instead of Broker. All method calls remain the same, but
>>since
>> we
>> > > > > > > return a new type, we break the API.
>> > > > > > >
>> > > > > > > Note that this breakage does not prevent upgrades - existing
>> > > > > > > SimpleConsumers will continue working (because we are
>> > > > > > > wire-compatible).
>> > > > > > > The only thing that won't work is building SimpleConsumers
>>with
>> > > > > > > dependency on Kafka versions higher than 0.8.2. Arguably, we
>> > don't
>> > > > > > > want anyone to do it anyway :)
>> > > > > > >
>> > > > > > > So:
>> > > > > > > Do we state that the highest release on which
>>SimpleConsumers
>> can
>> > > > > > > depend is 0.8.2? Or shall we keep Broker as is and create an
>> > > > > > > UberBroker which will contain multiple brokers as its
>> endpoints?
>> > > > > > >
>> > > > > > > 2.
>> > > > > > > The KIP suggests "use.new.wire.protocol" configuration to
>> decide
>> > > > which
>> > > > > > > protocols the brokers will use to talk to each other. The
>> problem
>> > > is
>> > > > > > > that after the next upgrade, the wire protocol is no longer
>> new,
>> > so
>> > > > > > > we'll have to reset it to false for the following upgrade,
>>then
>> > > > change
>> > > > > > > to true again... and upgrading more than a single version
>>will
>> be
>> > > > > > > impossible.
>> > > > > > > Bad idea :)
>> > > > > > >
>> > > > > > > As an alternative, we can have a property for each version
>>and
>> > set
>> > > > one
>> > > > > > > of them to true. Or (simple, I think) have
>> > "wire.protocol.version"
>> > > > > > > property and accept version numbers (0.8.2, 0.8.3, 0.9) as
>> > values.
>> > > > > > >
>> > > > > > > Please share your thoughts :)
>> > > > > > >
>> > > > > > > Gwen
>> > > > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Gwen Shapira <gs...@cloudera.com>]

I REALLY like the idea of supporting separate network for inter-broker
communication (and probably Zookeeper too).
I think its actually a pretty typical configuration in clusters, so I'm
surprised we didn't think of it before :)
Servers arrive with multiple cards specifically for "admin nic" vs.
"clients nic" vs "storage nic".

That said, I'd like to handle it in a separate patch. First because
KAFKA-1809 is big enough already, and second because this really deserve
its own requirement-gathering and design.

Does that make sense?

Gwen



On Thu, Feb 12, 2015 at 12:34 PM, Todd Palino <tp...@gmail.com> wrote:

> The idea is more about isolating the intra-cluster traffic from the normal
> clients as much as possible. There's a couple situations we've seen where
> this would be useful that I can think of immediately:
>
> 1) Normal operation - just having the intra-cluster traffic on a separate
> network interface would allow it to not get overwhelmed by something like a
> bootstrapping client who is saturating the network interface. We see this
> fairly often, where the replication falls behind because of heavy traffic
> from one application. We can always adjust the network threads, but
> segregating the traffic is the first step.
>
> 2) Isolation in case of an error - We have had situations, more than once,
> where we are needing to rebuild a cluster after a catastrophic problem and
> the clients are causing that process to take too long, or are causing
> additional failures. This has mostly come into play with file descriptor
> limits in the past, but it's certainly not the only situation. Constantly
> reconnecting clients continue to cause the brokers to fall over while we
> are trying to recover a down cluster. The only solution was to firewall off
> all the clients temporarily. This is a great deal more complicated if the
> brokers and the clients are all operating over the same port.
>
> Now, that said, quotas can be a partial solution to this. I don't want to
> jump the gun on that discussion (because it's going to come up separately
> and in more detail), but it is possible to structure quotas in a way that
> will allow the intra-cluster replication to continue to function in the
> case of high load. That would partially address case 1, but it does nothing
> for case 2. Additionally, I think it is also desirable to segregate the
> traffic even with quotas, so that regardless of the client load, the
> cluster itself is able to be healthy.
>
> -Todd
>
>
> On Thu, Feb 12, 2015 at 11:38 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Todd,
> >
> > Could you elaborate on the benefit for having a separate endpoint for
> > intra-cluster communication? Is it mainly for giving intra-cluster
> requests
> > a high priority? At this moment, having a separate endpoint just means
> that
> > the socket connection for the intra-cluster communication is handled by a
> > separate acceptor thread. The processing of the requests from the network
> > and the handling of the requests are still shared by a single thread
> pool.
> > So, if any of the thread pool is exhausted, the intra-cluster requests
> will
> > still be delayed. We can potentially change this model, but this requires
> > more work.
> >
> > An alternative is to just rely on quotas. Intra-cluster requests will be
> > exempt from any kind of throttling.
> >
> > Gwen,
> >
> > I agree that defaulting wire.protocol.version to the current version is
> > probably better. It just means that we need to document the migration
> path
> > for previous versions.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino <tp...@gmail.com> wrote:
> >
> > > Thanks, Gwen. This looks good to me as far as the wire protocol
> > versioning
> > > goes. I agree with you on defaulting to the new wire protocol version
> for
> > > new installs. I think it will also need to be very clear (to general
> > > installer of Kafka, and not just developers) in documentation when the
> > wire
> > > protocol version changes moving forwards, and what the risk/benefit of
> > > changing to the new version is.
> > >
> > > Since a rolling upgrade of the intra-cluster protocol is supported,
> will
> > a
> > > rolling downgrade work as well? Should a flaw (bug, security, or
> > otherwise)
> > > be discovered after upgrade, is it possible to change the
> > > wire.protocol.version
> > > back to 0.8.2 and do a rolling bounce?
> > >
> > > On the host/port/protocol specification, specifically the ZK config
> > format,
> > > is it possible to have an un-advertised endpoint? I would see this as
> > > potentially useful if you wanted to have an endpoint that you are
> > reserving
> > > for intra-cluster communication, and you would prefer to not have it
> > > advertised at all. Perhaps it is blocked by a firewall rule or other
> > > authentication method. This could also allow you to duplicate a
> security
> > > protocol type but segregate it on a different port or interface (if it
> is
> > > unadvertised, there is no ambiguity to the clients as to which endpoint
> > > should be selected). I believe I asked about that previously, and I
> > didn't
> > > track what the final outcome was or even if it was discussed further.
> > >
> > >
> > > -Todd
> > >
> > >
> > > On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira <gs...@cloudera.com>
> > > wrote:
> > >
> > > > Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun.
> I
> > > was
> > > > clearly struggling with this...) and removed the reference to
> > > > use.new.wire.protocol.
> > > >
> > > > On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy <jj...@gmail.com>
> > wrote:
> > > >
> > > > > The description that Jun gave for (2) was the detail I was looking
> > for
> > > > > - Gwen can you update the KIP with that for completeness/clarity?
> > > > >
> > > > > I'm +1 as well overall. However, I think it would be good if we
> also
> > > > > get an ack from someone who is more experienced on the operations
> > side
> > > > > (say, Todd) to review especially the upgrade plan.
> > > > >
> > > > > On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote:
> > > > > > +1 for proposed changes in 1 and 2.
> > > > > >
> > > > > > 1. The impact is that if someone uses SimpleConsumer and
> references
> > > > > Broker
> > > > > > explicitly, the application needs code change to compile with
> > 0.8.3.
> > > > > Since
> > > > > > SimpleConsumer is not widely used, breaking the API in
> > SimpleConsumer
> > > > but
> > > > > > maintaining overall code cleanness seems to be a better tradeoff.
> > > > > >
> > > > > > 2. For clarification, the issue is the following. In 0.8.3, we
> will
> > > be
> > > > > > evolving the wire protocol of UpdateMedataRequest (to send info
> > about
> > > > > > endpoints for different security protocols). Since this is used
> in
> > > > > > intra-cluster communication, we need to do the upgrade in two
> > steps.
> > > > The
> > > > > > idea is that in 0.8.3, we will default wire.protocol.version to
> > > 0.8.2.
> > > > > When
> > > > > > upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3.
> > > After
> > > > > step
> > > > > > 1, all brokers will be capable for processing the new protocol in
> > > > 0.8.3,
> > > > > > but without actually using it. In step 2, we
> > > > > > configure wire.protocol.version to 0.8.3 in each broker and do
> > > another
> > > > > > rolling restart. After step 2, all brokers will start using the
> new
> > > > > > protocol in 0.8.3. Let's say that in the next release 0.9, we are
> > > > > changing
> > > > > > the intra-cluster wire protocol again. We will do the similar
> > thing:
> > > > > > defaulting wire.protocol.version to 0.8.3 in 0.9 so that people
> can
> > > > > upgrade
> > > > > > from 0.8.3 to 0.9 in two steps. For people who want to upgrade
> from
> > > > 0.8.2
> > > > > > to 0.9 directly, they will have to configure
> wire.protocol.version
> > to
> > > > > 0.8.2
> > > > > > first and then do the two-step upgrade to 0.9.
> > > > > >
> > > > > > Gwen,
> > > > > >
> > > > > > In KIP2, there is still a reference to use.new.protocol. This
> needs
> > > to
> > > > be
> > > > > > removed. Also, would it be better to use
> > > > > intra.cluster.wire.protocol.version
> > > > > > since this only applies to the wire protocol among brokers?
> > > > > >
> > > > > > Others,
> > > > > >
> > > > > > The patch in KAFKA-1809 is almost ready. It would be good to wrap
> > up
> > > > the
> > > > > > discussion on KIP2 soon. So, if you haven't looked at this KIP,
> > > please
> > > > > take
> > > > > > a look and send your comments.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <
> > gshapira@cloudera.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Kafka Devs,
> > > > > > >
> > > > > > > While reviewing the patch for KAFKA-1809, we came across two
> > > > questions
> > > > > > > that we are interested in hearing the community out on.
> > > > > > >
> > > > > > > 1. This patch changes the Broker class and adds a new class
> > > > > > > BrokerEndPoint that behaves like the previous broker.
> > > > > > >
> > > > > > > While technically kafka.cluster.Broker is not part of the
> public
> > > API,
> > > > > > > it is returned by javaapi, used with the SimpleConsumer.
> > > > > > >
> > > > > > > Getting replicas from PartitionMetadata will now return
> > > > BrokerEndPoint
> > > > > > > instead of Broker. All method calls remain the same, but since
> we
> > > > > > > return a new type, we break the API.
> > > > > > >
> > > > > > > Note that this breakage does not prevent upgrades - existing
> > > > > > > SimpleConsumers will continue working (because we are
> > > > > > > wire-compatible).
> > > > > > > The only thing that won't work is building SimpleConsumers with
> > > > > > > dependency on Kafka versions higher than 0.8.2. Arguably, we
> > don't
> > > > > > > want anyone to do it anyway :)
> > > > > > >
> > > > > > > So:
> > > > > > > Do we state that the highest release on which SimpleConsumers
> can
> > > > > > > depend is 0.8.2? Or shall we keep Broker as is and create an
> > > > > > > UberBroker which will contain multiple brokers as its
> endpoints?
> > > > > > >
> > > > > > > 2.
> > > > > > > The KIP suggests "use.new.wire.protocol" configuration to
> decide
> > > > which
> > > > > > > protocols the brokers will use to talk to each other. The
> problem
> > > is
> > > > > > > that after the next upgrade, the wire protocol is no longer
> new,
> > so
> > > > > > > we'll have to reset it to false for the following upgrade, then
> > > > change
> > > > > > > to true again... and upgrading more than a single version will
> be
> > > > > > > impossible.
> > > > > > > Bad idea :)
> > > > > > >
> > > > > > > As an alternative, we can have a property for each version and
> > set
> > > > one
> > > > > > > of them to true. Or (simple, I think) have
> > "wire.protocol.version"
> > > > > > > property and accept version numbers (0.8.2, 0.8.3, 0.9) as
> > values.
> > > > > > >
> > > > > > > Please share your thoughts :)
> > > > > > >
> > > > > > > Gwen
> > > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Todd Palino <tp...@gmail.com>]

The idea is more about isolating the intra-cluster traffic from the normal
clients as much as possible. There's a couple situations we've seen where
this would be useful that I can think of immediately:

1) Normal operation - just having the intra-cluster traffic on a separate
network interface would allow it to not get overwhelmed by something like a
bootstrapping client who is saturating the network interface. We see this
fairly often, where the replication falls behind because of heavy traffic
from one application. We can always adjust the network threads, but
segregating the traffic is the first step.

2) Isolation in case of an error - We have had situations, more than once,
where we are needing to rebuild a cluster after a catastrophic problem and
the clients are causing that process to take too long, or are causing
additional failures. This has mostly come into play with file descriptor
limits in the past, but it's certainly not the only situation. Constantly
reconnecting clients continue to cause the brokers to fall over while we
are trying to recover a down cluster. The only solution was to firewall off
all the clients temporarily. This is a great deal more complicated if the
brokers and the clients are all operating over the same port.

Now, that said, quotas can be a partial solution to this. I don't want to
jump the gun on that discussion (because it's going to come up separately
and in more detail), but it is possible to structure quotas in a way that
will allow the intra-cluster replication to continue to function in the
case of high load. That would partially address case 1, but it does nothing
for case 2. Additionally, I think it is also desirable to segregate the
traffic even with quotas, so that regardless of the client load, the
cluster itself is able to be healthy.

-Todd


On Thu, Feb 12, 2015 at 11:38 AM, Jun Rao <ju...@confluent.io> wrote:

> Todd,
>
> Could you elaborate on the benefit for having a separate endpoint for
> intra-cluster communication? Is it mainly for giving intra-cluster requests
> a high priority? At this moment, having a separate endpoint just means that
> the socket connection for the intra-cluster communication is handled by a
> separate acceptor thread. The processing of the requests from the network
> and the handling of the requests are still shared by a single thread pool.
> So, if any of the thread pool is exhausted, the intra-cluster requests will
> still be delayed. We can potentially change this model, but this requires
> more work.
>
> An alternative is to just rely on quotas. Intra-cluster requests will be
> exempt from any kind of throttling.
>
> Gwen,
>
> I agree that defaulting wire.protocol.version to the current version is
> probably better. It just means that we need to document the migration path
> for previous versions.
>
> Thanks,
>
> Jun
>
>
> On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino <tp...@gmail.com> wrote:
>
> > Thanks, Gwen. This looks good to me as far as the wire protocol
> versioning
> > goes. I agree with you on defaulting to the new wire protocol version for
> > new installs. I think it will also need to be very clear (to general
> > installer of Kafka, and not just developers) in documentation when the
> wire
> > protocol version changes moving forwards, and what the risk/benefit of
> > changing to the new version is.
> >
> > Since a rolling upgrade of the intra-cluster protocol is supported, will
> a
> > rolling downgrade work as well? Should a flaw (bug, security, or
> otherwise)
> > be discovered after upgrade, is it possible to change the
> > wire.protocol.version
> > back to 0.8.2 and do a rolling bounce?
> >
> > On the host/port/protocol specification, specifically the ZK config
> format,
> > is it possible to have an un-advertised endpoint? I would see this as
> > potentially useful if you wanted to have an endpoint that you are
> reserving
> > for intra-cluster communication, and you would prefer to not have it
> > advertised at all. Perhaps it is blocked by a firewall rule or other
> > authentication method. This could also allow you to duplicate a security
> > protocol type but segregate it on a different port or interface (if it is
> > unadvertised, there is no ambiguity to the clients as to which endpoint
> > should be selected). I believe I asked about that previously, and I
> didn't
> > track what the final outcome was or even if it was discussed further.
> >
> >
> > -Todd
> >
> >
> > On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira <gs...@cloudera.com>
> > wrote:
> >
> > > Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun. I
> > was
> > > clearly struggling with this...) and removed the reference to
> > > use.new.wire.protocol.
> > >
> > > On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy <jj...@gmail.com>
> wrote:
> > >
> > > > The description that Jun gave for (2) was the detail I was looking
> for
> > > > - Gwen can you update the KIP with that for completeness/clarity?
> > > >
> > > > I'm +1 as well overall. However, I think it would be good if we also
> > > > get an ack from someone who is more experienced on the operations
> side
> > > > (say, Todd) to review especially the upgrade plan.
> > > >
> > > > On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote:
> > > > > +1 for proposed changes in 1 and 2.
> > > > >
> > > > > 1. The impact is that if someone uses SimpleConsumer and references
> > > > Broker
> > > > > explicitly, the application needs code change to compile with
> 0.8.3.
> > > > Since
> > > > > SimpleConsumer is not widely used, breaking the API in
> SimpleConsumer
> > > but
> > > > > maintaining overall code cleanness seems to be a better tradeoff.
> > > > >
> > > > > 2. For clarification, the issue is the following. In 0.8.3, we will
> > be
> > > > > evolving the wire protocol of UpdateMedataRequest (to send info
> about
> > > > > endpoints for different security protocols). Since this is used in
> > > > > intra-cluster communication, we need to do the upgrade in two
> steps.
> > > The
> > > > > idea is that in 0.8.3, we will default wire.protocol.version to
> > 0.8.2.
> > > > When
> > > > > upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3.
> > After
> > > > step
> > > > > 1, all brokers will be capable for processing the new protocol in
> > > 0.8.3,
> > > > > but without actually using it. In step 2, we
> > > > > configure wire.protocol.version to 0.8.3 in each broker and do
> > another
> > > > > rolling restart. After step 2, all brokers will start using the new
> > > > > protocol in 0.8.3. Let's say that in the next release 0.9, we are
> > > > changing
> > > > > the intra-cluster wire protocol again. We will do the similar
> thing:
> > > > > defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can
> > > > upgrade
> > > > > from 0.8.3 to 0.9 in two steps. For people who want to upgrade from
> > > 0.8.2
> > > > > to 0.9 directly, they will have to configure wire.protocol.version
> to
> > > > 0.8.2
> > > > > first and then do the two-step upgrade to 0.9.
> > > > >
> > > > > Gwen,
> > > > >
> > > > > In KIP2, there is still a reference to use.new.protocol. This needs
> > to
> > > be
> > > > > removed. Also, would it be better to use
> > > > intra.cluster.wire.protocol.version
> > > > > since this only applies to the wire protocol among brokers?
> > > > >
> > > > > Others,
> > > > >
> > > > > The patch in KAFKA-1809 is almost ready. It would be good to wrap
> up
> > > the
> > > > > discussion on KIP2 soon. So, if you haven't looked at this KIP,
> > please
> > > > take
> > > > > a look and send your comments.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <
> gshapira@cloudera.com
> > >
> > > > wrote:
> > > > >
> > > > > > Hi Kafka Devs,
> > > > > >
> > > > > > While reviewing the patch for KAFKA-1809, we came across two
> > > questions
> > > > > > that we are interested in hearing the community out on.
> > > > > >
> > > > > > 1. This patch changes the Broker class and adds a new class
> > > > > > BrokerEndPoint that behaves like the previous broker.
> > > > > >
> > > > > > While technically kafka.cluster.Broker is not part of the public
> > API,
> > > > > > it is returned by javaapi, used with the SimpleConsumer.
> > > > > >
> > > > > > Getting replicas from PartitionMetadata will now return
> > > BrokerEndPoint
> > > > > > instead of Broker. All method calls remain the same, but since we
> > > > > > return a new type, we break the API.
> > > > > >
> > > > > > Note that this breakage does not prevent upgrades - existing
> > > > > > SimpleConsumers will continue working (because we are
> > > > > > wire-compatible).
> > > > > > The only thing that won't work is building SimpleConsumers with
> > > > > > dependency on Kafka versions higher than 0.8.2. Arguably, we
> don't
> > > > > > want anyone to do it anyway :)
> > > > > >
> > > > > > So:
> > > > > > Do we state that the highest release on which SimpleConsumers can
> > > > > > depend is 0.8.2? Or shall we keep Broker as is and create an
> > > > > > UberBroker which will contain multiple brokers as its endpoints?
> > > > > >
> > > > > > 2.
> > > > > > The KIP suggests "use.new.wire.protocol" configuration to decide
> > > which
> > > > > > protocols the brokers will use to talk to each other. The problem
> > is
> > > > > > that after the next upgrade, the wire protocol is no longer new,
> so
> > > > > > we'll have to reset it to false for the following upgrade, then
> > > change
> > > > > > to true again... and upgrading more than a single version will be
> > > > > > impossible.
> > > > > > Bad idea :)
> > > > > >
> > > > > > As an alternative, we can have a property for each version and
> set
> > > one
> > > > > > of them to true. Or (simple, I think) have
> "wire.protocol.version"
> > > > > > property and accept version numbers (0.8.2, 0.8.3, 0.9) as
> values.
> > > > > >
> > > > > > Please share your thoughts :)
> > > > > >
> > > > > > Gwen
> > > > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Jun Rao <ju...@confluent.io>]

Todd,

Could you elaborate on the benefit for having a separate endpoint for
intra-cluster communication? Is it mainly for giving intra-cluster requests
a high priority? At this moment, having a separate endpoint just means that
the socket connection for the intra-cluster communication is handled by a
separate acceptor thread. The processing of the requests from the network
and the handling of the requests are still shared by a single thread pool.
So, if any of the thread pool is exhausted, the intra-cluster requests will
still be delayed. We can potentially change this model, but this requires
more work.

An alternative is to just rely on quotas. Intra-cluster requests will be
exempt from any kind of throttling.

Gwen,

I agree that defaulting wire.protocol.version to the current version is
probably better. It just means that we need to document the migration path
for previous versions.

Thanks,

Jun


On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino <tp...@gmail.com> wrote:

> Thanks, Gwen. This looks good to me as far as the wire protocol versioning
> goes. I agree with you on defaulting to the new wire protocol version for
> new installs. I think it will also need to be very clear (to general
> installer of Kafka, and not just developers) in documentation when the wire
> protocol version changes moving forwards, and what the risk/benefit of
> changing to the new version is.
>
> Since a rolling upgrade of the intra-cluster protocol is supported, will a
> rolling downgrade work as well? Should a flaw (bug, security, or otherwise)
> be discovered after upgrade, is it possible to change the
> wire.protocol.version
> back to 0.8.2 and do a rolling bounce?
>
> On the host/port/protocol specification, specifically the ZK config format,
> is it possible to have an un-advertised endpoint? I would see this as
> potentially useful if you wanted to have an endpoint that you are reserving
> for intra-cluster communication, and you would prefer to not have it
> advertised at all. Perhaps it is blocked by a firewall rule or other
> authentication method. This could also allow you to duplicate a security
> protocol type but segregate it on a different port or interface (if it is
> unadvertised, there is no ambiguity to the clients as to which endpoint
> should be selected). I believe I asked about that previously, and I didn't
> track what the final outcome was or even if it was discussed further.
>
>
> -Todd
>
>
> On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
> > Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun. I
> was
> > clearly struggling with this...) and removed the reference to
> > use.new.wire.protocol.
> >
> > On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy <jj...@gmail.com> wrote:
> >
> > > The description that Jun gave for (2) was the detail I was looking for
> > > - Gwen can you update the KIP with that for completeness/clarity?
> > >
> > > I'm +1 as well overall. However, I think it would be good if we also
> > > get an ack from someone who is more experienced on the operations side
> > > (say, Todd) to review especially the upgrade plan.
> > >
> > > On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote:
> > > > +1 for proposed changes in 1 and 2.
> > > >
> > > > 1. The impact is that if someone uses SimpleConsumer and references
> > > Broker
> > > > explicitly, the application needs code change to compile with 0.8.3.
> > > Since
> > > > SimpleConsumer is not widely used, breaking the API in SimpleConsumer
> > but
> > > > maintaining overall code cleanness seems to be a better tradeoff.
> > > >
> > > > 2. For clarification, the issue is the following. In 0.8.3, we will
> be
> > > > evolving the wire protocol of UpdateMedataRequest (to send info about
> > > > endpoints for different security protocols). Since this is used in
> > > > intra-cluster communication, we need to do the upgrade in two steps.
> > The
> > > > idea is that in 0.8.3, we will default wire.protocol.version to
> 0.8.2.
> > > When
> > > > upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3.
> After
> > > step
> > > > 1, all brokers will be capable for processing the new protocol in
> > 0.8.3,
> > > > but without actually using it. In step 2, we
> > > > configure wire.protocol.version to 0.8.3 in each broker and do
> another
> > > > rolling restart. After step 2, all brokers will start using the new
> > > > protocol in 0.8.3. Let's say that in the next release 0.9, we are
> > > changing
> > > > the intra-cluster wire protocol again. We will do the similar thing:
> > > > defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can
> > > upgrade
> > > > from 0.8.3 to 0.9 in two steps. For people who want to upgrade from
> > 0.8.2
> > > > to 0.9 directly, they will have to configure wire.protocol.version to
> > > 0.8.2
> > > > first and then do the two-step upgrade to 0.9.
> > > >
> > > > Gwen,
> > > >
> > > > In KIP2, there is still a reference to use.new.protocol. This needs
> to
> > be
> > > > removed. Also, would it be better to use
> > > intra.cluster.wire.protocol.version
> > > > since this only applies to the wire protocol among brokers?
> > > >
> > > > Others,
> > > >
> > > > The patch in KAFKA-1809 is almost ready. It would be good to wrap up
> > the
> > > > discussion on KIP2 soon. So, if you haven't looked at this KIP,
> please
> > > take
> > > > a look and send your comments.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <gshapira@cloudera.com
> >
> > > wrote:
> > > >
> > > > > Hi Kafka Devs,
> > > > >
> > > > > While reviewing the patch for KAFKA-1809, we came across two
> > questions
> > > > > that we are interested in hearing the community out on.
> > > > >
> > > > > 1. This patch changes the Broker class and adds a new class
> > > > > BrokerEndPoint that behaves like the previous broker.
> > > > >
> > > > > While technically kafka.cluster.Broker is not part of the public
> API,
> > > > > it is returned by javaapi, used with the SimpleConsumer.
> > > > >
> > > > > Getting replicas from PartitionMetadata will now return
> > BrokerEndPoint
> > > > > instead of Broker. All method calls remain the same, but since we
> > > > > return a new type, we break the API.
> > > > >
> > > > > Note that this breakage does not prevent upgrades - existing
> > > > > SimpleConsumers will continue working (because we are
> > > > > wire-compatible).
> > > > > The only thing that won't work is building SimpleConsumers with
> > > > > dependency on Kafka versions higher than 0.8.2. Arguably, we don't
> > > > > want anyone to do it anyway :)
> > > > >
> > > > > So:
> > > > > Do we state that the highest release on which SimpleConsumers can
> > > > > depend is 0.8.2? Or shall we keep Broker as is and create an
> > > > > UberBroker which will contain multiple brokers as its endpoints?
> > > > >
> > > > > 2.
> > > > > The KIP suggests "use.new.wire.protocol" configuration to decide
> > which
> > > > > protocols the brokers will use to talk to each other. The problem
> is
> > > > > that after the next upgrade, the wire protocol is no longer new, so
> > > > > we'll have to reset it to false for the following upgrade, then
> > change
> > > > > to true again... and upgrading more than a single version will be
> > > > > impossible.
> > > > > Bad idea :)
> > > > >
> > > > > As an alternative, we can have a property for each version and set
> > one
> > > > > of them to true. Or (simple, I think) have "wire.protocol.version"
> > > > > property and accept version numbers (0.8.2, 0.8.3, 0.9) as values.
> > > > >
> > > > > Please share your thoughts :)
> > > > >
> > > > > Gwen
> > > > >
> > >
> > >
> >
>

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Todd Palino <tp...@gmail.com>]

Thanks, Gwen. This looks good to me as far as the wire protocol versioning
goes. I agree with you on defaulting to the new wire protocol version for
new installs. I think it will also need to be very clear (to general
installer of Kafka, and not just developers) in documentation when the wire
protocol version changes moving forwards, and what the risk/benefit of
changing to the new version is.

Since a rolling upgrade of the intra-cluster protocol is supported, will a
rolling downgrade work as well? Should a flaw (bug, security, or otherwise)
be discovered after upgrade, is it possible to change the wire.protocol.version
back to 0.8.2 and do a rolling bounce?

On the host/port/protocol specification, specifically the ZK config format,
is it possible to have an un-advertised endpoint? I would see this as
potentially useful if you wanted to have an endpoint that you are reserving
for intra-cluster communication, and you would prefer to not have it
advertised at all. Perhaps it is blocked by a firewall rule or other
authentication method. This could also allow you to duplicate a security
protocol type but segregate it on a different port or interface (if it is
unadvertised, there is no ambiguity to the clients as to which endpoint
should be selected). I believe I asked about that previously, and I didn't
track what the final outcome was or even if it was discussed further.


-Todd


On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira <gs...@cloudera.com> wrote:

> Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun. I was
> clearly struggling with this...) and removed the reference to
> use.new.wire.protocol.
>
> On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy <jj...@gmail.com> wrote:
>
> > The description that Jun gave for (2) was the detail I was looking for
> > - Gwen can you update the KIP with that for completeness/clarity?
> >
> > I'm +1 as well overall. However, I think it would be good if we also
> > get an ack from someone who is more experienced on the operations side
> > (say, Todd) to review especially the upgrade plan.
> >
> > On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote:
> > > +1 for proposed changes in 1 and 2.
> > >
> > > 1. The impact is that if someone uses SimpleConsumer and references
> > Broker
> > > explicitly, the application needs code change to compile with 0.8.3.
> > Since
> > > SimpleConsumer is not widely used, breaking the API in SimpleConsumer
> but
> > > maintaining overall code cleanness seems to be a better tradeoff.
> > >
> > > 2. For clarification, the issue is the following. In 0.8.3, we will be
> > > evolving the wire protocol of UpdateMedataRequest (to send info about
> > > endpoints for different security protocols). Since this is used in
> > > intra-cluster communication, we need to do the upgrade in two steps.
> The
> > > idea is that in 0.8.3, we will default wire.protocol.version to 0.8.2.
> > When
> > > upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3. After
> > step
> > > 1, all brokers will be capable for processing the new protocol in
> 0.8.3,
> > > but without actually using it. In step 2, we
> > > configure wire.protocol.version to 0.8.3 in each broker and do another
> > > rolling restart. After step 2, all brokers will start using the new
> > > protocol in 0.8.3. Let's say that in the next release 0.9, we are
> > changing
> > > the intra-cluster wire protocol again. We will do the similar thing:
> > > defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can
> > upgrade
> > > from 0.8.3 to 0.9 in two steps. For people who want to upgrade from
> 0.8.2
> > > to 0.9 directly, they will have to configure wire.protocol.version to
> > 0.8.2
> > > first and then do the two-step upgrade to 0.9.
> > >
> > > Gwen,
> > >
> > > In KIP2, there is still a reference to use.new.protocol. This needs to
> be
> > > removed. Also, would it be better to use
> > intra.cluster.wire.protocol.version
> > > since this only applies to the wire protocol among brokers?
> > >
> > > Others,
> > >
> > > The patch in KAFKA-1809 is almost ready. It would be good to wrap up
> the
> > > discussion on KIP2 soon. So, if you haven't looked at this KIP, please
> > take
> > > a look and send your comments.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <gs...@cloudera.com>
> > wrote:
> > >
> > > > Hi Kafka Devs,
> > > >
> > > > While reviewing the patch for KAFKA-1809, we came across two
> questions
> > > > that we are interested in hearing the community out on.
> > > >
> > > > 1. This patch changes the Broker class and adds a new class
> > > > BrokerEndPoint that behaves like the previous broker.
> > > >
> > > > While technically kafka.cluster.Broker is not part of the public API,
> > > > it is returned by javaapi, used with the SimpleConsumer.
> > > >
> > > > Getting replicas from PartitionMetadata will now return
> BrokerEndPoint
> > > > instead of Broker. All method calls remain the same, but since we
> > > > return a new type, we break the API.
> > > >
> > > > Note that this breakage does not prevent upgrades - existing
> > > > SimpleConsumers will continue working (because we are
> > > > wire-compatible).
> > > > The only thing that won't work is building SimpleConsumers with
> > > > dependency on Kafka versions higher than 0.8.2. Arguably, we don't
> > > > want anyone to do it anyway :)
> > > >
> > > > So:
> > > > Do we state that the highest release on which SimpleConsumers can
> > > > depend is 0.8.2? Or shall we keep Broker as is and create an
> > > > UberBroker which will contain multiple brokers as its endpoints?
> > > >
> > > > 2.
> > > > The KIP suggests "use.new.wire.protocol" configuration to decide
> which
> > > > protocols the brokers will use to talk to each other. The problem is
> > > > that after the next upgrade, the wire protocol is no longer new, so
> > > > we'll have to reset it to false for the following upgrade, then
> change
> > > > to true again... and upgrading more than a single version will be
> > > > impossible.
> > > > Bad idea :)
> > > >
> > > > As an alternative, we can have a property for each version and set
> one
> > > > of them to true. Or (simple, I think) have "wire.protocol.version"
> > > > property and accept version numbers (0.8.2, 0.8.3, 0.9) as values.
> > > >
> > > > Please share your thoughts :)
> > > >
> > > > Gwen
> > > >
> >
> >
>

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Gwen Shapira <gs...@cloudera.com>]

Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun. I was
clearly struggling with this...) and removed the reference to
use.new.wire.protocol.

On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy <jj...@gmail.com> wrote:

> The description that Jun gave for (2) was the detail I was looking for
> - Gwen can you update the KIP with that for completeness/clarity?
>
> I'm +1 as well overall. However, I think it would be good if we also
> get an ack from someone who is more experienced on the operations side
> (say, Todd) to review especially the upgrade plan.
>
> On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote:
> > +1 for proposed changes in 1 and 2.
> >
> > 1. The impact is that if someone uses SimpleConsumer and references
> Broker
> > explicitly, the application needs code change to compile with 0.8.3.
> Since
> > SimpleConsumer is not widely used, breaking the API in SimpleConsumer but
> > maintaining overall code cleanness seems to be a better tradeoff.
> >
> > 2. For clarification, the issue is the following. In 0.8.3, we will be
> > evolving the wire protocol of UpdateMedataRequest (to send info about
> > endpoints for different security protocols). Since this is used in
> > intra-cluster communication, we need to do the upgrade in two steps. The
> > idea is that in 0.8.3, we will default wire.protocol.version to 0.8.2.
> When
> > upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3. After
> step
> > 1, all brokers will be capable for processing the new protocol in 0.8.3,
> > but without actually using it. In step 2, we
> > configure wire.protocol.version to 0.8.3 in each broker and do another
> > rolling restart. After step 2, all brokers will start using the new
> > protocol in 0.8.3. Let's say that in the next release 0.9, we are
> changing
> > the intra-cluster wire protocol again. We will do the similar thing:
> > defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can
> upgrade
> > from 0.8.3 to 0.9 in two steps. For people who want to upgrade from 0.8.2
> > to 0.9 directly, they will have to configure wire.protocol.version to
> 0.8.2
> > first and then do the two-step upgrade to 0.9.
> >
> > Gwen,
> >
> > In KIP2, there is still a reference to use.new.protocol. This needs to be
> > removed. Also, would it be better to use
> intra.cluster.wire.protocol.version
> > since this only applies to the wire protocol among brokers?
> >
> > Others,
> >
> > The patch in KAFKA-1809 is almost ready. It would be good to wrap up the
> > discussion on KIP2 soon. So, if you haven't looked at this KIP, please
> take
> > a look and send your comments.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
> >
> > > Hi Kafka Devs,
> > >
> > > While reviewing the patch for KAFKA-1809, we came across two questions
> > > that we are interested in hearing the community out on.
> > >
> > > 1. This patch changes the Broker class and adds a new class
> > > BrokerEndPoint that behaves like the previous broker.
> > >
> > > While technically kafka.cluster.Broker is not part of the public API,
> > > it is returned by javaapi, used with the SimpleConsumer.
> > >
> > > Getting replicas from PartitionMetadata will now return BrokerEndPoint
> > > instead of Broker. All method calls remain the same, but since we
> > > return a new type, we break the API.
> > >
> > > Note that this breakage does not prevent upgrades - existing
> > > SimpleConsumers will continue working (because we are
> > > wire-compatible).
> > > The only thing that won't work is building SimpleConsumers with
> > > dependency on Kafka versions higher than 0.8.2. Arguably, we don't
> > > want anyone to do it anyway :)
> > >
> > > So:
> > > Do we state that the highest release on which SimpleConsumers can
> > > depend is 0.8.2? Or shall we keep Broker as is and create an
> > > UberBroker which will contain multiple brokers as its endpoints?
> > >
> > > 2.
> > > The KIP suggests "use.new.wire.protocol" configuration to decide which
> > > protocols the brokers will use to talk to each other. The problem is
> > > that after the next upgrade, the wire protocol is no longer new, so
> > > we'll have to reset it to false for the following upgrade, then change
> > > to true again... and upgrading more than a single version will be
> > > impossible.
> > > Bad idea :)
> > >
> > > As an alternative, we can have a property for each version and set one
> > > of them to true. Or (simple, I think) have "wire.protocol.version"
> > > property and accept version numbers (0.8.2, 0.8.3, 0.9) as values.
> > >
> > > Please share your thoughts :)
> > >
> > > Gwen
> > >
>
>

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Joel Koshy <jj...@gmail.com>]

The description that Jun gave for (2) was the detail I was looking for
- Gwen can you update the KIP with that for completeness/clarity?

I'm +1 as well overall. However, I think it would be good if we also
get an ack from someone who is more experienced on the operations side
(say, Todd) to review especially the upgrade plan.

On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote:
> +1 for proposed changes in 1 and 2.
> 
> 1. The impact is that if someone uses SimpleConsumer and references Broker
> explicitly, the application needs code change to compile with 0.8.3. Since
> SimpleConsumer is not widely used, breaking the API in SimpleConsumer but
> maintaining overall code cleanness seems to be a better tradeoff.
> 
> 2. For clarification, the issue is the following. In 0.8.3, we will be
> evolving the wire protocol of UpdateMedataRequest (to send info about
> endpoints for different security protocols). Since this is used in
> intra-cluster communication, we need to do the upgrade in two steps. The
> idea is that in 0.8.3, we will default wire.protocol.version to 0.8.2. When
> upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3. After step
> 1, all brokers will be capable for processing the new protocol in 0.8.3,
> but without actually using it. In step 2, we
> configure wire.protocol.version to 0.8.3 in each broker and do another
> rolling restart. After step 2, all brokers will start using the new
> protocol in 0.8.3. Let's say that in the next release 0.9, we are changing
> the intra-cluster wire protocol again. We will do the similar thing:
> defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can upgrade
> from 0.8.3 to 0.9 in two steps. For people who want to upgrade from 0.8.2
> to 0.9 directly, they will have to configure wire.protocol.version to 0.8.2
> first and then do the two-step upgrade to 0.9.
> 
> Gwen,
> 
> In KIP2, there is still a reference to use.new.protocol. This needs to be
> removed. Also, would it be better to use intra.cluster.wire.protocol.version
> since this only applies to the wire protocol among brokers?
> 
> Others,
> 
> The patch in KAFKA-1809 is almost ready. It would be good to wrap up the
> discussion on KIP2 soon. So, if you haven't looked at this KIP, please take
> a look and send your comments.
> 
> Thanks,
> 
> Jun
> 
> 
> On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> 
> > Hi Kafka Devs,
> >
> > While reviewing the patch for KAFKA-1809, we came across two questions
> > that we are interested in hearing the community out on.
> >
> > 1. This patch changes the Broker class and adds a new class
> > BrokerEndPoint that behaves like the previous broker.
> >
> > While technically kafka.cluster.Broker is not part of the public API,
> > it is returned by javaapi, used with the SimpleConsumer.
> >
> > Getting replicas from PartitionMetadata will now return BrokerEndPoint
> > instead of Broker. All method calls remain the same, but since we
> > return a new type, we break the API.
> >
> > Note that this breakage does not prevent upgrades - existing
> > SimpleConsumers will continue working (because we are
> > wire-compatible).
> > The only thing that won't work is building SimpleConsumers with
> > dependency on Kafka versions higher than 0.8.2. Arguably, we don't
> > want anyone to do it anyway :)
> >
> > So:
> > Do we state that the highest release on which SimpleConsumers can
> > depend is 0.8.2? Or shall we keep Broker as is and create an
> > UberBroker which will contain multiple brokers as its endpoints?
> >
> > 2.
> > The KIP suggests "use.new.wire.protocol" configuration to decide which
> > protocols the brokers will use to talk to each other. The problem is
> > that after the next upgrade, the wire protocol is no longer new, so
> > we'll have to reset it to false for the following upgrade, then change
> > to true again... and upgrading more than a single version will be
> > impossible.
> > Bad idea :)
> >
> > As an alternative, we can have a property for each version and set one
> > of them to true. Or (simple, I think) have "wire.protocol.version"
> > property and accept version numbers (0.8.2, 0.8.3, 0.9) as values.
> >
> > Please share your thoughts :)
> >
> > Gwen
> >

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Jun Rao <ju...@confluent.io>]

+1 for proposed changes in 1 and 2.

1. The impact is that if someone uses SimpleConsumer and references Broker
explicitly, the application needs code change to compile with 0.8.3. Since
SimpleConsumer is not widely used, breaking the API in SimpleConsumer but
maintaining overall code cleanness seems to be a better tradeoff.

2. For clarification, the issue is the following. In 0.8.3, we will be
evolving the wire protocol of UpdateMedataRequest (to send info about
endpoints for different security protocols). Since this is used in
intra-cluster communication, we need to do the upgrade in two steps. The
idea is that in 0.8.3, we will default wire.protocol.version to 0.8.2. When
upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3. After step
1, all brokers will be capable for processing the new protocol in 0.8.3,
but without actually using it. In step 2, we
configure wire.protocol.version to 0.8.3 in each broker and do another
rolling restart. After step 2, all brokers will start using the new
protocol in 0.8.3. Let's say that in the next release 0.9, we are changing
the intra-cluster wire protocol again. We will do the similar thing:
defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can upgrade
from 0.8.3 to 0.9 in two steps. For people who want to upgrade from 0.8.2
to 0.9 directly, they will have to configure wire.protocol.version to 0.8.2
first and then do the two-step upgrade to 0.9.

Gwen,

In KIP2, there is still a reference to use.new.protocol. This needs to be
removed. Also, would it be better to use intra.cluster.wire.protocol.version
since this only applies to the wire protocol among brokers?

Others,

The patch in KAFKA-1809 is almost ready. It would be good to wrap up the
discussion on KIP2 soon. So, if you haven't looked at this KIP, please take
a look and send your comments.

Thanks,

Jun


On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <gs...@cloudera.com> wrote:

> Hi Kafka Devs,
>
> While reviewing the patch for KAFKA-1809, we came across two questions
> that we are interested in hearing the community out on.
>
> 1. This patch changes the Broker class and adds a new class
> BrokerEndPoint that behaves like the previous broker.
>
> While technically kafka.cluster.Broker is not part of the public API,
> it is returned by javaapi, used with the SimpleConsumer.
>
> Getting replicas from PartitionMetadata will now return BrokerEndPoint
> instead of Broker. All method calls remain the same, but since we
> return a new type, we break the API.
>
> Note that this breakage does not prevent upgrades - existing
> SimpleConsumers will continue working (because we are
> wire-compatible).
> The only thing that won't work is building SimpleConsumers with
> dependency on Kafka versions higher than 0.8.2. Arguably, we don't
> want anyone to do it anyway :)
>
> So:
> Do we state that the highest release on which SimpleConsumers can
> depend is 0.8.2? Or shall we keep Broker as is and create an
> UberBroker which will contain multiple brokers as its endpoints?
>
> 2.
> The KIP suggests "use.new.wire.protocol" configuration to decide which
> protocols the brokers will use to talk to each other. The problem is
> that after the next upgrade, the wire protocol is no longer new, so
> we'll have to reset it to false for the following upgrade, then change
> to true again... and upgrading more than a single version will be
> impossible.
> Bad idea :)
>
> As an alternative, we can have a property for each version and set one
> of them to true. Or (simple, I think) have "wire.protocol.version"
> property and accept version numbers (0.8.2, 0.8.3, 0.9) as values.
>
> Please share your thoughts :)
>
> Gwen
>

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Gwen Shapira <gs...@cloudera.com>]

On Mon, Feb 9, 2015 at 5:47 PM, Joel Koshy <jj...@gmail.com> wrote:

> For (1) - +1 especially since the existing clients will keep working.
> For (2) - I'm less clear on the proposal. Can you incorporate it into
> the KIP and/or linked wiki?
>

Added detail on wire.protocol.version to the KIP (under upgrade plan). Let
me know if still unclear.


>
> Also, on the KIP itself, can you clarify what the TRACE protocol is?
> The RB has a brief comment ("plan is to add instrumentation in the
> future") but I'm not sure what that means.
>

Added details in KIP


>
> (I already started looking at an earlier version of your RB couple of
> days ago, but did not finish. I'll look at your latest RB.)
>
>
Thank you!


> Thanks,
>
> Joel
>
> On Wed, Jan 28, 2015 at 10:28:39AM -0800, Gwen Shapira wrote:
> > Bumping :)
> >
> > If there are no objections, I'd like to go with the following:
> >
> > 1. Do not support javaapi (SimpleConsumer) with dependency on versions
> > higher than 0.8.2. Existing clients will keep working.
> > 2. The configuration parameter for upgrades will be
> > inter.broker.protocol.version={0.8.2.0, 0.8.3.0, 0.9.0.0...}
> >
> > Gwen
> >
> > On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
> > > Hi Kafka Devs,
> > >
> > > While reviewing the patch for KAFKA-1809, we came across two questions
> > > that we are interested in hearing the community out on.
> > >
> > > 1. This patch changes the Broker class and adds a new class
> > > BrokerEndPoint that behaves like the previous broker.
> > >
> > > While technically kafka.cluster.Broker is not part of the public API,
> > > it is returned by javaapi, used with the SimpleConsumer.
> > >
> > > Getting replicas from PartitionMetadata will now return BrokerEndPoint
> > > instead of Broker. All method calls remain the same, but since we
> > > return a new type, we break the API.
> > >
> > > Note that this breakage does not prevent upgrades - existing
> > > SimpleConsumers will continue working (because we are
> > > wire-compatible).
> > > The only thing that won't work is building SimpleConsumers with
> > > dependency on Kafka versions higher than 0.8.2. Arguably, we don't
> > > want anyone to do it anyway :)
> > >
> > > So:
> > > Do we state that the highest release on which SimpleConsumers can
> > > depend is 0.8.2? Or shall we keep Broker as is and create an
> > > UberBroker which will contain multiple brokers as its endpoints?
> > >
> > > 2.
> > > The KIP suggests "use.new.wire.protocol" configuration to decide which
> > > protocols the brokers will use to talk to each other. The problem is
> > > that after the next upgrade, the wire protocol is no longer new, so
> > > we'll have to reset it to false for the following upgrade, then change
> > > to true again... and upgrading more than a single version will be
> > > impossible.
> > > Bad idea :)
> > >
> > > As an alternative, we can have a property for each version and set one
> > > of them to true. Or (simple, I think) have "wire.protocol.version"
> > > property and accept version numbers (0.8.2, 0.8.3, 0.9) as values.
> > >
> > > Please share your thoughts :)
> > >
> > > Gwen
>
>

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Joel Koshy <jj...@gmail.com>]

For (1) - +1 especially since the existing clients will keep working.
For (2) - I'm less clear on the proposal. Can you incorporate it into
the KIP and/or linked wiki?

Also, on the KIP itself, can you clarify what the TRACE protocol is?
The RB has a brief comment ("plan is to add instrumentation in the
future") but I'm not sure what that means.

(I already started looking at an earlier version of your RB couple of
days ago, but did not finish. I'll look at your latest RB.)

Thanks,

Joel

On Wed, Jan 28, 2015 at 10:28:39AM -0800, Gwen Shapira wrote:
> Bumping :)
> 
> If there are no objections, I'd like to go with the following:
> 
> 1. Do not support javaapi (SimpleConsumer) with dependency on versions
> higher than 0.8.2. Existing clients will keep working.
> 2. The configuration parameter for upgrades will be
> inter.broker.protocol.version={0.8.2.0, 0.8.3.0, 0.9.0.0...}
> 
> Gwen
> 
> On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> > Hi Kafka Devs,
> >
> > While reviewing the patch for KAFKA-1809, we came across two questions
> > that we are interested in hearing the community out on.
> >
> > 1. This patch changes the Broker class and adds a new class
> > BrokerEndPoint that behaves like the previous broker.
> >
> > While technically kafka.cluster.Broker is not part of the public API,
> > it is returned by javaapi, used with the SimpleConsumer.
> >
> > Getting replicas from PartitionMetadata will now return BrokerEndPoint
> > instead of Broker. All method calls remain the same, but since we
> > return a new type, we break the API.
> >
> > Note that this breakage does not prevent upgrades - existing
> > SimpleConsumers will continue working (because we are
> > wire-compatible).
> > The only thing that won't work is building SimpleConsumers with
> > dependency on Kafka versions higher than 0.8.2. Arguably, we don't
> > want anyone to do it anyway :)
> >
> > So:
> > Do we state that the highest release on which SimpleConsumers can
> > depend is 0.8.2? Or shall we keep Broker as is and create an
> > UberBroker which will contain multiple brokers as its endpoints?
> >
> > 2.
> > The KIP suggests "use.new.wire.protocol" configuration to decide which
> > protocols the brokers will use to talk to each other. The problem is
> > that after the next upgrade, the wire protocol is no longer new, so
> > we'll have to reset it to false for the following upgrade, then change
> > to true again... and upgrading more than a single version will be
> > impossible.
> > Bad idea :)
> >
> > As an alternative, we can have a property for each version and set one
> > of them to true. Or (simple, I think) have "wire.protocol.version"
> > property and accept version numbers (0.8.2, 0.8.3, 0.9) as values.
> >
> > Please share your thoughts :)
> >
> > Gwen

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Gwen Shapira <gs...@cloudera.com>]

Bumping :)

If there are no objections, I'd like to go with the following:

1. Do not support javaapi (SimpleConsumer) with dependency on versions
higher than 0.8.2. Existing clients will keep working.
2. The configuration parameter for upgrades will be
inter.broker.protocol.version={0.8.2.0, 0.8.3.0, 0.9.0.0...}

Gwen

On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> Hi Kafka Devs,
>
> While reviewing the patch for KAFKA-1809, we came across two questions
> that we are interested in hearing the community out on.
>
> 1. This patch changes the Broker class and adds a new class
> BrokerEndPoint that behaves like the previous broker.
>
> While technically kafka.cluster.Broker is not part of the public API,
> it is returned by javaapi, used with the SimpleConsumer.
>
> Getting replicas from PartitionMetadata will now return BrokerEndPoint
> instead of Broker. All method calls remain the same, but since we
> return a new type, we break the API.
>
> Note that this breakage does not prevent upgrades - existing
> SimpleConsumers will continue working (because we are
> wire-compatible).
> The only thing that won't work is building SimpleConsumers with
> dependency on Kafka versions higher than 0.8.2. Arguably, we don't
> want anyone to do it anyway :)
>
> So:
> Do we state that the highest release on which SimpleConsumers can
> depend is 0.8.2? Or shall we keep Broker as is and create an
> UberBroker which will contain multiple brokers as its endpoints?
>
> 2.
> The KIP suggests "use.new.wire.protocol" configuration to decide which
> protocols the brokers will use to talk to each other. The problem is
> that after the next upgrade, the wire protocol is no longer new, so
> we'll have to reset it to false for the following upgrade, then change
> to true again... and upgrading more than a single version will be
> impossible.
> Bad idea :)
>
> As an alternative, we can have a property for each version and set one
> of them to true. Or (simple, I think) have "wire.protocol.version"
> property and accept version numbers (0.8.2, 0.8.3, 0.9) as values.
>
> Please share your thoughts :)
>
> Gwen