Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Joel Koshy <jj...@gmail.com>]

For (1) - +1 especially since the existing clients will keep working.
For (2) - I'm less clear on the proposal. Can you incorporate it into
the KIP and/or linked wiki?

Also, on the KIP itself, can you clarify what the TRACE protocol is?
The RB has a brief comment ("plan is to add instrumentation in the
future") but I'm not sure what that means.

(I already started looking at an earlier version of your RB couple of
days ago, but did not finish. I'll look at your latest RB.)

Thanks,

Joel

On Wed, Jan 28, 2015 at 10:28:39AM -0800, Gwen Shapira wrote:
> Bumping :)
> 
> If there are no objections, I'd like to go with the following:
> 
> 1. Do not support javaapi (SimpleConsumer) with dependency on versions
> higher than 0.8.2. Existing clients will keep working.
> 2. The configuration parameter for upgrades will be
> inter.broker.protocol.version={0.8.2.0, 0.8.3.0, 0.9.0.0...}
> 
> Gwen
> 
> On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> > Hi Kafka Devs,
> >
> > While reviewing the patch for KAFKA-1809, we came across two questions
> > that we are interested in hearing the community out on.
> >
> > 1. This patch changes the Broker class and adds a new class
> > BrokerEndPoint that behaves like the previous broker.
> >
> > While technically kafka.cluster.Broker is not part of the public API,
> > it is returned by javaapi, used with the SimpleConsumer.
> >
> > Getting replicas from PartitionMetadata will now return BrokerEndPoint
> > instead of Broker. All method calls remain the same, but since we
> > return a new type, we break the API.
> >
> > Note that this breakage does not prevent upgrades - existing
> > SimpleConsumers will continue working (because we are
> > wire-compatible).
> > The only thing that won't work is building SimpleConsumers with
> > dependency on Kafka versions higher than 0.8.2. Arguably, we don't
> > want anyone to do it anyway :)
> >
> > So:
> > Do we state that the highest release on which SimpleConsumers can
> > depend is 0.8.2? Or shall we keep Broker as is and create an
> > UberBroker which will contain multiple brokers as its endpoints?
> >
> > 2.
> > The KIP suggests "use.new.wire.protocol" configuration to decide which
> > protocols the brokers will use to talk to each other. The problem is
> > that after the next upgrade, the wire protocol is no longer new, so
> > we'll have to reset it to false for the following upgrade, then change
> > to true again... and upgrading more than a single version will be
> > impossible.
> > Bad idea :)
> >
> > As an alternative, we can have a property for each version and set one
> > of them to true. Or (simple, I think) have "wire.protocol.version"
> > property and accept version numbers (0.8.2, 0.8.3, 0.9) as values.
> >
> > Please share your thoughts :)
> >
> > Gwen

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

[Gwen Shapira <gs...@cloudera.com>]

On Mon, Feb 9, 2015 at 5:47 PM, Joel Koshy <jj...@gmail.com> wrote:

> For (1) - +1 especially since the existing clients will keep working.
> For (2) - I'm less clear on the proposal. Can you incorporate it into
> the KIP and/or linked wiki?
>

Added detail on wire.protocol.version to the KIP (under upgrade plan). Let
me know if still unclear.


>
> Also, on the KIP itself, can you clarify what the TRACE protocol is?
> The RB has a brief comment ("plan is to add instrumentation in the
> future") but I'm not sure what that means.
>

Added details in KIP


>
> (I already started looking at an earlier version of your RB couple of
> days ago, but did not finish. I'll look at your latest RB.)
>
>
Thank you!


> Thanks,
>
> Joel
>
> On Wed, Jan 28, 2015 at 10:28:39AM -0800, Gwen Shapira wrote:
> > Bumping :)
> >
> > If there are no objections, I'd like to go with the following:
> >
> > 1. Do not support javaapi (SimpleConsumer) with dependency on versions
> > higher than 0.8.2. Existing clients will keep working.
> > 2. The configuration parameter for upgrades will be
> > inter.broker.protocol.version={0.8.2.0, 0.8.3.0, 0.9.0.0...}
> >
> > Gwen
> >
> > On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
> > > Hi Kafka Devs,
> > >
> > > While reviewing the patch for KAFKA-1809, we came across two questions
> > > that we are interested in hearing the community out on.
> > >
> > > 1. This patch changes the Broker class and adds a new class
> > > BrokerEndPoint that behaves like the previous broker.
> > >
> > > While technically kafka.cluster.Broker is not part of the public API,
> > > it is returned by javaapi, used with the SimpleConsumer.
> > >
> > > Getting replicas from PartitionMetadata will now return BrokerEndPoint
> > > instead of Broker. All method calls remain the same, but since we
> > > return a new type, we break the API.
> > >
> > > Note that this breakage does not prevent upgrades - existing
> > > SimpleConsumers will continue working (because we are
> > > wire-compatible).
> > > The only thing that won't work is building SimpleConsumers with
> > > dependency on Kafka versions higher than 0.8.2. Arguably, we don't
> > > want anyone to do it anyway :)
> > >
> > > So:
> > > Do we state that the highest release on which SimpleConsumers can
> > > depend is 0.8.2? Or shall we keep Broker as is and create an
> > > UberBroker which will contain multiple brokers as its endpoints?
> > >
> > > 2.
> > > The KIP suggests "use.new.wire.protocol" configuration to decide which
> > > protocols the brokers will use to talk to each other. The problem is
> > > that after the next upgrade, the wire protocol is no longer new, so
> > > we'll have to reset it to false for the following upgrade, then change
> > > to true again... and upgrading more than a single version will be
> > > impossible.
> > > Bad idea :)
> > >
> > > As an alternative, we can have a property for each version and set one
> > > of them to true. Or (simple, I think) have "wire.protocol.version"
> > > property and accept version numbers (0.8.2, 0.8.3, 0.9) as values.
> > >
> > > Please share your thoughts :)
> > >
> > > Gwen
>
>