Update Kafka to 2.0.0

[Markus Thömmes <ma...@apache.org>]

Dear OpenWhiskers,

James Dubee, Sven Lange-Last and I have been on a Kafka-related bug
hunt throughout the past days and found several issues in our current
KafkaConsumerConnector. Several PRs have been opened since and several
are yet to come.

Currently, I'm trying to make the Kafka 2.0.0 clients work with our
old-dated Kafka 0.11 line Kafka broker. I'm wondering if we should
just update the brokers to 2.0.0 as well. It has been released today
and is admittedly fairly young. I feel like we should start tracking
more recent Kafka versions though, to profit from bug and stability
fixes.

One wrinkle: Today we don't support a graceful update of the Kafka
brokers in-place. Is any of the production deployments based on
OpenWhisk relying on being able to make an in-place update or is a
blue/green style full replacement of the whole cluster fine for
everybody?

Cheers,
Markus

Re: Update Kafka to 2.0.0

[Markus Thömmes <ma...@apache.org>]

Hi Dragos,

Yes, the newer client works even on our current version, got it to
work with SSL finally as well: Check
https://github.com/apache/incubator-openwhisk/pull/3913 for da codez.

Cheers,
Markus
Am Di., 31. Juli 2018 um 03:56 Uhr schrieb Dascalita Dragos
<dd...@gmail.com>:
>
> Hi Markus,
> Big +1 for upgrading.
>
> In my case we use blue/green deployments so that would be fine.
>
> The only potential problem is on the DCOS package for Mesos, for supporting
> the latest Kafka version; i think it should work OOTB, but until it’s
> tested I’m leaving room for unknowns. Would you know if the Kafka client
> library is able to handle both versions ?
>
> Thanks,
> Dragos
> On Mon, Jul 30, 2018 at 12:57 PM Markus Thömmes <ma...@apache.org>
> wrote:
>
> > Dear OpenWhiskers,
> >
> > James Dubee, Sven Lange-Last and I have been on a Kafka-related bug
> > hunt throughout the past days and found several issues in our current
> > KafkaConsumerConnector. Several PRs have been opened since and several
> > are yet to come.
> >
> > Currently, I'm trying to make the Kafka 2.0.0 clients work with our
> > old-dated Kafka 0.11 line Kafka broker. I'm wondering if we should
> > just update the brokers to 2.0.0 as well. It has been released today
> > and is admittedly fairly young. I feel like we should start tracking
> > more recent Kafka versions though, to profit from bug and stability
> > fixes.
> >
> > One wrinkle: Today we don't support a graceful update of the Kafka
> > brokers in-place. Is any of the production deployments based on
> > OpenWhisk relying on being able to make an in-place update or is a
> > blue/green style full replacement of the whole cluster fine for
> > everybody?
> >
> > Cheers,
> > Markus
> >

Re: Update Kafka to 2.0.0

[Dascalita Dragos <dd...@gmail.com>]

Hi Markus,
Big +1 for upgrading.

In my case we use blue/green deployments so that would be fine.

The only potential problem is on the DCOS package for Mesos, for supporting
the latest Kafka version; i think it should work OOTB, but until it’s
tested I’m leaving room for unknowns. Would you know if the Kafka client
library is able to handle both versions ?

Thanks,
Dragos
On Mon, Jul 30, 2018 at 12:57 PM Markus Thömmes <ma...@apache.org>
wrote:

> Dear OpenWhiskers,
>
> James Dubee, Sven Lange-Last and I have been on a Kafka-related bug
> hunt throughout the past days and found several issues in our current
> KafkaConsumerConnector. Several PRs have been opened since and several
> are yet to come.
>
> Currently, I'm trying to make the Kafka 2.0.0 clients work with our
> old-dated Kafka 0.11 line Kafka broker. I'm wondering if we should
> just update the brokers to 2.0.0 as well. It has been released today
> and is admittedly fairly young. I feel like we should start tracking
> more recent Kafka versions though, to profit from bug and stability
> fixes.
>
> One wrinkle: Today we don't support a graceful update of the Kafka
> brokers in-place. Is any of the production deployments based on
> OpenWhisk relying on being able to make an in-place update or is a
> blue/green style full replacement of the whole cluster fine for
> everybody?
>
> Cheers,
> Markus
>