Kafka support - with Kubernetes ?

[Jim Langston <jl...@resolutebi.com>]

Hi All,

I’m currently working on our direction with Kafka and whether to move our cluster
into Kubernetes. Flume is an integral part of delivering data to the Kafka cluster.

I’m looking at moving to Kafka 2.2.1 (current release) , and notice that this bug is
applicable to the whole update process ..

https://issues.apache.org/jira/browse/KAFKA-7755


This bug relates to clients that would be affected by Kubernetes needing to handle reconnecting
to brokers where Kubernetes gives them another IP address.

My questions to the community:

What version of Flume do I need to be on ? 1.9 last tested with Kafka 2.0.x

Can I pull in a Kafka 2.2.0 client for Flume ?, where this bug is fixed

Thanks,

Jim

Re: Kafka support - with Kubernetes ?

[Ferenc Szabo <sz...@apache.org>]

Hi Jim,

1.9 Flume should be able to work with Kafka 2.2.0 or any 2.x if Kafka does
not break backward compatibility as they should not in a minor version.
as long as you add the extra "client.dns.lookup" parameter in the flume
configuration as well.

Let us know if any issue comes up.

Thanks,
Ferenc

On Tue, Jun 4, 2019 at 2:32 PM Jim Langston <jl...@resolutebi.com>
wrote:

> Hi All,
>
> I’m currently working on our direction with Kafka and whether to move our
> cluster
> into Kubernetes. Flume is an integral part of delivering data to the Kafka
> cluster.
>
> I’m looking at moving to Kafka 2.2.1 (current release) , and notice that
> this bug is
> applicable to the whole update process ..
>
> https://issues.apache.org/jira/browse/KAFKA-7755
>
>
> This bug relates to clients that would be affected by Kubernetes needing
> to handle reconnecting
> to brokers where Kubernetes gives them another IP address.
>
> My questions to the community:
>
> What version of Flume do I need to be on ? 1.9 last tested with Kafka 2.0.x
>
> Can I pull in a Kafka 2.2.0 client for Flume ?, where this bug is fixed
>
> Thanks,
>
> Jim
>