Kafka 0.8/VIP/SSL

[Reiner Stach <re...@gmail.com>]

I'm looking for advice on running Kafka 0.8 behind VIPs. The goal is to
support SSL traffic, with encryption and decryption being performed by
back-to-back VIPs at the client and in front of the broker. That is:

Kafka client
    --> vip1a.myco.com:8080 (SSL encrypt) --- WAN ---> VIP 1b (SSL decrypt)
--> Kafka broker 1
    --> vip2a.myco.com:8080 (SSL encrypt) --- WAN ---> VIP 2b (SSL decrypt)
--> Kafka broker 2

Note there is a separate VIP pipeline per broker, no multiplexing involved.
We don't care about ordering of messages within partitions or elsewhere.

I've configured each broker to register in ZK and advertise itself as being
the VIP endpoint:
broker 1:
    advertised.host.name=vip1a.myco.com
    advertised.port=8080
broker 2:
    advertised.host.name=vip2a.myco.com
    advertised.port=8080

Should this work? Any issues?

And since it *isn't* working for me yet, any idea why I might consistently
see a LeaderNotAvailableException when trying to publish or consume?

One possible issue: the brokers need to talk directly to each other,
broker-to-broker, right? And they will try to talk to each other via the
VIP endpoints (vip1a, vip2a)? Seems like I will need to either make the VIP
address routeable from the Kafka broker hosts to allow the broker-to-broker
comms, or else hack the Kafka code so the brokers present the advertised
host/port in response to metadata requests from clients - but talk to each
other on their 'real' addresses.

Many thanks
Reiner

Re: Kafka 0.8/VIP/SSL

[Neha Narkhede <ne...@gmail.com>]

One possible issue: the brokers need to talk directly to each other,
broker-to-broker, right? And they will try to talk to each other via the
VIP endpoints (vip1a, vip2a)?

The brokers communicate with each other and they use the
advertised.host.name for the same. So you will need to ensure that is
possible in your setup.

Thanks,
Neha


On Tue, Jun 24, 2014 at 3:35 PM, Reiner Stach <re...@gmail.com>
wrote:

> I'm looking for advice on running Kafka 0.8 behind VIPs. The goal is to
> support SSL traffic, with encryption and decryption being performed by
> back-to-back VIPs at the client and in front of the broker. That is:
>
> Kafka client
>     --> vip1a.myco.com:8080 (SSL encrypt) --- WAN ---> VIP 1b (SSL
> decrypt)
> --> Kafka broker 1
>     --> vip2a.myco.com:8080 (SSL encrypt) --- WAN ---> VIP 2b (SSL
> decrypt)
> --> Kafka broker 2
>
> Note there is a separate VIP pipeline per broker, no multiplexing involved.
> We don't care about ordering of messages within partitions or elsewhere.
>
> I've configured each broker to register in ZK and advertise itself as being
> the VIP endpoint:
> broker 1:
>     advertised.host.name=vip1a.myco.com
>     advertised.port=8080
> broker 2:
>     advertised.host.name=vip2a.myco.com
>     advertised.port=8080
>
> Should this work? Any issues?
>
> And since it *isn't* working for me yet, any idea why I might consistently
> see a LeaderNotAvailableException when trying to publish or consume?
>
> One possible issue: the brokers need to talk directly to each other,
> broker-to-broker, right? And they will try to talk to each other via the
> VIP endpoints (vip1a, vip2a)? Seems like I will need to either make the VIP
> address routeable from the Kafka broker hosts to allow the broker-to-broker
> comms, or else hack the Kafka code so the brokers present the advertised
> host/port in response to metadata requests from clients - but talk to each
> other on their 'real' addresses.
>
> Many thanks
> Reiner
>