Kubernetes: Using Load Balancers

[Phillip Neumann <pn...@ned.cl>]

Hi all!

I was trying to deploy a little Kafka cluster over Kubernetes (k8s), and
noticed this wonderful helm charts from here
https://github.com/confluentinc/cp-helm-charts
The Kafka chart included there, will try to expose the brokers so they can
be accesible not only from within the k8s cluster, but from outside too.
Great!

Lets (try to) draw this:

+---VPC-------------------+
|                         |
|  +--------+  +--------+ |                .-~~~-.
|  | Node 1 |  | Node 2 | |        .- ~ ~-(       )_ _
|  +--------+  +--------+ |       /                     ~ -.
|  +--------+  +--------+ |      |      The internetz        \
|  | Node 3 |  | Node 4 | |       \                         .'
|  +--------+  +--------+ |         ~- . _____________ . -~
|                         |
|  +-------------------+  |
|  |    A nice VM      |  |
|  +-------------------+  |
|                         |
+-------------------------+

Legend:
*Node X*:        K8s nodes. They live inside a VPC. Pods lives in the nodes.
*A nice VM*:     A VM that lives in the same VPC as k8s nodes, but its not
part of it.
*The internetz*: Things outside the VPC


Lets separate this in 3 parts:

1) Conectivity from within the cluster
Easy, the communicate via the POD's IPs.
K8s is in charge of route the packets around.

2) Conectivity from a nice VM.
When enabled, the kafka chart will expose the borkers using NodePort  (A)
By default it uses the IP of the Host, on ports 31090, 31091 and 31092 (B)

So clients that sits on this VM, need just to point to one of these and
start doing its job.

3) Conectivity from the internet.
Not possible. :(

Clouds. They typical to have 2 IP's: a local to the VPC one, and a
externally routable one, which can be volatile or permanent/elastic
The variable hostIP used by the chart (C), will use the local to the VPC
IP, which makes it possible to connect a VM from the same VPC subnet to
kafka.
But not from other places.

On k8s i'm not sure if it makes too much sense to have elastic IPs on the
nodes, but anyway they would be assigned after the node is created, and
could be manually changed by the operator. Probably it will be a better
idea to use LoadBalancers instead.

If we wanted to implement LoadBalancer in the chart, how do you think would
be a good way to do it?
What would be better?
- To make the kafka chart wait until the load-balancers are created, get
their URL, setup advertised.listeners based on that and then start the
broker? (maybe using a initContainer (?))
- Or maybe just listen to when it's ready, setup the advertised.listeners,
and then make the the brokers restart with the new values?

Thanks!
----
(A)
https://github.com/confluentinc/cp-helm-charts/blob/master/charts/cp-kafka/templates/nodeport-service.yaml#L21
(B)
https://github.com/confluentinc/cp-helm-charts/blob/master/charts/cp-kafka/values.yaml#L106
(C)
https://github.com/confluentinc/cp-helm-charts/blob/master/charts/cp-kafka/templates/statefulset.yaml#L94