Kafka Broker Config (logs.dir) on Kubernetes

["M. Manna" <ma...@gmail.com>]

 Hello,

The question might trigger people to reply with "Confluent" - but it's not
related to confluent as the kubernetes offering is not for publi/community
edition. So, discussing Helm charts and intro to Confluent isn't our
objective here.

What I am trying to understand is how does the log files (kafka message
logs, consumer offsets) are managed in Kubernetes (e.g. persistent volume,
statefulsets, etc). I have a 3 node cluster running over 3 physical Linux
VMs, and would like to move this setup to Kubernetes.

The only part where we are strugging is with the following:

1) how does logs.dir configuration work per Pod?
2) Assuming I have 3 PODs (3 brokers), and one of the Pod Goes down - how
do I manage the message log and offset files ? If a POD goes does, that
means it will delete everything in the logs.dir location, won't it?
3) I am assuming broker.id will be supplied using some form of configMap,
but if there is anything worth knowing here, please do share.

We have a reliable service on bare metal cloud, so we don't want to disrupt
it unless we are sure about the changes.

Does anyone have any experience with this? If so, it would be great if you
can share any gists or configs.

Much appreciated.

Regards,

Re: Kafka Broker Config (logs.dir) on Kubernetes

[Raman Gupta <ro...@gmail.com>]

Note you shouldn't cross-post to both the users and dev list -- this
kind of question belongs on the user list.

The fundamental things you need to go investigate:
* Kubernetes Stateful Sets
* Kafka packaged for use on Kubernetes -- I have been happy with
https://github.com/Yolean/kubernetes-kafka. Even if you're interested
in a more DIY solution, you can get an understanding about how they
manage things like `broker.id` assignment (see the config in
https://github.com/Yolean/kubernetes-kafka/blob/master/kafka/10broker-config.yml
for example).

If you want a more automated solution, then look into Kubernetes
operators for managing Kafka on Kubernetes. There are quite a few of
these. An open source one is Strimzi
(https://github.com/strimzi/strimzi-kafka-operator), and Confluent is
working on one too.

Hope that gets you started.

Regards,
Raman

On Thu, Apr 4, 2019 at 7:21 AM M. Manna <ma...@gmail.com> wrote:
>
>  Hello,
>
> The question might trigger people to reply with "Confluent" - but it's not
> related to confluent as the kubernetes offering is not for publi/community
> edition. So, discussing Helm charts and intro to Confluent isn't our
> objective here.
>
> What I am trying to understand is how does the log files (kafka message
> logs, consumer offsets) are managed in Kubernetes (e.g. persistent volume,
> statefulsets, etc). I have a 3 node cluster running over 3 physical Linux
> VMs, and would like to move this setup to Kubernetes.
>
> The only part where we are strugging is with the following:
>
> 1) how does logs.dir configuration work per Pod?
> 2) Assuming I have 3 PODs (3 brokers), and one of the Pod Goes down - how
> do I manage the message log and offset files ? If a POD goes does, that
> means it will delete everything in the logs.dir location, won't it?
> 3) I am assuming broker.id will be supplied using some form of configMap,
> but if there is anything worth knowing here, please do share.
>
> We have a reliable service on bare metal cloud, so we don't want to disrupt
> it unless we are sure about the changes.
>
> Does anyone have any experience with this? If so, it would be great if you
> can share any gists or configs.
>
> Much appreciated.
>
> Regards,