Kafka Deplopyment Using Kubernetes (on Cloud) - settings for log.dirs

["M. Manna" <ma...@gmail.com>]

Hello,

We are thinking of rolling out Kafka on Kubernetes deployed on public cloud
(AWS or GCP, or other). We were hoping if someone could provide some
suggestion or insight here.

What we are trying to understand is how logs.dir property is affected when
we run Pods in a specific worker node? if we are running 3 broker PODs with
1 ZK Pod, how are we treating the logs.dir property for brokers (i.e. their
location for offsets/data etc.)? Are the log.dirs mounted to different
locations? Or, are they sharing the same path i.e. same set of files?

We understand that people are quite busy with the ongoing release. So any
insight you can provide will be highly appreciated.

Thanks,

Re: Kafka Deplopyment Using Kubernetes (on Cloud) - settings for log.dirs

["M. Manna" <ma...@gmail.com>]

Yes I have. But it lacks detailed explanation - even the document which I
have obtained through a webinar.

Log.dirs impact, imho, is the most important and relevant item to elaborate
for such publications. And I mean, by using diagrams and details that
apache Kafka and Confluent have done so far (and they’re very good). I’m
sure it’ll be done soon :)

We can give a lot of names e.g. SatefulSets, but I was looking for an
detailed explanation for how this will be different in K8s.


Thanks,

On Mon, 22 Oct 2018 at 19:12, Mark Anderson <ma...@gmail.com> wrote:

> Have you reviewed
> https://www.confluent.io/blog/getting-started-apache-kafka-kubernetes/ as
> a
> starting point?
>
> On Mon, 22 Oct 2018, 18:07 M. Manna, <ma...@gmail.com> wrote:
>
> > Thanks a lot for your prompt answer. This is what I was expecting.
> >
> > So, if we had three pods where volumes are mapped as the following
> >
> > Pod 1 = (log.dirs=/some/directory1)
> > Pod 2 = (log.dirs=/some/directory2)
> > Pod 3 = (log.dirs=/some/directory3)
> >
> > if something bad happens to Pod 3 and goes down, would we be able to spin
> > up a new pod with the same logs.dir location?
> >
> > Sorry if I am thinking of something impossible, but kafka and zk are
> > essentially databases and has some stateful items. So just trying to
> > understand how it all becomes different when deployed with K8s on public
> > cloud.
> >
> > Thanks,
> >
> > On Mon, 22 Oct 2018 at 17:55, Svante Karlsson <sv...@csi.se>
> > wrote:
> >
> > > Different directories, they cannot share path. A broker will delete
> > > everything under the log directory that it does not know about
> > >
> > > Den mån 22 okt. 2018 kl 17:47 skrev M. Manna <ma...@gmail.com>:
> > >
> > > > Hello,
> > > >
> > > > We are thinking of rolling out Kafka on Kubernetes deployed on public
> > > cloud
> > > > (AWS or GCP, or other). We were hoping if someone could provide some
> > > > suggestion or insight here.
> > > >
> > > > What we are trying to understand is how logs.dir property is affected
> > > when
> > > > we run Pods in a specific worker node? if we are running 3 broker
> PODs
> > > with
> > > > 1 ZK Pod, how are we treating the logs.dir property for brokers (i.e.
> > > their
> > > > location for offsets/data etc.)? Are the log.dirs mounted to
> different
> > > > locations? Or, are they sharing the same path i.e. same set of files?
> > > >
> > > > We understand that people are quite busy with the ongoing release. So
> > any
> > > > insight you can provide will be highly appreciated.
> > > >
> > > > Thanks,
> > > >
> > >
> >
>

Re: Kafka Deplopyment Using Kubernetes (on Cloud) - settings for log.dirs

[Mark Anderson <ma...@gmail.com>]

Have you reviewed
https://www.confluent.io/blog/getting-started-apache-kafka-kubernetes/ as a
starting point?

On Mon, 22 Oct 2018, 18:07 M. Manna, <ma...@gmail.com> wrote:

> Thanks a lot for your prompt answer. This is what I was expecting.
>
> So, if we had three pods where volumes are mapped as the following
>
> Pod 1 = (log.dirs=/some/directory1)
> Pod 2 = (log.dirs=/some/directory2)
> Pod 3 = (log.dirs=/some/directory3)
>
> if something bad happens to Pod 3 and goes down, would we be able to spin
> up a new pod with the same logs.dir location?
>
> Sorry if I am thinking of something impossible, but kafka and zk are
> essentially databases and has some stateful items. So just trying to
> understand how it all becomes different when deployed with K8s on public
> cloud.
>
> Thanks,
>
> On Mon, 22 Oct 2018 at 17:55, Svante Karlsson <sv...@csi.se>
> wrote:
>
> > Different directories, they cannot share path. A broker will delete
> > everything under the log directory that it does not know about
> >
> > Den mån 22 okt. 2018 kl 17:47 skrev M. Manna <ma...@gmail.com>:
> >
> > > Hello,
> > >
> > > We are thinking of rolling out Kafka on Kubernetes deployed on public
> > cloud
> > > (AWS or GCP, or other). We were hoping if someone could provide some
> > > suggestion or insight here.
> > >
> > > What we are trying to understand is how logs.dir property is affected
> > when
> > > we run Pods in a specific worker node? if we are running 3 broker PODs
> > with
> > > 1 ZK Pod, how are we treating the logs.dir property for brokers (i.e.
> > their
> > > location for offsets/data etc.)? Are the log.dirs mounted to different
> > > locations? Or, are they sharing the same path i.e. same set of files?
> > >
> > > We understand that people are quite busy with the ongoing release. So
> any
> > > insight you can provide will be highly appreciated.
> > >
> > > Thanks,
> > >
> >
>

Re: Kafka Deplopyment Using Kubernetes (on Cloud) - settings for log.dirs

["M. Manna" <ma...@gmail.com>]

Thanks a lot for your prompt answer. This is what I was expecting.

So, if we had three pods where volumes are mapped as the following

Pod 1 = (log.dirs=/some/directory1)
Pod 2 = (log.dirs=/some/directory2)
Pod 3 = (log.dirs=/some/directory3)

if something bad happens to Pod 3 and goes down, would we be able to spin
up a new pod with the same logs.dir location?

Sorry if I am thinking of something impossible, but kafka and zk are
essentially databases and has some stateful items. So just trying to
understand how it all becomes different when deployed with K8s on public
cloud.

Thanks,

On Mon, 22 Oct 2018 at 17:55, Svante Karlsson <sv...@csi.se>
wrote:

> Different directories, they cannot share path. A broker will delete
> everything under the log directory that it does not know about
>
> Den mån 22 okt. 2018 kl 17:47 skrev M. Manna <ma...@gmail.com>:
>
> > Hello,
> >
> > We are thinking of rolling out Kafka on Kubernetes deployed on public
> cloud
> > (AWS or GCP, or other). We were hoping if someone could provide some
> > suggestion or insight here.
> >
> > What we are trying to understand is how logs.dir property is affected
> when
> > we run Pods in a specific worker node? if we are running 3 broker PODs
> with
> > 1 ZK Pod, how are we treating the logs.dir property for brokers (i.e.
> their
> > location for offsets/data etc.)? Are the log.dirs mounted to different
> > locations? Or, are they sharing the same path i.e. same set of files?
> >
> > We understand that people are quite busy with the ongoing release. So any
> > insight you can provide will be highly appreciated.
> >
> > Thanks,
> >
>

Re: Kafka Deplopyment Using Kubernetes (on Cloud) - settings for log.dirs

[Svante Karlsson <sv...@csi.se>]

Different directories, they cannot share path. A broker will delete
everything under the log directory that it does not know about

Den mån 22 okt. 2018 kl 17:47 skrev M. Manna <ma...@gmail.com>:

> Hello,
>
> We are thinking of rolling out Kafka on Kubernetes deployed on public cloud
> (AWS or GCP, or other). We were hoping if someone could provide some
> suggestion or insight here.
>
> What we are trying to understand is how logs.dir property is affected when
> we run Pods in a specific worker node? if we are running 3 broker PODs with
> 1 ZK Pod, how are we treating the logs.dir property for brokers (i.e. their
> location for offsets/data etc.)? Are the log.dirs mounted to different
> locations? Or, are they sharing the same path i.e. same set of files?
>
> We understand that people are quite busy with the ongoing release. So any
> insight you can provide will be highly appreciated.
>
> Thanks,
>