Running Apache Samza on Kubernetes cluster with Zookeeper

[Stefano Rebora <st...@live.it>]

Hello,

I’m trying to develop Apache Samza on a Kubernetes cluster using Zookeeper for coordination.
Is there a suggested or working example to follow?

Thank you in advance,

Stefano

Re: Running Apache Samza on Kubernetes cluster with Zookeeper

[Jagadish Venkatraman <ja...@gmail.com>]

+Weiqing Yang, who gave a community talk on this at KubeCon this year.

Hi Stefano,

Running Samza-standalone + Zk on Kubernetes should be no different than
running any other application on Kubernetes.

At a high-level, you would:


   1. Package your application as a container image - just as you would for
   any other application.
   2. Deploy it using Kubectl

From this post <https://github.com/sonian/samza-kubernetes> sharing how to
run Samza + Zk on K8s (It is prior to Samza 1.1 - please skip the section
on low-level jobs)

*"By comparison, using the high-level StreamApplication API is considerably
simpler. There's actually very little to it: just package the application
in a way that can be executed and deploy it using StatefulSet directly,
instead of via the KubernetesJob.*

*Running gradle dockerDistTar will produce a
folder example/build/docker which contains a Dockerfile and everything
necessary to run the example application in Kubernetes. Running gradle
dockerBuildImage will install said image locally, at which point you can
run kubectl create -f example/k8s/app.yaml*

*If you're planning to use the KV store with a volume mount, keep in mind
that Samza hard-codes the location to {user.dir}/state. The combination of
the generated Docker image and app.yaml mount point works because the image
runs the application from / and the mount point is /state."*
A related (a bit-outdated) email thread:
http://mail-archives.apache.org/mod_mbox/samza-dev/201802.mbox/%3CCAMd3yjgPanNxXnwYzkZ-EQNvJ3_ukpf8aAK7Qubt9CK9zJh_yQ@mail.gmail.com%3E


On Tue, Apr 30, 2019 at 8:46 AM Stefano Rebora <st...@live.it> wrote:

> Hello,
>
> I’m trying to develop Apache Samza on a Kubernetes cluster using Zookeeper
> for coordination.
> Is there a suggested or working example to follow?
>
> Thank you in advance,
>
> Stefano
>
>