Docker to deploy NiFi

[Stéphane Maarek <st...@gmail.com>]

Hi,

Has anyone deployed NiFi using docker? I see many images on dockerhub, but
I have a few questions regarding usage:

1) How do you handle overall configuration (java memory options, etc) ?
2) If you add a docker process, does the spun up NiFi get automatically
added to your cluster?
3) How do you handle persistence of data ?

Thanks!
Stephane

Re: Docker to deploy NiFi

[Koji Kawamura <ij...@gmail.com>]

Hi Stephane,

I've been using Docker compose to form a NiFi cluster for testing purpose.
It's not on dockerhub, but I've shared it as a Github project. [1]

My knowledge of Docker is fairly limited, but hope this helps.

1) Although I haven't played with those yet, docker-compose has
resource allocation parameters, such as cpu_quota or mem_limit [2]. I
assume these can be helpful to allocate resources in a container pool
using Swarm together.

From NiFi's perspective,  conf/bootstrap.conf has JVM options. It's
possible to tweak the bootstrap.conf when NiFi starts on a container,
please see the example shell script I used to replace nifi.properties
hostname setting [3]

2) docker-compose has 'scale' command. With that, you can spawn new
NiFi nodes within the same network and get it automatically added to
the cluster, e.g. $ docker-compose scale nifi-nodes=2
I have configured a Zk node, and a NiFi seed node to expose ports, and
nifi-nodes as services in docker-compose.yml [4]

3) Docker container can map local file system to container volume. If
you'd like to preserve NiFi data dirs such as content-repository
across container's life cycle, then I think you can use this feature.
[5]

[1] https://github.com/ijokarumawak/docker-compose-nifi-cluster
[2] https://docs.docker.com/compose/compose-file/#/cpushares-cpuquota-cpuset-domainname-hostname-ipc-macaddress-memlimit-memswaplimit-privileged-readonly-restart-shmsize-stdinopen-tty-user-workingdir
[3] https://github.com/ijokarumawak/docker-compose-nifi-cluster/blob/master/nifi-node/start-nifi.sh
[4] https://github.com/ijokarumawak/docker-compose-nifi-cluster/blob/master/docker-compose.yml
[5] https://docs.docker.com/engine/tutorials/dockervolumes/

Thanks,
Koji

On Tue, Oct 11, 2016 at 3:03 PM, Stéphane Maarek
<st...@gmail.com> wrote:
> Hi,
>
> Has anyone deployed NiFi using docker? I see many images on dockerhub, but I
> have a few questions regarding usage:
>
> 1) How do you handle overall configuration (java memory options, etc) ?
> 2) If you add a docker process, does the spun up NiFi get automatically
> added to your cluster?
> 3) How do you handle persistence of data ?
>
> Thanks!
> Stephane