Customizing NiFi in a Docker Container on EC2

[James McMahon <js...@gmail.com>]

The NiFi System Administration Guide makes many recommendations for
configuration changes to optimize nifi performance. These "best
practices", for example:
* https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices
Placement of repos on separate disk devices is another big one; here
is an example:
* https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#content-repository

I have nifi installed in a docker container on an EC2 instance.

As far as I can tell, it is not optimized and I am writing today to
ask if anyone has done that with success? If so, I'd like to learn
more about that..


I use the command
docker exec -it nifi /bin/bash
to review what I understand to be the nifi directories and config files in
the container.

I notice nifi in the container is in many ways not configured to Apache
NiFi recommendations for optimal performance. For example, the
nifi.properties repo params for the docker installation all show repos
placed on one common disk
device (the one where the container lives, presumably).

I've configured external ebs volumes that I've mounted on my instance.
My intention: one
for content_repository, one for flowfile_repository, and likewise for
database and provenance repositories. I'd like to have the containerized
nifi write to and read from those so that I don't bottleneck performance
reading and writing to the same device for repos.

I need to persist my changes to nifi config files. How does one avoid
making changes in nifi.properties and the like that are lost when the
docker container is stopped, deleted, and a new one instantiated?

I need to engage with external repo resources when nifi runs within my
container.
How do we direct nifi in the container to use those external resources
outside of the container to host content_repository, etc etc?

Thank you in advance for any help.
Jim

Re: Customizing NiFi in a Docker Container on EC2

[Mike Thomsen <mi...@gmail.com>]

If you're just slapping Docker on the box and not trying something
fancy like running it on EKS, I would just switch to running NiFi
directly. That allows you to use something like Chef or Ansible to
manage your configurations which is a lot more powerful than manually
tweaking Docker.

On Fri, Nov 11, 2022 at 6:20 AM James McMahon <js...@gmail.com> wrote:
>
> The NiFi System Administration Guide makes many recommendations for configuration changes to optimize nifi performance. These "best practices", for example:
> * https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices
> Placement of repos on separate disk devices is another big one; here is an example:
> * https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#content-repository
>
> I have nifi installed in a docker container on an EC2 instance.
>
> As far as I can tell, it is not optimized and I am writing today to ask if anyone has done that with success? If so, I'd like to learn more about that..
>
>
> I use the command
> docker exec -it nifi /bin/bash
> to review what I understand to be the nifi directories and config files in
> the container.
>
> I notice nifi in the container is in many ways not configured to Apache
> NiFi recommendations for optimal performance. For example, the
> nifi.properties repo params for the docker installation all show repos placed on one common disk
> device (the one where the container lives, presumably).
>
> I've configured external ebs volumes that I've mounted on my instance. My intention: one
> for content_repository, one for flowfile_repository, and likewise for
> database and provenance repositories. I'd like to have the containerized
> nifi write to and read from those so that I don't bottleneck performance
> reading and writing to the same device for repos.
>
> I need to persist my changes to nifi config files. How does one avoid
> making changes in nifi.properties and the like that are lost when the
> docker container is stopped, deleted, and a new one instantiated?
>
> I need to engage with external repo resources when nifi runs within my container.
> How do we direct nifi in the container to use those external resources
> outside of the container to host content_repository, etc etc?
>
> Thank you in advance for any help.
> Jim