You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Stephen Greszczyszyn <sg...@gmail.com> on 2018/10/24 20:04:18 UTC
Recommended NiFi Docker volume mappings?
Hi there,
I'm trying to get a working configuration for the official vanilla NiFi
docker image where it can read existing SFTP incoming data as well as allow
me to pass in any necessary configuration files.
The problem seems to be that by default the docker container picks up
userID 1000 to run the nifi process, which is OK since I mapped my
/etc/passwd and /etc/group volumes and I'm managing the directory
read/write access through my underlying OS (Ubuntu 18.04).
Where I am having problems is mapping the docker NiFi /opt/nifi directory
to a local directory, despite the permissions looking OK. I've even set my
local /data/nifi directory to chmod 777, but the docker container fails to
start.
Any suggestions on how to resolve this? Also any best practices for
mapping the NiFi internal docker volumes to the local OS would be
appreciated.
Thanks,
Stephen
Re: Recommended NiFi Docker volume mappings?
Posted by Stephen Greszczyszyn <sg...@gmail.com>.
@Juan, Thanks for the tip, probably not going to mount logs as using
Elastic filebeat to tail any live docker container logs - assuming that
nifi docker is configured to use the standard "docker log" API.
I've also found from experience that it is safer to move the default root
Docker volume location from /var/lib to a larger drive as it can blow up
and fill up root very quickly depending on how the internal docker app
behaves or writes logs.
On Thu, 25 Oct 2018 at 14:15, Juan Pablo Gardella <
gardellajuanpablo@gmail.com> wrote:
> I suggest to be careful when mount log directory. In one day fills some
> Gigabytes. If you want to mount logs, adjust the logging.
>
> On Thu, 25 Oct 2018 at 10:07 Stephen Greszczyszyn <sg...@gmail.com>
> wrote:
>
>>
>>
>> On Thu, 25 Oct 2018 at 12:50, Peter Wilcsinszky <
>> peterwilcsinszky@gmail.com> wrote:
>>
>> But even with 1.8 I'll need to declare the host mount directory somehow
>> via docker-compose, as how will the built docker image on dockerhub know
>> where to locally mount the internal $(NIFI_HOME) volumes as described below?
>>
>> VOLUME ${NIFI_LOG_DIR} \
>>>> ${NIFI_HOME}/conf \
>>>> ${NIFI_HOME}/database_repository \
>>>> ${NIFI_HOME}/flowfile_repository \
>>>> ${NIFI_HOME}/content_repository \
>>>> ${NIFI_HOME}/provenance_repository \
>>>> ${NIFI_HOME}/state
>>>>
>>>
>>> Yes you should specify volumes explicitly if you use 1.7.1, but also you
>>> should specify an extra separate volume to use for your incoming SFTP data.
>>>
>>>
Re: Recommended NiFi Docker volume mappings?
Posted by Juan Pablo Gardella <ga...@gmail.com>.
I suggest to be careful when mount log directory. In one day fills some
Gigabytes. If you want to mount logs, adjust the logging.
On Thu, 25 Oct 2018 at 10:07 Stephen Greszczyszyn <sg...@gmail.com>
wrote:
>
>
> On Thu, 25 Oct 2018 at 12:50, Peter Wilcsinszky <
> peterwilcsinszky@gmail.com> wrote:
>
> But even with 1.8 I'll need to declare the host mount directory somehow
> via docker-compose, as how will the built docker image on dockerhub know
> where to locally mount the internal $(NIFI_HOME) volumes as described below?
>
> VOLUME ${NIFI_LOG_DIR} \
>>> ${NIFI_HOME}/conf \
>>> ${NIFI_HOME}/database_repository \
>>> ${NIFI_HOME}/flowfile_repository \
>>> ${NIFI_HOME}/content_repository \
>>> ${NIFI_HOME}/provenance_repository \
>>> ${NIFI_HOME}/state
>>>
>>
>> Yes you should specify volumes explicitly if you use 1.7.1, but also you
>> should specify an extra separate volume to use for your incoming SFTP data.
>>
>>
Re: Recommended NiFi Docker volume mappings?
Posted by Peter Wilcsinszky <pe...@gmail.com>.
If you want them to be on your host machine then you have to declare those
yes. By default docker will create directories for those volumes on the
docker host under /var/lib/docker/volumes/. Note: the docker host is
typically running in a VM, at least this is the case on Docker for Mac.
On Thu, Oct 25, 2018 at 3:07 PM Stephen Greszczyszyn <sg...@gmail.com>
wrote:
>
>
> On Thu, 25 Oct 2018 at 12:50, Peter Wilcsinszky <
> peterwilcsinszky@gmail.com> wrote:
>
> But even with 1.8 I'll need to declare the host mount directory somehow
> via docker-compose, as how will the built docker image on dockerhub know
> where to locally mount the internal $(NIFI_HOME) volumes as described below?
>
> VOLUME ${NIFI_LOG_DIR} \
>>> ${NIFI_HOME}/conf \
>>> ${NIFI_HOME}/database_repository \
>>> ${NIFI_HOME}/flowfile_repository \
>>> ${NIFI_HOME}/content_repository \
>>> ${NIFI_HOME}/provenance_repository \
>>> ${NIFI_HOME}/state
>>>
>>
>> Yes you should specify volumes explicitly if you use 1.7.1, but also you
>> should specify an extra separate volume to use for your incoming SFTP data.
>>
>>
Re: Recommended NiFi Docker volume mappings?
Posted by Stephen Greszczyszyn <sg...@gmail.com>.
On Thu, 25 Oct 2018 at 12:50, Peter Wilcsinszky <pe...@gmail.com>
wrote:
But even with 1.8 I'll need to declare the host mount directory somehow via
docker-compose, as how will the built docker image on dockerhub know where
to locally mount the internal $(NIFI_HOME) volumes as described below?
VOLUME ${NIFI_LOG_DIR} \
>> ${NIFI_HOME}/conf \
>> ${NIFI_HOME}/database_repository \
>> ${NIFI_HOME}/flowfile_repository \
>> ${NIFI_HOME}/content_repository \
>> ${NIFI_HOME}/provenance_repository \
>> ${NIFI_HOME}/state
>>
>
> Yes you should specify volumes explicitly if you use 1.7.1, but also you
> should specify an extra separate volume to use for your incoming SFTP data.
>
>
Re: Recommended NiFi Docker volume mappings?
Posted by Peter Wilcsinszky <pe...@gmail.com>.
On Thu, Oct 25, 2018 at 1:01 PM Stephen Greszczyszyn <sg...@gmail.com>
wrote:
> Thanks for the reply Peter,
>
> You are right, last night when I tried mapping just /opt/nifi from NiFi
> version 1.7.1 the container wasn't happy starting up and I couldn't figure
> out what folders were needed to store state and manage any configurations.
>
> Just to be clear, should I be mapping the following volumes to local
> folders that have read/write access for host user ID 1000 (or a Linux group
> that user 1000 is a member of) for the internal docker user nifi (UID 1000)
> to be able to access? I guess there is no way to change the UID of docker
> user nifi without doing a custom docker build. For security/LDAP, I'm
> assuming I can just pass the environment variables through as documented on
> the README.md?
>
> VOLUME ${NIFI_LOG_DIR} \
> ${NIFI_HOME}/conf \
> ${NIFI_HOME}/database_repository \
> ${NIFI_HOME}/flowfile_repository \
> ${NIFI_HOME}/content_repository \
> ${NIFI_HOME}/provenance_repository \
> ${NIFI_HOME}/state
>
Yes you should specify volumes explicitly if you use 1.7.1, but also you
should specify an extra separate volume to use for your incoming SFTP data.
>
> I'm trying to automate the docker config using docker-compose via ansible,
> so normally I use a framework like this:
>
> - name: Create local host nifi state directories in /data/nifi/
> file:
> path: "{{ item }}"
> state: directory
> owner: 1000
> group: 1000
> mode: 0775
> with_items:
> - /data/nifi
> - /data/nifi/conf
> - /data/nifi/state
> - /data/nifi/database_repository
> - /data/nifi/flowfile_repository
> - /data/nifi/content_repository
> - /data/nifi/provenance_repository
>
> - name: Build NiFi Docker Image
> docker_service:
> project_name: nifi
> definition:
> version: '2'
> services:
> nifi:
> image: apache/nifi:{{ nifi_version }}
> container_name: nifi
> restart: on-failure
> # environment:
>
> volumes:
> # take uid/gid lists from host to give same user/group
> permissions mapping as host
> # - /etc/passwd:/etc/passwd
> # - /etc/group:/etc/group
>
> # Give NiFi access to read/write in /data
> - /data:/data
>
> # Expose NiFi config and state directories
> - /data/nifi/conf:/opt/nifi/conf
> - /data/nifi/state:/data/nifi/state
> -
> /data/nifi/database_repository:/opt/nifi/database_repository
> -
> /data/nifi/flowfile_repository:/opt/nifi/flowfile_repository
> -
> /data/nifi/content_repository:/opt/nifi/content_repository
> -
> /data/nifi/provenance_repository:/opt/nifi/provenance_repository
>
> ports:
> - 8080:8080
> - 8443:8443
> - 10000:10000
>
> On Thu, 25 Oct 2018 at 11:02, Peter Wilcsinszky <
> peterwilcsinszky@gmail.com> wrote:
>
>> Hi Stephen,
>>
>> I don't recommend mounting /opt/nifi directly as it will copy all the
>> NiFi binaries over to the volume as well, which is unnecessary I beleive.
>> The latest dockerfile that will be used to build the docker image for the
>> upcoming release already declares volumes that I recommend to leverage:
>>
>> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/Dockerfile#L73
>>
>> However if you have special needs you can always tweak the dockerfile and
>> build you own image from it.
>>
>> On Wed, Oct 24, 2018 at 10:04 PM Stephen Greszczyszyn <sg...@gmail.com>
>> wrote:
>>
>>> Hi there,
>>>
>>> I'm trying to get a working configuration for the official vanilla NiFi
>>> docker image where it can read existing SFTP incoming data as well as allow
>>> me to pass in any necessary configuration files.
>>>
>>> The problem seems to be that by default the docker container picks up
>>> userID 1000 to run the nifi process, which is OK since I mapped my
>>> /etc/passwd and /etc/group volumes and I'm managing the directory
>>> read/write access through my underlying OS (Ubuntu 18.04).
>>>
>>> Where I am having problems is mapping the docker NiFi /opt/nifi
>>> directory to a local directory, despite the permissions looking OK. I've
>>> even set my local /data/nifi directory to chmod 777, but the docker
>>> container fails to start.
>>>
>>> Any suggestions on how to resolve this? Also any best practices for
>>> mapping the NiFi internal docker volumes to the local OS would be
>>> appreciated.
>>>
>>> Thanks,
>>>
>>> Stephen
>>>
>>
Re: Recommended NiFi Docker volume mappings?
Posted by Stephen Greszczyszyn <sg...@gmail.com>.
Thanks for the reply Peter,
You are right, last night when I tried mapping just /opt/nifi from NiFi
version 1.7.1 the container wasn't happy starting up and I couldn't figure
out what folders were needed to store state and manage any configurations.
Just to be clear, should I be mapping the following volumes to local
folders that have read/write access for host user ID 1000 (or a Linux group
that user 1000 is a member of) for the internal docker user nifi (UID 1000)
to be able to access? I guess there is no way to change the UID of docker
user nifi without doing a custom docker build. For security/LDAP, I'm
assuming I can just pass the environment variables through as documented on
the README.md?
VOLUME ${NIFI_LOG_DIR} \
${NIFI_HOME}/conf \
${NIFI_HOME}/database_repository \
${NIFI_HOME}/flowfile_repository \
${NIFI_HOME}/content_repository \
${NIFI_HOME}/provenance_repository \
${NIFI_HOME}/state
I'm trying to automate the docker config using docker-compose via ansible,
so normally I use a framework like this:
- name: Create local host nifi state directories in /data/nifi/
file:
path: "{{ item }}"
state: directory
owner: 1000
group: 1000
mode: 0775
with_items:
- /data/nifi
- /data/nifi/conf
- /data/nifi/state
- /data/nifi/database_repository
- /data/nifi/flowfile_repository
- /data/nifi/content_repository
- /data/nifi/provenance_repository
- name: Build NiFi Docker Image
docker_service:
project_name: nifi
definition:
version: '2'
services:
nifi:
image: apache/nifi:{{ nifi_version }}
container_name: nifi
restart: on-failure
# environment:
volumes:
# take uid/gid lists from host to give same user/group
permissions mapping as host
# - /etc/passwd:/etc/passwd
# - /etc/group:/etc/group
# Give NiFi access to read/write in /data
- /data:/data
# Expose NiFi config and state directories
- /data/nifi/conf:/opt/nifi/conf
- /data/nifi/state:/data/nifi/state
-
/data/nifi/database_repository:/opt/nifi/database_repository
-
/data/nifi/flowfile_repository:/opt/nifi/flowfile_repository
- /data/nifi/content_repository:/opt/nifi/content_repository
-
/data/nifi/provenance_repository:/opt/nifi/provenance_repository
ports:
- 8080:8080
- 8443:8443
- 10000:10000
On Thu, 25 Oct 2018 at 11:02, Peter Wilcsinszky <pe...@gmail.com>
wrote:
> Hi Stephen,
>
> I don't recommend mounting /opt/nifi directly as it will copy all the NiFi
> binaries over to the volume as well, which is unnecessary I beleive. The
> latest dockerfile that will be used to build the docker image for the
> upcoming release already declares volumes that I recommend to leverage:
>
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/Dockerfile#L73
>
> However if you have special needs you can always tweak the dockerfile and
> build you own image from it.
>
> On Wed, Oct 24, 2018 at 10:04 PM Stephen Greszczyszyn <sg...@gmail.com>
> wrote:
>
>> Hi there,
>>
>> I'm trying to get a working configuration for the official vanilla NiFi
>> docker image where it can read existing SFTP incoming data as well as allow
>> me to pass in any necessary configuration files.
>>
>> The problem seems to be that by default the docker container picks up
>> userID 1000 to run the nifi process, which is OK since I mapped my
>> /etc/passwd and /etc/group volumes and I'm managing the directory
>> read/write access through my underlying OS (Ubuntu 18.04).
>>
>> Where I am having problems is mapping the docker NiFi /opt/nifi directory
>> to a local directory, despite the permissions looking OK. I've even set my
>> local /data/nifi directory to chmod 777, but the docker container fails to
>> start.
>>
>> Any suggestions on how to resolve this? Also any best practices for
>> mapping the NiFi internal docker volumes to the local OS would be
>> appreciated.
>>
>> Thanks,
>>
>> Stephen
>>
>
Re: Recommended NiFi Docker volume mappings?
Posted by Peter Wilcsinszky <pe...@gmail.com>.
Hi Stephen,
I don't recommend mounting /opt/nifi directly as it will copy all the NiFi
binaries over to the volume as well, which is unnecessary I beleive. The
latest dockerfile that will be used to build the docker image for the
upcoming release already declares volumes that I recommend to leverage:
https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/Dockerfile#L73
However if you have special needs you can always tweak the dockerfile and
build you own image from it.
On Wed, Oct 24, 2018 at 10:04 PM Stephen Greszczyszyn <sg...@gmail.com>
wrote:
> Hi there,
>
> I'm trying to get a working configuration for the official vanilla NiFi
> docker image where it can read existing SFTP incoming data as well as allow
> me to pass in any necessary configuration files.
>
> The problem seems to be that by default the docker container picks up
> userID 1000 to run the nifi process, which is OK since I mapped my
> /etc/passwd and /etc/group volumes and I'm managing the directory
> read/write access through my underlying OS (Ubuntu 18.04).
>
> Where I am having problems is mapping the docker NiFi /opt/nifi directory
> to a local directory, despite the permissions looking OK. I've even set my
> local /data/nifi directory to chmod 777, but the docker container fails to
> start.
>
> Any suggestions on how to resolve this? Also any best practices for
> mapping the NiFi internal docker volumes to the local OS would be
> appreciated.
>
> Thanks,
>
> Stephen
>