You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by pepov <gi...@git.apache.org> on 2018/05/30 08:00:36 UTC
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
GitHub user pepov opened a pull request:
https://github.com/apache/nifi/pull/2747
NIFI-5249 Dockerfile enhancements
- Createdata repository and log directories
- Add explicit environment variable declarations to replace nifi-env.sh
- Remove obsolete chown commands after user has already been switched to nifi
How I tested the changes:
- I've ran `mvn package -P docker` in the nifi-docker folder to build the image.
- I've made sure that `docker run --rm -ti apache/nifi` runs and start nifi using the new image
- Made sure that all files are owned by the nifi user under /opt/nifi: `docker run --rm -ti --entrypoint /bin/bash apache/nifi -c "find /opt/nifi ! -user nifi"`
- Checked that the expected environment variables are set:
```
docker run --rm -ti --entrypoint /bin/bash apache/nifi -c "env | grep NIFI"
NIFI_HOME=/opt/nifi/nifi-1.7.0-SNAPSHOT
NIFI_LOG_DIR=/opt/nifi/nifi-1.7.0-SNAPSHOT/logs
NIFI_PID_DIR=/opt/nifi/nifi-1.7.0-SNAPSHOT/run
NIFI_BASE_DIR=/opt/nifi
```
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pepov/nifi NIFI-5249
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2747.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2747
----
commit fb0b515c565cc532169a03eb7c4a59027d818718
Author: pepov <pe...@...>
Date: 2018-05-29T11:27:45Z
NIFI-5249 Dockerfile enhancements: create data repository and log directories, explicit environment variable declarations to replace nifi-env.sh
----
---
[GitHub] nifi issue #2747: NIFI-5249 Dockerfile enhancements
Posted by pepov <gi...@git.apache.org>.
Github user pepov commented on the issue:
https://github.com/apache/nifi/pull/2747
No, sorry! I meant integration-test not package. I beleive this would do it, at least works for me, just tested:
```
mvn package -pl nifi-assembly
cd nifi-docker
mvn integration-test -P docker
```
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by zenfenan <gi...@git.apache.org>.
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r192756076
--- Diff: nifi-docker/dockermaven/pom.xml ---
@@ -44,7 +44,6 @@
</buildArgs>
<repository>apache/nifi</repository>
<tag>${project.version}</tag>
- <tag>latest</tag>
--- End diff --
Why was this removed? When we do a release, I believe we want to keep that release as the latest.
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r195304620
--- Diff: nifi-docker/dockermaven/Dockerfile ---
@@ -26,23 +26,33 @@ ARG NIFI_BINARY
ENV NIFI_BASE_DIR /opt/nifi
ENV NIFI_HOME $NIFI_BASE_DIR/nifi-$NIFI_VERSION
-
-# Setup NiFi user
-RUN groupadd -g $GID nifi || groupmod -n nifi `getent group $GID | cut -d: -f1` \
- && useradd --shell /bin/bash -u $UID -g $GID -m nifi \
- && mkdir -p $NIFI_HOME/conf/templates \
- && chown -R nifi:nifi $NIFI_BASE_DIR
+ENV NIFI_PID_DIR=${NIFI_HOME}/run
+ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ADD $NIFI_BINARY $NIFI_BASE_DIR
-RUN chown -R nifi:nifi $NIFI_HOME
+# Setup NiFi user and create necessary directories
+RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d: -f1` \
+ && useradd --shell /bin/bash -u ${UID} -g ${GID} -m nifi \
+ && mkdir -p ${NIFI_HOME}/conf/templates \
+ && mkdir -p $NIFI_BASE_DIR/data \
+ && mkdir -p $NIFI_BASE_DIR/flowfile_repository \
+ && mkdir -p $NIFI_BASE_DIR/content_repository \
+ && mkdir -p $NIFI_BASE_DIR/provenance_repository \
+ && mkdir -p $NIFI_LOG_DIR \
+ && chown -R nifi:nifi ${NIFI_BASE_DIR} \
+ && apt-get update \
+ && apt-get install -y jq xmlstarlet procps
USER nifi
-# Web HTTP Port & Remote Site-to-Site Ports
-EXPOSE 8080 8181
+# Clear nifi-env.sh in favour of configuring all environment variables in the Dockerfile
+RUN echo "#!/bin/sh\n" > $NIFI_HOME/bin/nifi-env.sh
+
+# Web HTTP(s) & Socket Site-to-Site Ports
+EXPOSE 8080 8443 10000
-WORKDIR $NIFI_HOME
+WORKDIR ${NIFI_HOME}
# Startup NiFi
ENTRYPOINT ["bin/nifi.sh"]
-CMD ["run"]
+CMD ["run"]
--- End diff --
I tried to use Ctrl-C after NiFi was successfully up and running to kill the container. I had to open a new shell and use docker kill to bring it down. Not a big deal, I still think that's due to docker and having started the container in a non-interactive and non-detached method.
For the logging, I'm not sure if there's a reason why the two docker modules have different wrapper scripts. You could check with @apiri, but most likely it'd be good to bring them in line. It doesn't have to be done in this PR, though it'd be nice to get this into the NiFi 1.7.0 release if no one disagrees with it.
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by pepov <gi...@git.apache.org>.
Github user pepov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r195312869
--- Diff: nifi-docker/dockermaven/Dockerfile ---
@@ -26,23 +26,33 @@ ARG NIFI_BINARY
ENV NIFI_BASE_DIR /opt/nifi
ENV NIFI_HOME $NIFI_BASE_DIR/nifi-$NIFI_VERSION
-
-# Setup NiFi user
-RUN groupadd -g $GID nifi || groupmod -n nifi `getent group $GID | cut -d: -f1` \
- && useradd --shell /bin/bash -u $UID -g $GID -m nifi \
- && mkdir -p $NIFI_HOME/conf/templates \
- && chown -R nifi:nifi $NIFI_BASE_DIR
+ENV NIFI_PID_DIR=${NIFI_HOME}/run
+ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ADD $NIFI_BINARY $NIFI_BASE_DIR
-RUN chown -R nifi:nifi $NIFI_HOME
+# Setup NiFi user and create necessary directories
+RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d: -f1` \
+ && useradd --shell /bin/bash -u ${UID} -g ${GID} -m nifi \
+ && mkdir -p ${NIFI_HOME}/conf/templates \
+ && mkdir -p $NIFI_BASE_DIR/data \
+ && mkdir -p $NIFI_BASE_DIR/flowfile_repository \
+ && mkdir -p $NIFI_BASE_DIR/content_repository \
+ && mkdir -p $NIFI_BASE_DIR/provenance_repository \
+ && mkdir -p $NIFI_LOG_DIR \
+ && chown -R nifi:nifi ${NIFI_BASE_DIR} \
+ && apt-get update \
+ && apt-get install -y jq xmlstarlet procps
USER nifi
-# Web HTTP Port & Remote Site-to-Site Ports
-EXPOSE 8080 8181
+# Clear nifi-env.sh in favour of configuring all environment variables in the Dockerfile
+RUN echo "#!/bin/sh\n" > $NIFI_HOME/bin/nifi-env.sh
+
+# Web HTTP(s) & Socket Site-to-Site Ports
+EXPOSE 8080 8443 10000
-WORKDIR $NIFI_HOME
+WORKDIR ${NIFI_HOME}
# Startup NiFi
ENTRYPOINT ["bin/nifi.sh"]
-CMD ["run"]
+CMD ["run"]
--- End diff --
I've just pushed what hopefully fixes this issue once and for all. Entrypoint and command instructions have their gotchas, and we have to be careful with those. Also added a comment what and why I changed around the entrypoint, please see in the commit: https://github.com/apache/nifi/pull/2747/commits/8aef89bfd3ec7d1771e6dd835c53a1ba1f61dda3#diff-2cef119cd914e1b710d41b387a0b72b2R61
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by zenfenan <gi...@git.apache.org>.
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r194404103
--- Diff: nifi-docker/dockermaven/pom.xml ---
@@ -44,7 +44,6 @@
</buildArgs>
<repository>apache/nifi</repository>
<tag>${project.version}</tag>
- <tag>latest</tag>
--- End diff --
Understood. Thanks.
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by pepov <gi...@git.apache.org>.
Github user pepov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r192981828
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -25,28 +25,37 @@ ARG GID=1000
ARG NIFI_VERSION=1.7.0
ARG MIRROR=https://archive.apache.org/dist
-ENV NIFI_BASE_DIR /opt/nifi
+ENV NIFI_BASE_DIR /opt/nifi
ENV NIFI_HOME=${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} \
NIFI_BINARY_URL=/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.tar.gz
+ENV NIFI_PID_DIR=${NIFI_HOME}/run
+ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ADD sh/ /opt/nifi/scripts/
-# Setup NiFi user
+# Setup NiFi user and create necessary directories
RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d: -f1` \
&& useradd --shell /bin/bash -u ${UID} -g ${GID} -m nifi \
&& mkdir -p ${NIFI_HOME}/conf/templates \
+ && mkdir -p $NIFI_BASE_DIR/data \
+ && mkdir -p $NIFI_BASE_DIR/flowfile_repository \
+ && mkdir -p $NIFI_BASE_DIR/content_repository \
+ && mkdir -p $NIFI_BASE_DIR/provenance_repository \
+ && mkdir -p $NIFI_LOG_DIR \
&& chown -R nifi:nifi ${NIFI_BASE_DIR} \
&& apt-get update \
- && apt-get install -y jq xmlstarlet
+ && apt-get install -y jq xmlstarlet procps
USER nifi
# Download, validate, and expand Apache NiFi binary.
RUN curl -fSL ${MIRROR}/${NIFI_BINARY_URL} -o ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.tar.gz \
- && echo "$(curl https://archive.apache.org/dist/${NIFI_BINARY_URL}.sha256) *${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.tar.gz" | sha256sum -c - \
+ && echo "$(curl ${MIRROR}/${NIFI_BINARY_URL}.sha256) *${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.tar.gz" | sha256sum -c - \
--- End diff --
I've restored the hardcoded url
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by pepov <gi...@git.apache.org>.
Github user pepov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r195301228
--- Diff: nifi-docker/dockermaven/Dockerfile ---
@@ -26,23 +26,33 @@ ARG NIFI_BINARY
ENV NIFI_BASE_DIR /opt/nifi
ENV NIFI_HOME $NIFI_BASE_DIR/nifi-$NIFI_VERSION
-
-# Setup NiFi user
-RUN groupadd -g $GID nifi || groupmod -n nifi `getent group $GID | cut -d: -f1` \
- && useradd --shell /bin/bash -u $UID -g $GID -m nifi \
- && mkdir -p $NIFI_HOME/conf/templates \
- && chown -R nifi:nifi $NIFI_BASE_DIR
+ENV NIFI_PID_DIR=${NIFI_HOME}/run
+ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ADD $NIFI_BINARY $NIFI_BASE_DIR
-RUN chown -R nifi:nifi $NIFI_HOME
+# Setup NiFi user and create necessary directories
+RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d: -f1` \
+ && useradd --shell /bin/bash -u ${UID} -g ${GID} -m nifi \
+ && mkdir -p ${NIFI_HOME}/conf/templates \
+ && mkdir -p $NIFI_BASE_DIR/data \
+ && mkdir -p $NIFI_BASE_DIR/flowfile_repository \
+ && mkdir -p $NIFI_BASE_DIR/content_repository \
+ && mkdir -p $NIFI_BASE_DIR/provenance_repository \
+ && mkdir -p $NIFI_LOG_DIR \
+ && chown -R nifi:nifi ${NIFI_BASE_DIR} \
+ && apt-get update \
+ && apt-get install -y jq xmlstarlet procps
USER nifi
-# Web HTTP Port & Remote Site-to-Site Ports
-EXPOSE 8080 8181
+# Clear nifi-env.sh in favour of configuring all environment variables in the Dockerfile
+RUN echo "#!/bin/sh\n" > $NIFI_HOME/bin/nifi-env.sh
+
+# Web HTTP(s) & Socket Site-to-Site Ports
+EXPOSE 8080 8443 10000
-WORKDIR $NIFI_HOME
+WORKDIR ${NIFI_HOME}
# Startup NiFi
ENTRYPOINT ["bin/nifi.sh"]
-CMD ["run"]
+CMD ["run"]
--- End diff --
Ctrl-C works for me, even if it's somewhat slow while nifi is booting (faster when already booted).
The dockermaven image does not contain the wrapper scripts that the dockerhub one does, this is why tailing and sending all logs to stdout doesn't work. I wasn't sure that was intentional or not but I can add it to make the two environments more similar.
---
[GitHub] nifi issue #2747: NIFI-5249 Dockerfile enhancements
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/nifi/pull/2747
Thanks, I'd like to try this and help with the review
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by pepov <gi...@git.apache.org>.
Github user pepov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r195311227
--- Diff: nifi-docker/dockermaven/Dockerfile ---
@@ -26,23 +26,33 @@ ARG NIFI_BINARY
ENV NIFI_BASE_DIR /opt/nifi
ENV NIFI_HOME $NIFI_BASE_DIR/nifi-$NIFI_VERSION
-
-# Setup NiFi user
-RUN groupadd -g $GID nifi || groupmod -n nifi `getent group $GID | cut -d: -f1` \
- && useradd --shell /bin/bash -u $UID -g $GID -m nifi \
- && mkdir -p $NIFI_HOME/conf/templates \
- && chown -R nifi:nifi $NIFI_BASE_DIR
+ENV NIFI_PID_DIR=${NIFI_HOME}/run
+ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ADD $NIFI_BINARY $NIFI_BASE_DIR
-RUN chown -R nifi:nifi $NIFI_HOME
+# Setup NiFi user and create necessary directories
+RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d: -f1` \
+ && useradd --shell /bin/bash -u ${UID} -g ${GID} -m nifi \
+ && mkdir -p ${NIFI_HOME}/conf/templates \
+ && mkdir -p $NIFI_BASE_DIR/data \
+ && mkdir -p $NIFI_BASE_DIR/flowfile_repository \
+ && mkdir -p $NIFI_BASE_DIR/content_repository \
+ && mkdir -p $NIFI_BASE_DIR/provenance_repository \
+ && mkdir -p $NIFI_LOG_DIR \
+ && chown -R nifi:nifi ${NIFI_BASE_DIR} \
+ && apt-get update \
+ && apt-get install -y jq xmlstarlet procps
USER nifi
-# Web HTTP Port & Remote Site-to-Site Ports
-EXPOSE 8080 8181
+# Clear nifi-env.sh in favour of configuring all environment variables in the Dockerfile
+RUN echo "#!/bin/sh\n" > $NIFI_HOME/bin/nifi-env.sh
+
+# Web HTTP(s) & Socket Site-to-Site Ports
+EXPOSE 8080 8443 10000
-WORKDIR $NIFI_HOME
+WORKDIR ${NIFI_HOME}
# Startup NiFi
ENTRYPOINT ["bin/nifi.sh"]
-CMD ["run"]
+CMD ["run"]
--- End diff --
You're right I totally missed that. The biggest problem is that it is even the case with the dockerhub image, which is much more painful. I think I know what the problem is and working on the fix.
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r192803756
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -25,28 +25,37 @@ ARG GID=1000
ARG NIFI_VERSION=1.7.0
ARG MIRROR=https://archive.apache.org/dist
-ENV NIFI_BASE_DIR /opt/nifi
+ENV NIFI_BASE_DIR /opt/nifi
ENV NIFI_HOME=${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} \
NIFI_BINARY_URL=/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.tar.gz
+ENV NIFI_PID_DIR=${NIFI_HOME}/run
+ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ADD sh/ /opt/nifi/scripts/
-# Setup NiFi user
+# Setup NiFi user and create necessary directories
RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d: -f1` \
&& useradd --shell /bin/bash -u ${UID} -g ${GID} -m nifi \
&& mkdir -p ${NIFI_HOME}/conf/templates \
+ && mkdir -p $NIFI_BASE_DIR/data \
+ && mkdir -p $NIFI_BASE_DIR/flowfile_repository \
+ && mkdir -p $NIFI_BASE_DIR/content_repository \
+ && mkdir -p $NIFI_BASE_DIR/provenance_repository \
+ && mkdir -p $NIFI_LOG_DIR \
&& chown -R nifi:nifi ${NIFI_BASE_DIR} \
&& apt-get update \
- && apt-get install -y jq xmlstarlet
+ && apt-get install -y jq xmlstarlet procps
USER nifi
# Download, validate, and expand Apache NiFi binary.
RUN curl -fSL ${MIRROR}/${NIFI_BINARY_URL} -o ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.tar.gz \
- && echo "$(curl https://archive.apache.org/dist/${NIFI_BINARY_URL}.sha256) *${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.tar.gz" | sha256sum -c - \
+ && echo "$(curl ${MIRROR}/${NIFI_BINARY_URL}.sha256) *${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.tar.gz" | sha256sum -c - \
--- End diff --
@pepov It is an automated build in Docker Hub. The Dockerfile is used as shown without any external arguments when a rel/nifi-<version> tag is created
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r192208160
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -25,28 +25,37 @@ ARG GID=1000
ARG NIFI_VERSION=1.7.0
ARG MIRROR=https://archive.apache.org/dist
-ENV NIFI_BASE_DIR /opt/nifi
+ENV NIFI_BASE_DIR /opt/nifi
ENV NIFI_HOME=${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} \
NIFI_BINARY_URL=/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.tar.gz
+ENV NIFI_PID_DIR=${NIFI_HOME}/run
+ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ADD sh/ /opt/nifi/scripts/
-# Setup NiFi user
+# Setup NiFi user and create necessary directories
RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d: -f1` \
&& useradd --shell /bin/bash -u ${UID} -g ${GID} -m nifi \
&& mkdir -p ${NIFI_HOME}/conf/templates \
+ && mkdir -p $NIFI_BASE_DIR/data \
+ && mkdir -p $NIFI_BASE_DIR/flowfile_repository \
+ && mkdir -p $NIFI_BASE_DIR/content_repository \
+ && mkdir -p $NIFI_BASE_DIR/provenance_repository \
+ && mkdir -p $NIFI_LOG_DIR \
&& chown -R nifi:nifi ${NIFI_BASE_DIR} \
&& apt-get update \
- && apt-get install -y jq xmlstarlet
+ && apt-get install -y jq xmlstarlet procps
USER nifi
# Download, validate, and expand Apache NiFi binary.
RUN curl -fSL ${MIRROR}/${NIFI_BINARY_URL} -o ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.tar.gz \
- && echo "$(curl https://archive.apache.org/dist/${NIFI_BINARY_URL}.sha256) *${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.tar.gz" | sha256sum -c - \
+ && echo "$(curl ${MIRROR}/${NIFI_BINARY_URL}.sha256) *${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.tar.gz" | sha256sum -c - \
--- End diff --
@pepov I would suggest removing the MIRROR build arg from this line and reverting back to the apache archive, since from what @apiri has told me, only the Apache archive will host the SHA files to verify the archive. A mirror will not contain those.
Also, there's a caveat with using a mirror. If you're not building a version that still exists on the mirror (which should be current and current-1), the build will fail, if that version has been removed/rolled off from the mirror.
---
[GitHub] nifi issue #2747: NIFI-5249 Dockerfile enhancements
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/nifi/pull/2747
+1 fwiw|lgtm.
Ran the above steps to build and test the image from the maven snapshots ( not built locally ).
Everything ran fine.
Your integration test is awesome. I'm totally going to steal it.
Super stuff.
---
[GitHub] nifi issue #2747: NIFI-5249 Dockerfile enhancements
Posted by pepov <gi...@git.apache.org>.
Github user pepov commented on the issue:
https://github.com/apache/nifi/pull/2747
Are there any issues left I did not address?
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by pepov <gi...@git.apache.org>.
Github user pepov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r192787369
--- Diff: nifi-docker/dockermaven/pom.xml ---
@@ -44,7 +44,6 @@
</buildArgs>
<repository>apache/nifi</repository>
<tag>${project.version}</tag>
- <tag>latest</tag>
--- End diff --
I removed it because it did not build both tags, just the last one, so I had a choice. Also this is only a dev/test config and does not affect the public image.
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r195240352
--- Diff: nifi-docker/dockermaven/integration-test.sh ---
@@ -0,0 +1,35 @@
+#!/bin/bash
--- End diff --
This script is missing the license, can you please add it?
---
[GitHub] nifi issue #2747: NIFI-5249 Dockerfile enhancements
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/nifi/pull/2747
So to test this?
- mvn package
- mvn package -P docker from nifi-docker
- ???
- docker run --rm -ti --entrypoint /bin/bash apache/nifi -c "env | grep NIFI" ? from nifi-docker dir?
- docker run --rm -ti --entrypoint /bin/bash apache/nifi -c "find /opt/nifi ! -user nifi" from nifi-docker dir?
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by pepov <gi...@git.apache.org>.
Github user pepov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r192781212
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -25,28 +25,37 @@ ARG GID=1000
ARG NIFI_VERSION=1.7.0
ARG MIRROR=https://archive.apache.org/dist
-ENV NIFI_BASE_DIR /opt/nifi
+ENV NIFI_BASE_DIR /opt/nifi
ENV NIFI_HOME=${NIFI_BASE_DIR}/nifi-${NIFI_VERSION} \
NIFI_BINARY_URL=/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.tar.gz
+ENV NIFI_PID_DIR=${NIFI_HOME}/run
+ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ADD sh/ /opt/nifi/scripts/
-# Setup NiFi user
+# Setup NiFi user and create necessary directories
RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d: -f1` \
&& useradd --shell /bin/bash -u ${UID} -g ${GID} -m nifi \
&& mkdir -p ${NIFI_HOME}/conf/templates \
+ && mkdir -p $NIFI_BASE_DIR/data \
+ && mkdir -p $NIFI_BASE_DIR/flowfile_repository \
+ && mkdir -p $NIFI_BASE_DIR/content_repository \
+ && mkdir -p $NIFI_BASE_DIR/provenance_repository \
+ && mkdir -p $NIFI_LOG_DIR \
&& chown -R nifi:nifi ${NIFI_BASE_DIR} \
&& apt-get update \
- && apt-get install -y jq xmlstarlet
+ && apt-get install -y jq xmlstarlet procps
USER nifi
# Download, validate, and expand Apache NiFi binary.
RUN curl -fSL ${MIRROR}/${NIFI_BINARY_URL} -o ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.tar.gz \
- && echo "$(curl https://archive.apache.org/dist/${NIFI_BINARY_URL}.sha256) *${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.tar.gz" | sha256sum -c - \
+ && echo "$(curl ${MIRROR}/${NIFI_BINARY_URL}.sha256) *${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.tar.gz" | sha256sum -c - \
--- End diff --
Ok, I just borrowed it from @markap14 and it made sense since the value is the same. Where can we see the build config of the public docker image?
---
[GitHub] nifi issue #2747: NIFI-5249 Dockerfile enhancements
Posted by pepov <gi...@git.apache.org>.
Github user pepov commented on the issue:
https://github.com/apache/nifi/pull/2747
To reply to my own question: I think we should also test the dockerhub image almost the same way we test the maven one. We just have to use the latest public nifi archive to do that.
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r195262129
--- Diff: nifi-docker/dockermaven/Dockerfile ---
@@ -26,23 +26,33 @@ ARG NIFI_BINARY
ENV NIFI_BASE_DIR /opt/nifi
ENV NIFI_HOME $NIFI_BASE_DIR/nifi-$NIFI_VERSION
-
-# Setup NiFi user
-RUN groupadd -g $GID nifi || groupmod -n nifi `getent group $GID | cut -d: -f1` \
- && useradd --shell /bin/bash -u $UID -g $GID -m nifi \
- && mkdir -p $NIFI_HOME/conf/templates \
- && chown -R nifi:nifi $NIFI_BASE_DIR
+ENV NIFI_PID_DIR=${NIFI_HOME}/run
+ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ADD $NIFI_BINARY $NIFI_BASE_DIR
-RUN chown -R nifi:nifi $NIFI_HOME
+# Setup NiFi user and create necessary directories
+RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d: -f1` \
+ && useradd --shell /bin/bash -u ${UID} -g ${GID} -m nifi \
+ && mkdir -p ${NIFI_HOME}/conf/templates \
+ && mkdir -p $NIFI_BASE_DIR/data \
+ && mkdir -p $NIFI_BASE_DIR/flowfile_repository \
+ && mkdir -p $NIFI_BASE_DIR/content_repository \
+ && mkdir -p $NIFI_BASE_DIR/provenance_repository \
+ && mkdir -p $NIFI_LOG_DIR \
+ && chown -R nifi:nifi ${NIFI_BASE_DIR} \
+ && apt-get update \
+ && apt-get install -y jq xmlstarlet procps
USER nifi
-# Web HTTP Port & Remote Site-to-Site Ports
-EXPOSE 8080 8181
+# Clear nifi-env.sh in favour of configuring all environment variables in the Dockerfile
+RUN echo "#!/bin/sh\n" > $NIFI_HOME/bin/nifi-env.sh
+
+# Web HTTP(s) & Socket Site-to-Site Ports
+EXPOSE 8080 8443 10000
-WORKDIR $NIFI_HOME
+WORKDIR ${NIFI_HOME}
# Startup NiFi
ENTRYPOINT ["bin/nifi.sh"]
-CMD ["run"]
+CMD ["run"]
--- End diff --
Creating a container with:
`docker run -p 8080:8080 apache/nifi:1.7.0-SNAPSHOT-dockermaven`
results in NiFi starting successfully, but I'm unable to control-c out of the container. I'm not a docker expert, but I would expect that hitting control-c would kill the container. Although, since I didn't run it in interactive mode, this is probably a docker thing, not specific to this Dockerfile.
Starting the container with:
`docker run -d -p 8080:8080 apache/nifi:1.7.0-SNAPSHOT-dockermaven`
and then issuing:
`docker logs <container>`
I see the nifi-bootstrap output, but not the nifi-app.log and nifi-user.log output. Would it be preferable to have this behavior?
---
[GitHub] nifi issue #2747: NIFI-5249 Dockerfile enhancements
Posted by pepov <gi...@git.apache.org>.
Github user pepov commented on the issue:
https://github.com/apache/nifi/pull/2747
Now all the above described test steps can be executed with `mvn package -P docker` from the `nifi-docker` folder
---
[GitHub] nifi issue #2747: NIFI-5249 Dockerfile enhancements
Posted by pepov <gi...@git.apache.org>.
Github user pepov commented on the issue:
https://github.com/apache/nifi/pull/2747
Rebased and squashed commits
---
[GitHub] nifi issue #2747: NIFI-5249 Dockerfile enhancements
Posted by pepov <gi...@git.apache.org>.
Github user pepov commented on the issue:
https://github.com/apache/nifi/pull/2747
yes, working on it
---
[GitHub] nifi issue #2747: NIFI-5249 Dockerfile enhancements
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/nifi/pull/2747
Can you add explicit steps to test and verify, for those of use who don't work with docker as much?
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2747
---