You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by jdye64 <gi...@git.apache.org> on 2016/12/29 20:37:07 UTC
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
GitHub user jdye64 opened a pull request:
https://github.com/apache/nifi/pull/1372
NIFI-3260 Official Docker Image
Added baseline Docker image for NiFi version 1.1.1 and 1.1.0. These are
MVP images that we can massage as we go along once they have been
accepted in the formal official Dockerhub repo. Will also add all of
the older releases once our community agrees on the format for the
Dockerfiles.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jdye64/nifi NIFI-3260
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1372.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1372
----
commit 884f75cc50ebb2a72360139af7689e8f62d45860
Author: Jeremy Dyer <jd...@gmail.com>
Date: 2016-12-29T20:36:04Z
NIFI-3260 Official Docker Image
Added baseline Docker image for NiFi version 1.1.1 and 1.1.0. These are
MVP images that we can massage as we go along once they have been
accepted in the formal official Dockerhub repo. Will also add all of
the older releases once our community agrees on the format for the
Dockerfiles.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/1372
Hey @jdye64, thanks for starting this so quickly! It will be great to have an official nifi docker image.
I'm not sure the way you have it set up currently is the best solution for maintainability. It's weird because it would mean having the sources be self referencing ie. when we do 1.2.0 we couldn't vote with the docker image there because there would be no release yet.
I wonder if it's best to have this as an external repo? Similar to nifi-site[1] and nifi-maven[2]
[1] https://github.com/apache/nifi-site
[2] https://github.com/apache/nifi-maven
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by jdye64 <gi...@git.apache.org>.
Github user jdye64 commented on the issue:
https://github.com/apache/nifi/pull/1372
@apiri I didn't realize that the tags alone were sufficient without the directories but after reading a little deeper it seems like your right. Let me remove the directories and consolidate to a single Dockerfile and make another commit then we can continue discussion from there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r95401180
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+FROM java:8
+MAINTAINER Apache NiFi <de...@nifi.apache.org>
+
+ARG UID
+ARG GID
+ARG NIFI_VERSION
+
+ENV NIFI_BASE_DIR /opt/nifi
+ENV NIFI_HOME $NIFI_BASE_DIR/nifi-$NIFI_VERSION
+ENV NIFI_BINARY_URL https://archive.apache.org/dist/nifi/$NIFI_VERSION/nifi-$NIFI_VERSION-bin.tar.gz
+ENV NIFI_DOWNLOAD_SHA256 afb58050b0904dab329b19b14250b57d3978743a1108323bdc52aba0c2a75853
--- End diff --
This will need to be the file released with the source artifact as well. In this case, https://archive.apache.org/dist/nifi/$NIFI_VERSION/nifi-$NIFI_VERSION-bin.tar.gz.sha256
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by jdye64 <gi...@git.apache.org>.
Github user jdye64 commented on the issue:
https://github.com/apache/nifi/pull/1372
@achristianson wow. I saw the io.fabric8 one but over the spotify one. Thanks for pointing that out
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by jdye64 <gi...@git.apache.org>.
Github user jdye64 commented on the issue:
https://github.com/apache/nifi/pull/1372
Good point about Docker ARG. If the plugin doesn't currently support I'm sure it would be an easy add.
Sorry I wasn't very clear there. I was thinking about a more infrastructure level. For us to actually host our official image on Dockerhub there must be a unique tag, commit, and directory that points to our Dockerfile for a specific Docker tag. So if we had a minor release we would need to create a new directory. That is why in this commit I have the two directories "1.1.0" and "1.1.1" to illustrate that. I don't like that structure at all but it almost seems necessary to host via Dockerhub. This is somewhat of a limitation for "official" images only which most likely other users of the docker-maven-plugin are not doing but rather making internal release and sending to their own private docker repos.
So when we made a PR to https://github.com/docker-library/official-images our PR would be a single file looking something like ...
https://github.com/docker-library/official-images/blob/master/library/storm
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by achristianson <gi...@git.apache.org>.
Github user achristianson commented on the issue:
https://github.com/apache/nifi/pull/1372
@jdye64 If we do end up going the route of injecting the build #, Dockerfile ARG statements might be the way to go. The docker-maven-plugin would be the place to set the arg, if that plugin supports it. Unsure if they do.
What sort of image maintenance are you concerned with? Are you talking about going back and releasing minor releases? In my experience with docker, the way it would work is with image tags. E.g. the "centos7" tag of the official "centos" image tracks the 7.x release. This would be analogous to a "nifi1.0.x" or "nifi-1.1.x" tag, or however it would best be named. In that arrangement, it would come down to compatibility between versions. If I'm running a production release of nifi 1.1.x, I want to get the latest bugfixes/patches, but I do not want to be upgraded to an incompatible version, i.e. 1.2.x.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by jdye64 <gi...@git.apache.org>.
Github user jdye64 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r94867503
--- Diff: nifi-docker/docker/Dockerfile ---
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+FROM java:8u91-jdk
+MAINTAINER Apache NiFi <de...@nifi.apache.org>
+
+ARG UID
+ARG GID
+ARG NIFI_VERSION
+ARG NIFI_BINARY
+
+ENV NIFI_BASE_DIR /opt/nifi
+ENV NIFI_HOME $NIFI_BASE_DIR/nifi-$NIFI_VERSION
+
+# Setup NiFi user
+RUN groupadd -g $GID nifi || groupmod -n nifi `getent group $GID | cut -d: -f1`
+RUN useradd --shell /bin/bash -u $UID -g $GID -m nifi
+RUN mkdir -p $NIFI_HOME
+
+ADD $NIFI_BINARY $NIFI_BASE_DIR
--- End diff --
I did a little research and your right. It will have to download the binary since the built binary would not be referenceable. I don't think it would be the end of the world to have 2 Dockerfiles. Really the files could be identically aside from the piece that pulls in the binary. This one could be used for people doing local builds and development while the one that downloads the binary could be used for pushes to Dockerhub. I'll work something up for that here shortly and make another commit.
Thanks everyone for chiming in on this. There are a lot of ways to do this I know and things keep changing a little bit but we will get there =)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by achristianson <gi...@git.apache.org>.
Github user achristianson commented on the issue:
https://github.com/apache/nifi/pull/1372
As a counterpoint to the idea of putting it in a separate repo, would it be possible to produce the official image as part of a maven build stage? To reference an internet URL somewhat breaks the idea that the build system works of of package names/versions. Both maven and the OS (alpine, centos, etc.) reference dependencies by name/version only. If we can avoid this, we also improve portability of the builds and prevent issues when the upstream HTTP url of the referenced built nifi changes or goes down.
What the others say about the self-referencing version makes sense, too.
Basically, if building the image were another build stage, then the version of the docker image produced would just be in sync with the source tree as a whole. E.g. you build a 1.1.0 release, and the java build/package produces a 1.1.0 binary, then the final docker packaging stage produces a 1.1.0 image.
The docker-maven-plugin that @ddewaele references looks like a perfect way to add this as another maven stage.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by brosander <gi...@git.apache.org>.
Github user brosander commented on the issue:
https://github.com/apache/nifi/pull/1372
May need to add rat ignores for things that can't have license headers in them
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by jdye64 <gi...@git.apache.org>.
Github user jdye64 commented on the issue:
https://github.com/apache/nifi/pull/1372
@ddewaele The plan is to certainly add volumes. For this first pass however we might not add it just to get things like infrastructure ironed out and simple. It should be really easy to add volumes in our next iteration, which doesn't even necessarily have to wait until the next release.
As for the custom processors you could still extend this Apache image the same way you are doing today and add your custom processors. The official image will only contain the base processors however.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r94821158
--- Diff: nifi-docker/README.md ---
@@ -0,0 +1,33 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of Apache NiFi include:
+
+- Web-based user interface
+-- Seamless experience between design, control, feedback, and monitoring
+- Highly configurable
+-- Loss tolerant vs guaranteed delivery
--- End diff --
This markdown formatting is a little wonky on GitHub with the nested items: https://github.com/jdye64/nifi/blob/265cdbcfbd7ccb84aabb7b68e2451e813b1b60f3/nifi-docker/README.md
I think if it gets changed to a bulleted list it may be okay.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the issue:
https://github.com/apache/nifi/pull/1372
Sounds good to me. That's likely what I would do as well. Unfortunately the chown was a costly misstep for image size in this case. Mostly if you are interested in doing so feel free to throw a PR out there otherwise we will certainly scope it out and address. Just want to give you the credit for it if you would like. Thanks again!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r95184717
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+FROM java:8u91-jdk
--- End diff --
Perhaps we should just use version 8? This ensures we are always on the latest update for that build. Additionally, we shouldn't need the JDK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by jdye64 <gi...@git.apache.org>.
Github user jdye64 commented on the issue:
https://github.com/apache/nifi/pull/1372
@achristianson I have mixed emotions about incorporating this into the build process but I am ok with it if that is what most people think. My original thought was that this would be a more singular build that we as a community release for end users to download. However from this discussion it seems like most people might be more interested in the build process approach.
In that case it should be pretty simple to do a little maven magic to inject the current version into the Dockerfile and use the docker-maven-plugin.
I do believe this might make maintenance on older Docker images a little harder since they would be tightly coupled to a release/branch however. Does anyone more seasoned than myself have ideas about how to make that process easier?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by achristianson <gi...@git.apache.org>.
Github user achristianson commented on the issue:
https://github.com/apache/nifi/pull/1372
@jdye64 Looks like there are two in central; one by fabric8 and one by spotify:
https://search.maven.org/#artifactdetails|io.fabric8|docker-maven-plugin|0.19.0|maven-plugin
https://search.maven.org/#artifactdetails|com.spotify|docker-maven-plugin|0.4.13|maven-plugin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by cricket007 <gi...@git.apache.org>.
Github user cricket007 commented on the issue:
https://github.com/apache/nifi/pull/1372
I mean, it seems to be an open discussion in Docker community for some time.
I'm not too skilled in layer management to know the "correct approach" at this moment, but the first thing I would try would be to merge in `&& chown -R ...` into the previous layer. I'll copy your Dockerfile and see what I get.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r94825509
--- Diff: nifi-docker/docker/Dockerfile ---
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+FROM java:8u91-jdk
+MAINTAINER Apache NiFi <de...@nifi.apache.org>
+
+ARG UID
+ARG GID
+ARG NIFI_VERSION
+ARG NIFI_BINARY
+
+ENV NIFI_BASE_DIR /opt/nifi
+ENV NIFI_HOME $NIFI_BASE_DIR/nifi-$NIFI_VERSION
+
+# Setup NiFi user
+RUN groupadd -g $GID nifi || groupmod -n nifi `getent group $GID | cut -d: -f1`
+RUN useradd --shell /bin/bash -u $UID -g $GID -m nifi
+RUN mkdir -p $NIFI_HOME
+
+ADD $NIFI_BINARY $NIFI_BASE_DIR
--- End diff --
Will this work for Docker Hub? It was my understanding that process runs a docker build which wouldn't include a built binary referenceable from that environment. I think this is nice functionality to have, but wonder if we need to defer to pulling from a mirror upon release. Thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r95187906
--- Diff: nifi-docker/dockermaven/Dockerfile ---
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+FROM java:8u91-jdk
--- End diff --
Same as above
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the issue:
https://github.com/apache/nifi/pull/1372
reviewing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by achristianson <gi...@git.apache.org>.
Github user achristianson commented on the issue:
https://github.com/apache/nifi/pull/1372
@jdye64 Interesting, wasn't aware of the restrictions with the official docker hub. When I go look at some other projects for comparison, it looks roughly like the directory is used for compatibility releases. E.g. with postgres you have 9.5, 9.6 directories, but each minor release maps to several tags. The java image seems to break down the directories per major release (6, 7), and JDK vs JRE. Looking at centos, there is no directory: https://github.com/docker-library/official-images/blob/master/library/centos.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r95184676
--- Diff: nifi-docker/dockerhub/DockerBuild.sh ---
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#!/bin/bash
+
+DOCKER_UID=1000
+if [ -n "$1" ]; then
+ DOCKER_UID="$1"
+fi
+
+DOCKER_GID=50
+if [ -n "$2" ]; then
+ DOCKER_GID="$2"
+fi
+
+DOCKER_IMAGE="$(egrep -v '(^#|^\s*$|^\s*\t*#)' DockerImage.txt)"
+NIFI_IMAGE_VERSION="$(echo "apachenifi:1.1.1" | cut -d : -f 2)"
--- End diff --
Should this also be derived from DockerImage.txt?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by cricket007 <gi...@git.apache.org>.
Github user cricket007 commented on the issue:
https://github.com/apache/nifi/pull/1372
Why is this image so large? Did someone not realize that `chown` is doubling the layer that downloads the binary?
`docker history --no-trunc apache/nifi:1.3.0`
```
3 GID=50 NIFI_VERSION=1.3.0 UID=1000 /bin/sh -c chown -R nifi:nifi $NIFI_HOME => 983 MB
3 GID=50 NIFI_VERSION=1.3.0 UID=1000 /bin/sh -c curl -fSL $NIFI_BINARY_URL -o $NIFI_BASE_DIR/nifi-$NIFI_VERSION-bin.tar.gz ... => 983 MB
```
For comparison, I have built this myself (granted, using openjdk-alpine and without switching user accounts), but still, a 2.4x size difference!
```
local/nifi 1.3.0 f28ea17bf37b 15 minutes ago 1.07 GB
apache/nifi 1.3.0 ccd137e275bc 32 hours ago 2.58 GB
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by p1ck <gi...@git.apache.org>.
Github user p1ck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r96285950
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+FROM java:8
--- End diff --
The `java` image has been deprecated in favor of the `openjdk` image.
source: https://hub.docker.com/_/java/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the issue:
https://github.com/apache/nifi/pull/1372
@jdye64 looks good here and should be a nice starting point. will merge and then we can look into working with INFRA to get this incorporated with the Apache Docker Hub account.
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/1372
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r95191911
--- Diff: nifi-docker/README.md ---
@@ -0,0 +1,33 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of Apache NiFi include:
+
+* Web-based user interface
--- End diff --
Might want to capture details on exposed ports and their intended usage (HTTP vs Site to Site) and an overview of what scripts are provided. Not exactly sure how this will map with the verified repo, but the more information we can include, the better for when it manifests itself on Docker Hub.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r97082483
--- Diff: nifi-docker/1.1.1/DockerRun.sh ---
@@ -0,0 +1,3 @@
+#!/bin/bash
+DOCKER_IMAGE="$(cat DockerImage.txt)"
+docker run -it -d -p 8080:8080 -p 8181:8181 $DOCKER_IMAGE
--- End diff --
The port mapping should also be present when a given container is started again
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r96308936
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+FROM java:8
--- End diff --
Good catch. This was overlooked and #1419 has this queued up to be incorporated with some other changes to facilitate our builds on release.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by gauravgoyal86 <gi...@git.apache.org>.
Github user gauravgoyal86 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r96966952
--- Diff: nifi-docker/1.1.1/DockerRun.sh ---
@@ -0,0 +1,3 @@
+#!/bin/bash
+DOCKER_IMAGE="$(cat DockerImage.txt)"
+docker run -it -d -p 8080:8080 -p 8181:8181 $DOCKER_IMAGE
--- End diff --
Do we have to execute following command every time we want to start nifi docker instance?
docker run -it -d -p 8080:8080 -p 8181:8181 $DOCKER_IMAGE
or once docker container is created, we can do docker stop and docker start.
i am trying that with my docker container. It works well with docker run. but it does work well with docker stop and docker start.
Can you try that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by jdye64 <gi...@git.apache.org>.
Github user jdye64 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r95209790
--- Diff: nifi-docker/dockerhub/DockerBuild.sh ---
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#!/bin/bash
+
+DOCKER_UID=1000
+if [ -n "$1" ]; then
+ DOCKER_UID="$1"
+fi
+
+DOCKER_GID=50
+if [ -n "$2" ]; then
+ DOCKER_GID="$2"
+fi
+
+DOCKER_IMAGE="$(egrep -v '(^#|^\s*$|^\s*\t*#)' DockerImage.txt)"
+NIFI_IMAGE_VERSION="$(echo "apachenifi:1.1.1" | cut -d : -f 2)"
--- End diff --
it certainly should. I must have just overlooked that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the issue:
https://github.com/apache/nifi/pull/1372
Hey @gauravgoyal86,
Didn't realize you were referencing the image in the repository versus your custom one. I created an issue (https://issues.apache.org/jira/browse/NIFI-3835) such that we can incorporate this into the official image.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by gauravgoyal86 <gi...@git.apache.org>.
Github user gauravgoyal86 commented on the issue:
https://github.com/apache/nifi/pull/1372
Hey @apiri
Thanks for your quick response and creating a JIRA ticket
I am using my custom docker image but i get the same this with generic docker image.
We get 2 process.
Docker stop nifi kill PID1 but child process takes PID 1 and we could never close that process.
need your help to sort out this problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by brosander <gi...@git.apache.org>.
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r94178631
--- Diff: nifi-docker/1.1.0/nifi.properties ---
@@ -0,0 +1,204 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Core Properties #
+nifi.version=1.1.0
+nifi.flow.configuration.file=./conf/flow.xml.gz
+nifi.flow.configuration.archive.enabled=true
+nifi.flow.configuration.archive.dir=./conf/archive/
+nifi.flow.configuration.archive.max.time=30 days
+nifi.flow.configuration.archive.max.storage=500 MB
+nifi.flowcontroller.autoResumeState=true
+nifi.flowcontroller.graceful.shutdown.period=10 sec
+nifi.flowservice.writedelay.interval=500 ms
+nifi.administrative.yield.duration=30 sec
+# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
+nifi.bored.yield.duration=10 millis
+
+nifi.authorizer.configuration.file=./conf/authorizers.xml
+nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
+nifi.templates.directory=./conf/templates
+nifi.ui.banner.text=
+nifi.ui.autorefresh.interval=30 sec
+nifi.nar.library.directory=./lib
+nifi.nar.working.directory=./work/nar/
+nifi.documentation.working.directory=./work/docs/components
+
+####################
+# State Management #
+####################
+nifi.state.management.configuration.file=./conf/state-management.xml
+# The ID of the local state provider
+nifi.state.management.provider.local=local-provider
+# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
+nifi.state.management.provider.cluster=zk-provider
+# Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
+nifi.state.management.embedded.zookeeper.start=false
+# Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
+nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
+
+
+# H2 Settings
+nifi.database.directory=./database_repository
+nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
+
+# FlowFile Repository
+nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
+nifi.flowfile.repository.directory=./flowfile_repository
+nifi.flowfile.repository.partitions=256
+nifi.flowfile.repository.checkpoint.interval=2 mins
+nifi.flowfile.repository.always.sync=false
+
+nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
+nifi.queue.swap.threshold=20000
+nifi.swap.in.period=5 sec
+nifi.swap.in.threads=1
+nifi.swap.out.period=5 sec
+nifi.swap.out.threads=4
+
+# Content Repository
+nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
+nifi.content.claim.max.appendable.size=10 MB
+nifi.content.claim.max.flow.files=100
+nifi.content.repository.directory.default=./content_repository
+nifi.content.repository.archive.max.retention.period=12 hours
+nifi.content.repository.archive.max.usage.percentage=50%
+nifi.content.repository.archive.enabled=true
+nifi.content.repository.always.sync=false
+nifi.content.viewer.url=/nifi-content-viewer/
+
+# Provenance Repository Properties
+nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
+
+# Persistent Provenance Repository Properties
+nifi.provenance.repository.directory.default=./provenance_repository
+nifi.provenance.repository.max.storage.time=24 hours
+nifi.provenance.repository.max.storage.size=1 GB
+nifi.provenance.repository.rollover.time=30 secs
+nifi.provenance.repository.rollover.size=100 MB
+nifi.provenance.repository.query.threads=2
+nifi.provenance.repository.index.threads=1
+nifi.provenance.repository.compress.on.rollover=true
+nifi.provenance.repository.always.sync=false
+nifi.provenance.repository.journal.count=16
+# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
+# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
+nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
+# FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
+nifi.provenance.repository.indexed.attributes=
+# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
+# but should provide better performance
+nifi.provenance.repository.index.shard.size=500 MB
+# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
+# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
+nifi.provenance.repository.max.attribute.length=65536
+
+# Volatile Provenance Respository Properties
+nifi.provenance.repository.buffer.size=100000
+
+# Component Status Repository
+nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
+nifi.components.status.repository.buffer.size=1440
+nifi.components.status.snapshot.frequency=1 min
+
+# Site to Site properties
+nifi.remote.input.host=
+nifi.remote.input.secure=false
+nifi.remote.input.socket.port=8181
+nifi.remote.input.http.enabled=true
+nifi.remote.input.http.transaction.ttl=30 sec
+
+# web properties #
+nifi.web.war.directory=./lib
+nifi.web.http.host=
+nifi.web.http.port=8080
+nifi.web.https.host=
+nifi.web.https.port=
+nifi.web.jetty.working.directory=./work/jetty
+nifi.web.jetty.threads=200
+
+# security properties #
+nifi.sensitive.props.key=
+nifi.sensitive.props.key.protected=
+nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
+nifi.sensitive.props.provider=BC
+nifi.sensitive.props.additional.keys=
+
+nifi.security.keystore=
+nifi.security.keystoreType=
+nifi.security.keystorePasswd=
+nifi.security.keyPasswd=
+nifi.security.truststore=
+nifi.security.truststoreType=
+nifi.security.truststorePasswd=
+nifi.security.needClientAuth=
+nifi.security.user.authorizer=file-provider
+nifi.security.user.login.identity.provider=
+nifi.security.ocsp.responder.url=
+nifi.security.ocsp.responder.certificate=
+
+# Identity Mapping Properties #
+# These properties allow normalizing user identities such that identities coming from different identity providers
+# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
+# DNs from certificates and principals from Kerberos into a common identity string:
+#
+# nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
+# nifi.security.identity.mapping.value.dn=$1@$2
+# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
+# nifi.security.identity.mapping.value.kerb=$1@$2
+
+# cluster common properties (all nodes must have same values) #
+nifi.cluster.protocol.heartbeat.interval=5 sec
+nifi.cluster.protocol.is.secure=false
+
+# cluster node properties (only configure for cluster nodes) #
+nifi.cluster.is.node=false
+nifi.cluster.node.address=
+nifi.cluster.node.protocol.port=
+nifi.cluster.node.protocol.threads=10
+nifi.cluster.node.event.history.size=25
+nifi.cluster.node.connection.timeout=5 sec
+nifi.cluster.node.read.timeout=5 sec
+nifi.cluster.firewall.file=
+nifi.cluster.flow.election.max.wait.time=5 mins
+nifi.cluster.flow.election.max.candidates=
+
+# zookeeper properties, used for cluster management #
+nifi.zookeeper.connect.string=
+nifi.zookeeper.connect.timeout=3 secs
+nifi.zookeeper.session.timeout=3 secs
+nifi.zookeeper.root.node=/nifi
+
+# kerberos #
+nifi.kerberos.krb5.file=
+
+# kerberos service principal #
+nifi.kerberos.service.principal=
+nifi.kerberos.service.keytab.location=
+
+# kerberos spnego principal #
+nifi.kerberos.spnego.principal=
+nifi.kerberos.spnego.keytab.location=
+nifi.kerberos.spnego.authentication.expiration=12 hours
+
+# external properties files for variable registry
+# supports a comma delimited list of file locations
+nifi.variable.registry.properties=
+
+# Build info
--- End diff --
should this be in sync with the Dockerfile?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r95191837
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+FROM java:8u91-jdk
--- End diff --
Additionally, it seems that as of the end of last year, the java repository is deprecated and should use openjdk (https://hub.docker.com/_/openjdk/) instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r94821856
--- Diff: nifi-docker/LICENSE ---
@@ -0,0 +1,202 @@
+
--- End diff --
This LICENSE is unneeded. This code would be covered by the root LICENSE.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/1372
@apiri @jdye64 had replied to my comment on the JIRA here[1] probably best to keep the convo here though.
[1] https://issues.apache.org/jira/browse/NIFI-3260?focusedCommentId=15786103&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15786103
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by gauravgoyal86 <gi...@git.apache.org>.
Github user gauravgoyal86 commented on the issue:
https://github.com/apache/nifi/pull/1372
I have just tried using dumb-init
now i am getting following three processes
`15-096-Goyal:exanova-templates gaurav.goyal$ docker exec -it d053f9d31436 ps -ef
PID USER TIME COMMAND
1 root 0:00 dumb-init /opt/nifi/bin/nifi.sh run
7 root 0:00 {nifi.sh} /bin/sh /opt/nifi/bin/nifi.sh run
19 root 0:00 {nifi.sh} /bin/sh /opt/nifi/bin/nifi.sh run`
executing docker stop gave "ExitCode": 143.
is it correct way to use dumb-init?
`def run():
logger.info('Starting Nifi.')
os.execl('/usr/bin/dumb-init', 'dumb-init', '/opt/nifi/bin/nifi.sh', 'run')`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by gauravgoyal86 <gi...@git.apache.org>.
Github user gauravgoyal86 commented on the issue:
https://github.com/apache/nifi/pull/1372
Thanks for nifi docker image!!
I am using this docker image. i found 2 nifi.sh run processes running under nifi docker container
`15-096-Goyal:exanova-templates gaurav.goyal$ docker exec -it blissful_banach bash
bash-4.2$ ps -ef
UID PID PPID C STIME TTY TIME CMD
nifi 1 0 0 19:26 ? 00:00:00 /bin/sh bin/nifi.sh run
nifi 17 1 0 19:26 ? 00:00:00 /bin/sh bin/nifi.sh run`
Now, When we run docker nifi stop, it will send kill to PID 1, but it does not stop PID 17.
Thats why we do not get exit status code 0..
Infact we get code 137.
May we know how can we cleanly stop nifi data container including bg and fg processes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by brosander <gi...@git.apache.org>.
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r94179174
--- Diff: nifi-docker/1.1.0/Dockerfile ---
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+FROM java:8u91-jdk
+MAINTAINER Apache NiFi <de...@nifi.apache.org>
+
+ENV NIFI_VERSION=1.1.0
+ENV NIFI_HOME=/nifi-$NIFI_VERSION
+ENV NIFI_DOWNLOAD_SHA256 3190d5b27d2668cf9a61ad458f666a686b315241025b1b4fb2f4e40f121ffb81
+
+LABEL owner=Apache\ NiFi \
+ org.apache.nifi.version=$NIFI_VERSION \
+ org.apache.nifi.release-date="2016-11-29"
+
+# Download, validate, and expand Apache NiFi binary.
+RUN curl -fSL "https://archive.apache.org/dist/nifi/$NIFI_VERSION/nifi-$NIFI_VERSION-bin.tar.gz" -o /nifi-$NIFI_VERSION-bin.tar.gz \
+ && echo "$NIFI_DOWNLOAD_SHA256 */nifi-$NIFI_VERSION-bin.tar.gz" | sha256sum -c - \
+ && tar -xvzf /nifi-$NIFI_VERSION-bin.tar.gz \
+ && rm /nifi-$NIFI_VERSION-bin.tar.gz
+
+# Setup NiFi.properties
+ADD nifi.properties $NIFI_HOME/conf/
+
--- End diff --
Might be good to switch to a non-root user
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by cricket007 <gi...@git.apache.org>.
Github user cricket007 commented on the issue:
https://github.com/apache/nifi/pull/1372
That also seems like a good idea. mkdir on NIFI_HOME, then chown, curl, and extract.
I hit the download limit on Apache site last night (5 GB) testing some options, so I'll continue once that's resolved
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the issue:
https://github.com/apache/nifi/pull/1372
Hey @cricket007!
Good catch and was likely just a something that got overlooked. Would you like to submit a PR to correct this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the issue:
https://github.com/apache/nifi/pull/1372
Just some initial thoughts on the convo thus far:
From a release perspective, I don't think we should be maintaining versioned directories. Along the lines of what @JPercivall was mentioning, the code in here should coincide with the current release we have. When things are voted upon and released we can have a corresponding image. This would make things a bit tricky in the interim given that it's looking for a specific location. Not sure how those logistics would work out exactly, but short of having a separate repo and vote process, may be something that we need to work against.
There is nothing that precludes volumes being specified at container instantiation. It would be nice to have them as defaults, but is not explicitly required.
Custom processors would make sense to have an additional volume and update the config to provide one or more additional lib directories for custom NARs. These do not necessarily need to be in the lib directory, and from a maintainability standpoint, would be nice to have a dedicated location for these custom NARs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by jdye64 <gi...@git.apache.org>.
Github user jdye64 commented on the issue:
https://github.com/apache/nifi/pull/1372
@achristianson is the docker-maven-plugin in a public maven repository somewhere that you know of? I don't see it in maven central. I built locally and have it working but we would need it somewhere public.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by jdye64 <gi...@git.apache.org>.
Github user jdye64 commented on the issue:
https://github.com/apache/nifi/pull/1372
Refactored to a single Dockerfile and also incorporated the docker-maven-plugin for building the docker image. You can build the image now by invoking the profile "docker"
Ex: mvn clean install package -Pdocker from the $NIFI_HOME directory.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by taftster <gi...@git.apache.org>.
Github user taftster commented on the issue:
https://github.com/apache/nifi/pull/1372
One solution is just to move the chown and USER components _above_ the curl statement. Basically you want curl to be executed as the NIFI user, not as root.
Just a quick test, rearranging the statements:
Before: 2.58 GB
After: 1.59 GB
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r95191672
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+FROM java:8u91-jdk
+MAINTAINER Apache NiFi <de...@nifi.apache.org>
+
+ARG UID
+ARG GID
+ARG NIFI_VERSION
+
+ENV NIFI_BASE_DIR /opt/nifi
+ENV NIFI_HOME $NIFI_BASE_DIR/nifi-$NIFI_VERSION
+ENV NIFI_DOWNLOAD_SHA256 afb58050b0904dab329b19b14250b57d3978743a1108323bdc52aba0c2a75853
--- End diff --
This should likely come from the published file as part of the release (nifi-$NIFI_VERSION-bin.tar.gz.sha256). We would have no way to actually know the hash for a given release a priori as this would be part of the source that would be tagged to generate a given artifact.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1372: NIFI-3260 Official Docker Image
Posted by gauravgoyal86 <gi...@git.apache.org>.
Github user gauravgoyal86 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1372#discussion_r96966950
--- Diff: nifi-docker/1.1.1/DockerRun.sh ---
@@ -0,0 +1,3 @@
+#!/bin/bash
+DOCKER_IMAGE="$(cat DockerImage.txt)"
+docker run -it -d -p 8080:8080 -p 8181:8181 $DOCKER_IMAGE
--- End diff --
Do we have to execute following command every time we want to start nifi docker instance?
docker run -it -d -p 8080:8080 -p 8181:8181 $DOCKER_IMAGE
or once docker container is created, we can do docker stop and docker start.
i am trying that with my docker container. It works well with docker run. but it does work well with docker stop and docker start.
Can you try that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by ddewaele <gi...@git.apache.org>.
Github user ddewaele commented on the issue:
https://github.com/apache/nifi/pull/1372
Hi @jdye64 ,
We also started using Nifi in a docker container using one of the unofficial images (https://github.com/mkobit/docker-nifi).
Being able to use an official and supported Nifi Docker image would be great.
Are you planning on adding data volumes for the repositories and the config ? (to avoid losing state / config).
We typically map
- database_repository
- flowfile_repository
- content_repository
- provenance_repository
- conf
- external config (variable registry)
How are you planning to handle custom processors in docker images ? We're currently using a [docker maven plugin](https://github.com/spotify/docker-maven-plugin) in Jenkins to build our custom nifi image (extending the mkobit one), and we add our custom processors in the docker image during build time (as everything needs to be in ${NIFI_HOME}/lib).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1372: NIFI-3260 Official Docker Image
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the issue:
https://github.com/apache/nifi/pull/1372
I think the tags should be fine and it is okay to continuously version the one, true Dockerfile with each release. From a release perspective, it would be on the RM to kick off the various processes that would generate a build with the new tag, but our core, source release is the Dockerfile and whatever periphery we may choose to include (in future releases, this could include things like compose files). We could likely additionally make use of maven filtering to have the needed versions get updated.
Insofar as tagging/versions, I have taken a similar approach with the image I have been maintaining: https://hub.docker.com/r/aldrin/apache-nifi/tags/. Upon each update, I tag that source and push both that commit and tag to the repository. This manifests itself as the latest version being tagged latest and its respective version. When a new version comes out, latest is advanced to correspond to the same image generated from the new tag.
For the official repo as provided, we could just add the new version to that list and keep it there as long as supported. Given our release structure, this seems to coincide pretty well with the guidance provided at https://github.com/docker-library/official-images#library-definition-files. As certain versions become obsolete/unsupported, those associated entries would be purged. I think this also allows us to get away from the directory structure as every release we perform will have a new commit which we can reference whilst keeping a consistent path to the Dockerfile.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---