You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Kevin Doran <kd...@gmail.com> on 2022/01/13 16:38:13 UTC
Re: apache/nifi convenience Docker Image - publish JDK 11 version?
Thanks, Chris! I’ve incorporated these ideas into the Confluence page. I’ll look into getting you edit permissions - what is your username there?
> On Dec 18, 2021, at 13:35, Chris Sampson <ch...@naimuri.com.INVALID> wrote:
>
> Kevin,
>
> Good idea to write up a list of ideas - unfortunately I don't appear to
> have permission to edit the page, but a few other ideas are:
>
> - Provide an "alpine" based image (could be the
> "openjdk:<version>-alpine" image used as a base) to reduce image size and
> O/S footprint of the container
> - Change the Docker start scripts to allow for all nifi.properties (and
> other config file properties) to be set by Environment Variables - this
> could be a case of using a common Env Var prefix to and iterating through
> them within the script, e.g. anything that is NIFI_* is assumed to be a
> nifi.properties entry, etc. - this would make the convenience image all the
> more convenient for people and follow suit with some other applications
> using Docker wrappers
> - Docker-focussed default config, for example logback.xml that directs
> output to STDOUT/STDERR by default, possibly in JSON format, allowing
> easier out-of-the-box use of the logs in Docker-based environments that
> already process Docker log output
> - Allow for the nifi.properties/config file changes during start.sh to
> be skipped (some people inject the config files into their containers but
> want them to be read-only, which then means the start.sh script fails
> because the files can't be edited)
> - Change the default config file locations within the Docker images,
> e.g. the flow.xml.gz and other system generated config that should be
> persisted through container restart would be better in a different folder
> to things like nifi.properties, then the directory can be externalised as a
> Volume (without running into read-only/mutability issues like the start.sh
> point above)
> - Make it easier for Docker-based instances to join an existing cluster
> - this might be a case of relocating some of the files to make better use
> of ephemeral vs. persistent directories/Volumes and/or having a
> configurable mechanism that allows for the cluster config files to be
> deleted during startup if the instance expects to join a cluster but cannot
> because it is unable to overwrite existing files with those being offered
> by the cluster (not sure how much of an issue this still is after some
> recent changes)
>
> Please feel free to add any/all of the above to the Confluence page that
> you think would be worthwhile to add to the pile for consideration.
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.sampson@naimuri.com
>
>
> On Fri, 17 Dec 2021 at 18:01, Kevin Doran <kd...@apache.org> wrote:
>
>> Hi Chris,
>>
>> I like this idea and have been thinking of a number of other improvements
>> we could make for container images of Apache NiFi, including (I think
>> discussed on another thread) consolidating Dockerfiles for dockerhub /
>> dockermaven, as the only significant difference is where the nifi assembly
>> archive comes from, arm64 images that run native on ARM platforms,
>> including newer M1 Macs (requires some minor changes to NiFi source itself
>> to compile).
>>
>> I’ve started a Wiki page [1] to collect ideas, and once we reach consensus
>> on the improvements we’d like to make and possible approaches, we can turn
>> these into Jiras.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements
>>
>> Thanks!
>> Kevin
>>
>>> On Nov 26, 2021, at 07:46, Chris Sampson <ch...@naimuri.com.INVALID>
>> wrote:
>>>
>>> I'm not sure whether I've seen this discussed previously, but I thought
>> it
>>> worth asking whether we could/should publish a JDK 11 based version of
>> the
>>> apache/nifi Docker Image to Docker Hub as part of the release process in
>>> future?
>>>
>>> I realise NiFi 1.x is/will remain based on JDK 8 but with all the work
>> gone
>>> into enabling JDK 11 compilation and deployment, along with the planned
>>> move to JDK 11 in the future for NiFi 2.x, it would seem sensible to
>> start
>>> making it easier for people to move towards using JDK 11 now that it's
>>> available.
>>>
>>> The Dockerfiles used to build NiFi (I've not looked at Toolkit, MiNiFi or
>>> Registry) can be given a build ARG to use different base images/versions,
>>> so it should be relatively straightforward to do this and maybe publish
>> the
>>> Image to Docker Hub as *apache/nifi:<version>-jdk11*?
>>>
>>> Users can, of course, download the NiFi source to build this image
>>> themselves, but to make it easier for the community it would seem
>> sensible
>>> to consider the additional release artifact. This avoids lots of people
>>> having to pull all of the code (it's not a small repo) or build the
>> image,
>>> which can take a long time, etc.
>>>
>>>
>>> ---
>>> *Chris Sampson*
>>> IT Consultant
>>> chris.sampson@naimuri.com
>>
>>
Re: apache/nifi convenience Docker Image - publish JDK 11 version?
Posted by Kevin Doran <kd...@apache.org>.
Hi Chris,
I forgot to mention - Bryan Bende was able to help me with this, so you should have edit permissions in Confluence now.
Thanks,
Kevin
> On Jan 13, 2022, at 12:09, Chris Sampson <ch...@naimuri.com.INVALID> wrote:
>
> Thanks Kevin, that's great.
>
> My username is "chriss".
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.sampson@naimuri.com
>
>
> On Thu, 13 Jan 2022 at 16:38, Kevin Doran <kd...@gmail.com> wrote:
>
>> Thanks, Chris! I’ve incorporated these ideas into the Confluence page.
>> I’ll look into getting you edit permissions - what is your username there?
>>
>>> On Dec 18, 2021, at 13:35, Chris Sampson <ch...@naimuri.com.INVALID>
>> wrote:
>>>
>>> Kevin,
>>>
>>> Good idea to write up a list of ideas - unfortunately I don't appear to
>>> have permission to edit the page, but a few other ideas are:
>>>
>>> - Provide an "alpine" based image (could be the
>>> "openjdk:<version>-alpine" image used as a base) to reduce image size
>> and
>>> O/S footprint of the container
>>> - Change the Docker start scripts to allow for all nifi.properties (and
>>> other config file properties) to be set by Environment Variables - this
>>> could be a case of using a common Env Var prefix to and iterating
>> through
>>> them within the script, e.g. anything that is NIFI_* is assumed to be a
>>> nifi.properties entry, etc. - this would make the convenience image
>> all the
>>> more convenient for people and follow suit with some other applications
>>> using Docker wrappers
>>> - Docker-focussed default config, for example logback.xml that directs
>>> output to STDOUT/STDERR by default, possibly in JSON format, allowing
>>> easier out-of-the-box use of the logs in Docker-based environments that
>>> already process Docker log output
>>> - Allow for the nifi.properties/config file changes during start.sh to
>>> be skipped (some people inject the config files into their containers
>> but
>>> want them to be read-only, which then means the start.sh script fails
>>> because the files can't be edited)
>>> - Change the default config file locations within the Docker images,
>>> e.g. the flow.xml.gz and other system generated config that should be
>>> persisted through container restart would be better in a different
>> folder
>>> to things like nifi.properties, then the directory can be externalised
>> as a
>>> Volume (without running into read-only/mutability issues like the
>> start.sh
>>> point above)
>>> - Make it easier for Docker-based instances to join an existing cluster
>>> - this might be a case of relocating some of the files to make better
>> use
>>> of ephemeral vs. persistent directories/Volumes and/or having a
>>> configurable mechanism that allows for the cluster config files to be
>>> deleted during startup if the instance expects to join a cluster but
>> cannot
>>> because it is unable to overwrite existing files with those being
>> offered
>>> by the cluster (not sure how much of an issue this still is after some
>>> recent changes)
>>>
>>> Please feel free to add any/all of the above to the Confluence page that
>>> you think would be worthwhile to add to the pile for consideration.
>>>
>>>
>>> ---
>>> *Chris Sampson*
>>> IT Consultant
>>> chris.sampson@naimuri.com
>>>
>>>
>>> On Fri, 17 Dec 2021 at 18:01, Kevin Doran <kd...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> I like this idea and have been thinking of a number of other
>> improvements
>>>> we could make for container images of Apache NiFi, including (I think
>>>> discussed on another thread) consolidating Dockerfiles for dockerhub /
>>>> dockermaven, as the only significant difference is where the nifi
>> assembly
>>>> archive comes from, arm64 images that run native on ARM platforms,
>>>> including newer M1 Macs (requires some minor changes to NiFi source
>> itself
>>>> to compile).
>>>>
>>>> I’ve started a Wiki page [1] to collect ideas, and once we reach
>> consensus
>>>> on the improvements we’d like to make and possible approaches, we can
>> turn
>>>> these into Jiras.
>>>>
>>>> [1]
>>>>
>> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements
>>>>
>>>> Thanks!
>>>> Kevin
>>>>
>>>>> On Nov 26, 2021, at 07:46, Chris Sampson <chris.sampson@naimuri.com
>> .INVALID>
>>>> wrote:
>>>>>
>>>>> I'm not sure whether I've seen this discussed previously, but I thought
>>>> it
>>>>> worth asking whether we could/should publish a JDK 11 based version of
>>>> the
>>>>> apache/nifi Docker Image to Docker Hub as part of the release process
>> in
>>>>> future?
>>>>>
>>>>> I realise NiFi 1.x is/will remain based on JDK 8 but with all the work
>>>> gone
>>>>> into enabling JDK 11 compilation and deployment, along with the planned
>>>>> move to JDK 11 in the future for NiFi 2.x, it would seem sensible to
>>>> start
>>>>> making it easier for people to move towards using JDK 11 now that it's
>>>>> available.
>>>>>
>>>>> The Dockerfiles used to build NiFi (I've not looked at Toolkit, MiNiFi
>> or
>>>>> Registry) can be given a build ARG to use different base
>> images/versions,
>>>>> so it should be relatively straightforward to do this and maybe publish
>>>> the
>>>>> Image to Docker Hub as *apache/nifi:<version>-jdk11*?
>>>>>
>>>>> Users can, of course, download the NiFi source to build this image
>>>>> themselves, but to make it easier for the community it would seem
>>>> sensible
>>>>> to consider the additional release artifact. This avoids lots of people
>>>>> having to pull all of the code (it's not a small repo) or build the
>>>> image,
>>>>> which can take a long time, etc.
>>>>>
>>>>>
>>>>> ---
>>>>> *Chris Sampson*
>>>>> IT Consultant
>>>>> chris.sampson@naimuri.com
>>>>
>>>>
>>
>>
Re: apache/nifi convenience Docker Image - publish JDK 11 version?
Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
Thanks Kevin, that's great.
My username is "chriss".
---
*Chris Sampson*
IT Consultant
chris.sampson@naimuri.com
On Thu, 13 Jan 2022 at 16:38, Kevin Doran <kd...@gmail.com> wrote:
> Thanks, Chris! I’ve incorporated these ideas into the Confluence page.
> I’ll look into getting you edit permissions - what is your username there?
>
> > On Dec 18, 2021, at 13:35, Chris Sampson <ch...@naimuri.com.INVALID>
> wrote:
> >
> > Kevin,
> >
> > Good idea to write up a list of ideas - unfortunately I don't appear to
> > have permission to edit the page, but a few other ideas are:
> >
> > - Provide an "alpine" based image (could be the
> > "openjdk:<version>-alpine" image used as a base) to reduce image size
> and
> > O/S footprint of the container
> > - Change the Docker start scripts to allow for all nifi.properties (and
> > other config file properties) to be set by Environment Variables - this
> > could be a case of using a common Env Var prefix to and iterating
> through
> > them within the script, e.g. anything that is NIFI_* is assumed to be a
> > nifi.properties entry, etc. - this would make the convenience image
> all the
> > more convenient for people and follow suit with some other applications
> > using Docker wrappers
> > - Docker-focussed default config, for example logback.xml that directs
> > output to STDOUT/STDERR by default, possibly in JSON format, allowing
> > easier out-of-the-box use of the logs in Docker-based environments that
> > already process Docker log output
> > - Allow for the nifi.properties/config file changes during start.sh to
> > be skipped (some people inject the config files into their containers
> but
> > want them to be read-only, which then means the start.sh script fails
> > because the files can't be edited)
> > - Change the default config file locations within the Docker images,
> > e.g. the flow.xml.gz and other system generated config that should be
> > persisted through container restart would be better in a different
> folder
> > to things like nifi.properties, then the directory can be externalised
> as a
> > Volume (without running into read-only/mutability issues like the
> start.sh
> > point above)
> > - Make it easier for Docker-based instances to join an existing cluster
> > - this might be a case of relocating some of the files to make better
> use
> > of ephemeral vs. persistent directories/Volumes and/or having a
> > configurable mechanism that allows for the cluster config files to be
> > deleted during startup if the instance expects to join a cluster but
> cannot
> > because it is unable to overwrite existing files with those being
> offered
> > by the cluster (not sure how much of an issue this still is after some
> > recent changes)
> >
> > Please feel free to add any/all of the above to the Confluence page that
> > you think would be worthwhile to add to the pile for consideration.
> >
> >
> > ---
> > *Chris Sampson*
> > IT Consultant
> > chris.sampson@naimuri.com
> >
> >
> > On Fri, 17 Dec 2021 at 18:01, Kevin Doran <kd...@apache.org> wrote:
> >
> >> Hi Chris,
> >>
> >> I like this idea and have been thinking of a number of other
> improvements
> >> we could make for container images of Apache NiFi, including (I think
> >> discussed on another thread) consolidating Dockerfiles for dockerhub /
> >> dockermaven, as the only significant difference is where the nifi
> assembly
> >> archive comes from, arm64 images that run native on ARM platforms,
> >> including newer M1 Macs (requires some minor changes to NiFi source
> itself
> >> to compile).
> >>
> >> I’ve started a Wiki page [1] to collect ideas, and once we reach
> consensus
> >> on the improvements we’d like to make and possible approaches, we can
> turn
> >> these into Jiras.
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements
> >>
> >> Thanks!
> >> Kevin
> >>
> >>> On Nov 26, 2021, at 07:46, Chris Sampson <chris.sampson@naimuri.com
> .INVALID>
> >> wrote:
> >>>
> >>> I'm not sure whether I've seen this discussed previously, but I thought
> >> it
> >>> worth asking whether we could/should publish a JDK 11 based version of
> >> the
> >>> apache/nifi Docker Image to Docker Hub as part of the release process
> in
> >>> future?
> >>>
> >>> I realise NiFi 1.x is/will remain based on JDK 8 but with all the work
> >> gone
> >>> into enabling JDK 11 compilation and deployment, along with the planned
> >>> move to JDK 11 in the future for NiFi 2.x, it would seem sensible to
> >> start
> >>> making it easier for people to move towards using JDK 11 now that it's
> >>> available.
> >>>
> >>> The Dockerfiles used to build NiFi (I've not looked at Toolkit, MiNiFi
> or
> >>> Registry) can be given a build ARG to use different base
> images/versions,
> >>> so it should be relatively straightforward to do this and maybe publish
> >> the
> >>> Image to Docker Hub as *apache/nifi:<version>-jdk11*?
> >>>
> >>> Users can, of course, download the NiFi source to build this image
> >>> themselves, but to make it easier for the community it would seem
> >> sensible
> >>> to consider the additional release artifact. This avoids lots of people
> >>> having to pull all of the code (it's not a small repo) or build the
> >> image,
> >>> which can take a long time, etc.
> >>>
> >>>
> >>> ---
> >>> *Chris Sampson*
> >>> IT Consultant
> >>> chris.sampson@naimuri.com
> >>
> >>
>
>