You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Chris Sampson <ch...@naimuri.com.INVALID> on 2021/11/26 12:46:13 UTC
apache/nifi convenience Docker Image - publish JDK 11 version?
I'm not sure whether I've seen this discussed previously, but I thought it
worth asking whether we could/should publish a JDK 11 based version of the
apache/nifi Docker Image to Docker Hub as part of the release process in
future?
I realise NiFi 1.x is/will remain based on JDK 8 but with all the work gone
into enabling JDK 11 compilation and deployment, along with the planned
move to JDK 11 in the future for NiFi 2.x, it would seem sensible to start
making it easier for people to move towards using JDK 11 now that it's
available.
The Dockerfiles used to build NiFi (I've not looked at Toolkit, MiNiFi or
Registry) can be given a build ARG to use different base images/versions,
so it should be relatively straightforward to do this and maybe publish the
Image to Docker Hub as *apache/nifi:<version>-jdk11*?
Users can, of course, download the NiFi source to build this image
themselves, but to make it easier for the community it would seem sensible
to consider the additional release artifact. This avoids lots of people
having to pull all of the code (it's not a small repo) or build the image,
which can take a long time, etc.
---
*Chris Sampson*
IT Consultant
chris.sampson@naimuri.com
Re: apache/nifi convenience Docker Image - publish JDK 11 version?
Posted by Kevin Doran <kd...@apache.org>.
Hi Chris,
I forgot to mention - Bryan Bende was able to help me with this, so you should have edit permissions in Confluence now.
Thanks,
Kevin
> On Jan 13, 2022, at 12:09, Chris Sampson <ch...@naimuri.com.INVALID> wrote:
>
> Thanks Kevin, that's great.
>
> My username is "chriss".
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.sampson@naimuri.com
>
>
> On Thu, 13 Jan 2022 at 16:38, Kevin Doran <kd...@gmail.com> wrote:
>
>> Thanks, Chris! I’ve incorporated these ideas into the Confluence page.
>> I’ll look into getting you edit permissions - what is your username there?
>>
>>> On Dec 18, 2021, at 13:35, Chris Sampson <ch...@naimuri.com.INVALID>
>> wrote:
>>>
>>> Kevin,
>>>
>>> Good idea to write up a list of ideas - unfortunately I don't appear to
>>> have permission to edit the page, but a few other ideas are:
>>>
>>> - Provide an "alpine" based image (could be the
>>> "openjdk:<version>-alpine" image used as a base) to reduce image size
>> and
>>> O/S footprint of the container
>>> - Change the Docker start scripts to allow for all nifi.properties (and
>>> other config file properties) to be set by Environment Variables - this
>>> could be a case of using a common Env Var prefix to and iterating
>> through
>>> them within the script, e.g. anything that is NIFI_* is assumed to be a
>>> nifi.properties entry, etc. - this would make the convenience image
>> all the
>>> more convenient for people and follow suit with some other applications
>>> using Docker wrappers
>>> - Docker-focussed default config, for example logback.xml that directs
>>> output to STDOUT/STDERR by default, possibly in JSON format, allowing
>>> easier out-of-the-box use of the logs in Docker-based environments that
>>> already process Docker log output
>>> - Allow for the nifi.properties/config file changes during start.sh to
>>> be skipped (some people inject the config files into their containers
>> but
>>> want them to be read-only, which then means the start.sh script fails
>>> because the files can't be edited)
>>> - Change the default config file locations within the Docker images,
>>> e.g. the flow.xml.gz and other system generated config that should be
>>> persisted through container restart would be better in a different
>> folder
>>> to things like nifi.properties, then the directory can be externalised
>> as a
>>> Volume (without running into read-only/mutability issues like the
>> start.sh
>>> point above)
>>> - Make it easier for Docker-based instances to join an existing cluster
>>> - this might be a case of relocating some of the files to make better
>> use
>>> of ephemeral vs. persistent directories/Volumes and/or having a
>>> configurable mechanism that allows for the cluster config files to be
>>> deleted during startup if the instance expects to join a cluster but
>> cannot
>>> because it is unable to overwrite existing files with those being
>> offered
>>> by the cluster (not sure how much of an issue this still is after some
>>> recent changes)
>>>
>>> Please feel free to add any/all of the above to the Confluence page that
>>> you think would be worthwhile to add to the pile for consideration.
>>>
>>>
>>> ---
>>> *Chris Sampson*
>>> IT Consultant
>>> chris.sampson@naimuri.com
>>>
>>>
>>> On Fri, 17 Dec 2021 at 18:01, Kevin Doran <kd...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> I like this idea and have been thinking of a number of other
>> improvements
>>>> we could make for container images of Apache NiFi, including (I think
>>>> discussed on another thread) consolidating Dockerfiles for dockerhub /
>>>> dockermaven, as the only significant difference is where the nifi
>> assembly
>>>> archive comes from, arm64 images that run native on ARM platforms,
>>>> including newer M1 Macs (requires some minor changes to NiFi source
>> itself
>>>> to compile).
>>>>
>>>> I’ve started a Wiki page [1] to collect ideas, and once we reach
>> consensus
>>>> on the improvements we’d like to make and possible approaches, we can
>> turn
>>>> these into Jiras.
>>>>
>>>> [1]
>>>>
>> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements
>>>>
>>>> Thanks!
>>>> Kevin
>>>>
>>>>> On Nov 26, 2021, at 07:46, Chris Sampson <chris.sampson@naimuri.com
>> .INVALID>
>>>> wrote:
>>>>>
>>>>> I'm not sure whether I've seen this discussed previously, but I thought
>>>> it
>>>>> worth asking whether we could/should publish a JDK 11 based version of
>>>> the
>>>>> apache/nifi Docker Image to Docker Hub as part of the release process
>> in
>>>>> future?
>>>>>
>>>>> I realise NiFi 1.x is/will remain based on JDK 8 but with all the work
>>>> gone
>>>>> into enabling JDK 11 compilation and deployment, along with the planned
>>>>> move to JDK 11 in the future for NiFi 2.x, it would seem sensible to
>>>> start
>>>>> making it easier for people to move towards using JDK 11 now that it's
>>>>> available.
>>>>>
>>>>> The Dockerfiles used to build NiFi (I've not looked at Toolkit, MiNiFi
>> or
>>>>> Registry) can be given a build ARG to use different base
>> images/versions,
>>>>> so it should be relatively straightforward to do this and maybe publish
>>>> the
>>>>> Image to Docker Hub as *apache/nifi:<version>-jdk11*?
>>>>>
>>>>> Users can, of course, download the NiFi source to build this image
>>>>> themselves, but to make it easier for the community it would seem
>>>> sensible
>>>>> to consider the additional release artifact. This avoids lots of people
>>>>> having to pull all of the code (it's not a small repo) or build the
>>>> image,
>>>>> which can take a long time, etc.
>>>>>
>>>>>
>>>>> ---
>>>>> *Chris Sampson*
>>>>> IT Consultant
>>>>> chris.sampson@naimuri.com
>>>>
>>>>
>>
>>
Re: apache/nifi convenience Docker Image - publish JDK 11 version?
Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
Thanks Kevin, that's great.
My username is "chriss".
---
*Chris Sampson*
IT Consultant
chris.sampson@naimuri.com
On Thu, 13 Jan 2022 at 16:38, Kevin Doran <kd...@gmail.com> wrote:
> Thanks, Chris! I’ve incorporated these ideas into the Confluence page.
> I’ll look into getting you edit permissions - what is your username there?
>
> > On Dec 18, 2021, at 13:35, Chris Sampson <ch...@naimuri.com.INVALID>
> wrote:
> >
> > Kevin,
> >
> > Good idea to write up a list of ideas - unfortunately I don't appear to
> > have permission to edit the page, but a few other ideas are:
> >
> > - Provide an "alpine" based image (could be the
> > "openjdk:<version>-alpine" image used as a base) to reduce image size
> and
> > O/S footprint of the container
> > - Change the Docker start scripts to allow for all nifi.properties (and
> > other config file properties) to be set by Environment Variables - this
> > could be a case of using a common Env Var prefix to and iterating
> through
> > them within the script, e.g. anything that is NIFI_* is assumed to be a
> > nifi.properties entry, etc. - this would make the convenience image
> all the
> > more convenient for people and follow suit with some other applications
> > using Docker wrappers
> > - Docker-focussed default config, for example logback.xml that directs
> > output to STDOUT/STDERR by default, possibly in JSON format, allowing
> > easier out-of-the-box use of the logs in Docker-based environments that
> > already process Docker log output
> > - Allow for the nifi.properties/config file changes during start.sh to
> > be skipped (some people inject the config files into their containers
> but
> > want them to be read-only, which then means the start.sh script fails
> > because the files can't be edited)
> > - Change the default config file locations within the Docker images,
> > e.g. the flow.xml.gz and other system generated config that should be
> > persisted through container restart would be better in a different
> folder
> > to things like nifi.properties, then the directory can be externalised
> as a
> > Volume (without running into read-only/mutability issues like the
> start.sh
> > point above)
> > - Make it easier for Docker-based instances to join an existing cluster
> > - this might be a case of relocating some of the files to make better
> use
> > of ephemeral vs. persistent directories/Volumes and/or having a
> > configurable mechanism that allows for the cluster config files to be
> > deleted during startup if the instance expects to join a cluster but
> cannot
> > because it is unable to overwrite existing files with those being
> offered
> > by the cluster (not sure how much of an issue this still is after some
> > recent changes)
> >
> > Please feel free to add any/all of the above to the Confluence page that
> > you think would be worthwhile to add to the pile for consideration.
> >
> >
> > ---
> > *Chris Sampson*
> > IT Consultant
> > chris.sampson@naimuri.com
> >
> >
> > On Fri, 17 Dec 2021 at 18:01, Kevin Doran <kd...@apache.org> wrote:
> >
> >> Hi Chris,
> >>
> >> I like this idea and have been thinking of a number of other
> improvements
> >> we could make for container images of Apache NiFi, including (I think
> >> discussed on another thread) consolidating Dockerfiles for dockerhub /
> >> dockermaven, as the only significant difference is where the nifi
> assembly
> >> archive comes from, arm64 images that run native on ARM platforms,
> >> including newer M1 Macs (requires some minor changes to NiFi source
> itself
> >> to compile).
> >>
> >> I’ve started a Wiki page [1] to collect ideas, and once we reach
> consensus
> >> on the improvements we’d like to make and possible approaches, we can
> turn
> >> these into Jiras.
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements
> >>
> >> Thanks!
> >> Kevin
> >>
> >>> On Nov 26, 2021, at 07:46, Chris Sampson <chris.sampson@naimuri.com
> .INVALID>
> >> wrote:
> >>>
> >>> I'm not sure whether I've seen this discussed previously, but I thought
> >> it
> >>> worth asking whether we could/should publish a JDK 11 based version of
> >> the
> >>> apache/nifi Docker Image to Docker Hub as part of the release process
> in
> >>> future?
> >>>
> >>> I realise NiFi 1.x is/will remain based on JDK 8 but with all the work
> >> gone
> >>> into enabling JDK 11 compilation and deployment, along with the planned
> >>> move to JDK 11 in the future for NiFi 2.x, it would seem sensible to
> >> start
> >>> making it easier for people to move towards using JDK 11 now that it's
> >>> available.
> >>>
> >>> The Dockerfiles used to build NiFi (I've not looked at Toolkit, MiNiFi
> or
> >>> Registry) can be given a build ARG to use different base
> images/versions,
> >>> so it should be relatively straightforward to do this and maybe publish
> >> the
> >>> Image to Docker Hub as *apache/nifi:<version>-jdk11*?
> >>>
> >>> Users can, of course, download the NiFi source to build this image
> >>> themselves, but to make it easier for the community it would seem
> >> sensible
> >>> to consider the additional release artifact. This avoids lots of people
> >>> having to pull all of the code (it's not a small repo) or build the
> >> image,
> >>> which can take a long time, etc.
> >>>
> >>>
> >>> ---
> >>> *Chris Sampson*
> >>> IT Consultant
> >>> chris.sampson@naimuri.com
> >>
> >>
>
>
Re: apache/nifi convenience Docker Image - publish JDK 11 version?
Posted by Kevin Doran <kd...@gmail.com>.
Thanks, Chris! I’ve incorporated these ideas into the Confluence page. I’ll look into getting you edit permissions - what is your username there?
> On Dec 18, 2021, at 13:35, Chris Sampson <ch...@naimuri.com.INVALID> wrote:
>
> Kevin,
>
> Good idea to write up a list of ideas - unfortunately I don't appear to
> have permission to edit the page, but a few other ideas are:
>
> - Provide an "alpine" based image (could be the
> "openjdk:<version>-alpine" image used as a base) to reduce image size and
> O/S footprint of the container
> - Change the Docker start scripts to allow for all nifi.properties (and
> other config file properties) to be set by Environment Variables - this
> could be a case of using a common Env Var prefix to and iterating through
> them within the script, e.g. anything that is NIFI_* is assumed to be a
> nifi.properties entry, etc. - this would make the convenience image all the
> more convenient for people and follow suit with some other applications
> using Docker wrappers
> - Docker-focussed default config, for example logback.xml that directs
> output to STDOUT/STDERR by default, possibly in JSON format, allowing
> easier out-of-the-box use of the logs in Docker-based environments that
> already process Docker log output
> - Allow for the nifi.properties/config file changes during start.sh to
> be skipped (some people inject the config files into their containers but
> want them to be read-only, which then means the start.sh script fails
> because the files can't be edited)
> - Change the default config file locations within the Docker images,
> e.g. the flow.xml.gz and other system generated config that should be
> persisted through container restart would be better in a different folder
> to things like nifi.properties, then the directory can be externalised as a
> Volume (without running into read-only/mutability issues like the start.sh
> point above)
> - Make it easier for Docker-based instances to join an existing cluster
> - this might be a case of relocating some of the files to make better use
> of ephemeral vs. persistent directories/Volumes and/or having a
> configurable mechanism that allows for the cluster config files to be
> deleted during startup if the instance expects to join a cluster but cannot
> because it is unable to overwrite existing files with those being offered
> by the cluster (not sure how much of an issue this still is after some
> recent changes)
>
> Please feel free to add any/all of the above to the Confluence page that
> you think would be worthwhile to add to the pile for consideration.
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.sampson@naimuri.com
>
>
> On Fri, 17 Dec 2021 at 18:01, Kevin Doran <kd...@apache.org> wrote:
>
>> Hi Chris,
>>
>> I like this idea and have been thinking of a number of other improvements
>> we could make for container images of Apache NiFi, including (I think
>> discussed on another thread) consolidating Dockerfiles for dockerhub /
>> dockermaven, as the only significant difference is where the nifi assembly
>> archive comes from, arm64 images that run native on ARM platforms,
>> including newer M1 Macs (requires some minor changes to NiFi source itself
>> to compile).
>>
>> I’ve started a Wiki page [1] to collect ideas, and once we reach consensus
>> on the improvements we’d like to make and possible approaches, we can turn
>> these into Jiras.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements
>>
>> Thanks!
>> Kevin
>>
>>> On Nov 26, 2021, at 07:46, Chris Sampson <ch...@naimuri.com.INVALID>
>> wrote:
>>>
>>> I'm not sure whether I've seen this discussed previously, but I thought
>> it
>>> worth asking whether we could/should publish a JDK 11 based version of
>> the
>>> apache/nifi Docker Image to Docker Hub as part of the release process in
>>> future?
>>>
>>> I realise NiFi 1.x is/will remain based on JDK 8 but with all the work
>> gone
>>> into enabling JDK 11 compilation and deployment, along with the planned
>>> move to JDK 11 in the future for NiFi 2.x, it would seem sensible to
>> start
>>> making it easier for people to move towards using JDK 11 now that it's
>>> available.
>>>
>>> The Dockerfiles used to build NiFi (I've not looked at Toolkit, MiNiFi or
>>> Registry) can be given a build ARG to use different base images/versions,
>>> so it should be relatively straightforward to do this and maybe publish
>> the
>>> Image to Docker Hub as *apache/nifi:<version>-jdk11*?
>>>
>>> Users can, of course, download the NiFi source to build this image
>>> themselves, but to make it easier for the community it would seem
>> sensible
>>> to consider the additional release artifact. This avoids lots of people
>>> having to pull all of the code (it's not a small repo) or build the
>> image,
>>> which can take a long time, etc.
>>>
>>>
>>> ---
>>> *Chris Sampson*
>>> IT Consultant
>>> chris.sampson@naimuri.com
>>
>>
Re: apache/nifi convenience Docker Image - publish JDK 11 version?
Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
Kevin,
Good idea to write up a list of ideas - unfortunately I don't appear to
have permission to edit the page, but a few other ideas are:
- Provide an "alpine" based image (could be the
"openjdk:<version>-alpine" image used as a base) to reduce image size and
O/S footprint of the container
- Change the Docker start scripts to allow for all nifi.properties (and
other config file properties) to be set by Environment Variables - this
could be a case of using a common Env Var prefix to and iterating through
them within the script, e.g. anything that is NIFI_* is assumed to be a
nifi.properties entry, etc. - this would make the convenience image all the
more convenient for people and follow suit with some other applications
using Docker wrappers
- Docker-focussed default config, for example logback.xml that directs
output to STDOUT/STDERR by default, possibly in JSON format, allowing
easier out-of-the-box use of the logs in Docker-based environments that
already process Docker log output
- Allow for the nifi.properties/config file changes during start.sh to
be skipped (some people inject the config files into their containers but
want them to be read-only, which then means the start.sh script fails
because the files can't be edited)
- Change the default config file locations within the Docker images,
e.g. the flow.xml.gz and other system generated config that should be
persisted through container restart would be better in a different folder
to things like nifi.properties, then the directory can be externalised as a
Volume (without running into read-only/mutability issues like the start.sh
point above)
- Make it easier for Docker-based instances to join an existing cluster
- this might be a case of relocating some of the files to make better use
of ephemeral vs. persistent directories/Volumes and/or having a
configurable mechanism that allows for the cluster config files to be
deleted during startup if the instance expects to join a cluster but cannot
because it is unable to overwrite existing files with those being offered
by the cluster (not sure how much of an issue this still is after some
recent changes)
Please feel free to add any/all of the above to the Confluence page that
you think would be worthwhile to add to the pile for consideration.
---
*Chris Sampson*
IT Consultant
chris.sampson@naimuri.com
On Fri, 17 Dec 2021 at 18:01, Kevin Doran <kd...@apache.org> wrote:
> Hi Chris,
>
> I like this idea and have been thinking of a number of other improvements
> we could make for container images of Apache NiFi, including (I think
> discussed on another thread) consolidating Dockerfiles for dockerhub /
> dockermaven, as the only significant difference is where the nifi assembly
> archive comes from, arm64 images that run native on ARM platforms,
> including newer M1 Macs (requires some minor changes to NiFi source itself
> to compile).
>
> I’ve started a Wiki page [1] to collect ideas, and once we reach consensus
> on the improvements we’d like to make and possible approaches, we can turn
> these into Jiras.
>
> [1]
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements
>
> Thanks!
> Kevin
>
> > On Nov 26, 2021, at 07:46, Chris Sampson <ch...@naimuri.com.INVALID>
> wrote:
> >
> > I'm not sure whether I've seen this discussed previously, but I thought
> it
> > worth asking whether we could/should publish a JDK 11 based version of
> the
> > apache/nifi Docker Image to Docker Hub as part of the release process in
> > future?
> >
> > I realise NiFi 1.x is/will remain based on JDK 8 but with all the work
> gone
> > into enabling JDK 11 compilation and deployment, along with the planned
> > move to JDK 11 in the future for NiFi 2.x, it would seem sensible to
> start
> > making it easier for people to move towards using JDK 11 now that it's
> > available.
> >
> > The Dockerfiles used to build NiFi (I've not looked at Toolkit, MiNiFi or
> > Registry) can be given a build ARG to use different base images/versions,
> > so it should be relatively straightforward to do this and maybe publish
> the
> > Image to Docker Hub as *apache/nifi:<version>-jdk11*?
> >
> > Users can, of course, download the NiFi source to build this image
> > themselves, but to make it easier for the community it would seem
> sensible
> > to consider the additional release artifact. This avoids lots of people
> > having to pull all of the code (it's not a small repo) or build the
> image,
> > which can take a long time, etc.
> >
> >
> > ---
> > *Chris Sampson*
> > IT Consultant
> > chris.sampson@naimuri.com
>
>
Re: apache/nifi convenience Docker Image - publish JDK 11 version?
Posted by Kevin Doran <kd...@apache.org>.
Hi Chris,
I like this idea and have been thinking of a number of other improvements we could make for container images of Apache NiFi, including (I think discussed on another thread) consolidating Dockerfiles for dockerhub / dockermaven, as the only significant difference is where the nifi assembly archive comes from, arm64 images that run native on ARM platforms, including newer M1 Macs (requires some minor changes to NiFi source itself to compile).
I’ve started a Wiki page [1] to collect ideas, and once we reach consensus on the improvements we’d like to make and possible approaches, we can turn these into Jiras.
[1] https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements
Thanks!
Kevin
> On Nov 26, 2021, at 07:46, Chris Sampson <ch...@naimuri.com.INVALID> wrote:
>
> I'm not sure whether I've seen this discussed previously, but I thought it
> worth asking whether we could/should publish a JDK 11 based version of the
> apache/nifi Docker Image to Docker Hub as part of the release process in
> future?
>
> I realise NiFi 1.x is/will remain based on JDK 8 but with all the work gone
> into enabling JDK 11 compilation and deployment, along with the planned
> move to JDK 11 in the future for NiFi 2.x, it would seem sensible to start
> making it easier for people to move towards using JDK 11 now that it's
> available.
>
> The Dockerfiles used to build NiFi (I've not looked at Toolkit, MiNiFi or
> Registry) can be given a build ARG to use different base images/versions,
> so it should be relatively straightforward to do this and maybe publish the
> Image to Docker Hub as *apache/nifi:<version>-jdk11*?
>
> Users can, of course, download the NiFi source to build this image
> themselves, but to make it easier for the community it would seem sensible
> to consider the additional release artifact. This avoids lots of people
> having to pull all of the code (it's not a small repo) or build the image,
> which can take a long time, etc.
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.sampson@naimuri.com