You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Joe Witt <jo...@apache.org> on 2021/10/25 15:59:19 UTC
[discuss] NiFi 1.15
Team,
I thought I had already started such a thread but I dont see it so here goes...
We have made a ton of progress again and there are some super
important fixes as well. It is definitely time to kick out a 1.15.
My intent will be to attempt to pull together an RC this week.
Haven't done the analysis yet of what is hanging out there but will do
so. A quick look at all the features and fixes already landed though
make it clear we have more than enough to work with.
Lets please use this thread for coordination on the RC rather than it
becoming the wish list. We have new features/fixes arriving all the
time - those can be addressed in normal channels.
Thanks
Re: [discuss] NiFi 1.15
Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
Kevin,
Sounds sensible to me, thanks!
---
*Chris Sampson*
IT Consultant
chris.sampson@naimuri.com
On Fri, 5 Nov 2021 at 14:10, Kevin Doran <kd...@apache.org> wrote:
> Hi Chris,
>
> Thanks for the summary of your findings.
>
> I agree that we should clear up our process for generating Docker images
> and that the process should be consistent as possible across NiFi, MiNiFi
> Java, Registry, and Toolkit given they are all part of the same repository
> and maven project. This will clear up confusion and improve
> maintainability.
>
> I think both of these methods are important and useful:
>
> 1. building from downloaded convenience binaries for previously released
> versions
> 2. building from a local assembly output
>
> That said, I think we could probably consolidate to a single Dockerfile
> that takes the binary location as an argument that supports either a URL or
> local file path (or a version which could default to the current behavior
> of inferring a URL location). This would let us continue to support the
> flexibility we have today while maintaining a single Docker file rather
> than dockermaven and dockerhub variants. If you and others on the list
> agree, I can open up a Jira summarizing what needs to be done.
>
> Thanks,
> Kevin
>
> > On Nov 5, 2021, at 05:00, Chris Sampson <ch...@naimuri.com.INVALID>
> wrote:
> >
> > So the good news is that I realised what I was doing wrong yesterday -
> I'd
> > started a local installation after building the RC, then not shut that
> down
> > before booting up the Docker instance, so the username/password I was
> > trying to use from the Docker logs was wrong against the local
> > installation, d'oh!
> >
> > Having corrected that this morning, I've successfully booted up and
> logged
> > into a Docker instance built using the RC (with my NIFI-8779 changes so I
> > could build from the Dev server instead of the main Apache Archive).
> >
> > The reason the dockermaven build works is that it uses the locally built
> > archive files (i.e. you have to do a full Maven build locally first to
> > generate the ZIP files and then create teh Docker image). The
> > dockerhub build actually downloads published artifacts from the Apache
> > servers - to do this the Dockerfile needs to know the exact path to the
> > artifacts it's trying to download, of course.
> >
> > There was a question recently in one of the Slack channels about whether
> > both of these builds (dockerhub and dockermaven) were still valid/needed
> -
> > I think Joe replied that he wasn't sure (Docker not being an area he
> > ventures into much, which is fair enough). It may be worth considering
> > whether these two modules are both still needed or can be rationalised
> and
> > if the latter, which approach should be used - download from the archive
> > server or build from local (both have dis/advantages, but the more
> > "official" way is arguably to download from the server).
> >
> > Also maybe worth noting:
> > * NiFi Registry only has the "build from local" Dockerfile setup
> > * MiniFi (Java) has both local and archive download options, but the
> latter
> > cannot be used to build an image from the Dev server (the BASE_URL cannot
> > be changed via a build arg/env var... this is the issue addressed by
> > NIFI-8779 for NiFi)
> > * NiFi Toolkit has both local and archive download options, but are
> located
> > in a single assembly module (instead of split into two like NiFi and
> MiniFi)
> >
> > Assuming the "nifi/<version>" path will be used for the actual release
> > artifacts, this shouldn't be a blocker, but it's one to note/remember
> when
> > the time comes (assuming the "dockerhub" module is what's used to publish
> > the "apache/nifi" image to Docker Hub that is).
> >
> >
> > ---
> > *Chris Sampson*
> > IT Consultant
> > chris.sampson@naimuri.com
> >
> >
> > On Fri, 5 Nov 2021 at 03:34, David Handermann <
> exceptionfactory@apache.org>
> > wrote:
> >
> >> Chris,
> >>
> >> I am running through several verification steps, but I built 1.15.0 RC 3
> >> from sources and then ran "mvn install -pl :dockermaven -P docker" to
> build
> >> the Docker image. When starting with the following command, I was able
> to
> >> use the generated username and password to access the running NiFi
> >> container:
> >>
> >> docker run --name nifi -p 8443:8443 apache/nifi:1.15.0-dockermaven
> >>
> >> If you are still having issues, it would be helpful to provide any logs
> or
> >> error messages you are seeing.
> >>
> >> Regards,
> >> David Handermann
> >>
> >> On Thu, Nov 4, 2021 at 8:10 PM Kevin Doran <kd...@gmail.com>
> >> wrote:
> >>
> >>> Oh meant to send a link to this too:
> >>>
> >>>> ARG
> >>>
> >>
> NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
> >>>
> >>>
> >>>
> >>
> https://github.com/apache/nifi/blob/373498445fe589e2d4855a0730fbb9127f0b4452/nifi-docker/dockerhub/Dockerfile#L30
> >>>
> >>>> On Nov 4, 2021, at 9:09 PM, Kevin Doran <kd...@gmail.com>
> >> wrote:
> >>>>
> >>>> Hi Joe and Chris,
> >>>>
> >>>> Our Dockerfile that we use to build the Dockerhub image defaults to
> >>> looking for 1.15.0 instead of NiFi-1.15.0, but it is a variable that we
> >> can
> >>> override, so this is easy to workaround incase the release folder does
> >>> change. Agree its nice to keep the tree structure consistent when we
> can.
> >>>>
> >>>> I’m about to do my verification and will also check the single user
> >> with
> >>> the docker image as part of that.
> >>>>
> >>>> Cheers,
> >>>> Kevin
> >>>>
> >>>>> On Nov 4, 2021, at 6:44 PM, Joe Witt <jo...@gmail.com> wrote:
> >>>>>
> >>>>> Chris,
> >>>>>
> >>>>> Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
> >>>>> Should generally not really matter so the docker angle there is
> >>>>> interesting. Not sure why our docker images would have any
> >>>>> relationship to our dist/dev storage. But when I move them into the
> >>>>> release folder I can try to ensure I place them only in 1.15.0
> instead
> >>>>> of nifi/1.15.0.
> >>>>>
> >>>>> That directory the prov stuff makes does linger and can be annoying
> so
> >>>>> thanks for tackling that. Saw the PR. Will take a look soon if
> >>>>> nobody else has.
> >>>>>
> >>>>> Will keep an eye on your findings related to docker builds not
> working
> >>>>> with username/password things. Hopefully others can chime in there.
> >>>>>
> >>>>> Thanks
> >>>>> Send
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
> >>>>> <ch...@naimuri.com.invalid> wrote:
> >>>>>>
> >>>>>> Worryingly, when I do get the Docker Image to build (further changes
> >>> to the
> >>>>>> Dockerfile), the auto-generated username and password from the
> >> startup
> >>> logs
> >>>>>> aren't being accepted for login via my browser.
> >>>>>>
> >>>>>> I'll try to spend a little more time looking at this (but await
> input
> >>> on my
> >>>>>> earlier question/concern also).
> >>>>>>
> >>>>>>
> >>>>>> ---
> >>>>>> *Chris Sampson*
> >>>>>> IT Consultant
> >>>>>> chris.sampson@naimuri.com
> >>>>>>
> >>>>>>
> >>>>>> On Thu, 4 Nov 2021 at 20:47, Chris Sampson <
> >> chris.sampson@naimuri.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I've got most of the way through the release check process in order
> >> to
> >>>>>>> vote for 1.15.0, but I wanted to check on a change in the
> >> distribution
> >>>>>>> release artifacts.
> >>>>>>>
> >>>>>>> For 1.14.0, the Dev artifacts were located at:
> >>>>>>> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
> >>>>>>> For 1.15.0, the Dev artifacts are located at:
> >>>>>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
> >>>>>>>
> >>>>>>> i.e. there's been a change of directory/path from <version> to
> >>>>>>> nifi-<version>.
> >>>>>>>
> >>>>>>> The reason I raise this is that I can no longer build a Docker
> Image
> >>> using
> >>>>>>> the dockerhub/DockerBuild.sh script because it cannot find the
> >>> artifacts to
> >>>>>>> download. This may not be a problem if the path change is only for
> >>> the Dev
> >>>>>>> artifacts, but if the same change is going to happen for the
> >> released
> >>>>>>> artifacts, then the apache/nifi image (and presumably the
> >>>>>>> apache/nifi-registry, apache/nifi-toolkit and any minifi)
> >> convenience
> >>>>>>> images will no longer be possible as part of the Release, which
> will
> >>> likely
> >>>>>>> be an issue for a number of users that have deployments using these
> >>> images.
> >>>>>>>
> >>>>>>> I spotted this after rebasing my outstanding PR [1] for NIFI-8779
> >> [2]
> >>>>>>>
> >>>>>>>
> >>>>>>> Additionally, I noted NIFI-9366 [3] after an unwanted directory was
> >>>>>>> created by the unit tests executed during building and testing
> >>> 1.15.0-RC3.
> >>>>>>> I raised a PR [4] - this is a minor issue and not a reason to block
> >>> any
> >>>>>>> release.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> [1] https://github.com/apache/nifi/pull/5213
> >>>>>>> [2] https://issues.apache.org/jira/browse/NIFI-8779
> >>>>>>>
> >>>>>>> [3] https://issues.apache.org/jira/browse/NIFI-9366
> >>>>>>> [4] https://github.com/apache/nifi/pull/5510
> >>>>>>>
> >>>>>>> ---
> >>>>>>> *Chris Sampson*
> >>>>>>> IT Consultant
> >>>>>>> chris.sampson@naimuri.com
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sat, 30 Oct 2021 at 01:52, Joe Witt <jo...@apache.org> wrote:
> >>>>>>>
> >>>>>>>> Otto
> >>>>>>>>
> >>>>>>>> Got ya. Yeah it was this way in 1.14 as well. And to be clear
> >> with
> >>> every
> >>>>>>>> release what we are actually voting upon is the source release.
> >> Now
> >>> the
> >>>>>>>> source release includes nifi, minifi, nifi registry, stateless
> nifi
> >>> and
> >>>>>>>> toolkits among all the other having always been there goodies.
> >>>>>>>>
> >>>>>>>> Some of these things we make available in the form of convenience
> >>> binaries
> >>>>>>>> to make it easier on folks to consume.
> >>>>>>>>
> >>>>>>>> I think you dont need to do any verification you would not have
> >> done
> >>>>>>>> before.
> >>>>>>>>
> >>>>>>>> But I do hope folks help maintain various ways of easing more
> folks
> >>>>>>>> knowing
> >>>>>>>> what to vet with a given release
> >>>>>>>>
> >>>>>>>> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <
> >> ottobackwards@gmail.com
> >>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> This release, we are verifying not only Nifi, but also Minifi
> >> Java.
> >>> At
> >>>>>>>>> least that is my understanding.
> >>>>>>>>>
> >>>>>>>>> This would be my first time having *anything* to do with minifi,
> >>> i’ve
> >>>>>>>> not
> >>>>>>>>> even run it before.
> >>>>>>>>>
> >>>>>>>>> As such, I think the RC validation guide needs to be update to
> >>> include
> >>>>>>>> the
> >>>>>>>>> information about now validating nifi and minifi together.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> From: Joe Witt <jo...@apache.org> <jo...@apache.org>
> >>>>>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> >>> dev@nifi.apache.org>
> >>>>>>>>> Date: October 25, 2021 at 11:59:39
> >>>>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
> >> dev@nifi.apache.org>
> >>>>>>>>> Subject: [discuss] NiFi 1.15
> >>>>>>>>>
> >>>>>>>>> Team,
> >>>>>>>>>
> >>>>>>>>> I thought I had already started such a thread but I dont see it
> so
> >>> here
> >>>>>>>>> goes...
> >>>>>>>>>
> >>>>>>>>> We have made a ton of progress again and there are some super
> >>>>>>>>> important fixes as well. It is definitely time to kick out a
> 1.15.
> >>>>>>>>> My intent will be to attempt to pull together an RC this week.
> >>>>>>>>> Haven't done the analysis yet of what is hanging out there but
> >> will
> >>> do
> >>>>>>>>> so. A quick look at all the features and fixes already landed
> >> though
> >>>>>>>>> make it clear we have more than enough to work with.
> >>>>>>>>>
> >>>>>>>>> Lets please use this thread for coordination on the RC rather
> than
> >>> it
> >>>>>>>>> becoming the wish list. We have new features/fixes arriving all
> >> the
> >>>>>>>>> time - those can be addressed in normal channels.
> >>>>>>>>>
> >>>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>>
> >>
>
>
Re: [discuss] NiFi 1.15
Posted by Kevin Doran <kd...@apache.org>.
Thanks, Chris! I’ll take a look at that PR.
Regarding this:
> Might be worth considering building the docker build & push into the CI
> pipeline for the release if it's not already (and if that would fit how the
> release process works).
I’ve looked into automating in the past, and it might be possible to setup as a (manual) GitHub Action workflow with some help from ASF Infra. There is some concern about leaking credentials, so we left it at needing to do some investigation for if GitHub Actions Secrets meets their criteria for security.
In any case, I intent to write up the manual process I use now, which could be a starting point for how to automate in the future. Will share this next week most likely.
Cheers,
Kevin
> On Nov 6, 2021, at 15:58, Chris Sampson <ch...@naimuri.com.INVALID> wrote:
>
> I've updated NIFI-8779 PR [1] to at least remove the duplicated scripts
> between nifi docker hub & maven builds (as well as update the plugin
> versions and make the integration tests work again).
>
> There's still work to do to rationalise the builds properly in future, but
> this might be a step in the right direction at least (although we might
> decide to use the learnings from the PR and this discussion and instead
> rework things more fully in future).
>
> Might be worth considering building the docker build & push into the CI
> pipeline for the release if it's not already (and if that would fit how the
> release process works).
>
>
> [1] https://github.com/apache/nifi/pull/5213
>
>
> Cheers,
>
> Chris Sampson
>
> On Fri, 5 Nov 2021, 14:15 David Handermann, <ex...@apache.org>
> wrote:
>
>> Thanks for the update Chris!
>>
>> I agree with the suggestion to streamline the Dockerfiles for NiFi builds,
>> there is a lot of duplication between dockermaven and dockerhub, so
>> unifying the modules and parameterizing necessary differences would be a
>> helpful improvement.
>>
>> Regards,
>> David Handermann
>>
>> On Fri, Nov 5, 2021 at 9:10 AM Kevin Doran <kd...@apache.org> wrote:
>>
>>> Hi Chris,
>>>
>>> Thanks for the summary of your findings.
>>>
>>> I agree that we should clear up our process for generating Docker images
>>> and that the process should be consistent as possible across NiFi, MiNiFi
>>> Java, Registry, and Toolkit given they are all part of the same
>> repository
>>> and maven project. This will clear up confusion and improve
>>> maintainability.
>>>
>>> I think both of these methods are important and useful:
>>>
>>> 1. building from downloaded convenience binaries for previously released
>>> versions
>>> 2. building from a local assembly output
>>>
>>> That said, I think we could probably consolidate to a single Dockerfile
>>> that takes the binary location as an argument that supports either a URL
>> or
>>> local file path (or a version which could default to the current behavior
>>> of inferring a URL location). This would let us continue to support the
>>> flexibility we have today while maintaining a single Docker file rather
>>> than dockermaven and dockerhub variants. If you and others on the list
>>> agree, I can open up a Jira summarizing what needs to be done.
>>>
>>> Thanks,
>>> Kevin
>>>
>>>> On Nov 5, 2021, at 05:00, Chris Sampson <chris.sampson@naimuri.com
>> .INVALID>
>>> wrote:
>>>>
>>>> So the good news is that I realised what I was doing wrong yesterday -
>>> I'd
>>>> started a local installation after building the RC, then not shut that
>>> down
>>>> before booting up the Docker instance, so the username/password I was
>>>> trying to use from the Docker logs was wrong against the local
>>>> installation, d'oh!
>>>>
>>>> Having corrected that this morning, I've successfully booted up and
>>> logged
>>>> into a Docker instance built using the RC (with my NIFI-8779 changes
>> so I
>>>> could build from the Dev server instead of the main Apache Archive).
>>>>
>>>> The reason the dockermaven build works is that it uses the locally
>> built
>>>> archive files (i.e. you have to do a full Maven build locally first to
>>>> generate the ZIP files and then create teh Docker image). The
>>>> dockerhub build actually downloads published artifacts from the Apache
>>>> servers - to do this the Dockerfile needs to know the exact path to the
>>>> artifacts it's trying to download, of course.
>>>>
>>>> There was a question recently in one of the Slack channels about
>> whether
>>>> both of these builds (dockerhub and dockermaven) were still
>> valid/needed
>>> -
>>>> I think Joe replied that he wasn't sure (Docker not being an area he
>>>> ventures into much, which is fair enough). It may be worth considering
>>>> whether these two modules are both still needed or can be rationalised
>>> and
>>>> if the latter, which approach should be used - download from the
>> archive
>>>> server or build from local (both have dis/advantages, but the more
>>>> "official" way is arguably to download from the server).
>>>>
>>>> Also maybe worth noting:
>>>> * NiFi Registry only has the "build from local" Dockerfile setup
>>>> * MiniFi (Java) has both local and archive download options, but the
>>> latter
>>>> cannot be used to build an image from the Dev server (the BASE_URL
>> cannot
>>>> be changed via a build arg/env var... this is the issue addressed by
>>>> NIFI-8779 for NiFi)
>>>> * NiFi Toolkit has both local and archive download options, but are
>>> located
>>>> in a single assembly module (instead of split into two like NiFi and
>>> MiniFi)
>>>>
>>>> Assuming the "nifi/<version>" path will be used for the actual release
>>>> artifacts, this shouldn't be a blocker, but it's one to note/remember
>>> when
>>>> the time comes (assuming the "dockerhub" module is what's used to
>> publish
>>>> the "apache/nifi" image to Docker Hub that is).
>>>>
>>>>
>>>> ---
>>>> *Chris Sampson*
>>>> IT Consultant
>>>> chris.sampson@naimuri.com
>>>>
>>>>
>>>> On Fri, 5 Nov 2021 at 03:34, David Handermann <
>>> exceptionfactory@apache.org>
>>>> wrote:
>>>>
>>>>> Chris,
>>>>>
>>>>> I am running through several verification steps, but I built 1.15.0
>> RC 3
>>>>> from sources and then ran "mvn install -pl :dockermaven -P docker" to
>>> build
>>>>> the Docker image. When starting with the following command, I was
>> able
>>> to
>>>>> use the generated username and password to access the running NiFi
>>>>> container:
>>>>>
>>>>> docker run --name nifi -p 8443:8443 apache/nifi:1.15.0-dockermaven
>>>>>
>>>>> If you are still having issues, it would be helpful to provide any
>> logs
>>> or
>>>>> error messages you are seeing.
>>>>>
>>>>> Regards,
>>>>> David Handermann
>>>>>
>>>>> On Thu, Nov 4, 2021 at 8:10 PM Kevin Doran <kd...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Oh meant to send a link to this too:
>>>>>>
>>>>>>> ARG
>>>>>>
>>>>>
>>>
>> NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>> https://github.com/apache/nifi/blob/373498445fe589e2d4855a0730fbb9127f0b4452/nifi-docker/dockerhub/Dockerfile#L30
>>>>>>
>>>>>>> On Nov 4, 2021, at 9:09 PM, Kevin Doran <kd...@gmail.com>
>>>>> wrote:
>>>>>>>
>>>>>>> Hi Joe and Chris,
>>>>>>>
>>>>>>> Our Dockerfile that we use to build the Dockerhub image defaults to
>>>>>> looking for 1.15.0 instead of NiFi-1.15.0, but it is a variable that
>> we
>>>>> can
>>>>>> override, so this is easy to workaround incase the release folder
>> does
>>>>>> change. Agree its nice to keep the tree structure consistent when we
>>> can.
>>>>>>>
>>>>>>> I’m about to do my verification and will also check the single user
>>>>> with
>>>>>> the docker image as part of that.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Kevin
>>>>>>>
>>>>>>>> On Nov 4, 2021, at 6:44 PM, Joe Witt <jo...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Chris,
>>>>>>>>
>>>>>>>> Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
>>>>>>>> Should generally not really matter so the docker angle there is
>>>>>>>> interesting. Not sure why our docker images would have any
>>>>>>>> relationship to our dist/dev storage. But when I move them into
>> the
>>>>>>>> release folder I can try to ensure I place them only in 1.15.0
>>> instead
>>>>>>>> of nifi/1.15.0.
>>>>>>>>
>>>>>>>> That directory the prov stuff makes does linger and can be annoying
>>> so
>>>>>>>> thanks for tackling that. Saw the PR. Will take a look soon if
>>>>>>>> nobody else has.
>>>>>>>>
>>>>>>>> Will keep an eye on your findings related to docker builds not
>>> working
>>>>>>>> with username/password things. Hopefully others can chime in
>> there.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Send
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
>>>>>>>> <ch...@naimuri.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>> Worryingly, when I do get the Docker Image to build (further
>> changes
>>>>>> to the
>>>>>>>>> Dockerfile), the auto-generated username and password from the
>>>>> startup
>>>>>> logs
>>>>>>>>> aren't being accepted for login via my browser.
>>>>>>>>>
>>>>>>>>> I'll try to spend a little more time looking at this (but await
>>> input
>>>>>> on my
>>>>>>>>> earlier question/concern also).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> *Chris Sampson*
>>>>>>>>> IT Consultant
>>>>>>>>> chris.sampson@naimuri.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, 4 Nov 2021 at 20:47, Chris Sampson <
>>>>> chris.sampson@naimuri.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I've got most of the way through the release check process in
>> order
>>>>> to
>>>>>>>>>> vote for 1.15.0, but I wanted to check on a change in the
>>>>> distribution
>>>>>>>>>> release artifacts.
>>>>>>>>>>
>>>>>>>>>> For 1.14.0, the Dev artifacts were located at:
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
>>>>>>>>>> For 1.15.0, the Dev artifacts are located at:
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
>>>>>>>>>>
>>>>>>>>>> i.e. there's been a change of directory/path from <version> to
>>>>>>>>>> nifi-<version>.
>>>>>>>>>>
>>>>>>>>>> The reason I raise this is that I can no longer build a Docker
>>> Image
>>>>>> using
>>>>>>>>>> the dockerhub/DockerBuild.sh script because it cannot find the
>>>>>> artifacts to
>>>>>>>>>> download. This may not be a problem if the path change is only
>> for
>>>>>> the Dev
>>>>>>>>>> artifacts, but if the same change is going to happen for the
>>>>> released
>>>>>>>>>> artifacts, then the apache/nifi image (and presumably the
>>>>>>>>>> apache/nifi-registry, apache/nifi-toolkit and any minifi)
>>>>> convenience
>>>>>>>>>> images will no longer be possible as part of the Release, which
>>> will
>>>>>> likely
>>>>>>>>>> be an issue for a number of users that have deployments using
>> these
>>>>>> images.
>>>>>>>>>>
>>>>>>>>>> I spotted this after rebasing my outstanding PR [1] for NIFI-8779
>>>>> [2]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Additionally, I noted NIFI-9366 [3] after an unwanted directory
>> was
>>>>>>>>>> created by the unit tests executed during building and testing
>>>>>> 1.15.0-RC3.
>>>>>>>>>> I raised a PR [4] - this is a minor issue and not a reason to
>> block
>>>>>> any
>>>>>>>>>> release.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [1] https://github.com/apache/nifi/pull/5213
>>>>>>>>>> [2] https://issues.apache.org/jira/browse/NIFI-8779
>>>>>>>>>>
>>>>>>>>>> [3] https://issues.apache.org/jira/browse/NIFI-9366
>>>>>>>>>> [4] https://github.com/apache/nifi/pull/5510
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> *Chris Sampson*
>>>>>>>>>> IT Consultant
>>>>>>>>>> chris.sampson@naimuri.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, 30 Oct 2021 at 01:52, Joe Witt <jo...@apache.org>
>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Otto
>>>>>>>>>>>
>>>>>>>>>>> Got ya. Yeah it was this way in 1.14 as well. And to be clear
>>>>> with
>>>>>> every
>>>>>>>>>>> release what we are actually voting upon is the source release.
>>>>> Now
>>>>>> the
>>>>>>>>>>> source release includes nifi, minifi, nifi registry, stateless
>>> nifi
>>>>>> and
>>>>>>>>>>> toolkits among all the other having always been there goodies.
>>>>>>>>>>>
>>>>>>>>>>> Some of these things we make available in the form of
>> convenience
>>>>>> binaries
>>>>>>>>>>> to make it easier on folks to consume.
>>>>>>>>>>>
>>>>>>>>>>> I think you dont need to do any verification you would not have
>>>>> done
>>>>>>>>>>> before.
>>>>>>>>>>>
>>>>>>>>>>> But I do hope folks help maintain various ways of easing more
>>> folks
>>>>>>>>>>> knowing
>>>>>>>>>>> what to vet with a given release
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <
>>>>> ottobackwards@gmail.com
>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> This release, we are verifying not only Nifi, but also Minifi
>>>>> Java.
>>>>>> At
>>>>>>>>>>>> least that is my understanding.
>>>>>>>>>>>>
>>>>>>>>>>>> This would be my first time having *anything* to do with
>> minifi,
>>>>>> i’ve
>>>>>>>>>>> not
>>>>>>>>>>>> even run it before.
>>>>>>>>>>>>
>>>>>>>>>>>> As such, I think the RC validation guide needs to be update to
>>>>>> include
>>>>>>>>>>> the
>>>>>>>>>>>> information about now validating nifi and minifi together.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> From: Joe Witt <jo...@apache.org> <jo...@apache.org>
>>>>>>>>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
>>>>>> dev@nifi.apache.org>
>>>>>>>>>>>> Date: October 25, 2021 at 11:59:39
>>>>>>>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
>>>>> dev@nifi.apache.org>
>>>>>>>>>>>> Subject: [discuss] NiFi 1.15
>>>>>>>>>>>>
>>>>>>>>>>>> Team,
>>>>>>>>>>>>
>>>>>>>>>>>> I thought I had already started such a thread but I dont see it
>>> so
>>>>>> here
>>>>>>>>>>>> goes...
>>>>>>>>>>>>
>>>>>>>>>>>> We have made a ton of progress again and there are some super
>>>>>>>>>>>> important fixes as well. It is definitely time to kick out a
>>> 1.15.
>>>>>>>>>>>> My intent will be to attempt to pull together an RC this week.
>>>>>>>>>>>> Haven't done the analysis yet of what is hanging out there but
>>>>> will
>>>>>> do
>>>>>>>>>>>> so. A quick look at all the features and fixes already landed
>>>>> though
>>>>>>>>>>>> make it clear we have more than enough to work with.
>>>>>>>>>>>>
>>>>>>>>>>>> Lets please use this thread for coordination on the RC rather
>>> than
>>>>>> it
>>>>>>>>>>>> becoming the wish list. We have new features/fixes arriving all
>>>>> the
>>>>>>>>>>>> time - those can be addressed in normal channels.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>
Re: [discuss] NiFi 1.15
Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
I've updated NIFI-8779 PR [1] to at least remove the duplicated scripts
between nifi docker hub & maven builds (as well as update the plugin
versions and make the integration tests work again).
There's still work to do to rationalise the builds properly in future, but
this might be a step in the right direction at least (although we might
decide to use the learnings from the PR and this discussion and instead
rework things more fully in future).
Might be worth considering building the docker build & push into the CI
pipeline for the release if it's not already (and if that would fit how the
release process works).
[1] https://github.com/apache/nifi/pull/5213
Cheers,
Chris Sampson
On Fri, 5 Nov 2021, 14:15 David Handermann, <ex...@apache.org>
wrote:
> Thanks for the update Chris!
>
> I agree with the suggestion to streamline the Dockerfiles for NiFi builds,
> there is a lot of duplication between dockermaven and dockerhub, so
> unifying the modules and parameterizing necessary differences would be a
> helpful improvement.
>
> Regards,
> David Handermann
>
> On Fri, Nov 5, 2021 at 9:10 AM Kevin Doran <kd...@apache.org> wrote:
>
> > Hi Chris,
> >
> > Thanks for the summary of your findings.
> >
> > I agree that we should clear up our process for generating Docker images
> > and that the process should be consistent as possible across NiFi, MiNiFi
> > Java, Registry, and Toolkit given they are all part of the same
> repository
> > and maven project. This will clear up confusion and improve
> > maintainability.
> >
> > I think both of these methods are important and useful:
> >
> > 1. building from downloaded convenience binaries for previously released
> > versions
> > 2. building from a local assembly output
> >
> > That said, I think we could probably consolidate to a single Dockerfile
> > that takes the binary location as an argument that supports either a URL
> or
> > local file path (or a version which could default to the current behavior
> > of inferring a URL location). This would let us continue to support the
> > flexibility we have today while maintaining a single Docker file rather
> > than dockermaven and dockerhub variants. If you and others on the list
> > agree, I can open up a Jira summarizing what needs to be done.
> >
> > Thanks,
> > Kevin
> >
> > > On Nov 5, 2021, at 05:00, Chris Sampson <chris.sampson@naimuri.com
> .INVALID>
> > wrote:
> > >
> > > So the good news is that I realised what I was doing wrong yesterday -
> > I'd
> > > started a local installation after building the RC, then not shut that
> > down
> > > before booting up the Docker instance, so the username/password I was
> > > trying to use from the Docker logs was wrong against the local
> > > installation, d'oh!
> > >
> > > Having corrected that this morning, I've successfully booted up and
> > logged
> > > into a Docker instance built using the RC (with my NIFI-8779 changes
> so I
> > > could build from the Dev server instead of the main Apache Archive).
> > >
> > > The reason the dockermaven build works is that it uses the locally
> built
> > > archive files (i.e. you have to do a full Maven build locally first to
> > > generate the ZIP files and then create teh Docker image). The
> > > dockerhub build actually downloads published artifacts from the Apache
> > > servers - to do this the Dockerfile needs to know the exact path to the
> > > artifacts it's trying to download, of course.
> > >
> > > There was a question recently in one of the Slack channels about
> whether
> > > both of these builds (dockerhub and dockermaven) were still
> valid/needed
> > -
> > > I think Joe replied that he wasn't sure (Docker not being an area he
> > > ventures into much, which is fair enough). It may be worth considering
> > > whether these two modules are both still needed or can be rationalised
> > and
> > > if the latter, which approach should be used - download from the
> archive
> > > server or build from local (both have dis/advantages, but the more
> > > "official" way is arguably to download from the server).
> > >
> > > Also maybe worth noting:
> > > * NiFi Registry only has the "build from local" Dockerfile setup
> > > * MiniFi (Java) has both local and archive download options, but the
> > latter
> > > cannot be used to build an image from the Dev server (the BASE_URL
> cannot
> > > be changed via a build arg/env var... this is the issue addressed by
> > > NIFI-8779 for NiFi)
> > > * NiFi Toolkit has both local and archive download options, but are
> > located
> > > in a single assembly module (instead of split into two like NiFi and
> > MiniFi)
> > >
> > > Assuming the "nifi/<version>" path will be used for the actual release
> > > artifacts, this shouldn't be a blocker, but it's one to note/remember
> > when
> > > the time comes (assuming the "dockerhub" module is what's used to
> publish
> > > the "apache/nifi" image to Docker Hub that is).
> > >
> > >
> > > ---
> > > *Chris Sampson*
> > > IT Consultant
> > > chris.sampson@naimuri.com
> > >
> > >
> > > On Fri, 5 Nov 2021 at 03:34, David Handermann <
> > exceptionfactory@apache.org>
> > > wrote:
> > >
> > >> Chris,
> > >>
> > >> I am running through several verification steps, but I built 1.15.0
> RC 3
> > >> from sources and then ran "mvn install -pl :dockermaven -P docker" to
> > build
> > >> the Docker image. When starting with the following command, I was
> able
> > to
> > >> use the generated username and password to access the running NiFi
> > >> container:
> > >>
> > >> docker run --name nifi -p 8443:8443 apache/nifi:1.15.0-dockermaven
> > >>
> > >> If you are still having issues, it would be helpful to provide any
> logs
> > or
> > >> error messages you are seeing.
> > >>
> > >> Regards,
> > >> David Handermann
> > >>
> > >> On Thu, Nov 4, 2021 at 8:10 PM Kevin Doran <kd...@gmail.com>
> > >> wrote:
> > >>
> > >>> Oh meant to send a link to this too:
> > >>>
> > >>>> ARG
> > >>>
> > >>
> >
> NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://github.com/apache/nifi/blob/373498445fe589e2d4855a0730fbb9127f0b4452/nifi-docker/dockerhub/Dockerfile#L30
> > >>>
> > >>>> On Nov 4, 2021, at 9:09 PM, Kevin Doran <kd...@gmail.com>
> > >> wrote:
> > >>>>
> > >>>> Hi Joe and Chris,
> > >>>>
> > >>>> Our Dockerfile that we use to build the Dockerhub image defaults to
> > >>> looking for 1.15.0 instead of NiFi-1.15.0, but it is a variable that
> we
> > >> can
> > >>> override, so this is easy to workaround incase the release folder
> does
> > >>> change. Agree its nice to keep the tree structure consistent when we
> > can.
> > >>>>
> > >>>> I’m about to do my verification and will also check the single user
> > >> with
> > >>> the docker image as part of that.
> > >>>>
> > >>>> Cheers,
> > >>>> Kevin
> > >>>>
> > >>>>> On Nov 4, 2021, at 6:44 PM, Joe Witt <jo...@gmail.com> wrote:
> > >>>>>
> > >>>>> Chris,
> > >>>>>
> > >>>>> Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
> > >>>>> Should generally not really matter so the docker angle there is
> > >>>>> interesting. Not sure why our docker images would have any
> > >>>>> relationship to our dist/dev storage. But when I move them into
> the
> > >>>>> release folder I can try to ensure I place them only in 1.15.0
> > instead
> > >>>>> of nifi/1.15.0.
> > >>>>>
> > >>>>> That directory the prov stuff makes does linger and can be annoying
> > so
> > >>>>> thanks for tackling that. Saw the PR. Will take a look soon if
> > >>>>> nobody else has.
> > >>>>>
> > >>>>> Will keep an eye on your findings related to docker builds not
> > working
> > >>>>> with username/password things. Hopefully others can chime in
> there.
> > >>>>>
> > >>>>> Thanks
> > >>>>> Send
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
> > >>>>> <ch...@naimuri.com.invalid> wrote:
> > >>>>>>
> > >>>>>> Worryingly, when I do get the Docker Image to build (further
> changes
> > >>> to the
> > >>>>>> Dockerfile), the auto-generated username and password from the
> > >> startup
> > >>> logs
> > >>>>>> aren't being accepted for login via my browser.
> > >>>>>>
> > >>>>>> I'll try to spend a little more time looking at this (but await
> > input
> > >>> on my
> > >>>>>> earlier question/concern also).
> > >>>>>>
> > >>>>>>
> > >>>>>> ---
> > >>>>>> *Chris Sampson*
> > >>>>>> IT Consultant
> > >>>>>> chris.sampson@naimuri.com
> > >>>>>>
> > >>>>>>
> > >>>>>> On Thu, 4 Nov 2021 at 20:47, Chris Sampson <
> > >> chris.sampson@naimuri.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> I've got most of the way through the release check process in
> order
> > >> to
> > >>>>>>> vote for 1.15.0, but I wanted to check on a change in the
> > >> distribution
> > >>>>>>> release artifacts.
> > >>>>>>>
> > >>>>>>> For 1.14.0, the Dev artifacts were located at:
> > >>>>>>> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
> > >>>>>>> For 1.15.0, the Dev artifacts are located at:
> > >>>>>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
> > >>>>>>>
> > >>>>>>> i.e. there's been a change of directory/path from <version> to
> > >>>>>>> nifi-<version>.
> > >>>>>>>
> > >>>>>>> The reason I raise this is that I can no longer build a Docker
> > Image
> > >>> using
> > >>>>>>> the dockerhub/DockerBuild.sh script because it cannot find the
> > >>> artifacts to
> > >>>>>>> download. This may not be a problem if the path change is only
> for
> > >>> the Dev
> > >>>>>>> artifacts, but if the same change is going to happen for the
> > >> released
> > >>>>>>> artifacts, then the apache/nifi image (and presumably the
> > >>>>>>> apache/nifi-registry, apache/nifi-toolkit and any minifi)
> > >> convenience
> > >>>>>>> images will no longer be possible as part of the Release, which
> > will
> > >>> likely
> > >>>>>>> be an issue for a number of users that have deployments using
> these
> > >>> images.
> > >>>>>>>
> > >>>>>>> I spotted this after rebasing my outstanding PR [1] for NIFI-8779
> > >> [2]
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Additionally, I noted NIFI-9366 [3] after an unwanted directory
> was
> > >>>>>>> created by the unit tests executed during building and testing
> > >>> 1.15.0-RC3.
> > >>>>>>> I raised a PR [4] - this is a minor issue and not a reason to
> block
> > >>> any
> > >>>>>>> release.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> [1] https://github.com/apache/nifi/pull/5213
> > >>>>>>> [2] https://issues.apache.org/jira/browse/NIFI-8779
> > >>>>>>>
> > >>>>>>> [3] https://issues.apache.org/jira/browse/NIFI-9366
> > >>>>>>> [4] https://github.com/apache/nifi/pull/5510
> > >>>>>>>
> > >>>>>>> ---
> > >>>>>>> *Chris Sampson*
> > >>>>>>> IT Consultant
> > >>>>>>> chris.sampson@naimuri.com
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Sat, 30 Oct 2021 at 01:52, Joe Witt <jo...@apache.org>
> wrote:
> > >>>>>>>
> > >>>>>>>> Otto
> > >>>>>>>>
> > >>>>>>>> Got ya. Yeah it was this way in 1.14 as well. And to be clear
> > >> with
> > >>> every
> > >>>>>>>> release what we are actually voting upon is the source release.
> > >> Now
> > >>> the
> > >>>>>>>> source release includes nifi, minifi, nifi registry, stateless
> > nifi
> > >>> and
> > >>>>>>>> toolkits among all the other having always been there goodies.
> > >>>>>>>>
> > >>>>>>>> Some of these things we make available in the form of
> convenience
> > >>> binaries
> > >>>>>>>> to make it easier on folks to consume.
> > >>>>>>>>
> > >>>>>>>> I think you dont need to do any verification you would not have
> > >> done
> > >>>>>>>> before.
> > >>>>>>>>
> > >>>>>>>> But I do hope folks help maintain various ways of easing more
> > folks
> > >>>>>>>> knowing
> > >>>>>>>> what to vet with a given release
> > >>>>>>>>
> > >>>>>>>> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <
> > >> ottobackwards@gmail.com
> > >>>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> This release, we are verifying not only Nifi, but also Minifi
> > >> Java.
> > >>> At
> > >>>>>>>>> least that is my understanding.
> > >>>>>>>>>
> > >>>>>>>>> This would be my first time having *anything* to do with
> minifi,
> > >>> i’ve
> > >>>>>>>> not
> > >>>>>>>>> even run it before.
> > >>>>>>>>>
> > >>>>>>>>> As such, I think the RC validation guide needs to be update to
> > >>> include
> > >>>>>>>> the
> > >>>>>>>>> information about now validating nifi and minifi together.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> From: Joe Witt <jo...@apache.org> <jo...@apache.org>
> > >>>>>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > >>> dev@nifi.apache.org>
> > >>>>>>>>> Date: October 25, 2021 at 11:59:39
> > >>>>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
> > >> dev@nifi.apache.org>
> > >>>>>>>>> Subject: [discuss] NiFi 1.15
> > >>>>>>>>>
> > >>>>>>>>> Team,
> > >>>>>>>>>
> > >>>>>>>>> I thought I had already started such a thread but I dont see it
> > so
> > >>> here
> > >>>>>>>>> goes...
> > >>>>>>>>>
> > >>>>>>>>> We have made a ton of progress again and there are some super
> > >>>>>>>>> important fixes as well. It is definitely time to kick out a
> > 1.15.
> > >>>>>>>>> My intent will be to attempt to pull together an RC this week.
> > >>>>>>>>> Haven't done the analysis yet of what is hanging out there but
> > >> will
> > >>> do
> > >>>>>>>>> so. A quick look at all the features and fixes already landed
> > >> though
> > >>>>>>>>> make it clear we have more than enough to work with.
> > >>>>>>>>>
> > >>>>>>>>> Lets please use this thread for coordination on the RC rather
> > than
> > >>> it
> > >>>>>>>>> becoming the wish list. We have new features/fixes arriving all
> > >> the
> > >>>>>>>>> time - those can be addressed in normal channels.
> > >>>>>>>>>
> > >>>>>>>>> Thanks
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>>
> > >>
> >
> >
>
Re: [discuss] NiFi 1.15
Posted by David Handermann <ex...@apache.org>.
Thanks for the update Chris!
I agree with the suggestion to streamline the Dockerfiles for NiFi builds,
there is a lot of duplication between dockermaven and dockerhub, so
unifying the modules and parameterizing necessary differences would be a
helpful improvement.
Regards,
David Handermann
On Fri, Nov 5, 2021 at 9:10 AM Kevin Doran <kd...@apache.org> wrote:
> Hi Chris,
>
> Thanks for the summary of your findings.
>
> I agree that we should clear up our process for generating Docker images
> and that the process should be consistent as possible across NiFi, MiNiFi
> Java, Registry, and Toolkit given they are all part of the same repository
> and maven project. This will clear up confusion and improve
> maintainability.
>
> I think both of these methods are important and useful:
>
> 1. building from downloaded convenience binaries for previously released
> versions
> 2. building from a local assembly output
>
> That said, I think we could probably consolidate to a single Dockerfile
> that takes the binary location as an argument that supports either a URL or
> local file path (or a version which could default to the current behavior
> of inferring a URL location). This would let us continue to support the
> flexibility we have today while maintaining a single Docker file rather
> than dockermaven and dockerhub variants. If you and others on the list
> agree, I can open up a Jira summarizing what needs to be done.
>
> Thanks,
> Kevin
>
> > On Nov 5, 2021, at 05:00, Chris Sampson <ch...@naimuri.com.INVALID>
> wrote:
> >
> > So the good news is that I realised what I was doing wrong yesterday -
> I'd
> > started a local installation after building the RC, then not shut that
> down
> > before booting up the Docker instance, so the username/password I was
> > trying to use from the Docker logs was wrong against the local
> > installation, d'oh!
> >
> > Having corrected that this morning, I've successfully booted up and
> logged
> > into a Docker instance built using the RC (with my NIFI-8779 changes so I
> > could build from the Dev server instead of the main Apache Archive).
> >
> > The reason the dockermaven build works is that it uses the locally built
> > archive files (i.e. you have to do a full Maven build locally first to
> > generate the ZIP files and then create teh Docker image). The
> > dockerhub build actually downloads published artifacts from the Apache
> > servers - to do this the Dockerfile needs to know the exact path to the
> > artifacts it's trying to download, of course.
> >
> > There was a question recently in one of the Slack channels about whether
> > both of these builds (dockerhub and dockermaven) were still valid/needed
> -
> > I think Joe replied that he wasn't sure (Docker not being an area he
> > ventures into much, which is fair enough). It may be worth considering
> > whether these two modules are both still needed or can be rationalised
> and
> > if the latter, which approach should be used - download from the archive
> > server or build from local (both have dis/advantages, but the more
> > "official" way is arguably to download from the server).
> >
> > Also maybe worth noting:
> > * NiFi Registry only has the "build from local" Dockerfile setup
> > * MiniFi (Java) has both local and archive download options, but the
> latter
> > cannot be used to build an image from the Dev server (the BASE_URL cannot
> > be changed via a build arg/env var... this is the issue addressed by
> > NIFI-8779 for NiFi)
> > * NiFi Toolkit has both local and archive download options, but are
> located
> > in a single assembly module (instead of split into two like NiFi and
> MiniFi)
> >
> > Assuming the "nifi/<version>" path will be used for the actual release
> > artifacts, this shouldn't be a blocker, but it's one to note/remember
> when
> > the time comes (assuming the "dockerhub" module is what's used to publish
> > the "apache/nifi" image to Docker Hub that is).
> >
> >
> > ---
> > *Chris Sampson*
> > IT Consultant
> > chris.sampson@naimuri.com
> >
> >
> > On Fri, 5 Nov 2021 at 03:34, David Handermann <
> exceptionfactory@apache.org>
> > wrote:
> >
> >> Chris,
> >>
> >> I am running through several verification steps, but I built 1.15.0 RC 3
> >> from sources and then ran "mvn install -pl :dockermaven -P docker" to
> build
> >> the Docker image. When starting with the following command, I was able
> to
> >> use the generated username and password to access the running NiFi
> >> container:
> >>
> >> docker run --name nifi -p 8443:8443 apache/nifi:1.15.0-dockermaven
> >>
> >> If you are still having issues, it would be helpful to provide any logs
> or
> >> error messages you are seeing.
> >>
> >> Regards,
> >> David Handermann
> >>
> >> On Thu, Nov 4, 2021 at 8:10 PM Kevin Doran <kd...@gmail.com>
> >> wrote:
> >>
> >>> Oh meant to send a link to this too:
> >>>
> >>>> ARG
> >>>
> >>
> NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
> >>>
> >>>
> >>>
> >>
> https://github.com/apache/nifi/blob/373498445fe589e2d4855a0730fbb9127f0b4452/nifi-docker/dockerhub/Dockerfile#L30
> >>>
> >>>> On Nov 4, 2021, at 9:09 PM, Kevin Doran <kd...@gmail.com>
> >> wrote:
> >>>>
> >>>> Hi Joe and Chris,
> >>>>
> >>>> Our Dockerfile that we use to build the Dockerhub image defaults to
> >>> looking for 1.15.0 instead of NiFi-1.15.0, but it is a variable that we
> >> can
> >>> override, so this is easy to workaround incase the release folder does
> >>> change. Agree its nice to keep the tree structure consistent when we
> can.
> >>>>
> >>>> I’m about to do my verification and will also check the single user
> >> with
> >>> the docker image as part of that.
> >>>>
> >>>> Cheers,
> >>>> Kevin
> >>>>
> >>>>> On Nov 4, 2021, at 6:44 PM, Joe Witt <jo...@gmail.com> wrote:
> >>>>>
> >>>>> Chris,
> >>>>>
> >>>>> Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
> >>>>> Should generally not really matter so the docker angle there is
> >>>>> interesting. Not sure why our docker images would have any
> >>>>> relationship to our dist/dev storage. But when I move them into the
> >>>>> release folder I can try to ensure I place them only in 1.15.0
> instead
> >>>>> of nifi/1.15.0.
> >>>>>
> >>>>> That directory the prov stuff makes does linger and can be annoying
> so
> >>>>> thanks for tackling that. Saw the PR. Will take a look soon if
> >>>>> nobody else has.
> >>>>>
> >>>>> Will keep an eye on your findings related to docker builds not
> working
> >>>>> with username/password things. Hopefully others can chime in there.
> >>>>>
> >>>>> Thanks
> >>>>> Send
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
> >>>>> <ch...@naimuri.com.invalid> wrote:
> >>>>>>
> >>>>>> Worryingly, when I do get the Docker Image to build (further changes
> >>> to the
> >>>>>> Dockerfile), the auto-generated username and password from the
> >> startup
> >>> logs
> >>>>>> aren't being accepted for login via my browser.
> >>>>>>
> >>>>>> I'll try to spend a little more time looking at this (but await
> input
> >>> on my
> >>>>>> earlier question/concern also).
> >>>>>>
> >>>>>>
> >>>>>> ---
> >>>>>> *Chris Sampson*
> >>>>>> IT Consultant
> >>>>>> chris.sampson@naimuri.com
> >>>>>>
> >>>>>>
> >>>>>> On Thu, 4 Nov 2021 at 20:47, Chris Sampson <
> >> chris.sampson@naimuri.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I've got most of the way through the release check process in order
> >> to
> >>>>>>> vote for 1.15.0, but I wanted to check on a change in the
> >> distribution
> >>>>>>> release artifacts.
> >>>>>>>
> >>>>>>> For 1.14.0, the Dev artifacts were located at:
> >>>>>>> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
> >>>>>>> For 1.15.0, the Dev artifacts are located at:
> >>>>>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
> >>>>>>>
> >>>>>>> i.e. there's been a change of directory/path from <version> to
> >>>>>>> nifi-<version>.
> >>>>>>>
> >>>>>>> The reason I raise this is that I can no longer build a Docker
> Image
> >>> using
> >>>>>>> the dockerhub/DockerBuild.sh script because it cannot find the
> >>> artifacts to
> >>>>>>> download. This may not be a problem if the path change is only for
> >>> the Dev
> >>>>>>> artifacts, but if the same change is going to happen for the
> >> released
> >>>>>>> artifacts, then the apache/nifi image (and presumably the
> >>>>>>> apache/nifi-registry, apache/nifi-toolkit and any minifi)
> >> convenience
> >>>>>>> images will no longer be possible as part of the Release, which
> will
> >>> likely
> >>>>>>> be an issue for a number of users that have deployments using these
> >>> images.
> >>>>>>>
> >>>>>>> I spotted this after rebasing my outstanding PR [1] for NIFI-8779
> >> [2]
> >>>>>>>
> >>>>>>>
> >>>>>>> Additionally, I noted NIFI-9366 [3] after an unwanted directory was
> >>>>>>> created by the unit tests executed during building and testing
> >>> 1.15.0-RC3.
> >>>>>>> I raised a PR [4] - this is a minor issue and not a reason to block
> >>> any
> >>>>>>> release.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> [1] https://github.com/apache/nifi/pull/5213
> >>>>>>> [2] https://issues.apache.org/jira/browse/NIFI-8779
> >>>>>>>
> >>>>>>> [3] https://issues.apache.org/jira/browse/NIFI-9366
> >>>>>>> [4] https://github.com/apache/nifi/pull/5510
> >>>>>>>
> >>>>>>> ---
> >>>>>>> *Chris Sampson*
> >>>>>>> IT Consultant
> >>>>>>> chris.sampson@naimuri.com
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sat, 30 Oct 2021 at 01:52, Joe Witt <jo...@apache.org> wrote:
> >>>>>>>
> >>>>>>>> Otto
> >>>>>>>>
> >>>>>>>> Got ya. Yeah it was this way in 1.14 as well. And to be clear
> >> with
> >>> every
> >>>>>>>> release what we are actually voting upon is the source release.
> >> Now
> >>> the
> >>>>>>>> source release includes nifi, minifi, nifi registry, stateless
> nifi
> >>> and
> >>>>>>>> toolkits among all the other having always been there goodies.
> >>>>>>>>
> >>>>>>>> Some of these things we make available in the form of convenience
> >>> binaries
> >>>>>>>> to make it easier on folks to consume.
> >>>>>>>>
> >>>>>>>> I think you dont need to do any verification you would not have
> >> done
> >>>>>>>> before.
> >>>>>>>>
> >>>>>>>> But I do hope folks help maintain various ways of easing more
> folks
> >>>>>>>> knowing
> >>>>>>>> what to vet with a given release
> >>>>>>>>
> >>>>>>>> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <
> >> ottobackwards@gmail.com
> >>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> This release, we are verifying not only Nifi, but also Minifi
> >> Java.
> >>> At
> >>>>>>>>> least that is my understanding.
> >>>>>>>>>
> >>>>>>>>> This would be my first time having *anything* to do with minifi,
> >>> i’ve
> >>>>>>>> not
> >>>>>>>>> even run it before.
> >>>>>>>>>
> >>>>>>>>> As such, I think the RC validation guide needs to be update to
> >>> include
> >>>>>>>> the
> >>>>>>>>> information about now validating nifi and minifi together.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> From: Joe Witt <jo...@apache.org> <jo...@apache.org>
> >>>>>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> >>> dev@nifi.apache.org>
> >>>>>>>>> Date: October 25, 2021 at 11:59:39
> >>>>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
> >> dev@nifi.apache.org>
> >>>>>>>>> Subject: [discuss] NiFi 1.15
> >>>>>>>>>
> >>>>>>>>> Team,
> >>>>>>>>>
> >>>>>>>>> I thought I had already started such a thread but I dont see it
> so
> >>> here
> >>>>>>>>> goes...
> >>>>>>>>>
> >>>>>>>>> We have made a ton of progress again and there are some super
> >>>>>>>>> important fixes as well. It is definitely time to kick out a
> 1.15.
> >>>>>>>>> My intent will be to attempt to pull together an RC this week.
> >>>>>>>>> Haven't done the analysis yet of what is hanging out there but
> >> will
> >>> do
> >>>>>>>>> so. A quick look at all the features and fixes already landed
> >> though
> >>>>>>>>> make it clear we have more than enough to work with.
> >>>>>>>>>
> >>>>>>>>> Lets please use this thread for coordination on the RC rather
> than
> >>> it
> >>>>>>>>> becoming the wish list. We have new features/fixes arriving all
> >> the
> >>>>>>>>> time - those can be addressed in normal channels.
> >>>>>>>>>
> >>>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>>
> >>
>
>
Re: [discuss] NiFi 1.15
Posted by Kevin Doran <kd...@apache.org>.
Hi Chris,
Thanks for the summary of your findings.
I agree that we should clear up our process for generating Docker images and that the process should be consistent as possible across NiFi, MiNiFi Java, Registry, and Toolkit given they are all part of the same repository and maven project. This will clear up confusion and improve maintainability.
I think both of these methods are important and useful:
1. building from downloaded convenience binaries for previously released versions
2. building from a local assembly output
That said, I think we could probably consolidate to a single Dockerfile that takes the binary location as an argument that supports either a URL or local file path (or a version which could default to the current behavior of inferring a URL location). This would let us continue to support the flexibility we have today while maintaining a single Docker file rather than dockermaven and dockerhub variants. If you and others on the list agree, I can open up a Jira summarizing what needs to be done.
Thanks,
Kevin
> On Nov 5, 2021, at 05:00, Chris Sampson <ch...@naimuri.com.INVALID> wrote:
>
> So the good news is that I realised what I was doing wrong yesterday - I'd
> started a local installation after building the RC, then not shut that down
> before booting up the Docker instance, so the username/password I was
> trying to use from the Docker logs was wrong against the local
> installation, d'oh!
>
> Having corrected that this morning, I've successfully booted up and logged
> into a Docker instance built using the RC (with my NIFI-8779 changes so I
> could build from the Dev server instead of the main Apache Archive).
>
> The reason the dockermaven build works is that it uses the locally built
> archive files (i.e. you have to do a full Maven build locally first to
> generate the ZIP files and then create teh Docker image). The
> dockerhub build actually downloads published artifacts from the Apache
> servers - to do this the Dockerfile needs to know the exact path to the
> artifacts it's trying to download, of course.
>
> There was a question recently in one of the Slack channels about whether
> both of these builds (dockerhub and dockermaven) were still valid/needed -
> I think Joe replied that he wasn't sure (Docker not being an area he
> ventures into much, which is fair enough). It may be worth considering
> whether these two modules are both still needed or can be rationalised and
> if the latter, which approach should be used - download from the archive
> server or build from local (both have dis/advantages, but the more
> "official" way is arguably to download from the server).
>
> Also maybe worth noting:
> * NiFi Registry only has the "build from local" Dockerfile setup
> * MiniFi (Java) has both local and archive download options, but the latter
> cannot be used to build an image from the Dev server (the BASE_URL cannot
> be changed via a build arg/env var... this is the issue addressed by
> NIFI-8779 for NiFi)
> * NiFi Toolkit has both local and archive download options, but are located
> in a single assembly module (instead of split into two like NiFi and MiniFi)
>
> Assuming the "nifi/<version>" path will be used for the actual release
> artifacts, this shouldn't be a blocker, but it's one to note/remember when
> the time comes (assuming the "dockerhub" module is what's used to publish
> the "apache/nifi" image to Docker Hub that is).
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.sampson@naimuri.com
>
>
> On Fri, 5 Nov 2021 at 03:34, David Handermann <ex...@apache.org>
> wrote:
>
>> Chris,
>>
>> I am running through several verification steps, but I built 1.15.0 RC 3
>> from sources and then ran "mvn install -pl :dockermaven -P docker" to build
>> the Docker image. When starting with the following command, I was able to
>> use the generated username and password to access the running NiFi
>> container:
>>
>> docker run --name nifi -p 8443:8443 apache/nifi:1.15.0-dockermaven
>>
>> If you are still having issues, it would be helpful to provide any logs or
>> error messages you are seeing.
>>
>> Regards,
>> David Handermann
>>
>> On Thu, Nov 4, 2021 at 8:10 PM Kevin Doran <kd...@gmail.com>
>> wrote:
>>
>>> Oh meant to send a link to this too:
>>>
>>>> ARG
>>>
>> NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
>>>
>>>
>>>
>> https://github.com/apache/nifi/blob/373498445fe589e2d4855a0730fbb9127f0b4452/nifi-docker/dockerhub/Dockerfile#L30
>>>
>>>> On Nov 4, 2021, at 9:09 PM, Kevin Doran <kd...@gmail.com>
>> wrote:
>>>>
>>>> Hi Joe and Chris,
>>>>
>>>> Our Dockerfile that we use to build the Dockerhub image defaults to
>>> looking for 1.15.0 instead of NiFi-1.15.0, but it is a variable that we
>> can
>>> override, so this is easy to workaround incase the release folder does
>>> change. Agree its nice to keep the tree structure consistent when we can.
>>>>
>>>> I’m about to do my verification and will also check the single user
>> with
>>> the docker image as part of that.
>>>>
>>>> Cheers,
>>>> Kevin
>>>>
>>>>> On Nov 4, 2021, at 6:44 PM, Joe Witt <jo...@gmail.com> wrote:
>>>>>
>>>>> Chris,
>>>>>
>>>>> Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
>>>>> Should generally not really matter so the docker angle there is
>>>>> interesting. Not sure why our docker images would have any
>>>>> relationship to our dist/dev storage. But when I move them into the
>>>>> release folder I can try to ensure I place them only in 1.15.0 instead
>>>>> of nifi/1.15.0.
>>>>>
>>>>> That directory the prov stuff makes does linger and can be annoying so
>>>>> thanks for tackling that. Saw the PR. Will take a look soon if
>>>>> nobody else has.
>>>>>
>>>>> Will keep an eye on your findings related to docker builds not working
>>>>> with username/password things. Hopefully others can chime in there.
>>>>>
>>>>> Thanks
>>>>> Send
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
>>>>> <ch...@naimuri.com.invalid> wrote:
>>>>>>
>>>>>> Worryingly, when I do get the Docker Image to build (further changes
>>> to the
>>>>>> Dockerfile), the auto-generated username and password from the
>> startup
>>> logs
>>>>>> aren't being accepted for login via my browser.
>>>>>>
>>>>>> I'll try to spend a little more time looking at this (but await input
>>> on my
>>>>>> earlier question/concern also).
>>>>>>
>>>>>>
>>>>>> ---
>>>>>> *Chris Sampson*
>>>>>> IT Consultant
>>>>>> chris.sampson@naimuri.com
>>>>>>
>>>>>>
>>>>>> On Thu, 4 Nov 2021 at 20:47, Chris Sampson <
>> chris.sampson@naimuri.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I've got most of the way through the release check process in order
>> to
>>>>>>> vote for 1.15.0, but I wanted to check on a change in the
>> distribution
>>>>>>> release artifacts.
>>>>>>>
>>>>>>> For 1.14.0, the Dev artifacts were located at:
>>>>>>> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
>>>>>>> For 1.15.0, the Dev artifacts are located at:
>>>>>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
>>>>>>>
>>>>>>> i.e. there's been a change of directory/path from <version> to
>>>>>>> nifi-<version>.
>>>>>>>
>>>>>>> The reason I raise this is that I can no longer build a Docker Image
>>> using
>>>>>>> the dockerhub/DockerBuild.sh script because it cannot find the
>>> artifacts to
>>>>>>> download. This may not be a problem if the path change is only for
>>> the Dev
>>>>>>> artifacts, but if the same change is going to happen for the
>> released
>>>>>>> artifacts, then the apache/nifi image (and presumably the
>>>>>>> apache/nifi-registry, apache/nifi-toolkit and any minifi)
>> convenience
>>>>>>> images will no longer be possible as part of the Release, which will
>>> likely
>>>>>>> be an issue for a number of users that have deployments using these
>>> images.
>>>>>>>
>>>>>>> I spotted this after rebasing my outstanding PR [1] for NIFI-8779
>> [2]
>>>>>>>
>>>>>>>
>>>>>>> Additionally, I noted NIFI-9366 [3] after an unwanted directory was
>>>>>>> created by the unit tests executed during building and testing
>>> 1.15.0-RC3.
>>>>>>> I raised a PR [4] - this is a minor issue and not a reason to block
>>> any
>>>>>>> release.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [1] https://github.com/apache/nifi/pull/5213
>>>>>>> [2] https://issues.apache.org/jira/browse/NIFI-8779
>>>>>>>
>>>>>>> [3] https://issues.apache.org/jira/browse/NIFI-9366
>>>>>>> [4] https://github.com/apache/nifi/pull/5510
>>>>>>>
>>>>>>> ---
>>>>>>> *Chris Sampson*
>>>>>>> IT Consultant
>>>>>>> chris.sampson@naimuri.com
>>>>>>>
>>>>>>>
>>>>>>> On Sat, 30 Oct 2021 at 01:52, Joe Witt <jo...@apache.org> wrote:
>>>>>>>
>>>>>>>> Otto
>>>>>>>>
>>>>>>>> Got ya. Yeah it was this way in 1.14 as well. And to be clear
>> with
>>> every
>>>>>>>> release what we are actually voting upon is the source release.
>> Now
>>> the
>>>>>>>> source release includes nifi, minifi, nifi registry, stateless nifi
>>> and
>>>>>>>> toolkits among all the other having always been there goodies.
>>>>>>>>
>>>>>>>> Some of these things we make available in the form of convenience
>>> binaries
>>>>>>>> to make it easier on folks to consume.
>>>>>>>>
>>>>>>>> I think you dont need to do any verification you would not have
>> done
>>>>>>>> before.
>>>>>>>>
>>>>>>>> But I do hope folks help maintain various ways of easing more folks
>>>>>>>> knowing
>>>>>>>> what to vet with a given release
>>>>>>>>
>>>>>>>> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <
>> ottobackwards@gmail.com
>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> This release, we are verifying not only Nifi, but also Minifi
>> Java.
>>> At
>>>>>>>>> least that is my understanding.
>>>>>>>>>
>>>>>>>>> This would be my first time having *anything* to do with minifi,
>>> i’ve
>>>>>>>> not
>>>>>>>>> even run it before.
>>>>>>>>>
>>>>>>>>> As such, I think the RC validation guide needs to be update to
>>> include
>>>>>>>> the
>>>>>>>>> information about now validating nifi and minifi together.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> From: Joe Witt <jo...@apache.org> <jo...@apache.org>
>>>>>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
>>> dev@nifi.apache.org>
>>>>>>>>> Date: October 25, 2021 at 11:59:39
>>>>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
>> dev@nifi.apache.org>
>>>>>>>>> Subject: [discuss] NiFi 1.15
>>>>>>>>>
>>>>>>>>> Team,
>>>>>>>>>
>>>>>>>>> I thought I had already started such a thread but I dont see it so
>>> here
>>>>>>>>> goes...
>>>>>>>>>
>>>>>>>>> We have made a ton of progress again and there are some super
>>>>>>>>> important fixes as well. It is definitely time to kick out a 1.15.
>>>>>>>>> My intent will be to attempt to pull together an RC this week.
>>>>>>>>> Haven't done the analysis yet of what is hanging out there but
>> will
>>> do
>>>>>>>>> so. A quick look at all the features and fixes already landed
>> though
>>>>>>>>> make it clear we have more than enough to work with.
>>>>>>>>>
>>>>>>>>> Lets please use this thread for coordination on the RC rather than
>>> it
>>>>>>>>> becoming the wish list. We have new features/fixes arriving all
>> the
>>>>>>>>> time - those can be addressed in normal channels.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>>>
>>>
>>
Re: [discuss] NiFi 1.15
Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
So the good news is that I realised what I was doing wrong yesterday - I'd
started a local installation after building the RC, then not shut that down
before booting up the Docker instance, so the username/password I was
trying to use from the Docker logs was wrong against the local
installation, d'oh!
Having corrected that this morning, I've successfully booted up and logged
into a Docker instance built using the RC (with my NIFI-8779 changes so I
could build from the Dev server instead of the main Apache Archive).
The reason the dockermaven build works is that it uses the locally built
archive files (i.e. you have to do a full Maven build locally first to
generate the ZIP files and then create teh Docker image). The
dockerhub build actually downloads published artifacts from the Apache
servers - to do this the Dockerfile needs to know the exact path to the
artifacts it's trying to download, of course.
There was a question recently in one of the Slack channels about whether
both of these builds (dockerhub and dockermaven) were still valid/needed -
I think Joe replied that he wasn't sure (Docker not being an area he
ventures into much, which is fair enough). It may be worth considering
whether these two modules are both still needed or can be rationalised and
if the latter, which approach should be used - download from the archive
server or build from local (both have dis/advantages, but the more
"official" way is arguably to download from the server).
Also maybe worth noting:
* NiFi Registry only has the "build from local" Dockerfile setup
* MiniFi (Java) has both local and archive download options, but the latter
cannot be used to build an image from the Dev server (the BASE_URL cannot
be changed via a build arg/env var... this is the issue addressed by
NIFI-8779 for NiFi)
* NiFi Toolkit has both local and archive download options, but are located
in a single assembly module (instead of split into two like NiFi and MiniFi)
Assuming the "nifi/<version>" path will be used for the actual release
artifacts, this shouldn't be a blocker, but it's one to note/remember when
the time comes (assuming the "dockerhub" module is what's used to publish
the "apache/nifi" image to Docker Hub that is).
---
*Chris Sampson*
IT Consultant
chris.sampson@naimuri.com
On Fri, 5 Nov 2021 at 03:34, David Handermann <ex...@apache.org>
wrote:
> Chris,
>
> I am running through several verification steps, but I built 1.15.0 RC 3
> from sources and then ran "mvn install -pl :dockermaven -P docker" to build
> the Docker image. When starting with the following command, I was able to
> use the generated username and password to access the running NiFi
> container:
>
> docker run --name nifi -p 8443:8443 apache/nifi:1.15.0-dockermaven
>
> If you are still having issues, it would be helpful to provide any logs or
> error messages you are seeing.
>
> Regards,
> David Handermann
>
> On Thu, Nov 4, 2021 at 8:10 PM Kevin Doran <kd...@gmail.com>
> wrote:
>
> > Oh meant to send a link to this too:
> >
> > > ARG
> >
> NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
> >
> >
> >
> https://github.com/apache/nifi/blob/373498445fe589e2d4855a0730fbb9127f0b4452/nifi-docker/dockerhub/Dockerfile#L30
> >
> > > On Nov 4, 2021, at 9:09 PM, Kevin Doran <kd...@gmail.com>
> wrote:
> > >
> > > Hi Joe and Chris,
> > >
> > > Our Dockerfile that we use to build the Dockerhub image defaults to
> > looking for 1.15.0 instead of NiFi-1.15.0, but it is a variable that we
> can
> > override, so this is easy to workaround incase the release folder does
> > change. Agree its nice to keep the tree structure consistent when we can.
> > >
> > > I’m about to do my verification and will also check the single user
> with
> > the docker image as part of that.
> > >
> > > Cheers,
> > > Kevin
> > >
> > >> On Nov 4, 2021, at 6:44 PM, Joe Witt <jo...@gmail.com> wrote:
> > >>
> > >> Chris,
> > >>
> > >> Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
> > >> Should generally not really matter so the docker angle there is
> > >> interesting. Not sure why our docker images would have any
> > >> relationship to our dist/dev storage. But when I move them into the
> > >> release folder I can try to ensure I place them only in 1.15.0 instead
> > >> of nifi/1.15.0.
> > >>
> > >> That directory the prov stuff makes does linger and can be annoying so
> > >> thanks for tackling that. Saw the PR. Will take a look soon if
> > >> nobody else has.
> > >>
> > >> Will keep an eye on your findings related to docker builds not working
> > >> with username/password things. Hopefully others can chime in there.
> > >>
> > >> Thanks
> > >> Send
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
> > >> <ch...@naimuri.com.invalid> wrote:
> > >>>
> > >>> Worryingly, when I do get the Docker Image to build (further changes
> > to the
> > >>> Dockerfile), the auto-generated username and password from the
> startup
> > logs
> > >>> aren't being accepted for login via my browser.
> > >>>
> > >>> I'll try to spend a little more time looking at this (but await input
> > on my
> > >>> earlier question/concern also).
> > >>>
> > >>>
> > >>> ---
> > >>> *Chris Sampson*
> > >>> IT Consultant
> > >>> chris.sampson@naimuri.com
> > >>>
> > >>>
> > >>> On Thu, 4 Nov 2021 at 20:47, Chris Sampson <
> chris.sampson@naimuri.com>
> > >>> wrote:
> > >>>
> > >>>> I've got most of the way through the release check process in order
> to
> > >>>> vote for 1.15.0, but I wanted to check on a change in the
> distribution
> > >>>> release artifacts.
> > >>>>
> > >>>> For 1.14.0, the Dev artifacts were located at:
> > >>>> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
> > >>>> For 1.15.0, the Dev artifacts are located at:
> > >>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
> > >>>>
> > >>>> i.e. there's been a change of directory/path from <version> to
> > >>>> nifi-<version>.
> > >>>>
> > >>>> The reason I raise this is that I can no longer build a Docker Image
> > using
> > >>>> the dockerhub/DockerBuild.sh script because it cannot find the
> > artifacts to
> > >>>> download. This may not be a problem if the path change is only for
> > the Dev
> > >>>> artifacts, but if the same change is going to happen for the
> released
> > >>>> artifacts, then the apache/nifi image (and presumably the
> > >>>> apache/nifi-registry, apache/nifi-toolkit and any minifi)
> convenience
> > >>>> images will no longer be possible as part of the Release, which will
> > likely
> > >>>> be an issue for a number of users that have deployments using these
> > images.
> > >>>>
> > >>>> I spotted this after rebasing my outstanding PR [1] for NIFI-8779
> [2]
> > >>>>
> > >>>>
> > >>>> Additionally, I noted NIFI-9366 [3] after an unwanted directory was
> > >>>> created by the unit tests executed during building and testing
> > 1.15.0-RC3.
> > >>>> I raised a PR [4] - this is a minor issue and not a reason to block
> > any
> > >>>> release.
> > >>>>
> > >>>>
> > >>>>
> > >>>> [1] https://github.com/apache/nifi/pull/5213
> > >>>> [2] https://issues.apache.org/jira/browse/NIFI-8779
> > >>>>
> > >>>> [3] https://issues.apache.org/jira/browse/NIFI-9366
> > >>>> [4] https://github.com/apache/nifi/pull/5510
> > >>>>
> > >>>> ---
> > >>>> *Chris Sampson*
> > >>>> IT Consultant
> > >>>> chris.sampson@naimuri.com
> > >>>>
> > >>>>
> > >>>> On Sat, 30 Oct 2021 at 01:52, Joe Witt <jo...@apache.org> wrote:
> > >>>>
> > >>>>> Otto
> > >>>>>
> > >>>>> Got ya. Yeah it was this way in 1.14 as well. And to be clear
> with
> > every
> > >>>>> release what we are actually voting upon is the source release.
> Now
> > the
> > >>>>> source release includes nifi, minifi, nifi registry, stateless nifi
> > and
> > >>>>> toolkits among all the other having always been there goodies.
> > >>>>>
> > >>>>> Some of these things we make available in the form of convenience
> > binaries
> > >>>>> to make it easier on folks to consume.
> > >>>>>
> > >>>>> I think you dont need to do any verification you would not have
> done
> > >>>>> before.
> > >>>>>
> > >>>>> But I do hope folks help maintain various ways of easing more folks
> > >>>>> knowing
> > >>>>> what to vet with a given release
> > >>>>>
> > >>>>> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <
> ottobackwards@gmail.com
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>>> This release, we are verifying not only Nifi, but also Minifi
> Java.
> > At
> > >>>>>> least that is my understanding.
> > >>>>>>
> > >>>>>> This would be my first time having *anything* to do with minifi,
> > i’ve
> > >>>>> not
> > >>>>>> even run it before.
> > >>>>>>
> > >>>>>> As such, I think the RC validation guide needs to be update to
> > include
> > >>>>> the
> > >>>>>> information about now validating nifi and minifi together.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> From: Joe Witt <jo...@apache.org> <jo...@apache.org>
> > >>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> > dev@nifi.apache.org>
> > >>>>>> Date: October 25, 2021 at 11:59:39
> > >>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <
> dev@nifi.apache.org>
> > >>>>>> Subject: [discuss] NiFi 1.15
> > >>>>>>
> > >>>>>> Team,
> > >>>>>>
> > >>>>>> I thought I had already started such a thread but I dont see it so
> > here
> > >>>>>> goes...
> > >>>>>>
> > >>>>>> We have made a ton of progress again and there are some super
> > >>>>>> important fixes as well. It is definitely time to kick out a 1.15.
> > >>>>>> My intent will be to attempt to pull together an RC this week.
> > >>>>>> Haven't done the analysis yet of what is hanging out there but
> will
> > do
> > >>>>>> so. A quick look at all the features and fixes already landed
> though
> > >>>>>> make it clear we have more than enough to work with.
> > >>>>>>
> > >>>>>> Lets please use this thread for coordination on the RC rather than
> > it
> > >>>>>> becoming the wish list. We have new features/fixes arriving all
> the
> > >>>>>> time - those can be addressed in normal channels.
> > >>>>>>
> > >>>>>> Thanks
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >
> >
> >
>
Re: [discuss] NiFi 1.15
Posted by David Handermann <ex...@apache.org>.
Chris,
I am running through several verification steps, but I built 1.15.0 RC 3
from sources and then ran "mvn install -pl :dockermaven -P docker" to build
the Docker image. When starting with the following command, I was able to
use the generated username and password to access the running NiFi
container:
docker run --name nifi -p 8443:8443 apache/nifi:1.15.0-dockermaven
If you are still having issues, it would be helpful to provide any logs or
error messages you are seeing.
Regards,
David Handermann
On Thu, Nov 4, 2021 at 8:10 PM Kevin Doran <kd...@gmail.com> wrote:
> Oh meant to send a link to this too:
>
> > ARG
> NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
>
>
> https://github.com/apache/nifi/blob/373498445fe589e2d4855a0730fbb9127f0b4452/nifi-docker/dockerhub/Dockerfile#L30
>
> > On Nov 4, 2021, at 9:09 PM, Kevin Doran <kd...@gmail.com> wrote:
> >
> > Hi Joe and Chris,
> >
> > Our Dockerfile that we use to build the Dockerhub image defaults to
> looking for 1.15.0 instead of NiFi-1.15.0, but it is a variable that we can
> override, so this is easy to workaround incase the release folder does
> change. Agree its nice to keep the tree structure consistent when we can.
> >
> > I’m about to do my verification and will also check the single user with
> the docker image as part of that.
> >
> > Cheers,
> > Kevin
> >
> >> On Nov 4, 2021, at 6:44 PM, Joe Witt <jo...@gmail.com> wrote:
> >>
> >> Chris,
> >>
> >> Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
> >> Should generally not really matter so the docker angle there is
> >> interesting. Not sure why our docker images would have any
> >> relationship to our dist/dev storage. But when I move them into the
> >> release folder I can try to ensure I place them only in 1.15.0 instead
> >> of nifi/1.15.0.
> >>
> >> That directory the prov stuff makes does linger and can be annoying so
> >> thanks for tackling that. Saw the PR. Will take a look soon if
> >> nobody else has.
> >>
> >> Will keep an eye on your findings related to docker builds not working
> >> with username/password things. Hopefully others can chime in there.
> >>
> >> Thanks
> >> Send
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
> >> <ch...@naimuri.com.invalid> wrote:
> >>>
> >>> Worryingly, when I do get the Docker Image to build (further changes
> to the
> >>> Dockerfile), the auto-generated username and password from the startup
> logs
> >>> aren't being accepted for login via my browser.
> >>>
> >>> I'll try to spend a little more time looking at this (but await input
> on my
> >>> earlier question/concern also).
> >>>
> >>>
> >>> ---
> >>> *Chris Sampson*
> >>> IT Consultant
> >>> chris.sampson@naimuri.com
> >>>
> >>>
> >>> On Thu, 4 Nov 2021 at 20:47, Chris Sampson <ch...@naimuri.com>
> >>> wrote:
> >>>
> >>>> I've got most of the way through the release check process in order to
> >>>> vote for 1.15.0, but I wanted to check on a change in the distribution
> >>>> release artifacts.
> >>>>
> >>>> For 1.14.0, the Dev artifacts were located at:
> >>>> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
> >>>> For 1.15.0, the Dev artifacts are located at:
> >>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
> >>>>
> >>>> i.e. there's been a change of directory/path from <version> to
> >>>> nifi-<version>.
> >>>>
> >>>> The reason I raise this is that I can no longer build a Docker Image
> using
> >>>> the dockerhub/DockerBuild.sh script because it cannot find the
> artifacts to
> >>>> download. This may not be a problem if the path change is only for
> the Dev
> >>>> artifacts, but if the same change is going to happen for the released
> >>>> artifacts, then the apache/nifi image (and presumably the
> >>>> apache/nifi-registry, apache/nifi-toolkit and any minifi) convenience
> >>>> images will no longer be possible as part of the Release, which will
> likely
> >>>> be an issue for a number of users that have deployments using these
> images.
> >>>>
> >>>> I spotted this after rebasing my outstanding PR [1] for NIFI-8779 [2]
> >>>>
> >>>>
> >>>> Additionally, I noted NIFI-9366 [3] after an unwanted directory was
> >>>> created by the unit tests executed during building and testing
> 1.15.0-RC3.
> >>>> I raised a PR [4] - this is a minor issue and not a reason to block
> any
> >>>> release.
> >>>>
> >>>>
> >>>>
> >>>> [1] https://github.com/apache/nifi/pull/5213
> >>>> [2] https://issues.apache.org/jira/browse/NIFI-8779
> >>>>
> >>>> [3] https://issues.apache.org/jira/browse/NIFI-9366
> >>>> [4] https://github.com/apache/nifi/pull/5510
> >>>>
> >>>> ---
> >>>> *Chris Sampson*
> >>>> IT Consultant
> >>>> chris.sampson@naimuri.com
> >>>>
> >>>>
> >>>> On Sat, 30 Oct 2021 at 01:52, Joe Witt <jo...@apache.org> wrote:
> >>>>
> >>>>> Otto
> >>>>>
> >>>>> Got ya. Yeah it was this way in 1.14 as well. And to be clear with
> every
> >>>>> release what we are actually voting upon is the source release. Now
> the
> >>>>> source release includes nifi, minifi, nifi registry, stateless nifi
> and
> >>>>> toolkits among all the other having always been there goodies.
> >>>>>
> >>>>> Some of these things we make available in the form of convenience
> binaries
> >>>>> to make it easier on folks to consume.
> >>>>>
> >>>>> I think you dont need to do any verification you would not have done
> >>>>> before.
> >>>>>
> >>>>> But I do hope folks help maintain various ways of easing more folks
> >>>>> knowing
> >>>>> what to vet with a given release
> >>>>>
> >>>>> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <ottobackwards@gmail.com
> >
> >>>>> wrote:
> >>>>>
> >>>>>> This release, we are verifying not only Nifi, but also Minifi Java.
> At
> >>>>>> least that is my understanding.
> >>>>>>
> >>>>>> This would be my first time having *anything* to do with minifi,
> i’ve
> >>>>> not
> >>>>>> even run it before.
> >>>>>>
> >>>>>> As such, I think the RC validation guide needs to be update to
> include
> >>>>> the
> >>>>>> information about now validating nifi and minifi together.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> From: Joe Witt <jo...@apache.org> <jo...@apache.org>
> >>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <
> dev@nifi.apache.org>
> >>>>>> Date: October 25, 2021 at 11:59:39
> >>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> >>>>>> Subject: [discuss] NiFi 1.15
> >>>>>>
> >>>>>> Team,
> >>>>>>
> >>>>>> I thought I had already started such a thread but I dont see it so
> here
> >>>>>> goes...
> >>>>>>
> >>>>>> We have made a ton of progress again and there are some super
> >>>>>> important fixes as well. It is definitely time to kick out a 1.15.
> >>>>>> My intent will be to attempt to pull together an RC this week.
> >>>>>> Haven't done the analysis yet of what is hanging out there but will
> do
> >>>>>> so. A quick look at all the features and fixes already landed though
> >>>>>> make it clear we have more than enough to work with.
> >>>>>>
> >>>>>> Lets please use this thread for coordination on the RC rather than
> it
> >>>>>> becoming the wish list. We have new features/fixes arriving all the
> >>>>>> time - those can be addressed in normal channels.
> >>>>>>
> >>>>>> Thanks
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >
>
>
Re: [discuss] NiFi 1.15
Posted by Kevin Doran <kd...@gmail.com>.
Oh meant to send a link to this too:
> ARG NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
https://github.com/apache/nifi/blob/373498445fe589e2d4855a0730fbb9127f0b4452/nifi-docker/dockerhub/Dockerfile#L30
> On Nov 4, 2021, at 9:09 PM, Kevin Doran <kd...@gmail.com> wrote:
>
> Hi Joe and Chris,
>
> Our Dockerfile that we use to build the Dockerhub image defaults to looking for 1.15.0 instead of NiFi-1.15.0, but it is a variable that we can override, so this is easy to workaround incase the release folder does change. Agree its nice to keep the tree structure consistent when we can.
>
> I’m about to do my verification and will also check the single user with the docker image as part of that.
>
> Cheers,
> Kevin
>
>> On Nov 4, 2021, at 6:44 PM, Joe Witt <jo...@gmail.com> wrote:
>>
>> Chris,
>>
>> Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
>> Should generally not really matter so the docker angle there is
>> interesting. Not sure why our docker images would have any
>> relationship to our dist/dev storage. But when I move them into the
>> release folder I can try to ensure I place them only in 1.15.0 instead
>> of nifi/1.15.0.
>>
>> That directory the prov stuff makes does linger and can be annoying so
>> thanks for tackling that. Saw the PR. Will take a look soon if
>> nobody else has.
>>
>> Will keep an eye on your findings related to docker builds not working
>> with username/password things. Hopefully others can chime in there.
>>
>> Thanks
>> Send
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
>> <ch...@naimuri.com.invalid> wrote:
>>>
>>> Worryingly, when I do get the Docker Image to build (further changes to the
>>> Dockerfile), the auto-generated username and password from the startup logs
>>> aren't being accepted for login via my browser.
>>>
>>> I'll try to spend a little more time looking at this (but await input on my
>>> earlier question/concern also).
>>>
>>>
>>> ---
>>> *Chris Sampson*
>>> IT Consultant
>>> chris.sampson@naimuri.com
>>>
>>>
>>> On Thu, 4 Nov 2021 at 20:47, Chris Sampson <ch...@naimuri.com>
>>> wrote:
>>>
>>>> I've got most of the way through the release check process in order to
>>>> vote for 1.15.0, but I wanted to check on a change in the distribution
>>>> release artifacts.
>>>>
>>>> For 1.14.0, the Dev artifacts were located at:
>>>> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
>>>> For 1.15.0, the Dev artifacts are located at:
>>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
>>>>
>>>> i.e. there's been a change of directory/path from <version> to
>>>> nifi-<version>.
>>>>
>>>> The reason I raise this is that I can no longer build a Docker Image using
>>>> the dockerhub/DockerBuild.sh script because it cannot find the artifacts to
>>>> download. This may not be a problem if the path change is only for the Dev
>>>> artifacts, but if the same change is going to happen for the released
>>>> artifacts, then the apache/nifi image (and presumably the
>>>> apache/nifi-registry, apache/nifi-toolkit and any minifi) convenience
>>>> images will no longer be possible as part of the Release, which will likely
>>>> be an issue for a number of users that have deployments using these images.
>>>>
>>>> I spotted this after rebasing my outstanding PR [1] for NIFI-8779 [2]
>>>>
>>>>
>>>> Additionally, I noted NIFI-9366 [3] after an unwanted directory was
>>>> created by the unit tests executed during building and testing 1.15.0-RC3.
>>>> I raised a PR [4] - this is a minor issue and not a reason to block any
>>>> release.
>>>>
>>>>
>>>>
>>>> [1] https://github.com/apache/nifi/pull/5213
>>>> [2] https://issues.apache.org/jira/browse/NIFI-8779
>>>>
>>>> [3] https://issues.apache.org/jira/browse/NIFI-9366
>>>> [4] https://github.com/apache/nifi/pull/5510
>>>>
>>>> ---
>>>> *Chris Sampson*
>>>> IT Consultant
>>>> chris.sampson@naimuri.com
>>>>
>>>>
>>>> On Sat, 30 Oct 2021 at 01:52, Joe Witt <jo...@apache.org> wrote:
>>>>
>>>>> Otto
>>>>>
>>>>> Got ya. Yeah it was this way in 1.14 as well. And to be clear with every
>>>>> release what we are actually voting upon is the source release. Now the
>>>>> source release includes nifi, minifi, nifi registry, stateless nifi and
>>>>> toolkits among all the other having always been there goodies.
>>>>>
>>>>> Some of these things we make available in the form of convenience binaries
>>>>> to make it easier on folks to consume.
>>>>>
>>>>> I think you dont need to do any verification you would not have done
>>>>> before.
>>>>>
>>>>> But I do hope folks help maintain various ways of easing more folks
>>>>> knowing
>>>>> what to vet with a given release
>>>>>
>>>>> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <ot...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> This release, we are verifying not only Nifi, but also Minifi Java. At
>>>>>> least that is my understanding.
>>>>>>
>>>>>> This would be my first time having *anything* to do with minifi, i’ve
>>>>> not
>>>>>> even run it before.
>>>>>>
>>>>>> As such, I think the RC validation guide needs to be update to include
>>>>> the
>>>>>> information about now validating nifi and minifi together.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From: Joe Witt <jo...@apache.org> <jo...@apache.org>
>>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
>>>>>> Date: October 25, 2021 at 11:59:39
>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
>>>>>> Subject: [discuss] NiFi 1.15
>>>>>>
>>>>>> Team,
>>>>>>
>>>>>> I thought I had already started such a thread but I dont see it so here
>>>>>> goes...
>>>>>>
>>>>>> We have made a ton of progress again and there are some super
>>>>>> important fixes as well. It is definitely time to kick out a 1.15.
>>>>>> My intent will be to attempt to pull together an RC this week.
>>>>>> Haven't done the analysis yet of what is hanging out there but will do
>>>>>> so. A quick look at all the features and fixes already landed though
>>>>>> make it clear we have more than enough to work with.
>>>>>>
>>>>>> Lets please use this thread for coordination on the RC rather than it
>>>>>> becoming the wish list. We have new features/fixes arriving all the
>>>>>> time - those can be addressed in normal channels.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>
>>>>
>
Re: [discuss] NiFi 1.15
Posted by Kevin Doran <kd...@gmail.com>.
Hi Joe and Chris,
Our Dockerfile that we use to build the Dockerhub image defaults to looking for 1.15.0 instead of NiFi-1.15.0, but it is a variable that we can override, so this is easy to workaround incase the release folder does change. Agree its nice to keep the tree structure consistent when we can.
I’m about to do my verification and will also check the single user with the docker image as part of that.
Cheers,
Kevin
> On Nov 4, 2021, at 6:44 PM, Joe Witt <jo...@gmail.com> wrote:
>
> Chris,
>
> Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
> Should generally not really matter so the docker angle there is
> interesting. Not sure why our docker images would have any
> relationship to our dist/dev storage. But when I move them into the
> release folder I can try to ensure I place them only in 1.15.0 instead
> of nifi/1.15.0.
>
> That directory the prov stuff makes does linger and can be annoying so
> thanks for tackling that. Saw the PR. Will take a look soon if
> nobody else has.
>
> Will keep an eye on your findings related to docker builds not working
> with username/password things. Hopefully others can chime in there.
>
> Thanks
> Send
>
>
>
>
>
>
>
>
>
>
> On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
> <ch...@naimuri.com.invalid> wrote:
>>
>> Worryingly, when I do get the Docker Image to build (further changes to the
>> Dockerfile), the auto-generated username and password from the startup logs
>> aren't being accepted for login via my browser.
>>
>> I'll try to spend a little more time looking at this (but await input on my
>> earlier question/concern also).
>>
>>
>> ---
>> *Chris Sampson*
>> IT Consultant
>> chris.sampson@naimuri.com
>>
>>
>> On Thu, 4 Nov 2021 at 20:47, Chris Sampson <ch...@naimuri.com>
>> wrote:
>>
>>> I've got most of the way through the release check process in order to
>>> vote for 1.15.0, but I wanted to check on a change in the distribution
>>> release artifacts.
>>>
>>> For 1.14.0, the Dev artifacts were located at:
>>> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
>>> For 1.15.0, the Dev artifacts are located at:
>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
>>>
>>> i.e. there's been a change of directory/path from <version> to
>>> nifi-<version>.
>>>
>>> The reason I raise this is that I can no longer build a Docker Image using
>>> the dockerhub/DockerBuild.sh script because it cannot find the artifacts to
>>> download. This may not be a problem if the path change is only for the Dev
>>> artifacts, but if the same change is going to happen for the released
>>> artifacts, then the apache/nifi image (and presumably the
>>> apache/nifi-registry, apache/nifi-toolkit and any minifi) convenience
>>> images will no longer be possible as part of the Release, which will likely
>>> be an issue for a number of users that have deployments using these images.
>>>
>>> I spotted this after rebasing my outstanding PR [1] for NIFI-8779 [2]
>>>
>>>
>>> Additionally, I noted NIFI-9366 [3] after an unwanted directory was
>>> created by the unit tests executed during building and testing 1.15.0-RC3.
>>> I raised a PR [4] - this is a minor issue and not a reason to block any
>>> release.
>>>
>>>
>>>
>>> [1] https://github.com/apache/nifi/pull/5213
>>> [2] https://issues.apache.org/jira/browse/NIFI-8779
>>>
>>> [3] https://issues.apache.org/jira/browse/NIFI-9366
>>> [4] https://github.com/apache/nifi/pull/5510
>>>
>>> ---
>>> *Chris Sampson*
>>> IT Consultant
>>> chris.sampson@naimuri.com
>>>
>>>
>>> On Sat, 30 Oct 2021 at 01:52, Joe Witt <jo...@apache.org> wrote:
>>>
>>>> Otto
>>>>
>>>> Got ya. Yeah it was this way in 1.14 as well. And to be clear with every
>>>> release what we are actually voting upon is the source release. Now the
>>>> source release includes nifi, minifi, nifi registry, stateless nifi and
>>>> toolkits among all the other having always been there goodies.
>>>>
>>>> Some of these things we make available in the form of convenience binaries
>>>> to make it easier on folks to consume.
>>>>
>>>> I think you dont need to do any verification you would not have done
>>>> before.
>>>>
>>>> But I do hope folks help maintain various ways of easing more folks
>>>> knowing
>>>> what to vet with a given release
>>>>
>>>> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <ot...@gmail.com>
>>>> wrote:
>>>>
>>>>> This release, we are verifying not only Nifi, but also Minifi Java. At
>>>>> least that is my understanding.
>>>>>
>>>>> This would be my first time having *anything* to do with minifi, i’ve
>>>> not
>>>>> even run it before.
>>>>>
>>>>> As such, I think the RC validation guide needs to be update to include
>>>> the
>>>>> information about now validating nifi and minifi together.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Joe Witt <jo...@apache.org> <jo...@apache.org>
>>>>> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
>>>>> Date: October 25, 2021 at 11:59:39
>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
>>>>> Subject: [discuss] NiFi 1.15
>>>>>
>>>>> Team,
>>>>>
>>>>> I thought I had already started such a thread but I dont see it so here
>>>>> goes...
>>>>>
>>>>> We have made a ton of progress again and there are some super
>>>>> important fixes as well. It is definitely time to kick out a 1.15.
>>>>> My intent will be to attempt to pull together an RC this week.
>>>>> Haven't done the analysis yet of what is hanging out there but will do
>>>>> so. A quick look at all the features and fixes already landed though
>>>>> make it clear we have more than enough to work with.
>>>>>
>>>>> Lets please use this thread for coordination on the RC rather than it
>>>>> becoming the wish list. We have new features/fixes arriving all the
>>>>> time - those can be addressed in normal channels.
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>
>>>
Re: [discuss] NiFi 1.15
Posted by Joe Witt <jo...@gmail.com>.
Chris,
Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
Should generally not really matter so the docker angle there is
interesting. Not sure why our docker images would have any
relationship to our dist/dev storage. But when I move them into the
release folder I can try to ensure I place them only in 1.15.0 instead
of nifi/1.15.0.
That directory the prov stuff makes does linger and can be annoying so
thanks for tackling that. Saw the PR. Will take a look soon if
nobody else has.
Will keep an eye on your findings related to docker builds not working
with username/password things. Hopefully others can chime in there.
Thanks
Send
On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
<ch...@naimuri.com.invalid> wrote:
>
> Worryingly, when I do get the Docker Image to build (further changes to the
> Dockerfile), the auto-generated username and password from the startup logs
> aren't being accepted for login via my browser.
>
> I'll try to spend a little more time looking at this (but await input on my
> earlier question/concern also).
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.sampson@naimuri.com
>
>
> On Thu, 4 Nov 2021 at 20:47, Chris Sampson <ch...@naimuri.com>
> wrote:
>
> > I've got most of the way through the release check process in order to
> > vote for 1.15.0, but I wanted to check on a change in the distribution
> > release artifacts.
> >
> > For 1.14.0, the Dev artifacts were located at:
> > https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
> > For 1.15.0, the Dev artifacts are located at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
> >
> > i.e. there's been a change of directory/path from <version> to
> > nifi-<version>.
> >
> > The reason I raise this is that I can no longer build a Docker Image using
> > the dockerhub/DockerBuild.sh script because it cannot find the artifacts to
> > download. This may not be a problem if the path change is only for the Dev
> > artifacts, but if the same change is going to happen for the released
> > artifacts, then the apache/nifi image (and presumably the
> > apache/nifi-registry, apache/nifi-toolkit and any minifi) convenience
> > images will no longer be possible as part of the Release, which will likely
> > be an issue for a number of users that have deployments using these images.
> >
> > I spotted this after rebasing my outstanding PR [1] for NIFI-8779 [2]
> >
> >
> > Additionally, I noted NIFI-9366 [3] after an unwanted directory was
> > created by the unit tests executed during building and testing 1.15.0-RC3.
> > I raised a PR [4] - this is a minor issue and not a reason to block any
> > release.
> >
> >
> >
> > [1] https://github.com/apache/nifi/pull/5213
> > [2] https://issues.apache.org/jira/browse/NIFI-8779
> >
> > [3] https://issues.apache.org/jira/browse/NIFI-9366
> > [4] https://github.com/apache/nifi/pull/5510
> >
> > ---
> > *Chris Sampson*
> > IT Consultant
> > chris.sampson@naimuri.com
> >
> >
> > On Sat, 30 Oct 2021 at 01:52, Joe Witt <jo...@apache.org> wrote:
> >
> >> Otto
> >>
> >> Got ya. Yeah it was this way in 1.14 as well. And to be clear with every
> >> release what we are actually voting upon is the source release. Now the
> >> source release includes nifi, minifi, nifi registry, stateless nifi and
> >> toolkits among all the other having always been there goodies.
> >>
> >> Some of these things we make available in the form of convenience binaries
> >> to make it easier on folks to consume.
> >>
> >> I think you dont need to do any verification you would not have done
> >> before.
> >>
> >> But I do hope folks help maintain various ways of easing more folks
> >> knowing
> >> what to vet with a given release
> >>
> >> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <ot...@gmail.com>
> >> wrote:
> >>
> >> > This release, we are verifying not only Nifi, but also Minifi Java. At
> >> > least that is my understanding.
> >> >
> >> > This would be my first time having *anything* to do with minifi, i’ve
> >> not
> >> > even run it before.
> >> >
> >> > As such, I think the RC validation guide needs to be update to include
> >> the
> >> > information about now validating nifi and minifi together.
> >> >
> >> >
> >> >
> >> >
> >> > From: Joe Witt <jo...@apache.org> <jo...@apache.org>
> >> > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> >> > Date: October 25, 2021 at 11:59:39
> >> > To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> >> > Subject: [discuss] NiFi 1.15
> >> >
> >> > Team,
> >> >
> >> > I thought I had already started such a thread but I dont see it so here
> >> > goes...
> >> >
> >> > We have made a ton of progress again and there are some super
> >> > important fixes as well. It is definitely time to kick out a 1.15.
> >> > My intent will be to attempt to pull together an RC this week.
> >> > Haven't done the analysis yet of what is hanging out there but will do
> >> > so. A quick look at all the features and fixes already landed though
> >> > make it clear we have more than enough to work with.
> >> >
> >> > Lets please use this thread for coordination on the RC rather than it
> >> > becoming the wish list. We have new features/fixes arriving all the
> >> > time - those can be addressed in normal channels.
> >> >
> >> > Thanks
> >> >
> >> >
> >>
> >
Re: [discuss] NiFi 1.15
Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
Worryingly, when I do get the Docker Image to build (further changes to the
Dockerfile), the auto-generated username and password from the startup logs
aren't being accepted for login via my browser.
I'll try to spend a little more time looking at this (but await input on my
earlier question/concern also).
---
*Chris Sampson*
IT Consultant
chris.sampson@naimuri.com
On Thu, 4 Nov 2021 at 20:47, Chris Sampson <ch...@naimuri.com>
wrote:
> I've got most of the way through the release check process in order to
> vote for 1.15.0, but I wanted to check on a change in the distribution
> release artifacts.
>
> For 1.14.0, the Dev artifacts were located at:
> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
> For 1.15.0, the Dev artifacts are located at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
>
> i.e. there's been a change of directory/path from <version> to
> nifi-<version>.
>
> The reason I raise this is that I can no longer build a Docker Image using
> the dockerhub/DockerBuild.sh script because it cannot find the artifacts to
> download. This may not be a problem if the path change is only for the Dev
> artifacts, but if the same change is going to happen for the released
> artifacts, then the apache/nifi image (and presumably the
> apache/nifi-registry, apache/nifi-toolkit and any minifi) convenience
> images will no longer be possible as part of the Release, which will likely
> be an issue for a number of users that have deployments using these images.
>
> I spotted this after rebasing my outstanding PR [1] for NIFI-8779 [2]
>
>
> Additionally, I noted NIFI-9366 [3] after an unwanted directory was
> created by the unit tests executed during building and testing 1.15.0-RC3.
> I raised a PR [4] - this is a minor issue and not a reason to block any
> release.
>
>
>
> [1] https://github.com/apache/nifi/pull/5213
> [2] https://issues.apache.org/jira/browse/NIFI-8779
>
> [3] https://issues.apache.org/jira/browse/NIFI-9366
> [4] https://github.com/apache/nifi/pull/5510
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.sampson@naimuri.com
>
>
> On Sat, 30 Oct 2021 at 01:52, Joe Witt <jo...@apache.org> wrote:
>
>> Otto
>>
>> Got ya. Yeah it was this way in 1.14 as well. And to be clear with every
>> release what we are actually voting upon is the source release. Now the
>> source release includes nifi, minifi, nifi registry, stateless nifi and
>> toolkits among all the other having always been there goodies.
>>
>> Some of these things we make available in the form of convenience binaries
>> to make it easier on folks to consume.
>>
>> I think you dont need to do any verification you would not have done
>> before.
>>
>> But I do hope folks help maintain various ways of easing more folks
>> knowing
>> what to vet with a given release
>>
>> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>> > This release, we are verifying not only Nifi, but also Minifi Java. At
>> > least that is my understanding.
>> >
>> > This would be my first time having *anything* to do with minifi, i’ve
>> not
>> > even run it before.
>> >
>> > As such, I think the RC validation guide needs to be update to include
>> the
>> > information about now validating nifi and minifi together.
>> >
>> >
>> >
>> >
>> > From: Joe Witt <jo...@apache.org> <jo...@apache.org>
>> > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
>> > Date: October 25, 2021 at 11:59:39
>> > To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
>> > Subject: [discuss] NiFi 1.15
>> >
>> > Team,
>> >
>> > I thought I had already started such a thread but I dont see it so here
>> > goes...
>> >
>> > We have made a ton of progress again and there are some super
>> > important fixes as well. It is definitely time to kick out a 1.15.
>> > My intent will be to attempt to pull together an RC this week.
>> > Haven't done the analysis yet of what is hanging out there but will do
>> > so. A quick look at all the features and fixes already landed though
>> > make it clear we have more than enough to work with.
>> >
>> > Lets please use this thread for coordination on the RC rather than it
>> > becoming the wish list. We have new features/fixes arriving all the
>> > time - those can be addressed in normal channels.
>> >
>> > Thanks
>> >
>> >
>>
>
Re: [discuss] NiFi 1.15
Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
I've got most of the way through the release check process in order to vote
for 1.15.0, but I wanted to check on a change in the distribution release
artifacts.
For 1.14.0, the Dev artifacts were located at:
https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
For 1.15.0, the Dev artifacts are located at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
i.e. there's been a change of directory/path from <version> to
nifi-<version>.
The reason I raise this is that I can no longer build a Docker Image using
the dockerhub/DockerBuild.sh script because it cannot find the artifacts to
download. This may not be a problem if the path change is only for the Dev
artifacts, but if the same change is going to happen for the released
artifacts, then the apache/nifi image (and presumably the
apache/nifi-registry, apache/nifi-toolkit and any minifi) convenience
images will no longer be possible as part of the Release, which will likely
be an issue for a number of users that have deployments using these images.
I spotted this after rebasing my outstanding PR [1] for NIFI-8779 [2]
Additionally, I noted NIFI-9366 [3] after an unwanted directory was created
by the unit tests executed during building and testing 1.15.0-RC3. I raised
a PR [4] - this is a minor issue and not a reason to block any release.
[1] https://github.com/apache/nifi/pull/5213
[2] https://issues.apache.org/jira/browse/NIFI-8779
[3] https://issues.apache.org/jira/browse/NIFI-9366
[4] https://github.com/apache/nifi/pull/5510
---
*Chris Sampson*
IT Consultant
chris.sampson@naimuri.com
On Sat, 30 Oct 2021 at 01:52, Joe Witt <jo...@apache.org> wrote:
> Otto
>
> Got ya. Yeah it was this way in 1.14 as well. And to be clear with every
> release what we are actually voting upon is the source release. Now the
> source release includes nifi, minifi, nifi registry, stateless nifi and
> toolkits among all the other having always been there goodies.
>
> Some of these things we make available in the form of convenience binaries
> to make it easier on folks to consume.
>
> I think you dont need to do any verification you would not have done
> before.
>
> But I do hope folks help maintain various ways of easing more folks knowing
> what to vet with a given release
>
> On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <ot...@gmail.com>
> wrote:
>
> > This release, we are verifying not only Nifi, but also Minifi Java. At
> > least that is my understanding.
> >
> > This would be my first time having *anything* to do with minifi, i’ve not
> > even run it before.
> >
> > As such, I think the RC validation guide needs to be update to include
> the
> > information about now validating nifi and minifi together.
> >
> >
> >
> >
> > From: Joe Witt <jo...@apache.org> <jo...@apache.org>
> > Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > Date: October 25, 2021 at 11:59:39
> > To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> > Subject: [discuss] NiFi 1.15
> >
> > Team,
> >
> > I thought I had already started such a thread but I dont see it so here
> > goes...
> >
> > We have made a ton of progress again and there are some super
> > important fixes as well. It is definitely time to kick out a 1.15.
> > My intent will be to attempt to pull together an RC this week.
> > Haven't done the analysis yet of what is hanging out there but will do
> > so. A quick look at all the features and fixes already landed though
> > make it clear we have more than enough to work with.
> >
> > Lets please use this thread for coordination on the RC rather than it
> > becoming the wish list. We have new features/fixes arriving all the
> > time - those can be addressed in normal channels.
> >
> > Thanks
> >
> >
>
Re: [discuss] NiFi 1.15
Posted by Joe Witt <jo...@apache.org>.
Otto
Got ya. Yeah it was this way in 1.14 as well. And to be clear with every
release what we are actually voting upon is the source release. Now the
source release includes nifi, minifi, nifi registry, stateless nifi and
toolkits among all the other having always been there goodies.
Some of these things we make available in the form of convenience binaries
to make it easier on folks to consume.
I think you dont need to do any verification you would not have done before.
But I do hope folks help maintain various ways of easing more folks knowing
what to vet with a given release
On Fri, Oct 29, 2021 at 5:34 PM Otto Fowler <ot...@gmail.com> wrote:
> This release, we are verifying not only Nifi, but also Minifi Java. At
> least that is my understanding.
>
> This would be my first time having *anything* to do with minifi, i’ve not
> even run it before.
>
> As such, I think the RC validation guide needs to be update to include the
> information about now validating nifi and minifi together.
>
>
>
>
> From: Joe Witt <jo...@apache.org> <jo...@apache.org>
> Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> Date: October 25, 2021 at 11:59:39
> To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
> Subject: [discuss] NiFi 1.15
>
> Team,
>
> I thought I had already started such a thread but I dont see it so here
> goes...
>
> We have made a ton of progress again and there are some super
> important fixes as well. It is definitely time to kick out a 1.15.
> My intent will be to attempt to pull together an RC this week.
> Haven't done the analysis yet of what is hanging out there but will do
> so. A quick look at all the features and fixes already landed though
> make it clear we have more than enough to work with.
>
> Lets please use this thread for coordination on the RC rather than it
> becoming the wish list. We have new features/fixes arriving all the
> time - those can be addressed in normal channels.
>
> Thanks
>
>
Re: [discuss] NiFi 1.15
Posted by Otto Fowler <ot...@gmail.com>.
This release, we are verifying not only Nifi, but also Minifi Java. At
least that is my understanding.
This would be my first time having *anything* to do with minifi, i’ve not
even run it before.
As such, I think the RC validation guide needs to be update to include the
information about now validating nifi and minifi together.
From: Joe Witt <jo...@apache.org> <jo...@apache.org>
Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
Date: October 25, 2021 at 11:59:39
To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
Subject: [discuss] NiFi 1.15
Team,
I thought I had already started such a thread but I dont see it so here
goes...
We have made a ton of progress again and there are some super
important fixes as well. It is definitely time to kick out a 1.15.
My intent will be to attempt to pull together an RC this week.
Haven't done the analysis yet of what is hanging out there but will do
so. A quick look at all the features and fixes already landed though
make it clear we have more than enough to work with.
Lets please use this thread for coordination on the RC rather than it
becoming the wish list. We have new features/fixes arriving all the
time - those can be addressed in normal channels.
Thanks
Re: [discuss] NiFi 1.15
Posted by Otto Fowler <ot...@gmail.com>.
Is there a discuss thread for the release? I only see the vote and I have a
question about the verification. My mail must be messed up, i have this
mail with no replies… then vote thread.
From: Joe Witt <jo...@apache.org> <jo...@apache.org>
Reply: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
Date: October 25, 2021 at 11:59:39
To: dev@nifi.apache.org <de...@nifi.apache.org> <de...@nifi.apache.org>
Subject: [discuss] NiFi 1.15
Team,
I thought I had already started such a thread but I dont see it so here
goes...
We have made a ton of progress again and there are some super
important fixes as well. It is definitely time to kick out a 1.15.
My intent will be to attempt to pull together an RC this week.
Haven't done the analysis yet of what is hanging out there but will do
so. A quick look at all the features and fixes already landed though
make it clear we have more than enough to work with.
Lets please use this thread for coordination on the RC rather than it
becoming the wish list. We have new features/fixes arriving all the
time - those can be addressed in normal channels.
Thanks