You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Russell Bateman <ru...@windofkeltia.com> on 2023/06/07 21:31:39 UTC
Possible Docker bug in 1.13.2
I downloaded sources to 1.13.2 in order to hand-spin my own container
image. When I got down to
/nifi-1.13.2/nifi-docker/dockerhub/Dockerfile/, I found:
...
ARG NIFI_VERSION=1.13.1
...
and the version is also wrong in /DockerImage.txt/ which
/DockerBuild.sh/ consumes.
Indeed, the image that is built appears to be versioned 1.13.1 and not
1.13.2 when listing local Docker repository images:
*REPOSITORY TAG IMAGE ID CREATED SIZE*
apache/nifi 1.13.1 8c18038f152a 30 minutes ago 2.06GB
Why am I juggling so ancient a version? Because I have custom processors
that cannot be rebuilt (source-code gone) and will not run on 1.14.0 and
later.
Russ
Re: Possible Docker bug in 1.13.2
Posted by Mathew Kiprop <ma...@gmail.com>.
Russel,
I faced a similar issue recently and reported on slack channel.
I am working on a fix and hopefully submit a PR in the coming hours.
Chris was resourceful pointing out the cause.
Thanks,
Mathew
On Thu, 8 Jun 2023 at 15:58, Russell Bateman <ru...@windofkeltia.com> wrote:
> Thanks, Chris.
>
> In fact, I merely hand-corrected the instances of the version in
> /Dockerfile/ and in /DockerImage.txt/, then ran /DockerBuild.sh/. I got
> exactly the image I wanted. It looked like a potential if irrelevant bug
> and I thought I'd report it.
>
> Best,
> Russ
>
> On 6/7/23 19:17, Chris Sampson wrote:
> > The DockerImage.txt file isn't always updated in the repo, a bit like the
> > README for Docker Hub - it probably should be, but is often forgotten
> (same
> > for all of the convenience images built after a release). Indeed, this is
> > currently set to 1.15.1 on `main`, but the images in Docker Hub since
> then
> > have certainly not used an incorrect version.
> >
> > Nifi 1.16.0 [1] updated the DockerBuild.sh script to allow easier
> > specification of the NIFI_IMAGE_VERSION from the command line. A small
> > issue has just been spotted with this script's use of the text file to
> > obtain the version's default value too [4].
> >
> > Before then, I think the process was for someone (with access to push
> > images to Docker Hub) to change the text file locally, build & push the
> > image, but not always then raise a PR to update the text file.
> >
> > To manually build the Docker Image locally containing the correct version
> > of the downloaded convenience binary from the Apache servers, simply
> > recreate the `docker build` command from within the DockerBuild.sh [2]
> > file, substituting the correct values for the `--build-arg`s.
> >
> > Alternatively, run the dockermaven [3] build for the repo (having checked
> > out the correct tag from GitHub), enabling the `-P docker` profile as you
> > do so in your `mvn` command. Note, however, that there were differences
> > between the dockermaven and dockerhub builds (both the included script
> > files and the Dockerfile structure) that weren't rationalised until NiFi
> > 1.16.0+.
> >
> >
> > [1]:
> >
> https://github.com/apache/nifi/blob/rel/nifi-1.16.0/nifi-docker/dockerhub/DockerBuild.sh
> >
> > [2]:
> >
> https://github.com/apache/nifi/blob/rel/nifi-1.13.2/nifi-docker/dockerhub/DockerBuild.sh#L36
> >
> > [3]:
> >
> https://github.com/apache/nifi/tree/rel/nifi-1.16.0/nifi-docker/dockermaven
> >
> > [4]:
> >
> https://apachenifi.slack.com/archives/CDGMCSDJT/p1686135561932289?thread_ts=1686135561.932289&cid=CDGMCSDJT
> >
> > On Thu, 8 Jun 2023, 00:10 Russell Bateman,<ru...@windofkeltia.com>
> wrote:
> >
> >> I'm re-rolling in order to update the Java inside to 11 in order to
> >> permit using the new Java HTTP client. This seems to work well; I fixed
> >> the bug locally.
> >>
> >> Maybe too old to be important, but NiFi 1.14.0 is a quantum step up for
> >> several aspects of processor writing requiring refactoring. So, until we
> >> can shed the old NAR we cannot rebuild, we're stuck at NiFi 1.13.2.
> >>
> >>
> >> On 6/7/23 15:31, Russell Bateman wrote:
> >>> I downloaded sources to 1.13.2 in order to hand-spin my own container
> >>> image. When I got down to
> >>> /nifi-1.13.2/nifi-docker/dockerhub/Dockerfile/, I found:
> >>>
> >>> ...
> >>> ARG NIFI_VERSION=1.13.1
> >>> ...
> >>>
> >>> and the version is also wrong in /DockerImage.txt/ which
> >>> /DockerBuild.sh/ consumes.
> >>>
> >>> Indeed, the image that is built appears to be versioned 1.13.1 and not
> >>> 1.13.2 when listing local Docker repository images:
> >>>
> >>> *REPOSITORY TAG IMAGE ID CREATED SIZE*
> >>> apache/nifi 1.13.1 8c18038f152a 30 minutes ago 2.06GB
> >>>
> >>> Why am I juggling so ancient a version? Because I have custom
> >>> processors that cannot be rebuilt (source-code gone) and will not run
> >>> on 1.14.0 and later.
> >>>
> >>> Russ
>
Re: Possible Docker bug in 1.13.2
Posted by Russell Bateman <ru...@windofkeltia.com>.
Thanks, Chris.
In fact, I merely hand-corrected the instances of the version in
/Dockerfile/ and in /DockerImage.txt/, then ran /DockerBuild.sh/. I got
exactly the image I wanted. It looked like a potential if irrelevant bug
and I thought I'd report it.
Best,
Russ
On 6/7/23 19:17, Chris Sampson wrote:
> The DockerImage.txt file isn't always updated in the repo, a bit like the
> README for Docker Hub - it probably should be, but is often forgotten (same
> for all of the convenience images built after a release). Indeed, this is
> currently set to 1.15.1 on `main`, but the images in Docker Hub since then
> have certainly not used an incorrect version.
>
> Nifi 1.16.0 [1] updated the DockerBuild.sh script to allow easier
> specification of the NIFI_IMAGE_VERSION from the command line. A small
> issue has just been spotted with this script's use of the text file to
> obtain the version's default value too [4].
>
> Before then, I think the process was for someone (with access to push
> images to Docker Hub) to change the text file locally, build & push the
> image, but not always then raise a PR to update the text file.
>
> To manually build the Docker Image locally containing the correct version
> of the downloaded convenience binary from the Apache servers, simply
> recreate the `docker build` command from within the DockerBuild.sh [2]
> file, substituting the correct values for the `--build-arg`s.
>
> Alternatively, run the dockermaven [3] build for the repo (having checked
> out the correct tag from GitHub), enabling the `-P docker` profile as you
> do so in your `mvn` command. Note, however, that there were differences
> between the dockermaven and dockerhub builds (both the included script
> files and the Dockerfile structure) that weren't rationalised until NiFi
> 1.16.0+.
>
>
> [1]:
> https://github.com/apache/nifi/blob/rel/nifi-1.16.0/nifi-docker/dockerhub/DockerBuild.sh
>
> [2]:
> https://github.com/apache/nifi/blob/rel/nifi-1.13.2/nifi-docker/dockerhub/DockerBuild.sh#L36
>
> [3]:
> https://github.com/apache/nifi/tree/rel/nifi-1.16.0/nifi-docker/dockermaven
>
> [4]:
> https://apachenifi.slack.com/archives/CDGMCSDJT/p1686135561932289?thread_ts=1686135561.932289&cid=CDGMCSDJT
>
> On Thu, 8 Jun 2023, 00:10 Russell Bateman,<ru...@windofkeltia.com> wrote:
>
>> I'm re-rolling in order to update the Java inside to 11 in order to
>> permit using the new Java HTTP client. This seems to work well; I fixed
>> the bug locally.
>>
>> Maybe too old to be important, but NiFi 1.14.0 is a quantum step up for
>> several aspects of processor writing requiring refactoring. So, until we
>> can shed the old NAR we cannot rebuild, we're stuck at NiFi 1.13.2.
>>
>>
>> On 6/7/23 15:31, Russell Bateman wrote:
>>> I downloaded sources to 1.13.2 in order to hand-spin my own container
>>> image. When I got down to
>>> /nifi-1.13.2/nifi-docker/dockerhub/Dockerfile/, I found:
>>>
>>> ...
>>> ARG NIFI_VERSION=1.13.1
>>> ...
>>>
>>> and the version is also wrong in /DockerImage.txt/ which
>>> /DockerBuild.sh/ consumes.
>>>
>>> Indeed, the image that is built appears to be versioned 1.13.1 and not
>>> 1.13.2 when listing local Docker repository images:
>>>
>>> *REPOSITORY TAG IMAGE ID CREATED SIZE*
>>> apache/nifi 1.13.1 8c18038f152a 30 minutes ago 2.06GB
>>>
>>> Why am I juggling so ancient a version? Because I have custom
>>> processors that cannot be rebuilt (source-code gone) and will not run
>>> on 1.14.0 and later.
>>>
>>> Russ
Re: Possible Docker bug in 1.13.2
Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
The DockerImage.txt file isn't always updated in the repo, a bit like the
README for Docker Hub - it probably should be, but is often forgotten (same
for all of the convenience images built after a release). Indeed, this is
currently set to 1.15.1 on `main`, but the images in Docker Hub since then
have certainly not used an incorrect version.
Nifi 1.16.0 [1] updated the DockerBuild.sh script to allow easier
specification of the NIFI_IMAGE_VERSION from the command line. A small
issue has just been spotted with this script's use of the text file to
obtain the version's default value too [4].
Before then, I think the process was for someone (with access to push
images to Docker Hub) to change the text file locally, build & push the
image, but not always then raise a PR to update the text file.
To manually build the Docker Image locally containing the correct version
of the downloaded convenience binary from the Apache servers, simply
recreate the `docker build` command from within the DockerBuild.sh [2]
file, substituting the correct values for the `--build-arg`s.
Alternatively, run the dockermaven [3] build for the repo (having checked
out the correct tag from GitHub), enabling the `-P docker` profile as you
do so in your `mvn` command. Note, however, that there were differences
between the dockermaven and dockerhub builds (both the included script
files and the Dockerfile structure) that weren't rationalised until NiFi
1.16.0+.
[1]:
https://github.com/apache/nifi/blob/rel/nifi-1.16.0/nifi-docker/dockerhub/DockerBuild.sh
[2]:
https://github.com/apache/nifi/blob/rel/nifi-1.13.2/nifi-docker/dockerhub/DockerBuild.sh#L36
[3]:
https://github.com/apache/nifi/tree/rel/nifi-1.16.0/nifi-docker/dockermaven
[4]:
https://apachenifi.slack.com/archives/CDGMCSDJT/p1686135561932289?thread_ts=1686135561.932289&cid=CDGMCSDJT
On Thu, 8 Jun 2023, 00:10 Russell Bateman, <ru...@windofkeltia.com> wrote:
> I'm re-rolling in order to update the Java inside to 11 in order to
> permit using the new Java HTTP client. This seems to work well; I fixed
> the bug locally.
>
> Maybe too old to be important, but NiFi 1.14.0 is a quantum step up for
> several aspects of processor writing requiring refactoring. So, until we
> can shed the old NAR we cannot rebuild, we're stuck at NiFi 1.13.2.
>
>
> On 6/7/23 15:31, Russell Bateman wrote:
> > I downloaded sources to 1.13.2 in order to hand-spin my own container
> > image. When I got down to
> > /nifi-1.13.2/nifi-docker/dockerhub/Dockerfile/, I found:
> >
> > ...
> > ARG NIFI_VERSION=1.13.1
> > ...
> >
> > and the version is also wrong in /DockerImage.txt/ which
> > /DockerBuild.sh/ consumes.
> >
> > Indeed, the image that is built appears to be versioned 1.13.1 and not
> > 1.13.2 when listing local Docker repository images:
> >
> > *REPOSITORY TAG IMAGE ID CREATED SIZE*
> > apache/nifi 1.13.1 8c18038f152a 30 minutes ago 2.06GB
> >
> > Why am I juggling so ancient a version? Because I have custom
> > processors that cannot be rebuilt (source-code gone) and will not run
> > on 1.14.0 and later.
> >
> > Russ
>
Re: Possible Docker bug in 1.13.2
Posted by Russell Bateman <ru...@windofkeltia.com>.
I'm re-rolling in order to update the Java inside to 11 in order to
permit using the new Java HTTP client. This seems to work well; I fixed
the bug locally.
Maybe too old to be important, but NiFi 1.14.0 is a quantum step up for
several aspects of processor writing requiring refactoring. So, until we
can shed the old NAR we cannot rebuild, we're stuck at NiFi 1.13.2.
On 6/7/23 15:31, Russell Bateman wrote:
> I downloaded sources to 1.13.2 in order to hand-spin my own container
> image. When I got down to
> /nifi-1.13.2/nifi-docker/dockerhub/Dockerfile/, I found:
>
> ...
> ARG NIFI_VERSION=1.13.1
> ...
>
> and the version is also wrong in /DockerImage.txt/ which
> /DockerBuild.sh/ consumes.
>
> Indeed, the image that is built appears to be versioned 1.13.1 and not
> 1.13.2 when listing local Docker repository images:
>
> *REPOSITORY TAG IMAGE ID CREATED SIZE*
> apache/nifi 1.13.1 8c18038f152a 30 minutes ago 2.06GB
>
> Why am I juggling so ancient a version? Because I have custom
> processors that cannot be rebuilt (source-code gone) and will not run
> on 1.14.0 and later.
>
> Russ
Re: Possible Docker bug in 1.13.2
Posted by Mike Thomsen <mi...@gmail.com>.
> Because I have custom processors that cannot be rebuilt (source-code
gone)
IntelliJ can probably give you a close version of the source code if you
expand the JAR and drag the class files for your custom code into the IDE.
On Wed, Jun 7, 2023 at 5:32 PM Russell Bateman <ru...@windofkeltia.com>
wrote:
> I downloaded sources to 1.13.2 in order to hand-spin my own container
> image. When I got down to
> /nifi-1.13.2/nifi-docker/dockerhub/Dockerfile/, I found:
>
> ...
> ARG NIFI_VERSION=1.13.1
> ...
>
> and the version is also wrong in /DockerImage.txt/ which
> /DockerBuild.sh/ consumes.
>
> Indeed, the image that is built appears to be versioned 1.13.1 and not
> 1.13.2 when listing local Docker repository images:
>
> *REPOSITORY TAG IMAGE ID CREATED SIZE*
> apache/nifi 1.13.1 8c18038f152a 30 minutes ago 2.06GB
>
> Why am I juggling so ancient a version? Because I have custom processors
> that cannot be rebuilt (source-code gone) and will not run on 1.14.0 and
> later.
>
> Russ