You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by James McMahon <js...@gmail.com> on 2022/11/08 19:53:50 UTC
NiFi on AWS EC2
Has anyone successfully configured NiFi on AWS, and accessed it from a
browser on a Windows desktop? I’ve tried following a few links to do this.
I’ve verified that my instance security group allows access to 8080 via its
inbound rules. I’ve putty’ed into the instance via ssh port 22 to verify
that there are no firewall restrictions. But still I get a message to the
effect that the server rejected the connection request. Can anyone
recommend a link that describes a success path for this?
Thanks in advance for your help.
Jim
Re: NiFi on AWS EC2
Posted by Mike Thomsen <mi...@gmail.com>.
> won't render if you're using Edge or IE to access.
The old, discontinued Edge had that problem, but Edge has worked just
fine for the last 2 years or so since it was redone on top of
Chromium.
IMO if you're going to use Chromium with NiFi, the best experience
seems to be Brave.
On Tue, Nov 8, 2022 at 4:21 PM Patrick Timmins <pt...@cox.net> wrote:
>
> In addition to the other suggestions, the last time I checked, the HTML5
> of the NiFi interface won't render if you're using Edge or IE to
> access. Brave, Chrome, Firefox etc will work, however.
>
> On 11/8/2022 1:53 PM, James McMahon wrote:
> > Has anyone successfully configured NiFi on AWS, and accessed it from a
> > browser on a Windows desktop? I’ve tried following a few links to do
> > this. I’ve verified that my instance security group allows access to
> > 8080 via its inbound rules. I’ve putty’ed into the instance via ssh
> > port 22 to verify that there are no firewall restrictions. But still I
> > get a message to the effect that the server rejected the connection
> > request. Can anyone recommend a link that describes a success path for
> > this?
> > Thanks in advance for your help.
> > Jim
Re: NiFi on AWS EC2
Posted by Patrick Timmins <pt...@cox.net>.
In addition to the other suggestions, the last time I checked, the HTML5
of the NiFi interface won't render if you're using Edge or IE to
access. Brave, Chrome, Firefox etc will work, however.
On 11/8/2022 1:53 PM, James McMahon wrote:
> Has anyone successfully configured NiFi on AWS, and accessed it from a
> browser on a Windows desktop? I’ve tried following a few links to do
> this. I’ve verified that my instance security group allows access to
> 8080 via its inbound rules. I’ve putty’ed into the instance via ssh
> port 22 to verify that there are no firewall restrictions. But still I
> get a message to the effect that the server rejected the connection
> request. Can anyone recommend a link that describes a success path for
> this?
> Thanks in advance for your help.
> Jim
Re: NiFi on AWS EC2
Posted by Dmitry Stepanov <dm...@dmitryds.com>.
We did.
Since NiFi is HTTPS by default (recent versions) you need to open ports
8443 or 443(default HTTPS)
Try opening those and see what happens
Cheers,
Dima Stepanov
Re: NiFi on AWS EC2
Posted by James McMahon <js...@gmail.com>.
Yes sir, I did. I used the full public domain name.
On Tue, Nov 8, 2022 at 8:08 PM Dmitry Stepanov <dm...@dmitryds.com> wrote:
> Make sure you use your full domain name
> ec2-3-238-27-220.compute-1.amazonaws.com
> David shorten it in his code
>
> On November 8, 2022 5:57:26 p.m. James McMahon <js...@gmail.com>
> wrote:
>
>> Thank you, David. I’ve made that change, adding the proxy host
>> specification on the docker command line. I continue to get the same error
>> message. Is it possible I need to indicate my key on the docker command
>> line too?
>>
>> Related, how can one access nifi.properties and the usual nifi config
>> files, as well as the family of nifi-app.log files and bootstrap.conf, when
>> nifi is running inside a docker container?
>>
>> Thanks again for sticking with this. I feel like we’re getting closer.
>> Jim
>>
>> On Tue, Nov 8, 2022 at 7:31 PM David Handermann <
>> exceptionfactory@apache.org> wrote:
>>
>>> Hi Jim,
>>>
>>> Good adjustment on the security group inbound rules.
>>>
>>> The error page is the result of NiFi receiving an unexpected HTTP Host
>>> header, not matching one of the expected values.
>>>
>>> For this to work, it is possible to pass the external DNS name as the
>>> value of the NIFI_WEB_PROXY_HOST environment variable. This can be
>>> specified in the docker run command as follows:
>>>
>>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
>>> amazonaws.com -d apache/nifi:latest
>>>
>>> That will allow NiFi to accept the Host header from the browser, and
>>> then present the login screen.
>>>
>>> Regards,
>>> David Handermann
>>>
>>> On Tue, Nov 8, 2022 at 6:06 PM James McMahon <js...@gmail.com>
>>> wrote:
>>>
>>>> Hi David. This is very helpful, thank you. I feel like I am close, but
>>>> I get an error. My Inbound Rules for my security group now include:
>>>> 8443 TCP (MyIP)/32
>>>> 443 TCP (MyIP)/32
>>>> 22 TCP (MyIP)/32
>>>>
>>>> In my browser - I tried both Edge and Chrome - I use this
>>>> URL:
>>>> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
>>>> I have also tried with /nifi at the tail end.
>>>>
>>>> I get this error:
>>>>
>>>> *System Error*
>>>>
>>>> *The request contained an invalid host header
>>>> [ec2-3-238-27-220.compute-1.amazonaws.com:8443
>>>> <http://ec2-3-238-27-220.compute-1.amazonaws.com:8443/>] in the request
>>>> [/]. Check for request manipulation or third-party intercept.*
>>>>
>>>> *Valid host headers are [empty] or:*
>>>>
>>>> - *127.0.0.1*
>>>> - *127.0.0.1:8443 <http://127.0.0.1:8443/>*
>>>> - *localhost*
>>>> - *localhost:8443*
>>>> - *[::1]*
>>>> - *[::1]:8443*
>>>> - *7f661ae687d7*
>>>> - *7f661ae687d7:8443*
>>>> - *172.17.0.2*
>>>> - *172.17.0.2:8443 <http://172.17.0.2:8443/>*
>>>>
>>>>
>>>> Does this mean I have formed the URL incorrectly?
>>>>
>>>> I also see that I had to add an exception to permit https. When I
>>>> created the instance, I created my own pem key pair. It is not signed by
>>>> any CA. For a self-signed key pair like this, do I need to install a key in
>>>> my browser security store to avoid adding that exception?
>>>>
>>>> Thank you for helping me get that much closer.
>>>> Jim
>>>>
>>>> On Tue, Nov 8, 2022 at 5:13 PM David Handermann <
>>>> exceptionfactory@apache.org> wrote:
>>>>
>>>>> Hi Jim,
>>>>>
>>>>> Thanks for the reply and additional background.
>>>>>
>>>>> The instructions are dated March 2021, which is prior to the release
>>>>> of NiFi 1.14.0. In particular, the run command is no longer accurate with
>>>>> the default NiFi container image.
>>>>>
>>>>> The current Docker Hub instructions [1] show the basic command needed
>>>>>
>>>>> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>>>>>
>>>>> In addition, any references to port 8080 in the AWS Security Group
>>>>> rules should be changed to 8443. The security group rules for port 80 and
>>>>> 18080 should be removed.
>>>>>
>>>>> The instructions that allow plain HTTP access to NiFi on port 8080
>>>>> should NEVER be followed, as this exposes unfiltered and unauthenticated
>>>>> access.
>>>>>
>>>>> Following those changes, it should be possible to access the NiFi UI
>>>>> using the AWS URL:
>>>>>
>>>>> https://ec2...amazonaws.com:8443
>>>>>
>>>>> The default installation will generate a username and password, which
>>>>> can be found in the container logs:
>>>>>
>>>>> docker logs nifi | grep Generated
>>>>>
>>>>> Regards,
>>>>> David Handermann
>>>>>
>>>>> [1] https://hub.docker.com/r/apache/nifi
>>>>>
>>>>> On Tue, Nov 8, 2022 at 4:00 PM James McMahon <js...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi and thank you, David and Dmitry. In my case I was following this
>>>>>> example,
>>>>>>
>>>>>> https://joeygoksu.com/software/apache-nifi-on-aws/
>>>>>>
>>>>>> which results in NiFi installed within a container. So to answer one
>>>>>> of your questions, I don’t yet know how or where to find nifi.properties in
>>>>>> the container framework. I don’t seem to have the usual /opt/nifi/…..
>>>>>> directories on my ec2 instance. Any idea where I need to look for that?
>>>>>>
>>>>>> These ports are open by my security group Inbound Rules: 22 to MyIP,
>>>>>> 80, 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.
>>>>>>
>>>>>> I am able to Putty into my instance as ec2-user with my ppk file,
>>>>>> which I created using putty tools from the original pem key pair. When I do
>>>>>> putty in, under /opt I find three subdirectories: aws, containerd, and rh.
>>>>>> Nothing nifi under any of the three that I can see so far.
>>>>>>
>>>>>> I start my docker instance with this command:
>>>>>> docker run —name nifi -p 18080:8080 -d apache/nifi:latest
>>>>>>
>>>>>> I can do a ps -ef and see running nifi processes. But I don’t yet
>>>>>> know how to get to the nifi logs or properties file.
>>>>>>
>>>>>> You mentioned using using localhost to get to the canvas UI. This
>>>>>> confuses me. Nifi is running on my EC2 instance - a linux host without a
>>>>>> browser. I’m in a browser on my laptop. How would localhost in my browser
>>>>>> get me to my EC2 instance running nifi?
>>>>>>
>>>>>> This is the URL I’m using in my browser:
>>>>>> http://ec2-3-238-27-220.compute-1.amazonaws.com
>>>>>> (that url changes with each Stop/Start of my instance. I’ve yet to
>>>>>> investigate how to get AWS to stop changing that IP, but I know it can be
>>>>>> done).
>>>>>>
>>>>>> The browser replies with: ec2…….amazonaws refused to connect.
>>>>>>
>>>>>> I can ping my laptop IP address from the putty terminal where I am
>>>>>> logged in to my instance. I cannot ping the Public DNS of my instance from
>>>>>> Powershell on my laptop. Again, that Public DNS is
>>>>>> ec2-3-238-27-220.compute-1.amazonaws.com
>>>>>>
>>>>>> Any help is much appreciated.
>>>>>> Jim
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 8, 2022 at 3:03 PM David Handermann <
>>>>>> exceptionfactory@apache.org> wrote:
>>>>>>
>>>>>>> Hi Jim,
>>>>>>>
>>>>>>> NiFi 1.14.0 and following default to HTTPS on port 8443, listening
>>>>>>> on the localhost address. The nifi.web.https.host can be changed to blank
>>>>>>> in order to listen on all interfaces, but the default HTTPS setting with
>>>>>>> authenticated required should be retained.
>>>>>>>
>>>>>>> Can you provide the version of NiFi and some additional details on
>>>>>>> the nifi.web values from nifi.properties?
>>>>>>>
>>>>>>> Regards,
>>>>>>> David Handermann
>>>>>>>
>>>>>>> On Tue, Nov 8, 2022 at 1:54 PM James McMahon <js...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Has anyone successfully configured NiFi on AWS, and accessed it
>>>>>>>> from a browser on a Windows desktop? I’ve tried following a few links to do
>>>>>>>> this. I’ve verified that my instance security group allows access to 8080
>>>>>>>> via its inbound rules. I’ve putty’ed into the instance via ssh port 22 to
>>>>>>>> verify that there are no firewall restrictions. But still I get a message
>>>>>>>> to the effect that the server rejected the connection request. Can anyone
>>>>>>>> recommend a link that describes a success path for this?
>>>>>>>> Thanks in advance for your help.
>>>>>>>> Jim
>>>>>>>>
>>>>>>>
>
Re: NiFi on AWS EC2
Posted by James McMahon <js...@gmail.com>.
I used the command
docker exec -it nifi /bin/bash
to review what I understand to be the nifi directories and config files in
the container.
I notice nifi in the container is in many ways not configured to Apache
NiFi recommendations for optimal performance. For example, the
nifi.properties repo params all refer to repos placed on one common disk
device (the one where the container lives, presumably).
I've configured external ebs volumes that I've mounted on my instance. One
for content_repository, one for flowfile_repository, and likewise for
database and provenance repositories. I'd like to have the containerized
nifi write to and read from those so that I don't bottleneck performance
reading and writing to the same device for repos.
I need to persist my changes to nifi config files. How does one avoid
making changes in nifi.properties and the like that are lost when the
docker container is stopped, deleted, and a new one instantiated?
I need to leverage external resources when nifi runs within my container.
How do we direct nifi in the container to use those external resources
outside of the container to host content_repository, etc etc?
Thank you in advance for any help.
Jim
On Tue, Nov 8, 2022 at 10:28 PM David Handermann <
exceptionfactory@apache.org> wrote:
> Jim,
>
> You're welcome! Thanks for following up and confirming the solution, great
> collaborative effort!
>
> Regard,
> David Handermann
>
>
>
>
> On Tue, Nov 8, 2022, 7:25 PM James McMahon <js...@gmail.com> wrote:
>
>> That was it. Adding the port to the docker run command proxy got me to
>> the promised land. I was then able to use the userid and password from the
>> docker log to access nifi on my ec2 instance.
>>
>> David, Dmitry - thank you so much. This was a huge help to me, and I hope
>> it will help others trying the same approach in the future.
>> Jim
>>
>> On Tue, Nov 8, 2022 at 8:13 PM David Handermann <
>> exceptionfactory@apache.org> wrote:
>>
>>> It may also be necessary to include the port in the host variable:
>>>
>>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=
>>> ec2-3-238-27-220.compute-1.amazonaws.com:8443 -d apache/nifi:latest
>>>
>>> It is possible to access the configuration and logs files using an
>>> interactive shell with the following Docker command:
>>>
>>> docker exec -it nifi /bin/bash
>>>
>>> Regards,
>>> David Handermann
>>>
>>> On Tue, Nov 8, 2022 at 7:09 PM Dmitry Stepanov <dm...@dmitryds.com>
>>> wrote:
>>>
>>>> Make sure you use your full domain name
>>>> ec2-3-238-27-220.compute-1.amazonaws.com
>>>> David shorten it in his code
>>>>
>>>> On November 8, 2022 5:57:26 p.m. James McMahon <js...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thank you, David. I’ve made that change, adding the proxy host
>>>>> specification on the docker command line. I continue to get the same error
>>>>> message. Is it possible I need to indicate my key on the docker command
>>>>> line too?
>>>>>
>>>>> Related, how can one access nifi.properties and the usual nifi config
>>>>> files, as well as the family of nifi-app.log files and bootstrap.conf, when
>>>>> nifi is running inside a docker container?
>>>>>
>>>>> Thanks again for sticking with this. I feel like we’re getting closer.
>>>>> Jim
>>>>>
>>>>> On Tue, Nov 8, 2022 at 7:31 PM David Handermann <
>>>>> exceptionfactory@apache.org> wrote:
>>>>>
>>>>>> Hi Jim,
>>>>>>
>>>>>> Good adjustment on the security group inbound rules.
>>>>>>
>>>>>> The error page is the result of NiFi receiving an unexpected HTTP
>>>>>> Host header, not matching one of the expected values.
>>>>>>
>>>>>> For this to work, it is possible to pass the external DNS name as the
>>>>>> value of the NIFI_WEB_PROXY_HOST environment variable. This can be
>>>>>> specified in the docker run command as follows:
>>>>>>
>>>>>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
>>>>>> amazonaws.com -d apache/nifi:latest
>>>>>>
>>>>>> That will allow NiFi to accept the Host header from the browser, and
>>>>>> then present the login screen.
>>>>>>
>>>>>> Regards,
>>>>>> David Handermann
>>>>>>
>>>>>> On Tue, Nov 8, 2022 at 6:06 PM James McMahon <js...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi David. This is very helpful, thank you. I feel like I am close,
>>>>>>> but I get an error. My Inbound Rules for my security group now include:
>>>>>>> 8443 TCP (MyIP)/32
>>>>>>> 443 TCP (MyIP)/32
>>>>>>> 22 TCP (MyIP)/32
>>>>>>>
>>>>>>> In my browser - I tried both Edge and Chrome - I use this
>>>>>>> URL:
>>>>>>> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
>>>>>>> I have also tried with /nifi at the tail end.
>>>>>>>
>>>>>>> I get this error:
>>>>>>>
>>>>>>> *System Error*
>>>>>>>
>>>>>>> *The request contained an invalid host header
>>>>>>> [ec2-3-238-27-220.compute-1.amazonaws.com:8443
>>>>>>> <http://ec2-3-238-27-220.compute-1.amazonaws.com:8443/>] in the request
>>>>>>> [/]. Check for request manipulation or third-party intercept.*
>>>>>>>
>>>>>>> *Valid host headers are [empty] or:*
>>>>>>>
>>>>>>> - *127.0.0.1*
>>>>>>> - *127.0.0.1:8443 <http://127.0.0.1:8443/>*
>>>>>>> - *localhost*
>>>>>>> - *localhost:8443*
>>>>>>> - *[::1]*
>>>>>>> - *[::1]:8443*
>>>>>>> - *7f661ae687d7*
>>>>>>> - *7f661ae687d7:8443*
>>>>>>> - *172.17.0.2*
>>>>>>> - *172.17.0.2:8443 <http://172.17.0.2:8443/>*
>>>>>>>
>>>>>>>
>>>>>>> Does this mean I have formed the URL incorrectly?
>>>>>>>
>>>>>>> I also see that I had to add an exception to permit https. When I
>>>>>>> created the instance, I created my own pem key pair. It is not signed by
>>>>>>> any CA. For a self-signed key pair like this, do I need to install a key in
>>>>>>> my browser security store to avoid adding that exception?
>>>>>>>
>>>>>>> Thank you for helping me get that much closer.
>>>>>>> Jim
>>>>>>>
>>>>>>> On Tue, Nov 8, 2022 at 5:13 PM David Handermann <
>>>>>>> exceptionfactory@apache.org> wrote:
>>>>>>>
>>>>>>>> Hi Jim,
>>>>>>>>
>>>>>>>> Thanks for the reply and additional background.
>>>>>>>>
>>>>>>>> The instructions are dated March 2021, which is prior to the
>>>>>>>> release of NiFi 1.14.0. In particular, the run command is no longer
>>>>>>>> accurate with the default NiFi container image.
>>>>>>>>
>>>>>>>> The current Docker Hub instructions [1] show the basic command
>>>>>>>> needed
>>>>>>>>
>>>>>>>> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>>>>>>>>
>>>>>>>> In addition, any references to port 8080 in the AWS Security Group
>>>>>>>> rules should be changed to 8443. The security group rules for port 80 and
>>>>>>>> 18080 should be removed.
>>>>>>>>
>>>>>>>> The instructions that allow plain HTTP access to NiFi on port 8080
>>>>>>>> should NEVER be followed, as this exposes unfiltered and unauthenticated
>>>>>>>> access.
>>>>>>>>
>>>>>>>> Following those changes, it should be possible to access the NiFi
>>>>>>>> UI using the AWS URL:
>>>>>>>>
>>>>>>>> https://ec2...amazonaws.com:8443
>>>>>>>>
>>>>>>>> The default installation will generate a username and password,
>>>>>>>> which can be found in the container logs:
>>>>>>>>
>>>>>>>> docker logs nifi | grep Generated
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> David Handermann
>>>>>>>>
>>>>>>>> [1] https://hub.docker.com/r/apache/nifi
>>>>>>>>
>>>>>>>> On Tue, Nov 8, 2022 at 4:00 PM James McMahon <js...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi and thank you, David and Dmitry. In my case I was following
>>>>>>>>> this example,
>>>>>>>>>
>>>>>>>>> https://joeygoksu.com/software/apache-nifi-on-aws/
>>>>>>>>>
>>>>>>>>> which results in NiFi installed within a container. So to answer
>>>>>>>>> one of your questions, I don’t yet know how or where to find
>>>>>>>>> nifi.properties in the container framework. I don’t seem to have the usual
>>>>>>>>> /opt/nifi/….. directories on my ec2 instance. Any idea where I need to look
>>>>>>>>> for that?
>>>>>>>>>
>>>>>>>>> These ports are open by my security group Inbound Rules: 22 to
>>>>>>>>> MyIP, 80, 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to
>>>>>>>>> MyIP.
>>>>>>>>>
>>>>>>>>> I am able to Putty into my instance as ec2-user with my ppk file,
>>>>>>>>> which I created using putty tools from the original pem key pair. When I do
>>>>>>>>> putty in, under /opt I find three subdirectories: aws, containerd, and rh.
>>>>>>>>> Nothing nifi under any of the three that I can see so far.
>>>>>>>>>
>>>>>>>>> I start my docker instance with this command:
>>>>>>>>> docker run —name nifi -p 18080:8080 -d apache/nifi:latest
>>>>>>>>>
>>>>>>>>> I can do a ps -ef and see running nifi processes. But I don’t yet
>>>>>>>>> know how to get to the nifi logs or properties file.
>>>>>>>>>
>>>>>>>>> You mentioned using using localhost to get to the canvas UI. This
>>>>>>>>> confuses me. Nifi is running on my EC2 instance - a linux host without a
>>>>>>>>> browser. I’m in a browser on my laptop. How would localhost in my browser
>>>>>>>>> get me to my EC2 instance running nifi?
>>>>>>>>>
>>>>>>>>> This is the URL I’m using in my browser:
>>>>>>>>> http://ec2-3-238-27-220.compute-1.amazonaws.com
>>>>>>>>> (that url changes with each Stop/Start of my instance. I’ve yet to
>>>>>>>>> investigate how to get AWS to stop changing that IP, but I know it can be
>>>>>>>>> done).
>>>>>>>>>
>>>>>>>>> The browser replies with: ec2…….amazonaws refused to connect.
>>>>>>>>>
>>>>>>>>> I can ping my laptop IP address from the putty terminal where I am
>>>>>>>>> logged in to my instance. I cannot ping the Public DNS of my instance from
>>>>>>>>> Powershell on my laptop. Again, that Public DNS is
>>>>>>>>> ec2-3-238-27-220.compute-1.amazonaws.com
>>>>>>>>>
>>>>>>>>> Any help is much appreciated.
>>>>>>>>> Jim
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Nov 8, 2022 at 3:03 PM David Handermann <
>>>>>>>>> exceptionfactory@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Jim,
>>>>>>>>>>
>>>>>>>>>> NiFi 1.14.0 and following default to HTTPS on port 8443,
>>>>>>>>>> listening on the localhost address. The nifi.web.https.host can be changed
>>>>>>>>>> to blank in order to listen on all interfaces, but the default HTTPS
>>>>>>>>>> setting with authenticated required should be retained.
>>>>>>>>>>
>>>>>>>>>> Can you provide the version of NiFi and some additional details
>>>>>>>>>> on the nifi.web values from nifi.properties?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> David Handermann
>>>>>>>>>>
>>>>>>>>>> On Tue, Nov 8, 2022 at 1:54 PM James McMahon <
>>>>>>>>>> jsmcmahon3@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Has anyone successfully configured NiFi on AWS, and accessed it
>>>>>>>>>>> from a browser on a Windows desktop? I’ve tried following a few links to do
>>>>>>>>>>> this. I’ve verified that my instance security group allows access to 8080
>>>>>>>>>>> via its inbound rules. I’ve putty’ed into the instance via ssh port 22 to
>>>>>>>>>>> verify that there are no firewall restrictions. But still I get a message
>>>>>>>>>>> to the effect that the server rejected the connection request. Can anyone
>>>>>>>>>>> recommend a link that describes a success path for this?
>>>>>>>>>>> Thanks in advance for your help.
>>>>>>>>>>> Jim
>>>>>>>>>>>
>>>>>>>>>>
>>>>
Re: NiFi on AWS EC2
Posted by David Handermann <ex...@apache.org>.
Jim,
You're welcome! Thanks for following up and confirming the solution, great
collaborative effort!
Regard,
David Handermann
On Tue, Nov 8, 2022, 7:25 PM James McMahon <js...@gmail.com> wrote:
> That was it. Adding the port to the docker run command proxy got me to the
> promised land. I was then able to use the userid and password from the
> docker log to access nifi on my ec2 instance.
>
> David, Dmitry - thank you so much. This was a huge help to me, and I hope
> it will help others trying the same approach in the future.
> Jim
>
> On Tue, Nov 8, 2022 at 8:13 PM David Handermann <
> exceptionfactory@apache.org> wrote:
>
>> It may also be necessary to include the port in the host variable:
>>
>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=
>> ec2-3-238-27-220.compute-1.amazonaws.com:8443 -d apache/nifi:latest
>>
>> It is possible to access the configuration and logs files using an
>> interactive shell with the following Docker command:
>>
>> docker exec -it nifi /bin/bash
>>
>> Regards,
>> David Handermann
>>
>> On Tue, Nov 8, 2022 at 7:09 PM Dmitry Stepanov <dm...@dmitryds.com>
>> wrote:
>>
>>> Make sure you use your full domain name
>>> ec2-3-238-27-220.compute-1.amazonaws.com
>>> David shorten it in his code
>>>
>>> On November 8, 2022 5:57:26 p.m. James McMahon <js...@gmail.com>
>>> wrote:
>>>
>>>> Thank you, David. I’ve made that change, adding the proxy host
>>>> specification on the docker command line. I continue to get the same error
>>>> message. Is it possible I need to indicate my key on the docker command
>>>> line too?
>>>>
>>>> Related, how can one access nifi.properties and the usual nifi config
>>>> files, as well as the family of nifi-app.log files and bootstrap.conf, when
>>>> nifi is running inside a docker container?
>>>>
>>>> Thanks again for sticking with this. I feel like we’re getting closer.
>>>> Jim
>>>>
>>>> On Tue, Nov 8, 2022 at 7:31 PM David Handermann <
>>>> exceptionfactory@apache.org> wrote:
>>>>
>>>>> Hi Jim,
>>>>>
>>>>> Good adjustment on the security group inbound rules.
>>>>>
>>>>> The error page is the result of NiFi receiving an unexpected HTTP Host
>>>>> header, not matching one of the expected values.
>>>>>
>>>>> For this to work, it is possible to pass the external DNS name as the
>>>>> value of the NIFI_WEB_PROXY_HOST environment variable. This can be
>>>>> specified in the docker run command as follows:
>>>>>
>>>>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
>>>>> amazonaws.com -d apache/nifi:latest
>>>>>
>>>>> That will allow NiFi to accept the Host header from the browser, and
>>>>> then present the login screen.
>>>>>
>>>>> Regards,
>>>>> David Handermann
>>>>>
>>>>> On Tue, Nov 8, 2022 at 6:06 PM James McMahon <js...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi David. This is very helpful, thank you. I feel like I am close,
>>>>>> but I get an error. My Inbound Rules for my security group now include:
>>>>>> 8443 TCP (MyIP)/32
>>>>>> 443 TCP (MyIP)/32
>>>>>> 22 TCP (MyIP)/32
>>>>>>
>>>>>> In my browser - I tried both Edge and Chrome - I use this
>>>>>> URL:
>>>>>> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
>>>>>> I have also tried with /nifi at the tail end.
>>>>>>
>>>>>> I get this error:
>>>>>>
>>>>>> *System Error*
>>>>>>
>>>>>> *The request contained an invalid host header
>>>>>> [ec2-3-238-27-220.compute-1.amazonaws.com:8443
>>>>>> <http://ec2-3-238-27-220.compute-1.amazonaws.com:8443/>] in the request
>>>>>> [/]. Check for request manipulation or third-party intercept.*
>>>>>>
>>>>>> *Valid host headers are [empty] or:*
>>>>>>
>>>>>> - *127.0.0.1*
>>>>>> - *127.0.0.1:8443 <http://127.0.0.1:8443/>*
>>>>>> - *localhost*
>>>>>> - *localhost:8443*
>>>>>> - *[::1]*
>>>>>> - *[::1]:8443*
>>>>>> - *7f661ae687d7*
>>>>>> - *7f661ae687d7:8443*
>>>>>> - *172.17.0.2*
>>>>>> - *172.17.0.2:8443 <http://172.17.0.2:8443/>*
>>>>>>
>>>>>>
>>>>>> Does this mean I have formed the URL incorrectly?
>>>>>>
>>>>>> I also see that I had to add an exception to permit https. When I
>>>>>> created the instance, I created my own pem key pair. It is not signed by
>>>>>> any CA. For a self-signed key pair like this, do I need to install a key in
>>>>>> my browser security store to avoid adding that exception?
>>>>>>
>>>>>> Thank you for helping me get that much closer.
>>>>>> Jim
>>>>>>
>>>>>> On Tue, Nov 8, 2022 at 5:13 PM David Handermann <
>>>>>> exceptionfactory@apache.org> wrote:
>>>>>>
>>>>>>> Hi Jim,
>>>>>>>
>>>>>>> Thanks for the reply and additional background.
>>>>>>>
>>>>>>> The instructions are dated March 2021, which is prior to the release
>>>>>>> of NiFi 1.14.0. In particular, the run command is no longer accurate with
>>>>>>> the default NiFi container image.
>>>>>>>
>>>>>>> The current Docker Hub instructions [1] show the basic command needed
>>>>>>>
>>>>>>> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>>>>>>>
>>>>>>> In addition, any references to port 8080 in the AWS Security Group
>>>>>>> rules should be changed to 8443. The security group rules for port 80 and
>>>>>>> 18080 should be removed.
>>>>>>>
>>>>>>> The instructions that allow plain HTTP access to NiFi on port 8080
>>>>>>> should NEVER be followed, as this exposes unfiltered and unauthenticated
>>>>>>> access.
>>>>>>>
>>>>>>> Following those changes, it should be possible to access the NiFi UI
>>>>>>> using the AWS URL:
>>>>>>>
>>>>>>> https://ec2...amazonaws.com:8443
>>>>>>>
>>>>>>> The default installation will generate a username and password,
>>>>>>> which can be found in the container logs:
>>>>>>>
>>>>>>> docker logs nifi | grep Generated
>>>>>>>
>>>>>>> Regards,
>>>>>>> David Handermann
>>>>>>>
>>>>>>> [1] https://hub.docker.com/r/apache/nifi
>>>>>>>
>>>>>>> On Tue, Nov 8, 2022 at 4:00 PM James McMahon <js...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi and thank you, David and Dmitry. In my case I was following this
>>>>>>>> example,
>>>>>>>>
>>>>>>>> https://joeygoksu.com/software/apache-nifi-on-aws/
>>>>>>>>
>>>>>>>> which results in NiFi installed within a container. So to answer
>>>>>>>> one of your questions, I don’t yet know how or where to find
>>>>>>>> nifi.properties in the container framework. I don’t seem to have the usual
>>>>>>>> /opt/nifi/….. directories on my ec2 instance. Any idea where I need to look
>>>>>>>> for that?
>>>>>>>>
>>>>>>>> These ports are open by my security group Inbound Rules: 22 to
>>>>>>>> MyIP, 80, 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.
>>>>>>>>
>>>>>>>> I am able to Putty into my instance as ec2-user with my ppk file,
>>>>>>>> which I created using putty tools from the original pem key pair. When I do
>>>>>>>> putty in, under /opt I find three subdirectories: aws, containerd, and rh.
>>>>>>>> Nothing nifi under any of the three that I can see so far.
>>>>>>>>
>>>>>>>> I start my docker instance with this command:
>>>>>>>> docker run —name nifi -p 18080:8080 -d apache/nifi:latest
>>>>>>>>
>>>>>>>> I can do a ps -ef and see running nifi processes. But I don’t yet
>>>>>>>> know how to get to the nifi logs or properties file.
>>>>>>>>
>>>>>>>> You mentioned using using localhost to get to the canvas UI. This
>>>>>>>> confuses me. Nifi is running on my EC2 instance - a linux host without a
>>>>>>>> browser. I’m in a browser on my laptop. How would localhost in my browser
>>>>>>>> get me to my EC2 instance running nifi?
>>>>>>>>
>>>>>>>> This is the URL I’m using in my browser:
>>>>>>>> http://ec2-3-238-27-220.compute-1.amazonaws.com
>>>>>>>> (that url changes with each Stop/Start of my instance. I’ve yet to
>>>>>>>> investigate how to get AWS to stop changing that IP, but I know it can be
>>>>>>>> done).
>>>>>>>>
>>>>>>>> The browser replies with: ec2…….amazonaws refused to connect.
>>>>>>>>
>>>>>>>> I can ping my laptop IP address from the putty terminal where I am
>>>>>>>> logged in to my instance. I cannot ping the Public DNS of my instance from
>>>>>>>> Powershell on my laptop. Again, that Public DNS is
>>>>>>>> ec2-3-238-27-220.compute-1.amazonaws.com
>>>>>>>>
>>>>>>>> Any help is much appreciated.
>>>>>>>> Jim
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Nov 8, 2022 at 3:03 PM David Handermann <
>>>>>>>> exceptionfactory@apache.org> wrote:
>>>>>>>>
>>>>>>>>> Hi Jim,
>>>>>>>>>
>>>>>>>>> NiFi 1.14.0 and following default to HTTPS on port 8443, listening
>>>>>>>>> on the localhost address. The nifi.web.https.host can be changed to blank
>>>>>>>>> in order to listen on all interfaces, but the default HTTPS setting with
>>>>>>>>> authenticated required should be retained.
>>>>>>>>>
>>>>>>>>> Can you provide the version of NiFi and some additional details on
>>>>>>>>> the nifi.web values from nifi.properties?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> David Handermann
>>>>>>>>>
>>>>>>>>> On Tue, Nov 8, 2022 at 1:54 PM James McMahon <js...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Has anyone successfully configured NiFi on AWS, and accessed it
>>>>>>>>>> from a browser on a Windows desktop? I’ve tried following a few links to do
>>>>>>>>>> this. I’ve verified that my instance security group allows access to 8080
>>>>>>>>>> via its inbound rules. I’ve putty’ed into the instance via ssh port 22 to
>>>>>>>>>> verify that there are no firewall restrictions. But still I get a message
>>>>>>>>>> to the effect that the server rejected the connection request. Can anyone
>>>>>>>>>> recommend a link that describes a success path for this?
>>>>>>>>>> Thanks in advance for your help.
>>>>>>>>>> Jim
>>>>>>>>>>
>>>>>>>>>
>>>
Re: NiFi on AWS EC2
Posted by James McMahon <js...@gmail.com>.
That was it. Adding the port to the docker run command proxy got me to the
promised land. I was then able to use the userid and password from the
docker log to access nifi on my ec2 instance.
David, Dmitry - thank you so much. This was a huge help to me, and I hope
it will help others trying the same approach in the future.
Jim
On Tue, Nov 8, 2022 at 8:13 PM David Handermann <ex...@apache.org>
wrote:
> It may also be necessary to include the port in the host variable:
>
> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=
> ec2-3-238-27-220.compute-1.amazonaws.com:8443 -d apache/nifi:latest
>
> It is possible to access the configuration and logs files using an
> interactive shell with the following Docker command:
>
> docker exec -it nifi /bin/bash
>
> Regards,
> David Handermann
>
> On Tue, Nov 8, 2022 at 7:09 PM Dmitry Stepanov <dm...@dmitryds.com>
> wrote:
>
>> Make sure you use your full domain name
>> ec2-3-238-27-220.compute-1.amazonaws.com
>> David shorten it in his code
>>
>> On November 8, 2022 5:57:26 p.m. James McMahon <js...@gmail.com>
>> wrote:
>>
>>> Thank you, David. I’ve made that change, adding the proxy host
>>> specification on the docker command line. I continue to get the same error
>>> message. Is it possible I need to indicate my key on the docker command
>>> line too?
>>>
>>> Related, how can one access nifi.properties and the usual nifi config
>>> files, as well as the family of nifi-app.log files and bootstrap.conf, when
>>> nifi is running inside a docker container?
>>>
>>> Thanks again for sticking with this. I feel like we’re getting closer.
>>> Jim
>>>
>>> On Tue, Nov 8, 2022 at 7:31 PM David Handermann <
>>> exceptionfactory@apache.org> wrote:
>>>
>>>> Hi Jim,
>>>>
>>>> Good adjustment on the security group inbound rules.
>>>>
>>>> The error page is the result of NiFi receiving an unexpected HTTP Host
>>>> header, not matching one of the expected values.
>>>>
>>>> For this to work, it is possible to pass the external DNS name as the
>>>> value of the NIFI_WEB_PROXY_HOST environment variable. This can be
>>>> specified in the docker run command as follows:
>>>>
>>>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
>>>> amazonaws.com -d apache/nifi:latest
>>>>
>>>> That will allow NiFi to accept the Host header from the browser, and
>>>> then present the login screen.
>>>>
>>>> Regards,
>>>> David Handermann
>>>>
>>>> On Tue, Nov 8, 2022 at 6:06 PM James McMahon <js...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi David. This is very helpful, thank you. I feel like I am close, but
>>>>> I get an error. My Inbound Rules for my security group now include:
>>>>> 8443 TCP (MyIP)/32
>>>>> 443 TCP (MyIP)/32
>>>>> 22 TCP (MyIP)/32
>>>>>
>>>>> In my browser - I tried both Edge and Chrome - I use this
>>>>> URL:
>>>>> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
>>>>> I have also tried with /nifi at the tail end.
>>>>>
>>>>> I get this error:
>>>>>
>>>>> *System Error*
>>>>>
>>>>> *The request contained an invalid host header
>>>>> [ec2-3-238-27-220.compute-1.amazonaws.com:8443
>>>>> <http://ec2-3-238-27-220.compute-1.amazonaws.com:8443/>] in the request
>>>>> [/]. Check for request manipulation or third-party intercept.*
>>>>>
>>>>> *Valid host headers are [empty] or:*
>>>>>
>>>>> - *127.0.0.1*
>>>>> - *127.0.0.1:8443 <http://127.0.0.1:8443/>*
>>>>> - *localhost*
>>>>> - *localhost:8443*
>>>>> - *[::1]*
>>>>> - *[::1]:8443*
>>>>> - *7f661ae687d7*
>>>>> - *7f661ae687d7:8443*
>>>>> - *172.17.0.2*
>>>>> - *172.17.0.2:8443 <http://172.17.0.2:8443/>*
>>>>>
>>>>>
>>>>> Does this mean I have formed the URL incorrectly?
>>>>>
>>>>> I also see that I had to add an exception to permit https. When I
>>>>> created the instance, I created my own pem key pair. It is not signed by
>>>>> any CA. For a self-signed key pair like this, do I need to install a key in
>>>>> my browser security store to avoid adding that exception?
>>>>>
>>>>> Thank you for helping me get that much closer.
>>>>> Jim
>>>>>
>>>>> On Tue, Nov 8, 2022 at 5:13 PM David Handermann <
>>>>> exceptionfactory@apache.org> wrote:
>>>>>
>>>>>> Hi Jim,
>>>>>>
>>>>>> Thanks for the reply and additional background.
>>>>>>
>>>>>> The instructions are dated March 2021, which is prior to the release
>>>>>> of NiFi 1.14.0. In particular, the run command is no longer accurate with
>>>>>> the default NiFi container image.
>>>>>>
>>>>>> The current Docker Hub instructions [1] show the basic command needed
>>>>>>
>>>>>> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>>>>>>
>>>>>> In addition, any references to port 8080 in the AWS Security Group
>>>>>> rules should be changed to 8443. The security group rules for port 80 and
>>>>>> 18080 should be removed.
>>>>>>
>>>>>> The instructions that allow plain HTTP access to NiFi on port 8080
>>>>>> should NEVER be followed, as this exposes unfiltered and unauthenticated
>>>>>> access.
>>>>>>
>>>>>> Following those changes, it should be possible to access the NiFi UI
>>>>>> using the AWS URL:
>>>>>>
>>>>>> https://ec2...amazonaws.com:8443
>>>>>>
>>>>>> The default installation will generate a username and password, which
>>>>>> can be found in the container logs:
>>>>>>
>>>>>> docker logs nifi | grep Generated
>>>>>>
>>>>>> Regards,
>>>>>> David Handermann
>>>>>>
>>>>>> [1] https://hub.docker.com/r/apache/nifi
>>>>>>
>>>>>> On Tue, Nov 8, 2022 at 4:00 PM James McMahon <js...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi and thank you, David and Dmitry. In my case I was following this
>>>>>>> example,
>>>>>>>
>>>>>>> https://joeygoksu.com/software/apache-nifi-on-aws/
>>>>>>>
>>>>>>> which results in NiFi installed within a container. So to answer one
>>>>>>> of your questions, I don’t yet know how or where to find nifi.properties in
>>>>>>> the container framework. I don’t seem to have the usual /opt/nifi/…..
>>>>>>> directories on my ec2 instance. Any idea where I need to look for that?
>>>>>>>
>>>>>>> These ports are open by my security group Inbound Rules: 22 to MyIP,
>>>>>>> 80, 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.
>>>>>>>
>>>>>>> I am able to Putty into my instance as ec2-user with my ppk file,
>>>>>>> which I created using putty tools from the original pem key pair. When I do
>>>>>>> putty in, under /opt I find three subdirectories: aws, containerd, and rh.
>>>>>>> Nothing nifi under any of the three that I can see so far.
>>>>>>>
>>>>>>> I start my docker instance with this command:
>>>>>>> docker run —name nifi -p 18080:8080 -d apache/nifi:latest
>>>>>>>
>>>>>>> I can do a ps -ef and see running nifi processes. But I don’t yet
>>>>>>> know how to get to the nifi logs or properties file.
>>>>>>>
>>>>>>> You mentioned using using localhost to get to the canvas UI. This
>>>>>>> confuses me. Nifi is running on my EC2 instance - a linux host without a
>>>>>>> browser. I’m in a browser on my laptop. How would localhost in my browser
>>>>>>> get me to my EC2 instance running nifi?
>>>>>>>
>>>>>>> This is the URL I’m using in my browser:
>>>>>>> http://ec2-3-238-27-220.compute-1.amazonaws.com
>>>>>>> (that url changes with each Stop/Start of my instance. I’ve yet to
>>>>>>> investigate how to get AWS to stop changing that IP, but I know it can be
>>>>>>> done).
>>>>>>>
>>>>>>> The browser replies with: ec2…….amazonaws refused to connect.
>>>>>>>
>>>>>>> I can ping my laptop IP address from the putty terminal where I am
>>>>>>> logged in to my instance. I cannot ping the Public DNS of my instance from
>>>>>>> Powershell on my laptop. Again, that Public DNS is
>>>>>>> ec2-3-238-27-220.compute-1.amazonaws.com
>>>>>>>
>>>>>>> Any help is much appreciated.
>>>>>>> Jim
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Nov 8, 2022 at 3:03 PM David Handermann <
>>>>>>> exceptionfactory@apache.org> wrote:
>>>>>>>
>>>>>>>> Hi Jim,
>>>>>>>>
>>>>>>>> NiFi 1.14.0 and following default to HTTPS on port 8443, listening
>>>>>>>> on the localhost address. The nifi.web.https.host can be changed to blank
>>>>>>>> in order to listen on all interfaces, but the default HTTPS setting with
>>>>>>>> authenticated required should be retained.
>>>>>>>>
>>>>>>>> Can you provide the version of NiFi and some additional details on
>>>>>>>> the nifi.web values from nifi.properties?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> David Handermann
>>>>>>>>
>>>>>>>> On Tue, Nov 8, 2022 at 1:54 PM James McMahon <js...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Has anyone successfully configured NiFi on AWS, and accessed it
>>>>>>>>> from a browser on a Windows desktop? I’ve tried following a few links to do
>>>>>>>>> this. I’ve verified that my instance security group allows access to 8080
>>>>>>>>> via its inbound rules. I’ve putty’ed into the instance via ssh port 22 to
>>>>>>>>> verify that there are no firewall restrictions. But still I get a message
>>>>>>>>> to the effect that the server rejected the connection request. Can anyone
>>>>>>>>> recommend a link that describes a success path for this?
>>>>>>>>> Thanks in advance for your help.
>>>>>>>>> Jim
>>>>>>>>>
>>>>>>>>
>>
Re: NiFi on AWS EC2
Posted by David Handermann <ex...@apache.org>.
It may also be necessary to include the port in the host variable:
docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=
ec2-3-238-27-220.compute-1.amazonaws.com:8443 -d apache/nifi:latest
It is possible to access the configuration and logs files using an
interactive shell with the following Docker command:
docker exec -it nifi /bin/bash
Regards,
David Handermann
On Tue, Nov 8, 2022 at 7:09 PM Dmitry Stepanov <dm...@dmitryds.com> wrote:
> Make sure you use your full domain name
> ec2-3-238-27-220.compute-1.amazonaws.com
> David shorten it in his code
>
> On November 8, 2022 5:57:26 p.m. James McMahon <js...@gmail.com>
> wrote:
>
>> Thank you, David. I’ve made that change, adding the proxy host
>> specification on the docker command line. I continue to get the same error
>> message. Is it possible I need to indicate my key on the docker command
>> line too?
>>
>> Related, how can one access nifi.properties and the usual nifi config
>> files, as well as the family of nifi-app.log files and bootstrap.conf, when
>> nifi is running inside a docker container?
>>
>> Thanks again for sticking with this. I feel like we’re getting closer.
>> Jim
>>
>> On Tue, Nov 8, 2022 at 7:31 PM David Handermann <
>> exceptionfactory@apache.org> wrote:
>>
>>> Hi Jim,
>>>
>>> Good adjustment on the security group inbound rules.
>>>
>>> The error page is the result of NiFi receiving an unexpected HTTP Host
>>> header, not matching one of the expected values.
>>>
>>> For this to work, it is possible to pass the external DNS name as the
>>> value of the NIFI_WEB_PROXY_HOST environment variable. This can be
>>> specified in the docker run command as follows:
>>>
>>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
>>> amazonaws.com -d apache/nifi:latest
>>>
>>> That will allow NiFi to accept the Host header from the browser, and
>>> then present the login screen.
>>>
>>> Regards,
>>> David Handermann
>>>
>>> On Tue, Nov 8, 2022 at 6:06 PM James McMahon <js...@gmail.com>
>>> wrote:
>>>
>>>> Hi David. This is very helpful, thank you. I feel like I am close, but
>>>> I get an error. My Inbound Rules for my security group now include:
>>>> 8443 TCP (MyIP)/32
>>>> 443 TCP (MyIP)/32
>>>> 22 TCP (MyIP)/32
>>>>
>>>> In my browser - I tried both Edge and Chrome - I use this
>>>> URL:
>>>> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
>>>> I have also tried with /nifi at the tail end.
>>>>
>>>> I get this error:
>>>>
>>>> *System Error*
>>>>
>>>> *The request contained an invalid host header
>>>> [ec2-3-238-27-220.compute-1.amazonaws.com:8443
>>>> <http://ec2-3-238-27-220.compute-1.amazonaws.com:8443/>] in the request
>>>> [/]. Check for request manipulation or third-party intercept.*
>>>>
>>>> *Valid host headers are [empty] or:*
>>>>
>>>> - *127.0.0.1*
>>>> - *127.0.0.1:8443 <http://127.0.0.1:8443/>*
>>>> - *localhost*
>>>> - *localhost:8443*
>>>> - *[::1]*
>>>> - *[::1]:8443*
>>>> - *7f661ae687d7*
>>>> - *7f661ae687d7:8443*
>>>> - *172.17.0.2*
>>>> - *172.17.0.2:8443 <http://172.17.0.2:8443/>*
>>>>
>>>>
>>>> Does this mean I have formed the URL incorrectly?
>>>>
>>>> I also see that I had to add an exception to permit https. When I
>>>> created the instance, I created my own pem key pair. It is not signed by
>>>> any CA. For a self-signed key pair like this, do I need to install a key in
>>>> my browser security store to avoid adding that exception?
>>>>
>>>> Thank you for helping me get that much closer.
>>>> Jim
>>>>
>>>> On Tue, Nov 8, 2022 at 5:13 PM David Handermann <
>>>> exceptionfactory@apache.org> wrote:
>>>>
>>>>> Hi Jim,
>>>>>
>>>>> Thanks for the reply and additional background.
>>>>>
>>>>> The instructions are dated March 2021, which is prior to the release
>>>>> of NiFi 1.14.0. In particular, the run command is no longer accurate with
>>>>> the default NiFi container image.
>>>>>
>>>>> The current Docker Hub instructions [1] show the basic command needed
>>>>>
>>>>> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>>>>>
>>>>> In addition, any references to port 8080 in the AWS Security Group
>>>>> rules should be changed to 8443. The security group rules for port 80 and
>>>>> 18080 should be removed.
>>>>>
>>>>> The instructions that allow plain HTTP access to NiFi on port 8080
>>>>> should NEVER be followed, as this exposes unfiltered and unauthenticated
>>>>> access.
>>>>>
>>>>> Following those changes, it should be possible to access the NiFi UI
>>>>> using the AWS URL:
>>>>>
>>>>> https://ec2...amazonaws.com:8443
>>>>>
>>>>> The default installation will generate a username and password, which
>>>>> can be found in the container logs:
>>>>>
>>>>> docker logs nifi | grep Generated
>>>>>
>>>>> Regards,
>>>>> David Handermann
>>>>>
>>>>> [1] https://hub.docker.com/r/apache/nifi
>>>>>
>>>>> On Tue, Nov 8, 2022 at 4:00 PM James McMahon <js...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi and thank you, David and Dmitry. In my case I was following this
>>>>>> example,
>>>>>>
>>>>>> https://joeygoksu.com/software/apache-nifi-on-aws/
>>>>>>
>>>>>> which results in NiFi installed within a container. So to answer one
>>>>>> of your questions, I don’t yet know how or where to find nifi.properties in
>>>>>> the container framework. I don’t seem to have the usual /opt/nifi/…..
>>>>>> directories on my ec2 instance. Any idea where I need to look for that?
>>>>>>
>>>>>> These ports are open by my security group Inbound Rules: 22 to MyIP,
>>>>>> 80, 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.
>>>>>>
>>>>>> I am able to Putty into my instance as ec2-user with my ppk file,
>>>>>> which I created using putty tools from the original pem key pair. When I do
>>>>>> putty in, under /opt I find three subdirectories: aws, containerd, and rh.
>>>>>> Nothing nifi under any of the three that I can see so far.
>>>>>>
>>>>>> I start my docker instance with this command:
>>>>>> docker run —name nifi -p 18080:8080 -d apache/nifi:latest
>>>>>>
>>>>>> I can do a ps -ef and see running nifi processes. But I don’t yet
>>>>>> know how to get to the nifi logs or properties file.
>>>>>>
>>>>>> You mentioned using using localhost to get to the canvas UI. This
>>>>>> confuses me. Nifi is running on my EC2 instance - a linux host without a
>>>>>> browser. I’m in a browser on my laptop. How would localhost in my browser
>>>>>> get me to my EC2 instance running nifi?
>>>>>>
>>>>>> This is the URL I’m using in my browser:
>>>>>> http://ec2-3-238-27-220.compute-1.amazonaws.com
>>>>>> (that url changes with each Stop/Start of my instance. I’ve yet to
>>>>>> investigate how to get AWS to stop changing that IP, but I know it can be
>>>>>> done).
>>>>>>
>>>>>> The browser replies with: ec2…….amazonaws refused to connect.
>>>>>>
>>>>>> I can ping my laptop IP address from the putty terminal where I am
>>>>>> logged in to my instance. I cannot ping the Public DNS of my instance from
>>>>>> Powershell on my laptop. Again, that Public DNS is
>>>>>> ec2-3-238-27-220.compute-1.amazonaws.com
>>>>>>
>>>>>> Any help is much appreciated.
>>>>>> Jim
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 8, 2022 at 3:03 PM David Handermann <
>>>>>> exceptionfactory@apache.org> wrote:
>>>>>>
>>>>>>> Hi Jim,
>>>>>>>
>>>>>>> NiFi 1.14.0 and following default to HTTPS on port 8443, listening
>>>>>>> on the localhost address. The nifi.web.https.host can be changed to blank
>>>>>>> in order to listen on all interfaces, but the default HTTPS setting with
>>>>>>> authenticated required should be retained.
>>>>>>>
>>>>>>> Can you provide the version of NiFi and some additional details on
>>>>>>> the nifi.web values from nifi.properties?
>>>>>>>
>>>>>>> Regards,
>>>>>>> David Handermann
>>>>>>>
>>>>>>> On Tue, Nov 8, 2022 at 1:54 PM James McMahon <js...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Has anyone successfully configured NiFi on AWS, and accessed it
>>>>>>>> from a browser on a Windows desktop? I’ve tried following a few links to do
>>>>>>>> this. I’ve verified that my instance security group allows access to 8080
>>>>>>>> via its inbound rules. I’ve putty’ed into the instance via ssh port 22 to
>>>>>>>> verify that there are no firewall restrictions. But still I get a message
>>>>>>>> to the effect that the server rejected the connection request. Can anyone
>>>>>>>> recommend a link that describes a success path for this?
>>>>>>>> Thanks in advance for your help.
>>>>>>>> Jim
>>>>>>>>
>>>>>>>
>
Re: NiFi on AWS EC2
Posted by Dmitry Stepanov <dm...@dmitryds.com>.
Make sure you use your full domain name
ec2-3-238-27-220.compute-1.amazonaws.com
David shorten it in his code
On November 8, 2022 5:57:26 p.m. James McMahon <js...@gmail.com> wrote:
> Thank you, David. I’ve made that change, adding the proxy host
> specification on the docker command line. I continue to get the same error
> message. Is it possible I need to indicate my key on the docker command
> line too?
>
> Related, how can one access nifi.properties and the usual nifi config
> files, as well as the family of nifi-app.log files and bootstrap.conf, when
> nifi is running inside a docker container?
>
> Thanks again for sticking with this. I feel like we’re getting closer.
> Jim
>
> On Tue, Nov 8, 2022 at 7:31 PM David Handermann
> <ex...@apache.org> wrote:
> Hi Jim,
>
> Good adjustment on the security group inbound rules.
>
> The error page is the result of NiFi receiving an unexpected HTTP Host
> header, not matching one of the expected values.
>
> For this to work, it is possible to pass the external DNS name as the value
> of the NIFI_WEB_PROXY_HOST environment variable. This can be specified in
> the docker run command as follows:
>
> docker run --name nifi -p 8443:8443 -e
> NIFI_WEB_PROXY_HOST=ec2...amazonaws.com -d apache/nifi:latest
>
> That will allow NiFi to accept the Host header from the browser, and then
> present the login screen.
>
> Regards,
> David Handermann
>
> On Tue, Nov 8, 2022 at 6:06 PM James McMahon <js...@gmail.com> wrote:
> Hi David. This is very helpful, thank you. I feel like I am close, but I
> get an error. My Inbound Rules for my security group now include:
> 8443 TCP (MyIP)/32
> 443 TCP (MyIP)/32
> 22 TCP (MyIP)/32
>
> In my browser - I tried both Edge and Chrome - I use this
> URL:
> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
> I have also tried with /nifi at the tail end.
>
> I get this error:
> System Error
> The request contained an invalid host header
> [ec2-3-238-27-220.compute-1.amazonaws.com:8443] in the request [/]. Check
> for request manipulation or third-party intercept.
> Valid host headers are [empty] or:
> 127.0.0.1
> 127.0.0.1:8443
> localhost
> localhost:8443
> [::1]
> [::1]:8443
> 7f661ae687d7
> 7f661ae687d7:8443
> 172.17.0.2
> 172.17.0.2:8443
>
>
> Does this mean I have formed the URL incorrectly?
>
>
> I also see that I had to add an exception to permit https. When I created
> the instance, I created my own pem key pair. It is not signed by any CA.
> For a self-signed key pair like this, do I need to install a key in my
> browser security store to avoid adding that exception?
>
>
> Thank you for helping me get that much closer.
> Jim
>
> On Tue, Nov 8, 2022 at 5:13 PM David Handermann
> <ex...@apache.org> wrote:
> Hi Jim,
>
> Thanks for the reply and additional background.
>
> The instructions are dated March 2021, which is prior to the release of
> NiFi 1.14.0. In particular, the run command is no longer accurate with the
> default NiFi container image.
>
> The current Docker Hub instructions [1] show the basic command needed
>
> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>
> In addition, any references to port 8080 in the AWS Security Group rules
> should be changed to 8443. The security group rules for port 80 and 18080
> should be removed.
>
> The instructions that allow plain HTTP access to NiFi on port 8080 should
> NEVER be followed, as this exposes unfiltered and unauthenticated access.
>
> Following those changes, it should be possible to access the NiFi UI using
> the AWS URL:
>
> https://ec2...amazonaws.com:8443
>
> The default installation will generate a username and password, which can
> be found in the container logs:
>
> docker logs nifi | grep Generated
>
> Regards,
> David Handermann
>
> [1] https://hub.docker.com/r/apache/nifi
>
> On Tue, Nov 8, 2022 at 4:00 PM James McMahon <js...@gmail.com> wrote:
> Hi and thank you, David and Dmitry. In my case I was following this example,
>
> https://joeygoksu.com/software/apache-nifi-on-aws/
>
> which results in NiFi installed within a container. So to answer one of
> your questions, I don’t yet know how or where to find nifi.properties in
> the container framework. I don’t seem to have the usual /opt/nifi/…..
> directories on my ec2 instance. Any idea where I need to look for that?
>
> These ports are open by my security group Inbound Rules: 22 to MyIP, 80,
> 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.
>
> I am able to Putty into my instance as ec2-user with my ppk file, which I
> created using putty tools from the original pem key pair. When I do putty
> in, under /opt I find three subdirectories: aws, containerd, and rh.
> Nothing nifi under any of the three that I can see so far.
>
> I start my docker instance with this command:
> docker run —name nifi -p 18080:8080 -d apache/nifi:latest
>
> I can do a ps -ef and see running nifi processes. But I don’t yet know how
> to get to the nifi logs or properties file.
>
> You mentioned using using localhost to get to the canvas UI. This confuses
> me. Nifi is running on my EC2 instance - a linux host without a browser.
> I’m in a browser on my laptop. How would localhost in my browser get me to
> my EC2 instance running nifi?
>
> This is the URL I’m using in my browser:
> http://ec2-3-238-27-220.compute-1.amazonaws.com
> (that url changes with each Stop/Start of my instance. I’ve yet to
> investigate how to get AWS to stop changing that IP, but I know it can be
> done).
>
> The browser replies with: ec2…….amazonaws refused to connect.
>
> I can ping my laptop IP address from the putty terminal where I am logged
> in to my instance. I cannot ping the Public DNS of my instance from
> Powershell on my laptop. Again, that Public DNS is
> ec2-3-238-27-220.compute-1.amazonaws.com
>
> Any help is much appreciated.
> Jim
>
>
>
> On Tue, Nov 8, 2022 at 3:03 PM David Handermann
> <ex...@apache.org> wrote:
> Hi Jim,
>
> NiFi 1.14.0 and following default to HTTPS on port 8443, listening on the
> localhost address. The nifi.web.https.host can be changed to blank in order
> to listen on all interfaces, but the default HTTPS setting with
> authenticated required should be retained.
>
> Can you provide the version of NiFi and some additional details on the
> nifi.web values from nifi.properties?
>
> Regards,
> David Handermann
>
> On Tue, Nov 8, 2022 at 1:54 PM James McMahon <js...@gmail.com> wrote:
> Has anyone successfully configured NiFi on AWS, and accessed it from a
> browser on a Windows desktop? I’ve tried following a few links to do this.
> I’ve verified that my instance security group allows access to 8080 via its
> inbound rules. I’ve putty’ed into the instance via ssh port 22 to verify
> that there are no firewall restrictions. But still I get a message to the
> effect that the server rejected the connection request. Can anyone
> recommend a link that describes a success path for this?
> Thanks in advance for your help.
>
> Jim
Re: NiFi on AWS EC2
Posted by James McMahon <js...@gmail.com>.
Thank you, David. I’ve made that change, adding the proxy host
specification on the docker command line. I continue to get the same error
message. Is it possible I need to indicate my key on the docker command
line too?
Related, how can one access nifi.properties and the usual nifi config
files, as well as the family of nifi-app.log files and bootstrap.conf, when
nifi is running inside a docker container?
Thanks again for sticking with this. I feel like we’re getting closer.
Jim
On Tue, Nov 8, 2022 at 7:31 PM David Handermann <ex...@apache.org>
wrote:
> Hi Jim,
>
> Good adjustment on the security group inbound rules.
>
> The error page is the result of NiFi receiving an unexpected HTTP Host
> header, not matching one of the expected values.
>
> For this to work, it is possible to pass the external DNS name as the
> value of the NIFI_WEB_PROXY_HOST environment variable. This can be
> specified in the docker run command as follows:
>
> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
> amazonaws.com -d apache/nifi:latest
>
> That will allow NiFi to accept the Host header from the browser, and then
> present the login screen.
>
> Regards,
> David Handermann
>
> On Tue, Nov 8, 2022 at 6:06 PM James McMahon <js...@gmail.com> wrote:
>
>> Hi David. This is very helpful, thank you. I feel like I am close, but I
>> get an error. My Inbound Rules for my security group now include:
>> 8443 TCP (MyIP)/32
>> 443 TCP (MyIP)/32
>> 22 TCP (MyIP)/32
>>
>> In my browser - I tried both Edge and Chrome - I use this
>> URL:
>> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
>> I have also tried with /nifi at the tail end.
>>
>> I get this error:
>>
>> *System Error*
>>
>> *The request contained an invalid host header
>> [ec2-3-238-27-220.compute-1.amazonaws.com:8443
>> <http://ec2-3-238-27-220.compute-1.amazonaws.com:8443/>] in the request
>> [/]. Check for request manipulation or third-party intercept.*
>>
>> *Valid host headers are [empty] or:*
>>
>> - *127.0.0.1*
>> - *127.0.0.1:8443 <http://127.0.0.1:8443/>*
>> - *localhost*
>> - *localhost:8443*
>> - *[::1]*
>> - *[::1]:8443*
>> - *7f661ae687d7*
>> - *7f661ae687d7:8443*
>> - *172.17.0.2*
>> - *172.17.0.2:8443 <http://172.17.0.2:8443/>*
>>
>>
>> Does this mean I have formed the URL incorrectly?
>>
>> I also see that I had to add an exception to permit https. When I created
>> the instance, I created my own pem key pair. It is not signed by any CA.
>> For a self-signed key pair like this, do I need to install a key in my
>> browser security store to avoid adding that exception?
>>
>> Thank you for helping me get that much closer.
>> Jim
>>
>> On Tue, Nov 8, 2022 at 5:13 PM David Handermann <
>> exceptionfactory@apache.org> wrote:
>>
>>> Hi Jim,
>>>
>>> Thanks for the reply and additional background.
>>>
>>> The instructions are dated March 2021, which is prior to the release of
>>> NiFi 1.14.0. In particular, the run command is no longer accurate with the
>>> default NiFi container image.
>>>
>>> The current Docker Hub instructions [1] show the basic command needed
>>>
>>> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>>>
>>> In addition, any references to port 8080 in the AWS Security Group rules
>>> should be changed to 8443. The security group rules for port 80 and 18080
>>> should be removed.
>>>
>>> The instructions that allow plain HTTP access to NiFi on port 8080
>>> should NEVER be followed, as this exposes unfiltered and unauthenticated
>>> access.
>>>
>>> Following those changes, it should be possible to access the NiFi UI
>>> using the AWS URL:
>>>
>>> https://ec2...amazonaws.com:8443
>>>
>>> The default installation will generate a username and password, which
>>> can be found in the container logs:
>>>
>>> docker logs nifi | grep Generated
>>>
>>> Regards,
>>> David Handermann
>>>
>>> [1] https://hub.docker.com/r/apache/nifi
>>>
>>> On Tue, Nov 8, 2022 at 4:00 PM James McMahon <js...@gmail.com>
>>> wrote:
>>>
>>>> Hi and thank you, David and Dmitry. In my case I was following this
>>>> example,
>>>>
>>>> https://joeygoksu.com/software/apache-nifi-on-aws/
>>>>
>>>> which results in NiFi installed within a container. So to answer one of
>>>> your questions, I don’t yet know how or where to find nifi.properties in
>>>> the container framework. I don’t seem to have the usual /opt/nifi/…..
>>>> directories on my ec2 instance. Any idea where I need to look for that?
>>>>
>>>> These ports are open by my security group Inbound Rules: 22 to MyIP,
>>>> 80, 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.
>>>>
>>>> I am able to Putty into my instance as ec2-user with my ppk file, which
>>>> I created using putty tools from the original pem key pair. When I do putty
>>>> in, under /opt I find three subdirectories: aws, containerd, and rh.
>>>> Nothing nifi under any of the three that I can see so far.
>>>>
>>>> I start my docker instance with this command:
>>>> docker run —name nifi -p 18080:8080 -d apache/nifi:latest
>>>>
>>>> I can do a ps -ef and see running nifi processes. But I don’t yet know
>>>> how to get to the nifi logs or properties file.
>>>>
>>>> You mentioned using using localhost to get to the canvas UI. This
>>>> confuses me. Nifi is running on my EC2 instance - a linux host without a
>>>> browser. I’m in a browser on my laptop. How would localhost in my browser
>>>> get me to my EC2 instance running nifi?
>>>>
>>>> This is the URL I’m using in my browser:
>>>> http://ec2-3-238-27-220.compute-1.amazonaws.com
>>>> (that url changes with each Stop/Start of my instance. I’ve yet to
>>>> investigate how to get AWS to stop changing that IP, but I know it can be
>>>> done).
>>>>
>>>> The browser replies with: ec2…….amazonaws refused to connect.
>>>>
>>>> I can ping my laptop IP address from the putty terminal where I am
>>>> logged in to my instance. I cannot ping the Public DNS of my instance from
>>>> Powershell on my laptop. Again, that Public DNS is
>>>> ec2-3-238-27-220.compute-1.amazonaws.com
>>>>
>>>> Any help is much appreciated.
>>>> Jim
>>>>
>>>>
>>>>
>>>> On Tue, Nov 8, 2022 at 3:03 PM David Handermann <
>>>> exceptionfactory@apache.org> wrote:
>>>>
>>>>> Hi Jim,
>>>>>
>>>>> NiFi 1.14.0 and following default to HTTPS on port 8443, listening on
>>>>> the localhost address. The nifi.web.https.host can be changed to blank in
>>>>> order to listen on all interfaces, but the default HTTPS setting with
>>>>> authenticated required should be retained.
>>>>>
>>>>> Can you provide the version of NiFi and some additional details on the
>>>>> nifi.web values from nifi.properties?
>>>>>
>>>>> Regards,
>>>>> David Handermann
>>>>>
>>>>> On Tue, Nov 8, 2022 at 1:54 PM James McMahon <js...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Has anyone successfully configured NiFi on AWS, and accessed it from
>>>>>> a browser on a Windows desktop? I’ve tried following a few links to do
>>>>>> this. I’ve verified that my instance security group allows access to 8080
>>>>>> via its inbound rules. I’ve putty’ed into the instance via ssh port 22 to
>>>>>> verify that there are no firewall restrictions. But still I get a message
>>>>>> to the effect that the server rejected the connection request. Can anyone
>>>>>> recommend a link that describes a success path for this?
>>>>>> Thanks in advance for your help.
>>>>>> Jim
>>>>>>
>>>>>
Re: NiFi on AWS EC2
Posted by David Handermann <ex...@apache.org>.
Hi Jim,
Good adjustment on the security group inbound rules.
The error page is the result of NiFi receiving an unexpected HTTP Host
header, not matching one of the expected values.
For this to work, it is possible to pass the external DNS name as the value
of the NIFI_WEB_PROXY_HOST environment variable. This can be specified in
the docker run command as follows:
docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2...
amazonaws.com -d apache/nifi:latest
That will allow NiFi to accept the Host header from the browser, and then
present the login screen.
Regards,
David Handermann
On Tue, Nov 8, 2022 at 6:06 PM James McMahon <js...@gmail.com> wrote:
> Hi David. This is very helpful, thank you. I feel like I am close, but I
> get an error. My Inbound Rules for my security group now include:
> 8443 TCP (MyIP)/32
> 443 TCP (MyIP)/32
> 22 TCP (MyIP)/32
>
> In my browser - I tried both Edge and Chrome - I use this
> URL:
> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
> I have also tried with /nifi at the tail end.
>
> I get this error:
>
> *System Error*
>
> *The request contained an invalid host header
> [ec2-3-238-27-220.compute-1.amazonaws.com:8443
> <http://ec2-3-238-27-220.compute-1.amazonaws.com:8443/>] in the request
> [/]. Check for request manipulation or third-party intercept.*
>
> *Valid host headers are [empty] or:*
>
> - *127.0.0.1*
> - *127.0.0.1:8443 <http://127.0.0.1:8443/>*
> - *localhost*
> - *localhost:8443*
> - *[::1]*
> - *[::1]:8443*
> - *7f661ae687d7*
> - *7f661ae687d7:8443*
> - *172.17.0.2*
> - *172.17.0.2:8443 <http://172.17.0.2:8443/>*
>
>
> Does this mean I have formed the URL incorrectly?
>
> I also see that I had to add an exception to permit https. When I created
> the instance, I created my own pem key pair. It is not signed by any CA.
> For a self-signed key pair like this, do I need to install a key in my
> browser security store to avoid adding that exception?
>
> Thank you for helping me get that much closer.
> Jim
>
> On Tue, Nov 8, 2022 at 5:13 PM David Handermann <
> exceptionfactory@apache.org> wrote:
>
>> Hi Jim,
>>
>> Thanks for the reply and additional background.
>>
>> The instructions are dated March 2021, which is prior to the release of
>> NiFi 1.14.0. In particular, the run command is no longer accurate with the
>> default NiFi container image.
>>
>> The current Docker Hub instructions [1] show the basic command needed
>>
>> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>>
>> In addition, any references to port 8080 in the AWS Security Group rules
>> should be changed to 8443. The security group rules for port 80 and 18080
>> should be removed.
>>
>> The instructions that allow plain HTTP access to NiFi on port 8080 should
>> NEVER be followed, as this exposes unfiltered and unauthenticated access.
>>
>> Following those changes, it should be possible to access the NiFi UI
>> using the AWS URL:
>>
>> https://ec2...amazonaws.com:8443
>>
>> The default installation will generate a username and password, which can
>> be found in the container logs:
>>
>> docker logs nifi | grep Generated
>>
>> Regards,
>> David Handermann
>>
>> [1] https://hub.docker.com/r/apache/nifi
>>
>> On Tue, Nov 8, 2022 at 4:00 PM James McMahon <js...@gmail.com>
>> wrote:
>>
>>> Hi and thank you, David and Dmitry. In my case I was following this
>>> example,
>>>
>>> https://joeygoksu.com/software/apache-nifi-on-aws/
>>>
>>> which results in NiFi installed within a container. So to answer one of
>>> your questions, I don’t yet know how or where to find nifi.properties in
>>> the container framework. I don’t seem to have the usual /opt/nifi/…..
>>> directories on my ec2 instance. Any idea where I need to look for that?
>>>
>>> These ports are open by my security group Inbound Rules: 22 to MyIP, 80,
>>> 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.
>>>
>>> I am able to Putty into my instance as ec2-user with my ppk file, which
>>> I created using putty tools from the original pem key pair. When I do putty
>>> in, under /opt I find three subdirectories: aws, containerd, and rh.
>>> Nothing nifi under any of the three that I can see so far.
>>>
>>> I start my docker instance with this command:
>>> docker run —name nifi -p 18080:8080 -d apache/nifi:latest
>>>
>>> I can do a ps -ef and see running nifi processes. But I don’t yet know
>>> how to get to the nifi logs or properties file.
>>>
>>> You mentioned using using localhost to get to the canvas UI. This
>>> confuses me. Nifi is running on my EC2 instance - a linux host without a
>>> browser. I’m in a browser on my laptop. How would localhost in my browser
>>> get me to my EC2 instance running nifi?
>>>
>>> This is the URL I’m using in my browser:
>>> http://ec2-3-238-27-220.compute-1.amazonaws.com
>>> (that url changes with each Stop/Start of my instance. I’ve yet to
>>> investigate how to get AWS to stop changing that IP, but I know it can be
>>> done).
>>>
>>> The browser replies with: ec2…….amazonaws refused to connect.
>>>
>>> I can ping my laptop IP address from the putty terminal where I am
>>> logged in to my instance. I cannot ping the Public DNS of my instance from
>>> Powershell on my laptop. Again, that Public DNS is
>>> ec2-3-238-27-220.compute-1.amazonaws.com
>>>
>>> Any help is much appreciated.
>>> Jim
>>>
>>>
>>>
>>> On Tue, Nov 8, 2022 at 3:03 PM David Handermann <
>>> exceptionfactory@apache.org> wrote:
>>>
>>>> Hi Jim,
>>>>
>>>> NiFi 1.14.0 and following default to HTTPS on port 8443, listening on
>>>> the localhost address. The nifi.web.https.host can be changed to blank in
>>>> order to listen on all interfaces, but the default HTTPS setting with
>>>> authenticated required should be retained.
>>>>
>>>> Can you provide the version of NiFi and some additional details on the
>>>> nifi.web values from nifi.properties?
>>>>
>>>> Regards,
>>>> David Handermann
>>>>
>>>> On Tue, Nov 8, 2022 at 1:54 PM James McMahon <js...@gmail.com>
>>>> wrote:
>>>>
>>>>> Has anyone successfully configured NiFi on AWS, and accessed it from a
>>>>> browser on a Windows desktop? I’ve tried following a few links to do this.
>>>>> I’ve verified that my instance security group allows access to 8080 via its
>>>>> inbound rules. I’ve putty’ed into the instance via ssh port 22 to verify
>>>>> that there are no firewall restrictions. But still I get a message to the
>>>>> effect that the server rejected the connection request. Can anyone
>>>>> recommend a link that describes a success path for this?
>>>>> Thanks in advance for your help.
>>>>> Jim
>>>>>
>>>>
Re: NiFi on AWS EC2
Posted by James McMahon <js...@gmail.com>.
Hi David. This is very helpful, thank you. I feel like I am close, but I
get an error. My Inbound Rules for my security group now include:
8443 TCP (MyIP)/32
443 TCP (MyIP)/32
22 TCP (MyIP)/32
In my browser - I tried both Edge and Chrome - I use this
URL:
https://ec2-3-238-27-230.compute-1.amazonaws.com:8443
I have also tried with /nifi at the tail end.
I get this error:
*System Error*
*The request contained an invalid host header
[ec2-3-238-27-220.compute-1.amazonaws.com:8443
<http://ec2-3-238-27-220.compute-1.amazonaws.com:8443/>] in the request
[/]. Check for request manipulation or third-party intercept.*
*Valid host headers are [empty] or:*
- *127.0.0.1*
- *127.0.0.1:8443 <http://127.0.0.1:8443/>*
- *localhost*
- *localhost:8443*
- *[::1]*
- *[::1]:8443*
- *7f661ae687d7*
- *7f661ae687d7:8443*
- *172.17.0.2*
- *172.17.0.2:8443 <http://172.17.0.2:8443/>*
Does this mean I have formed the URL incorrectly?
I also see that I had to add an exception to permit https. When I created
the instance, I created my own pem key pair. It is not signed by any CA.
For a self-signed key pair like this, do I need to install a key in my
browser security store to avoid adding that exception?
Thank you for helping me get that much closer.
Jim
On Tue, Nov 8, 2022 at 5:13 PM David Handermann <ex...@apache.org>
wrote:
> Hi Jim,
>
> Thanks for the reply and additional background.
>
> The instructions are dated March 2021, which is prior to the release of
> NiFi 1.14.0. In particular, the run command is no longer accurate with the
> default NiFi container image.
>
> The current Docker Hub instructions [1] show the basic command needed
>
> docker run --name nifi -p 8443:8443 -d apache/nifi:latest
>
> In addition, any references to port 8080 in the AWS Security Group rules
> should be changed to 8443. The security group rules for port 80 and 18080
> should be removed.
>
> The instructions that allow plain HTTP access to NiFi on port 8080 should
> NEVER be followed, as this exposes unfiltered and unauthenticated access.
>
> Following those changes, it should be possible to access the NiFi UI using
> the AWS URL:
>
> https://ec2...amazonaws.com:8443
>
> The default installation will generate a username and password, which can
> be found in the container logs:
>
> docker logs nifi | grep Generated
>
> Regards,
> David Handermann
>
> [1] https://hub.docker.com/r/apache/nifi
>
> On Tue, Nov 8, 2022 at 4:00 PM James McMahon <js...@gmail.com> wrote:
>
>> Hi and thank you, David and Dmitry. In my case I was following this
>> example,
>>
>> https://joeygoksu.com/software/apache-nifi-on-aws/
>>
>> which results in NiFi installed within a container. So to answer one of
>> your questions, I don’t yet know how or where to find nifi.properties in
>> the container framework. I don’t seem to have the usual /opt/nifi/…..
>> directories on my ec2 instance. Any idea where I need to look for that?
>>
>> These ports are open by my security group Inbound Rules: 22 to MyIP, 80,
>> 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.
>>
>> I am able to Putty into my instance as ec2-user with my ppk file, which I
>> created using putty tools from the original pem key pair. When I do putty
>> in, under /opt I find three subdirectories: aws, containerd, and rh.
>> Nothing nifi under any of the three that I can see so far.
>>
>> I start my docker instance with this command:
>> docker run —name nifi -p 18080:8080 -d apache/nifi:latest
>>
>> I can do a ps -ef and see running nifi processes. But I don’t yet know
>> how to get to the nifi logs or properties file.
>>
>> You mentioned using using localhost to get to the canvas UI. This
>> confuses me. Nifi is running on my EC2 instance - a linux host without a
>> browser. I’m in a browser on my laptop. How would localhost in my browser
>> get me to my EC2 instance running nifi?
>>
>> This is the URL I’m using in my browser:
>> http://ec2-3-238-27-220.compute-1.amazonaws.com
>> (that url changes with each Stop/Start of my instance. I’ve yet to
>> investigate how to get AWS to stop changing that IP, but I know it can be
>> done).
>>
>> The browser replies with: ec2…….amazonaws refused to connect.
>>
>> I can ping my laptop IP address from the putty terminal where I am logged
>> in to my instance. I cannot ping the Public DNS of my instance from
>> Powershell on my laptop. Again, that Public DNS is
>> ec2-3-238-27-220.compute-1.amazonaws.com
>>
>> Any help is much appreciated.
>> Jim
>>
>>
>>
>> On Tue, Nov 8, 2022 at 3:03 PM David Handermann <
>> exceptionfactory@apache.org> wrote:
>>
>>> Hi Jim,
>>>
>>> NiFi 1.14.0 and following default to HTTPS on port 8443, listening on
>>> the localhost address. The nifi.web.https.host can be changed to blank in
>>> order to listen on all interfaces, but the default HTTPS setting with
>>> authenticated required should be retained.
>>>
>>> Can you provide the version of NiFi and some additional details on the
>>> nifi.web values from nifi.properties?
>>>
>>> Regards,
>>> David Handermann
>>>
>>> On Tue, Nov 8, 2022 at 1:54 PM James McMahon <js...@gmail.com>
>>> wrote:
>>>
>>>> Has anyone successfully configured NiFi on AWS, and accessed it from a
>>>> browser on a Windows desktop? I’ve tried following a few links to do this.
>>>> I’ve verified that my instance security group allows access to 8080 via its
>>>> inbound rules. I’ve putty’ed into the instance via ssh port 22 to verify
>>>> that there are no firewall restrictions. But still I get a message to the
>>>> effect that the server rejected the connection request. Can anyone
>>>> recommend a link that describes a success path for this?
>>>> Thanks in advance for your help.
>>>> Jim
>>>>
>>>
Re: NiFi on AWS EC2
Posted by David Handermann <ex...@apache.org>.
Hi Jim,
Thanks for the reply and additional background.
The instructions are dated March 2021, which is prior to the release of
NiFi 1.14.0. In particular, the run command is no longer accurate with the
default NiFi container image.
The current Docker Hub instructions [1] show the basic command needed
docker run --name nifi -p 8443:8443 -d apache/nifi:latest
In addition, any references to port 8080 in the AWS Security Group rules
should be changed to 8443. The security group rules for port 80 and 18080
should be removed.
The instructions that allow plain HTTP access to NiFi on port 8080 should
NEVER be followed, as this exposes unfiltered and unauthenticated access.
Following those changes, it should be possible to access the NiFi UI using
the AWS URL:
https://ec2...amazonaws.com:8443
The default installation will generate a username and password, which can
be found in the container logs:
docker logs nifi | grep Generated
Regards,
David Handermann
[1] https://hub.docker.com/r/apache/nifi
On Tue, Nov 8, 2022 at 4:00 PM James McMahon <js...@gmail.com> wrote:
> Hi and thank you, David and Dmitry. In my case I was following this
> example,
>
> https://joeygoksu.com/software/apache-nifi-on-aws/
>
> which results in NiFi installed within a container. So to answer one of
> your questions, I don’t yet know how or where to find nifi.properties in
> the container framework. I don’t seem to have the usual /opt/nifi/…..
> directories on my ec2 instance. Any idea where I need to look for that?
>
> These ports are open by my security group Inbound Rules: 22 to MyIP, 80,
> 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.
>
> I am able to Putty into my instance as ec2-user with my ppk file, which I
> created using putty tools from the original pem key pair. When I do putty
> in, under /opt I find three subdirectories: aws, containerd, and rh.
> Nothing nifi under any of the three that I can see so far.
>
> I start my docker instance with this command:
> docker run —name nifi -p 18080:8080 -d apache/nifi:latest
>
> I can do a ps -ef and see running nifi processes. But I don’t yet know how
> to get to the nifi logs or properties file.
>
> You mentioned using using localhost to get to the canvas UI. This confuses
> me. Nifi is running on my EC2 instance - a linux host without a browser.
> I’m in a browser on my laptop. How would localhost in my browser get me to
> my EC2 instance running nifi?
>
> This is the URL I’m using in my browser:
> http://ec2-3-238-27-220.compute-1.amazonaws.com
> (that url changes with each Stop/Start of my instance. I’ve yet to
> investigate how to get AWS to stop changing that IP, but I know it can be
> done).
>
> The browser replies with: ec2…….amazonaws refused to connect.
>
> I can ping my laptop IP address from the putty terminal where I am logged
> in to my instance. I cannot ping the Public DNS of my instance from
> Powershell on my laptop. Again, that Public DNS is
> ec2-3-238-27-220.compute-1.amazonaws.com
>
> Any help is much appreciated.
> Jim
>
>
>
> On Tue, Nov 8, 2022 at 3:03 PM David Handermann <
> exceptionfactory@apache.org> wrote:
>
>> Hi Jim,
>>
>> NiFi 1.14.0 and following default to HTTPS on port 8443, listening on the
>> localhost address. The nifi.web.https.host can be changed to blank in order
>> to listen on all interfaces, but the default HTTPS setting with
>> authenticated required should be retained.
>>
>> Can you provide the version of NiFi and some additional details on the
>> nifi.web values from nifi.properties?
>>
>> Regards,
>> David Handermann
>>
>> On Tue, Nov 8, 2022 at 1:54 PM James McMahon <js...@gmail.com>
>> wrote:
>>
>>> Has anyone successfully configured NiFi on AWS, and accessed it from a
>>> browser on a Windows desktop? I’ve tried following a few links to do this.
>>> I’ve verified that my instance security group allows access to 8080 via its
>>> inbound rules. I’ve putty’ed into the instance via ssh port 22 to verify
>>> that there are no firewall restrictions. But still I get a message to the
>>> effect that the server rejected the connection request. Can anyone
>>> recommend a link that describes a success path for this?
>>> Thanks in advance for your help.
>>> Jim
>>>
>>
Re: NiFi on AWS EC2
Posted by James McMahon <js...@gmail.com>.
Hi and thank you, David and Dmitry. In my case I was following this
example,
https://joeygoksu.com/software/apache-nifi-on-aws/
which results in NiFi installed within a container. So to answer one of
your questions, I don’t yet know how or where to find nifi.properties in
the container framework. I don’t seem to have the usual /opt/nifi/…..
directories on my ec2 instance. Any idea where I need to look for that?
These ports are open by my security group Inbound Rules: 22 to MyIP, 80,
8080, and 18080 (per the link) to 0.0.0.0/0, 443 to MyIP.
I am able to Putty into my instance as ec2-user with my ppk file, which I
created using putty tools from the original pem key pair. When I do putty
in, under /opt I find three subdirectories: aws, containerd, and rh.
Nothing nifi under any of the three that I can see so far.
I start my docker instance with this command:
docker run —name nifi -p 18080:8080 -d apache/nifi:latest
I can do a ps -ef and see running nifi processes. But I don’t yet know how
to get to the nifi logs or properties file.
You mentioned using using localhost to get to the canvas UI. This confuses
me. Nifi is running on my EC2 instance - a linux host without a browser.
I’m in a browser on my laptop. How would localhost in my browser get me to
my EC2 instance running nifi?
This is the URL I’m using in my browser:
http://ec2-3-238-27-220.compute-1.amazonaws.com
(that url changes with each Stop/Start of my instance. I’ve yet to
investigate how to get AWS to stop changing that IP, but I know it can be
done).
The browser replies with: ec2…….amazonaws refused to connect.
I can ping my laptop IP address from the putty terminal where I am logged
in to my instance. I cannot ping the Public DNS of my instance from
Powershell on my laptop. Again, that Public DNS is
ec2-3-238-27-220.compute-1.amazonaws.com
Any help is much appreciated.
Jim
On Tue, Nov 8, 2022 at 3:03 PM David Handermann <ex...@apache.org>
wrote:
> Hi Jim,
>
> NiFi 1.14.0 and following default to HTTPS on port 8443, listening on the
> localhost address. The nifi.web.https.host can be changed to blank in order
> to listen on all interfaces, but the default HTTPS setting with
> authenticated required should be retained.
>
> Can you provide the version of NiFi and some additional details on the
> nifi.web values from nifi.properties?
>
> Regards,
> David Handermann
>
> On Tue, Nov 8, 2022 at 1:54 PM James McMahon <js...@gmail.com> wrote:
>
>> Has anyone successfully configured NiFi on AWS, and accessed it from a
>> browser on a Windows desktop? I’ve tried following a few links to do this.
>> I’ve verified that my instance security group allows access to 8080 via its
>> inbound rules. I’ve putty’ed into the instance via ssh port 22 to verify
>> that there are no firewall restrictions. But still I get a message to the
>> effect that the server rejected the connection request. Can anyone
>> recommend a link that describes a success path for this?
>> Thanks in advance for your help.
>> Jim
>>
>
Re: NiFi on AWS EC2
Posted by David Handermann <ex...@apache.org>.
Hi Jim,
NiFi 1.14.0 and following default to HTTPS on port 8443, listening on the
localhost address. The nifi.web.https.host can be changed to blank in order
to listen on all interfaces, but the default HTTPS setting with
authenticated required should be retained.
Can you provide the version of NiFi and some additional details on the
nifi.web values from nifi.properties?
Regards,
David Handermann
On Tue, Nov 8, 2022 at 1:54 PM James McMahon <js...@gmail.com> wrote:
> Has anyone successfully configured NiFi on AWS, and accessed it from a
> browser on a Windows desktop? I’ve tried following a few links to do this.
> I’ve verified that my instance security group allows access to 8080 via its
> inbound rules. I’ve putty’ed into the instance via ssh port 22 to verify
> that there are no firewall restrictions. But still I get a message to the
> effect that the server rejected the connection request. Can anyone
> recommend a link that describes a success path for this?
> Thanks in advance for your help.
> Jim
>