You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by William Gosse <Wi...@aifoundry.com> on 2018/10/23 13:27:54 UTC

Problem with remote groups reconnecting to input port after restart.

Hi,

I'm trying to implement a Listfile>FetchFile pattern similar to the one described on the following link:
https://pierrevillard.com/2017/02/23/listfetch-pattern-and-remote-process-group-in-apache-nifi/

I have multiple Listfiles process groups each containing a remote process group that point to a common input port.

The input port itself points to a common fetchfile process group.

My issue I'm having is that after I do a Nifi restart not all my remote process groups successfully connect to the input port.

Are there any setting I should be aware to help eliminate this problem? Should use RAW vs HTTP, I'm currently using HTTP.

Any help be greatly appreciated.

Regards,
Bill

Re: Problem with remote groups reconnecting to input port after restart.

Posted by Joe Witt <jo...@gmail.com>.
Bill

From the screenshots there is no data flowing/queing so possible sources of
difficultly there aren't seen.  The screenshots have a lot of potentially
helpful stuff cut off like URLs in the url bar, information from the top of
the nifi page that would tell me about clustering, etc.. I know sharing
screenshots at times can be sensitive so I get it but also it obviously
limits troubleshooting too.

From the lgos this is a windows system and with site to site engaged it
makes me assume this is a cluster (not just a single node).  I dont know
how well NiFi clusters on Windows will work.  NiFi nodes on Windows are
fine/common but full clustering in Windows is much less so.  Not sure of
the zookeeper implications there for instance.

In general site to site connections will have errors/warns in the logs as
nodes are coming on-line or having periodic issues and this is normal/ok
and the protocol overcomes them naturally.  As nodes join/leave/have
different loading it will handle that automatically.  You should *never*
have to restart anything for it to work and if you did we'd want to see
that.  If you're seeing data backlogged on a connection and not sending
then that would tell us more but the logs/screenshots dont suggest that.
If you see constant and enduring errors/warns in the logs for it then
certainly something is wrong.

When your original note said "My issue I’m having is that after I do a Nifi
restart not all my remote process groups successfully connect to the input
port" - what did that mean?  What were you seeing in the UI?  In the logs
to specifically suggest that it wasn't working?

Thanks

On Tue, Oct 23, 2018 at 3:54 PM William Gosse <Wi...@aifoundry.com>
wrote:

> So at my top level I have my nput port, common fetch process group and
> multiple list process groups:
>
>
>
> I set my input port to support 10 concurrent tasks which I though might
> help but I’m not really sure.
>
>
>
> Each list process group contains the following which includes the remote
> process group:
>
>
>
> I’ve tried both RAW and HTTP, RAW seem to be a little better on recovery
> be not perfect.
>
>
>
> My common fetch process group contains the following:
>
>
>
> No real issues here if the remote process group can find the the iinput
> port everything works as expected.
>
>
>
> In my nifi-app.log file I see things like this:
>
> *2018-10-23 15:19:46,403 INFO [main]
> o.apache.nifi.controller.FlowController Started 3 Remote Group Ports
> transmitting*
>
> *2018-10-23 15:19:46,403 INFO [main]
> o.a.n.w.c.ApplicationStartupContextListener Flow Controller started
> successfully.*
>
> *2018-10-23 15:19:46,650 WARN [Remote Process Group
> 763b0492-57ec-3bdb-71c0-b6c607183d9c Thread-1]
> o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from
> http://localhost:6080/nifi-api <http://localhost:6080/nifi-api> due to
> org.apache.http.conn.HttpHostConnectException: Connect to localhost:6080
> [localhost/127.0.0.1 <http://127.0.0.1>] failed: Connection refused:
> connect*
>
> *2018-10-23 15:19:46,653 WARN [Remote Process Group
> 763b0492-57ec-3bdb-71c0-b6c607183d9c Thread-1]
> o.a.n.remote.StandardRemoteProcessGroup Unable to communicate with remote
> instance RemoteProcessGroup[http://localhost:6080/nifi
> <http://localhost:6080/nifi>]*
>
> *org.apache.nifi.controller.exception.CommunicationsException:
> org.apache.nifi.controller.exception.CommunicationsException: Unable to
> communicate with Remote NiFi at URI http://localhost:6080/nifi
> <http://localhost:6080/nifi> due to: Connect to localhost:6080
> [localhost/127.0.0.1 <http://127.0.0.1>] failed: Connection refused:
> connect*
>
> *                at
> org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:883)*
>
> *                at
> org.apache.nifi.remote.StandardRemoteProcessGroup.lambda$initialize$0(StandardRemoteProcessGroup.java:193)*
>
> *                at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)*
>
> *                at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)*
>
> *                at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)*
>
> *                at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)*
>
> *                at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)*
>
> *                at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)*
>
> *                at java.lang.Thread.run(Thread.java:748)*
>
> *Caused by: org.apache.nifi.controller.exception.CommunicationsException:
> Unable to communicate with Remote NiFi at URI http://localhost:6080/nifi
> <http://localhost:6080/nifi> due to: Connect to localhost:6080
> [localhost/127.0.0.1 <http://127.0.0.1>] failed: Connection refused:
> connect*
>
> *                at
> org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:845)*
>
> *                ... 8 common frames omitted*
>
> *2018-10-23 15:19:47,664 WARN [NiFi Site-to-Site Connection Pool
> Maintenance] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller
> from http://localhost:6080/nifi-api <http://localhost:6080/nifi-api> due to
> org.apache.http.conn.HttpHostConnectException: Connect to localhost:6080
> [localhost/127.0.0.1 <http://127.0.0.1>] failed: Connection refused:
> connect*
>
> *2018-10-23 15:19:47,665 WARN [NiFi Site-to-Site Connection Pool
> Maintenance] o.apache.nifi.remote.client.PeerSelector
> org.apache.nifi.remote.client.PeerSelector@31c739e8 Unable to refresh
> Remote Group's peers due to Connect to localhost:6080 [localhost/127.0.0.1
> <http://127.0.0.1>] failed: Connection refused: connect*
>
> *2018-10-23 15:19:48,654 INFO [main]
> o.e.jetty.server.handler.ContextHandler Started
> o.e.j.w.WebAppContext@2c952e6{/nifi-api,file:///C:/Dev/aim_project_repos/nifi-1.7.1/work/jetty/nifi-web-api-1.7.1.war/webapp/,AVAILABLE}{.\work\nar\framework\nifi-framework-nar-1.7.1.nar-unpacked\META-INF\bundled-dependencies\nifi-web-api-1.7.1.war}*
>
> *2018-10-23 15:19:48,668 WARN [NiFi Site-to-Site Connection Pool
> Maintenance] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller
> from http://localhost:6080/nifi-api <http://localhost:6080/nifi-api> due to
> org.apache.http.conn.HttpHostConnectException: Connect to localhost:6080
> [localhost/127.0.0.1 <http://127.0.0.1>] failed: Connection refused:
> connect*
>
> *2018-10-23 15:19:48,668 WARN [NiFi Site-to-Site Connection Pool
> Maintenance] o.apache.nifi.remote.client.PeerSelector
> org.apache.nifi.remote.client.PeerSelector@750d934c Unable to refresh
> Remote Group's peers due to Connect to localhost:6080 [localhost/127.0.0.1
> <http://127.0.0.1>] failed: Connection refused: connect*
>
>
>
> Restarting sometimes clears the issue.
>
>
>
>
>
>
>
> *From:* Joe Witt <jo...@gmail.com>
> *Sent:* Tuesday, October 23, 2018 10:55 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: Problem with remote groups reconnecting to input port
> after restart.
>
>
>
> [CAUTION: This email originated from outside of Kodak Alaris. Do not click
> links or open attachments unless you recognize the sender and know the
> content is safe.]
> ------------------------------
>
> bill
>
>
>
> can you share more about what is observed on restart?  backlogs?  can you
> share screenshots and or logs.
>
>
>
> thanks
>
>
>
> On Tue, Oct 23, 2018, 7:28 AM William Gosse <Wi...@aifoundry.com>
> wrote:
>
> Hi,
>
>
>
> I’m trying to implement a Listfile>FetchFile pattern similar to the one
> described on the following link:
>
>
> https://pierrevillard.com/2017/02/23/listfetch-pattern-and-remote-process-group-in-apache-nifi/
> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpierrevillard.com%2F2017%2F02%2F23%2Flistfetch-pattern-and-remote-process-group-in-apache-nifi%2F&data=02%7C01%7Cwilliam.gosse%40aifoundry.com%7Cb67f81db843444fc9cef08d638f784d7%7Cd29b7a9b6edb472099a83c5c6c3eeeb0%7C0%7C0%7C636759033084574216&sdata=CV%2BozjbLevs565g6MDOGFHcnrUVnv2oQOBRQgZECx4A%3D&reserved=0>
>
>
>
> I have multiple Listfiles process groups each containing a remote process
> group that point to a common input port.
>
>
>
> The input port itself points to a common fetchfile process group.
>
>
>
> My issue I’m having is that after I do a Nifi restart not all my remote
> process groups successfully connect to the input port.
>
>
>
> Are there any setting I should be aware to help eliminate this problem?
> Should use RAW vs HTTP, I’m currently using HTTP.
>
>
>
> Any help be greatly appreciated.
>
>
>
> Regards,
>
> Bill
>
>

RE: Problem with remote groups reconnecting to input port after restart.

Posted by William Gosse <Wi...@aifoundry.com>.
So at my top level I have my nput port, common fetch process group and multiple list process groups:
[cid:image001.png@01D46AE7.9BB72E10]

I set my input port to support 10 concurrent tasks which I though might help but I’m not really sure.

Each list process group contains the following which includes the remote process group:
[cid:image002.png@01D46AE7.9BB72E10]

I’ve tried both RAW and HTTP, RAW seem to be a little better on recovery be not perfect.

My common fetch process group contains the following:
[cid:image003.png@01D46AE8.03379C00]

No real issues here if the remote process group can find the the iinput port everything works as expected.

In my nifi-app.log file I see things like this:
2018-10-23 15:19:46,403 INFO [main] o.apache.nifi.controller.FlowController Started 3 Remote Group Ports transmitting
2018-10-23 15:19:46,403 INFO [main] o.a.n.w.c.ApplicationStartupContextListener Flow Controller started successfully.
2018-10-23 15:19:46,650 WARN [Remote Process Group 763b0492-57ec-3bdb-71c0-b6c607183d9c Thread-1] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from http://localhost:6080/nifi-api due to org.apache.http.conn.HttpHostConnectException: Connect to localhost:6080 [localhost/127.0.0.1] failed: Connection refused: connect
2018-10-23 15:19:46,653 WARN [Remote Process Group 763b0492-57ec-3bdb-71c0-b6c607183d9c Thread-1] o.a.n.remote.StandardRemoteProcessGroup Unable to communicate with remote instance RemoteProcessGroup[http://localhost:6080/nifi]
org.apache.nifi.controller.exception.CommunicationsException: org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI http://localhost:6080/nifi due to: Connect to localhost:6080 [localhost/127.0.0.1] failed: Connection refused: connect
                at org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:883)
                at org.apache.nifi.remote.StandardRemoteProcessGroup.lambda$initialize$0(StandardRemoteProcessGroup.java:193)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI http://localhost:6080/nifi due to: Connect to localhost:6080 [localhost/127.0.0.1] failed: Connection refused: connect
                at org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:845)
                ... 8 common frames omitted
2018-10-23 15:19:47,664 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from http://localhost:6080/nifi-api due to org.apache.http.conn.HttpHostConnectException: Connect to localhost:6080 [localhost/127.0.0.1] failed: Connection refused: connect
2018-10-23 15:19:47,665 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@31c739e8 Unable to refresh Remote Group's peers due to Connect to localhost:6080 [localhost/127.0.0.1] failed: Connection refused: connect
2018-10-23 15:19:48,654 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@2c952e6{/nifi-api,file:///C:/Dev/aim_project_repos/nifi-1.7.1/work/jetty/nifi-web-api-1.7.1.war/webapp/,AVAILABLE}{.\work\nar\framework\nifi-framework-nar-1.7.1.nar-unpacked\META-INF\bundled-dependencies\nifi-web-api-1.7.1.war}
2018-10-23 15:19:48,668 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from http://localhost:6080/nifi-api due to org.apache.http.conn.HttpHostConnectException: Connect to localhost:6080 [localhost/127.0.0.1] failed: Connection refused: connect
2018-10-23 15:19:48,668 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@750d934c Unable to refresh Remote Group's peers due to Connect to localhost:6080 [localhost/127.0.0.1] failed: Connection refused: connect

Restarting sometimes clears the issue.



From: Joe Witt <jo...@gmail.com>
Sent: Tuesday, October 23, 2018 10:55 AM
To: users@nifi.apache.org
Subject: Re: Problem with remote groups reconnecting to input port after restart.

[CAUTION: This email originated from outside of Kodak Alaris. Do not click links or open attachments unless you recognize the sender and know the content is safe.]
________________________________
bill

can you share more about what is observed on restart?  backlogs?  can you share screenshots and or logs.

thanks

On Tue, Oct 23, 2018, 7:28 AM William Gosse <Wi...@aifoundry.com>> wrote:
Hi,

I’m trying to implement a Listfile>FetchFile pattern similar to the one described on the following link:
https://pierrevillard.com/2017/02/23/listfetch-pattern-and-remote-process-group-in-apache-nifi/<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpierrevillard.com%2F2017%2F02%2F23%2Flistfetch-pattern-and-remote-process-group-in-apache-nifi%2F&data=02%7C01%7Cwilliam.gosse%40aifoundry.com%7Cb67f81db843444fc9cef08d638f784d7%7Cd29b7a9b6edb472099a83c5c6c3eeeb0%7C0%7C0%7C636759033084574216&sdata=CV%2BozjbLevs565g6MDOGFHcnrUVnv2oQOBRQgZECx4A%3D&reserved=0>

I have multiple Listfiles process groups each containing a remote process group that point to a common input port.

The input port itself points to a common fetchfile process group.

My issue I’m having is that after I do a Nifi restart not all my remote process groups successfully connect to the input port.

Are there any setting I should be aware to help eliminate this problem? Should use RAW vs HTTP, I’m currently using HTTP.

Any help be greatly appreciated.

Regards,
Bill

Re: Problem with remote groups reconnecting to input port after restart.

Posted by Joe Witt <jo...@gmail.com>.
bill

can you share more about what is observed on restart?  backlogs?  can you
share screenshots and or logs.

thanks

On Tue, Oct 23, 2018, 7:28 AM William Gosse <Wi...@aifoundry.com>
wrote:

> Hi,
>
>
>
> I’m trying to implement a Listfile>FetchFile pattern similar to the one
> described on the following link:
>
>
> https://pierrevillard.com/2017/02/23/listfetch-pattern-and-remote-process-group-in-apache-nifi/
>
>
>
> I have multiple Listfiles process groups each containing a remote process
> group that point to a common input port.
>
>
>
> The input port itself points to a common fetchfile process group.
>
>
>
> My issue I’m having is that after I do a Nifi restart not all my remote
> process groups successfully connect to the input port.
>
>
>
> Are there any setting I should be aware to help eliminate this problem?
> Should use RAW vs HTTP, I’m currently using HTTP.
>
>
>
> Any help be greatly appreciated.
>
>
>
> Regards,
>
> Bill
>