You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Jens M. Kofoed" <jm...@gmail.com> on 2022/05/19 11:42:30 UTC

Load balancing just stopped working in NIFI 1.16.1

Hi

I have a 3 node test cluster running v. 1.16.1. Which has been working
fine, with no errors. But i doesn't do much, since it is my test cluster.
But now I am struggling with load balance connection refuse between nodes.
Both node 2 and 3 are refusing load balancing connections, even after
stopping the cluster. I have deleted all files at node 2 and 3, but they
still refuse connection.
I have not made any change to the configuration, but I have created a new
flow which got node 1 to die.
The new flow is using an ExecuteStreamCommand using stdin/stdout to a shell
command to manipulate with some data (1 GB files). I set the "Concurrent
Tasks" to high so node 1 ran out of memory and stopped. It continues to
stop if I tried to start it again. So I deleted the flow.gz file and the
run, state and work folders and started the node. Now node 1 was running
again. But after this node 2 and 3 are refusing load balancing connections.
I can't see what this flow have to do with this, but I have now tried to
remove all files at node 2 and 3 to get clean nodes. But they still doesn't
work

Does any one have an idea how to debug further?

From the log:
2022-05-19 13:22:46,268 ERROR [Load-Balanced Client Thread-2]
o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to
nifi-n03:8443 for load balancing
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect0(Native Method)
        at sun.nio.ch.Net.connect(Net.java:482)
        at sun.nio.ch.Net.connect(Net.java:474)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:107)
        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92)
        at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:497)
        at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:440)
        at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:234)
        at
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81)
        at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

 kind regards
Jens

Re: Load balancing just stopped working in NIFI 1.16.1

Posted by "Jens M. Kofoed" <jm...@gmail.com>.
Hi Mark

Thanks for the reply. The “funny” part is that it worked before and I haven’t made any change to the configuration. 
And I have deleted the state folder at all nodes and it doesn’t work. 

It is the same configuration as 3 other clusters except hostname/up adresses and certificates off course. 
All clusters are running 1.16.1. At the 3 other clusters don’t have any issues (yet).
It all started after node 1 went down do to memory problem. After that they just don’t like me anymore 🤔

Tomorrow I will give it a chance and if it doesn’t work I will reinstall the cluster from scratch. I would like to be able to fix it. If the same situations happened with one of the productions clusters. Hopefully all flows are in the registry.

Kind regards 
Jens 

> Den 19. maj 2022 kl. 15.21 skrev Mark Payne <ma...@hotmail.com>:
> 
> Jens,
> 
> So that would tell us that the hostname or the port is wrong, or that NiFi is not running/listening for load balanced connections. I would recommend the following:
> 
> - Check nifi.properties on all nodes to make sure that the nifi.cluster.load.balance.host property is set to the node’s hostname.
> - You could try deleting the state/ directory on the nodes and restarting. Perhaps somehow it’s got an old value cached and is trying to communicate with the wrong port?
> 
> Thanks
> -Mark
> 
> 
>> On May 19, 2022, at 7:42 AM, Jens M. Kofoed <jm...@gmail.com> wrote:
>> 
>> Hi
>> 
>> I have a 3 node test cluster running v. 1.16.1. Which has been working fine, with no errors. But i doesn't do much, since it is my test cluster.
>> But now I am struggling with load balance connection refuse between nodes.
>> Both node 2 and 3 are refusing load balancing connections, even after stopping the cluster. I have deleted all files at node 2 and 3, but they still refuse connection.
>> I have not made any change to the configuration, but I have created a new flow which got node 1 to die.
>> The new flow is using an ExecuteStreamCommand using stdin/stdout to a shell command to manipulate with some data (1 GB files). I set the "Concurrent Tasks" to high so node 1 ran out of memory and stopped. It continues to stop if I tried to start it again. So I deleted the flow.gz file and the run, state and work folders and started the node. Now node 1 was running again. But after this node 2 and 3 are refusing load balancing connections.
>> I can't see what this flow have to do with this, but I have now tried to remove all files at node 2 and 3 to get clean nodes. But they still doesn't work
>> 
>> Does any one have an idea how to debug further?
>> 
>> From the log:
>> 2022-05-19 13:22:46,268 ERROR [Load-Balanced Client Thread-2] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to nifi-n03:8443 for load balancing
>> java.net.ConnectException: Connection refused
>>        at sun.nio.ch.Net.connect0(Native Method)
>>        at sun.nio.ch.Net.connect(Net.java:482)
>>        at sun.nio.ch.Net.connect(Net.java:474)
>>        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
>>        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:107)
>>        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92)
>>        at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:497)
>>        at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:440)
>>        at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:234)
>>        at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81)
>>        at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>>        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>        at java.lang.Thread.run(Thread.java:748)
>> 
>> kind regards
>> Jens
>

Re: Load balancing just stopped working in NIFI 1.16.1

Posted by Mark Payne <ma...@hotmail.com>.
Jens,

So that would tell us that the hostname or the port is wrong, or that NiFi is not running/listening for load balanced connections. I would recommend the following:

- Check nifi.properties on all nodes to make sure that the nifi.cluster.load.balance.host property is set to the node’s hostname.
- You could try deleting the state/ directory on the nodes and restarting. Perhaps somehow it’s got an old value cached and is trying to communicate with the wrong port?

Thanks
-Mark


> On May 19, 2022, at 7:42 AM, Jens M. Kofoed <jm...@gmail.com> wrote:
> 
> Hi
> 
> I have a 3 node test cluster running v. 1.16.1. Which has been working fine, with no errors. But i doesn't do much, since it is my test cluster.
> But now I am struggling with load balance connection refuse between nodes.
> Both node 2 and 3 are refusing load balancing connections, even after stopping the cluster. I have deleted all files at node 2 and 3, but they still refuse connection.
> I have not made any change to the configuration, but I have created a new flow which got node 1 to die.
> The new flow is using an ExecuteStreamCommand using stdin/stdout to a shell command to manipulate with some data (1 GB files). I set the "Concurrent Tasks" to high so node 1 ran out of memory and stopped. It continues to stop if I tried to start it again. So I deleted the flow.gz file and the run, state and work folders and started the node. Now node 1 was running again. But after this node 2 and 3 are refusing load balancing connections.
> I can't see what this flow have to do with this, but I have now tried to remove all files at node 2 and 3 to get clean nodes. But they still doesn't work
> 
> Does any one have an idea how to debug further?
> 
> From the log:
> 2022-05-19 13:22:46,268 ERROR [Load-Balanced Client Thread-2] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to nifi-n03:8443 for load balancing
> java.net.ConnectException: Connection refused
>         at sun.nio.ch.Net.connect0(Native Method)
>         at sun.nio.ch.Net.connect(Net.java:482)
>         at sun.nio.ch.Net.connect(Net.java:474)
>         at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
>         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:107)
>         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92)
>         at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:497)
>         at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:440)
>         at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:234)
>         at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81)
>         at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> 
>  kind regards
> Jens