You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Axel Schwarz <Ax...@emailn.de> on 2021/07/27 06:07:05 UTC
No Load Balancing since 1.13.2
Dear Community,
we're running a secured 3 node Nifi Cluster on Java 8_u291 and Debian 7 and
experiencing
problems with load balancing since version 1.13.2.
I'm fully aware of Issue Nifi-8643 and tested alot around this, but gotta say,
that this
is not our problem. Mainly because the balance port never binds to localhost,
but also because I
implemented all workarounds under version 1.13.2 and even tried version 1.14.0
by now,
but load blancing still does not work.
What we experience is best described as "the primary node balances with
itself"...
So what it does is, opening the balancing connections to its own IP instead of
the IPs
of the other two nodes. And the other two nodes don't open balancing
connections at all.
When executing "ss | grep 6342" on the primary node, this is what it looks
like:
[root@nifiHost1 conf]# ss | grep 6342
tcp ESTAB 0 0 192.168.1.10:51380
192.168.1.10:6342
tcp ESTAB 0 0 192.168.1.10:51376
192.168.1.10:6342
tcp ESTAB 0 0 192.168.1.10:51378
192.168.1.10:6342
tcp ESTAB 0 0 192.168.1.10:51370
192.168.1.10:6342
tcp ESTAB 0 0 192.168.1.10:51372
192.168.1.10:6342
tcp ESTAB 0 0 192.168.1.10:6342
192.168.1.10:51376
tcp ESTAB 0 0 192.168.1.10:51374
192.168.1.10:6342
tcp ESTAB 0 0 192.168.1.10:6342
192.168.1.10:51374
tcp ESTAB 0 0 192.168.1.10:51366
192.168.1.10:6342
tcp ESTAB 0 0 192.168.1.10:6342
192.168.1.10:51370
tcp ESTAB 0 0 192.168.1.10:6342
192.168.1.10:51366
tcp ESTAB 0 0 192.168.1.10:51368
192.168.1.10:6342
tcp ESTAB 0 0 192.168.1.10:6342
192.168.1.10:51372
tcp ESTAB 0 0 192.168.1.10:6342
192.168.1.10:51378
tcp ESTAB 0 0 192.168.1.10:6342
192.168.1.10:51368
tcp ESTAB 0 0 192.168.1.10:6342
192.168.1.10:51380
Executing it on the other non primary nodes, just returns absolutely nothing.
Netstat show the following on each server:
[root@nifiHost1 conf]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp 0 0 192.168.1.10:6342 0.0.0.0:* LISTEN
10352/java
[root@nifiHost2 conf]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp 0 0 192.168.1.11:6342 0.0.0.0:* LISTEN
31562/java
[root@nifiHost3 conf]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp 0 0 192.168.1.12:6342 0.0.0.0:* LISTEN
31685/java
And here is what our load balancing properties look like:
# cluster load balancing properties #
nifi.cluster.load.balance.host=nifiHost1.contoso.com
nifi.cluster.load.balance.address=0.0.0.0
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=4
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec
When running Nifi in version 1.12.1 on the exact same setup in the exact same
environment, load balancing is working absolutely fine.
There was a time when load balancing even worked in version 1.13.2. But I'm
not able to reproduce this and it just stopped
working one day after some restart, without changing any property or
whatsoever.
If any more information would be helpful please let me know and I'll try to
provide it as fast as possible.
* * *
Versendet mit [Emailn.de](https://www.emailn.de/) \- Freemail
* Unbegrenzt Speicherplatz
* Eigenes Online-Büro
* 24h besten Mailempfang
* Spamschutz, Adressbuch
Re: Re: Re: Re: No Load Balancing since 1.13.2
Posted by Axel Schwarz <Ax...@emailn.de>.
Ok, so we played with these parameters again and got different results almost each time.
Once we actually had a functioning balancing cluster, but just once. This was with Version 1.14.0 and .host set to 0.0.0.0.
In this state NifiHost1 (Primary Node) had 8 connecctions established. 4 to each other host. And the other hosts each had 4 connections to NifiHost1.
Right after a restart it stopped working again and the primary node only had connections to itself.
The third result we saw was, that NifiHost1 (Primary Node) had 4 connections open to itself, NifiHost2 had 4 connections open to NifiHost1 and NifiHost3 had no connection at all. The result in the flow then was, that the cluster load balanced between NifiHost1 and NifiHost2, but not like in the right order (1. NifiHost1, 2. NifiHost2, 3. NifiHost3 and so on) but with every third flowfile to NifiHost2 and the rest to NifiHost1 (so it load balanced with itself indeed)
We can reproduce these outcomes randomly with Version 1.13.2 AND 1.14.0.
It's really hard to obtain conclusions out of that behaviour and I get a little desperate by now :(
--- Ursprüngliche Nachricht ---
Von: "Axel Schwarz" <Ax...@emailn.de>
Datum: 28.07.2021 13:10:11
An: users@nifi.apache.org
Betreff: Re: Re: Re: No Load Balancing since 1.13.2
> I've tried every single combination of values for nifi.cluster.load.balance.host
> and nifi.cluster.load.balance.address.
> But again, the problem described in Nifi-8643 is not what happens to us.
> The port always binds to the correct IP and NOT to localhost.
> As seen in the netstat-excerpts from my original post. Instead the primary
> node actually opens the desired count of connections for load balancing,
> but ALL to itself...
>
> --- Ursprüngliche Nachricht ---
> Von: Joe Gresock <jg...@gmail.com>
> Datum: 28.07.2021 12:09:42
> An: users@nifi.apache.org
> Betreff: Re: Re: No Load Balancing since 1.13.2
>
> > Also, make sure to use nifi.cluster.load.balance.host instead of .address
>
> >
> > if you're still running 1.14.0.
> >
> > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
>
> >
> > Virus-free.
> > www.avast.com
> > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
>
> >
> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >
> > On Wed, Jul 28, 2021 at 6:04 AM Jens M. Kofoed <jm...@gmail.com>
>
> >
> > wrote:
> >
> > > hi
> > >
> > > I can see that you have configured
> > > nifi.cluster.load.balance.address=0.0.0.0
> > > Have your tried to set the correct ip adress?
> > > node1: nifi.cluster.load.balance.address=192.168.1.10
> > > node2: nifi.cluster.load.balance.address=192.168.1.11
> > > node3: nifi.cluster.load.balance.address=192.168.1.12
> > >
> > > regards
> > > Jens M. Kofoed
> > >
> > > Den ons. 28. jul. 2021 kl. 11.17 skrev Axel Schwarz <Axelkopter@emailn.de
>
> >
> > > >:
> > >
> > >> Just tried Java 11. But still does not work. Nothing changed.
> :(
> >
> > >>
> > >> --- Ursprüngliche Nachricht ---
> > >> Von: Jorge Machado <jo...@me.com>
> > >> Datum: 27.07.2021 13:08:55
> > >> An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>
> >
> > >> Betreff: Re: No Load Balancing since 1.13.2
> > >>
> > >> > Did you tried java 11 ? I have a client running a similar
> setup
> > to yours
> > >> > but with a lower nigh version and it works fine. Maybe
> it is
> > worth to
> > >> try
> > >> > it.
> > >> >
> > >> >
> > >> > > On 27. Jul 2021, at 12:42, Axel Schwarz <Ax...@emailn.de>
>
> >
> > >> > wrote:
> > >> > >
> > >> > > I did indeed, but I updated from u161 to u291, as
> this
> > was the newest
> > >> > version at that time, because I thought it could help.
>
> > >> > > So the issue started under u161. But I just saw that
> u301
> > is out. I
> > >> > will try this as well.
> > >> > > --- Ursprüngliche Nachricht ---
> > >> > > Von: Pierre Villard <pi...@gmail.com>
>
> >
> > >> > > Datum: 27.07.2021 10:18:38
> > >> > > An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>
> >
> > >> >
> > >> > > Betreff: Re: No Load Balancing since 1.13.2
> > >> > >
> > >> > > Hi,
> > >> > >
> > >> > > I believe the minor u291 is known to have issues
> (for
> > some of its
> > >> early
> > >> > builds). Did you upgrade the Java version recently?
> > >> > >
> > >> > > Thanks,
> > >> > > Pierre
> > >> > >
> > >> > > Le mar. 27 juil. 2021 à 08:07, Axel Schwarz <Axelkopter@emailn.de
>
> >
> > >> > <ma...@emailn.de>> a écrit :
> > >> > > Dear Community,
> > >> > >
> > >> > > we're running a secured 3 node Nifi Cluster on Java
> 8_u291
> > and Debian
> > >> > 7 and experiencing
> > >> > > problems with load balancing since version 1.13.2.
>
> > >> > >
> > >> > > I'm fully aware of Issue Nifi-8643 and tested alot
> around
> > this, but
> > >> > gotta say, that this
> > >> > > is not our problem. Mainly because the balance port
> never
> > binds to
> > >> localhost,
> > >> > but also because I
> > >> > > implemented all workarounds under version 1.13.2
> and even
> > tried
> > >> version
> > >> > 1.14.0 by now,
> > >> > > but load blancing still does not work.
> > >> > > What we experience is best described as "the
> primary
> > node balances
> > >> > with itself"...
> > >> > >
> > >> > > So what it does is, opening the balancing connections
>
> > to its own IP
> > >> > instead of the IPs
> > >> > > of the other two nodes. And the other two nodes don't
>
> > open balancing
> > >> > connections at all.
> > >> > >
> > >> > > When executing "ss | grep 6342" on the
> primary
> > node, this
> > >> > is what it looks like:
> > >> > >
> > >> > > [root@nifiHost1 conf]# ss | grep 6342
> > >> > > tcp ESTAB 0 0 192.168.1.10:51380
> <
> >
> > >> http://192.168.1.10:51380/>
> > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:51376
> <
> >
> > >> http://192.168.1.10:51376/>
> > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:51378
> <
> >
> > >> http://192.168.1.10:51378/>
> > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:51370
> <
> >
> > >> http://192.168.1.10:51370/>
> > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:51372
> <
> >
> > >> http://192.168.1.10:51372/>
> > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:6342
> <
> >
> > >> http://192.168.1.10:6342/>
> > >> > 192.168.1.10:51376 <http://192.168.1.10:51376/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:51374
> <
> >
> > >> http://192.168.1.10:51374/>
> > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:6342
> <
> >
> > >> http://192.168.1.10:6342/>
> > >> > 192.168.1.10:51374 <http://192.168.1.10:51374/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:51366
> <
> >
> > >> http://192.168.1.10:51366/>
> > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:6342
> <
> >
> > >> http://192.168.1.10:6342/>
> > >> > 192.168.1.10:51370 <http://192.168.1.10:51370/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:6342
> <
> >
> > >> http://192.168.1.10:6342/>
> > >> > 192.168.1.10:51366 <http://192.168.1.10:51366/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:51368
> <
> >
> > >> http://192.168.1.10:51368/>
> > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:6342
> <
> >
> > >> http://192.168.1.10:6342/>
> > >> > 192.168.1.10:51372 <http://192.168.1.10:51372/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:6342
> <
> >
> > >> http://192.168.1.10:6342/>
> > >> > 192.168.1.10:51378 <http://192.168.1.10:51378/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:6342
> <
> >
> > >> http://192.168.1.10:6342/>
> > >> > 192.168.1.10:51368 <http://192.168.1.10:51368/>
>
> >
> > >> >
> > >> > > tcp ESTAB 0 0 192.168.1.10:6342
> <
> >
> > >> http://192.168.1.10:6342/>
> > >> > 192.168.1.10:51380 <http://192.168.1.10:51380/>
>
> >
> > >> > >
> > >> > > Executing it on the other non primary nodes, just
> returns
> > absolutely
> > >> > nothing.
> > >> > >
> > >> > > Netstat show the following on each server:
> > >> > >
> > >> > > [root@nifiHost1 conf]# netstat -tulpn
> > >> > > Active Internet connections (only servers)
> > >> > > Proto Recv-Q Send-Q Local Address Foreign
> Address
> >
> > >> > State PID/Program name
> > >> > > tcp 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >
> > >> > 0.0.0.0:* LISTEN 10352/java
>
> > >> > >
> > >> > > [root@nifiHost2 conf]# netstat -tulpn
> > >> > > Active Internet connections (only servers)
> > >> > > Proto Recv-Q Send-Q Local Address Foreign
> Address
> >
> > >> > State PID/Program name
> > >> > > tcp 0 0 192.168.1.11:6342 <http://192.168.1.11:6342/>
>
> >
> > >> > 0.0.0.0:* LISTEN 31562/java
>
> > >> > >
> > >> > > [root@nifiHost3 conf]# netstat -tulpn
> > >> > > Active Internet connections (only servers)
> > >> > > Proto Recv-Q Send-Q Local Address Foreign
> Address
> >
> > >> > State PID/Program name
> > >> > > tcp 0 0 192.168.1.12:6342 <http://192.168.1.12:6342/>
>
> >
> > >> > 0.0.0.0:* LISTEN 31685/java
>
> > >> > >
> > >> > > And here is what our load balancing properties look
> like:
> >
> > >> > >
> > >> > > # cluster load balancing properties #
> > >> > > nifi.cluster.load.balance.host=nifiHost1.contoso.com
> <
> >
> > >> http://nifihost1.contoso.com/>
> > >> >
> > >> > > nifi.cluster.load.balance.address=0.0.0.0
> > >> > > nifi.cluster.load.balance.port=6342
> > >> > > nifi.cluster.load.balance.connections.per.node=4
>
> > >> > > nifi.cluster.load.balance.max.thread.count=8
> > >> > > nifi.cluster.load.balance.comms.timeout=30 sec
> > >> > >
> > >> > > When running Nifi in version 1.12.1 on the exact
> same
> > setup in the
> > >> exact
> > >> > same environment, load balancing is working absolutely
> fine.
> >
> > >> > > There was a time when load balancing even worked
> in version
> > 1.13.2.
> > >> > But I'm not able to reproduce this and it just stopped
>
> > >> > > working one day after some restart, without changing
> any
> > property or
> > >> > whatsoever.
> > >> > >
> > >> > > If any more information would be helpful please let
> me
> > know and I'll
> > >> > try to provide it as fast as possible.
> > >> > >
> > >> > >
> > >> > >
> > >> > > Versendet mit Emailn.de <https://www.emailn.de/>
>
> > - Freemail
> > >> > >
> > >> > > * Unbegrenzt Speicherplatz
> > >> > > * Eigenes Online-Büro
> > >> > > * 24h besten Mailempfang
> > >> > > * Spamschutz, Adressbuch
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > Versendet mit Emailn.de <https://www.emailn.de/>
>
> > - Freemail
> > >> >
> > >> > >
> > >> > > * Unbegrenzt Speicherplatz
> > >> > > * Eigenes Online-Büro
> > >> > > * 24h besten Mailempfang
> > >> > > * Spamschutz, Adressbuch
> > >> > >
> > >> >
> > >> >
> > >>
> > >>
> > >>
> > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
>
> >
> > Virus-free.
> > www.avast.com
> > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
>
> >
> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >
>
>
>
Re: Re: Re: No Load Balancing since 1.13.2
Posted by Axel Schwarz <Ax...@emailn.de>.
I've tried every single combination of values for nifi.cluster.load.balance.host and nifi.cluster.load.balance.address.
But again, the problem described in Nifi-8643 is not what happens to us. The port always binds to the correct IP and NOT to localhost.
As seen in the netstat-excerpts from my original post. Instead the primary node actually opens the desired count of connections for load balancing, but ALL to itself...
--- Ursprüngliche Nachricht ---
Von: Joe Gresock <jg...@gmail.com>
Datum: 28.07.2021 12:09:42
An: users@nifi.apache.org
Betreff: Re: Re: No Load Balancing since 1.13.2
> Also, make sure to use nifi.cluster.load.balance.host instead of .address
>
> if you're still running 1.14.0.
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
>
> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Wed, Jul 28, 2021 at 6:04 AM Jens M. Kofoed <jm...@gmail.com>
>
> wrote:
>
> > hi
> >
> > I can see that you have configured
> > nifi.cluster.load.balance.address=0.0.0.0
> > Have your tried to set the correct ip adress?
> > node1: nifi.cluster.load.balance.address=192.168.1.10
> > node2: nifi.cluster.load.balance.address=192.168.1.11
> > node3: nifi.cluster.load.balance.address=192.168.1.12
> >
> > regards
> > Jens M. Kofoed
> >
> > Den ons. 28. jul. 2021 kl. 11.17 skrev Axel Schwarz <Axelkopter@emailn.de
>
> > >:
> >
> >> Just tried Java 11. But still does not work. Nothing changed. :(
>
> >>
> >> --- Ursprüngliche Nachricht ---
> >> Von: Jorge Machado <jo...@me.com>
> >> Datum: 27.07.2021 13:08:55
> >> An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>
> >> Betreff: Re: No Load Balancing since 1.13.2
> >>
> >> > Did you tried java 11 ? I have a client running a similar setup
> to yours
> >> > but with a lower nigh version and it works fine. Maybe it is
> worth to
> >> try
> >> > it.
> >> >
> >> >
> >> > > On 27. Jul 2021, at 12:42, Axel Schwarz <Ax...@emailn.de>
>
> >> > wrote:
> >> > >
> >> > > I did indeed, but I updated from u161 to u291, as this
> was the newest
> >> > version at that time, because I thought it could help.
> >> > > So the issue started under u161. But I just saw that u301
> is out. I
> >> > will try this as well.
> >> > > --- Ursprüngliche Nachricht ---
> >> > > Von: Pierre Villard <pi...@gmail.com>
>
> >> > > Datum: 27.07.2021 10:18:38
> >> > > An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>
> >> >
> >> > > Betreff: Re: No Load Balancing since 1.13.2
> >> > >
> >> > > Hi,
> >> > >
> >> > > I believe the minor u291 is known to have issues (for
> some of its
> >> early
> >> > builds). Did you upgrade the Java version recently?
> >> > >
> >> > > Thanks,
> >> > > Pierre
> >> > >
> >> > > Le mar. 27 juil. 2021 à 08:07, Axel Schwarz <Axelkopter@emailn.de
>
> >> > <ma...@emailn.de>> a écrit :
> >> > > Dear Community,
> >> > >
> >> > > we're running a secured 3 node Nifi Cluster on Java 8_u291
> and Debian
> >> > 7 and experiencing
> >> > > problems with load balancing since version 1.13.2.
> >> > >
> >> > > I'm fully aware of Issue Nifi-8643 and tested alot around
> this, but
> >> > gotta say, that this
> >> > > is not our problem. Mainly because the balance port never
> binds to
> >> localhost,
> >> > but also because I
> >> > > implemented all workarounds under version 1.13.2 and even
> tried
> >> version
> >> > 1.14.0 by now,
> >> > > but load blancing still does not work.
> >> > > What we experience is best described as "the primary
> node balances
> >> > with itself"...
> >> > >
> >> > > So what it does is, opening the balancing connections
> to its own IP
> >> > instead of the IPs
> >> > > of the other two nodes. And the other two nodes don't
> open balancing
> >> > connections at all.
> >> > >
> >> > > When executing "ss | grep 6342" on the primary
> node, this
> >> > is what it looks like:
> >> > >
> >> > > [root@nifiHost1 conf]# ss | grep 6342
> >> > > tcp ESTAB 0 0 192.168.1.10:51380 <
>
> >> http://192.168.1.10:51380/>
> >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:51376 <
>
> >> http://192.168.1.10:51376/>
> >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:51378 <
>
> >> http://192.168.1.10:51378/>
> >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:51370 <
>
> >> http://192.168.1.10:51370/>
> >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:51372 <
>
> >> http://192.168.1.10:51372/>
> >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>
> >> http://192.168.1.10:6342/>
> >> > 192.168.1.10:51376 <http://192.168.1.10:51376/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:51374 <
>
> >> http://192.168.1.10:51374/>
> >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>
> >> http://192.168.1.10:6342/>
> >> > 192.168.1.10:51374 <http://192.168.1.10:51374/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:51366 <
>
> >> http://192.168.1.10:51366/>
> >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>
> >> http://192.168.1.10:6342/>
> >> > 192.168.1.10:51370 <http://192.168.1.10:51370/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>
> >> http://192.168.1.10:6342/>
> >> > 192.168.1.10:51366 <http://192.168.1.10:51366/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:51368 <
>
> >> http://192.168.1.10:51368/>
> >> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>
> >> http://192.168.1.10:6342/>
> >> > 192.168.1.10:51372 <http://192.168.1.10:51372/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>
> >> http://192.168.1.10:6342/>
> >> > 192.168.1.10:51378 <http://192.168.1.10:51378/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>
> >> http://192.168.1.10:6342/>
> >> > 192.168.1.10:51368 <http://192.168.1.10:51368/>
>
> >> >
> >> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>
> >> http://192.168.1.10:6342/>
> >> > 192.168.1.10:51380 <http://192.168.1.10:51380/>
>
> >> > >
> >> > > Executing it on the other non primary nodes, just returns
> absolutely
> >> > nothing.
> >> > >
> >> > > Netstat show the following on each server:
> >> > >
> >> > > [root@nifiHost1 conf]# netstat -tulpn
> >> > > Active Internet connections (only servers)
> >> > > Proto Recv-Q Send-Q Local Address Foreign Address
>
> >> > State PID/Program name
> >> > > tcp 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> >> > 0.0.0.0:* LISTEN 10352/java
> >> > >
> >> > > [root@nifiHost2 conf]# netstat -tulpn
> >> > > Active Internet connections (only servers)
> >> > > Proto Recv-Q Send-Q Local Address Foreign Address
>
> >> > State PID/Program name
> >> > > tcp 0 0 192.168.1.11:6342 <http://192.168.1.11:6342/>
>
> >> > 0.0.0.0:* LISTEN 31562/java
> >> > >
> >> > > [root@nifiHost3 conf]# netstat -tulpn
> >> > > Active Internet connections (only servers)
> >> > > Proto Recv-Q Send-Q Local Address Foreign Address
>
> >> > State PID/Program name
> >> > > tcp 0 0 192.168.1.12:6342 <http://192.168.1.12:6342/>
>
> >> > 0.0.0.0:* LISTEN 31685/java
> >> > >
> >> > > And here is what our load balancing properties look like:
>
> >> > >
> >> > > # cluster load balancing properties #
> >> > > nifi.cluster.load.balance.host=nifiHost1.contoso.com <
>
> >> http://nifihost1.contoso.com/>
> >> >
> >> > > nifi.cluster.load.balance.address=0.0.0.0
> >> > > nifi.cluster.load.balance.port=6342
> >> > > nifi.cluster.load.balance.connections.per.node=4
> >> > > nifi.cluster.load.balance.max.thread.count=8
> >> > > nifi.cluster.load.balance.comms.timeout=30 sec
> >> > >
> >> > > When running Nifi in version 1.12.1 on the exact same
> setup in the
> >> exact
> >> > same environment, load balancing is working absolutely fine.
>
> >> > > There was a time when load balancing even worked in version
> 1.13.2.
> >> > But I'm not able to reproduce this and it just stopped
> >> > > working one day after some restart, without changing any
> property or
> >> > whatsoever.
> >> > >
> >> > > If any more information would be helpful please let me
> know and I'll
> >> > try to provide it as fast as possible.
> >> > >
> >> > >
> >> > >
> >> > > Versendet mit Emailn.de <https://www.emailn.de/>
> - Freemail
> >> > >
> >> > > * Unbegrenzt Speicherplatz
> >> > > * Eigenes Online-Büro
> >> > > * 24h besten Mailempfang
> >> > > * Spamschutz, Adressbuch
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > Versendet mit Emailn.de <https://www.emailn.de/>
> - Freemail
> >> >
> >> > >
> >> > > * Unbegrenzt Speicherplatz
> >> > > * Eigenes Online-Büro
> >> > > * 24h besten Mailempfang
> >> > > * Spamschutz, Adressbuch
> >> > >
> >> >
> >> >
> >>
> >>
> >>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
>
> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
Re: Re: No Load Balancing since 1.13.2
Posted by Joe Gresock <jg...@gmail.com>.
Also, make sure to use nifi.cluster.load.balance.host instead of .address
if you're still running 1.14.0.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
On Wed, Jul 28, 2021 at 6:04 AM Jens M. Kofoed <jm...@gmail.com>
wrote:
> hi
>
> I can see that you have configured
> nifi.cluster.load.balance.address=0.0.0.0
> Have your tried to set the correct ip adress?
> node1: nifi.cluster.load.balance.address=192.168.1.10
> node2: nifi.cluster.load.balance.address=192.168.1.11
> node3: nifi.cluster.load.balance.address=192.168.1.12
>
> regards
> Jens M. Kofoed
>
> Den ons. 28. jul. 2021 kl. 11.17 skrev Axel Schwarz <Axelkopter@emailn.de
> >:
>
>> Just tried Java 11. But still does not work. Nothing changed. :(
>>
>> --- Ursprüngliche Nachricht ---
>> Von: Jorge Machado <jo...@me.com>
>> Datum: 27.07.2021 13:08:55
>> An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>> Betreff: Re: No Load Balancing since 1.13.2
>>
>> > Did you tried java 11 ? I have a client running a similar setup to yours
>> > but with a lower nigh version and it works fine. Maybe it is worth to
>> try
>> > it.
>> >
>> >
>> > > On 27. Jul 2021, at 12:42, Axel Schwarz <Ax...@emailn.de>
>> > wrote:
>> > >
>> > > I did indeed, but I updated from u161 to u291, as this was the newest
>> > version at that time, because I thought it could help.
>> > > So the issue started under u161. But I just saw that u301 is out. I
>> > will try this as well.
>> > > --- Ursprüngliche Nachricht ---
>> > > Von: Pierre Villard <pi...@gmail.com>
>> > > Datum: 27.07.2021 10:18:38
>> > > An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>> >
>> > > Betreff: Re: No Load Balancing since 1.13.2
>> > >
>> > > Hi,
>> > >
>> > > I believe the minor u291 is known to have issues (for some of its
>> early
>> > builds). Did you upgrade the Java version recently?
>> > >
>> > > Thanks,
>> > > Pierre
>> > >
>> > > Le mar. 27 juil. 2021 à 08:07, Axel Schwarz <Axelkopter@emailn.de
>> > <ma...@emailn.de>> a écrit :
>> > > Dear Community,
>> > >
>> > > we're running a secured 3 node Nifi Cluster on Java 8_u291 and Debian
>> > 7 and experiencing
>> > > problems with load balancing since version 1.13.2.
>> > >
>> > > I'm fully aware of Issue Nifi-8643 and tested alot around this, but
>> > gotta say, that this
>> > > is not our problem. Mainly because the balance port never binds to
>> localhost,
>> > but also because I
>> > > implemented all workarounds under version 1.13.2 and even tried
>> version
>> > 1.14.0 by now,
>> > > but load blancing still does not work.
>> > > What we experience is best described as "the primary node balances
>> > with itself"...
>> > >
>> > > So what it does is, opening the balancing connections to its own IP
>> > instead of the IPs
>> > > of the other two nodes. And the other two nodes don't open balancing
>> > connections at all.
>> > >
>> > > When executing "ss | grep 6342" on the primary node, this
>> > is what it looks like:
>> > >
>> > > [root@nifiHost1 conf]# ss | grep 6342
>> > > tcp ESTAB 0 0 192.168.1.10:51380 <
>> http://192.168.1.10:51380/>
>> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:51376 <
>> http://192.168.1.10:51376/>
>> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:51378 <
>> http://192.168.1.10:51378/>
>> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:51370 <
>> http://192.168.1.10:51370/>
>> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:51372 <
>> http://192.168.1.10:51372/>
>> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> http://192.168.1.10:6342/>
>> > 192.168.1.10:51376 <http://192.168.1.10:51376/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:51374 <
>> http://192.168.1.10:51374/>
>> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> http://192.168.1.10:6342/>
>> > 192.168.1.10:51374 <http://192.168.1.10:51374/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:51366 <
>> http://192.168.1.10:51366/>
>> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> http://192.168.1.10:6342/>
>> > 192.168.1.10:51370 <http://192.168.1.10:51370/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> http://192.168.1.10:6342/>
>> > 192.168.1.10:51366 <http://192.168.1.10:51366/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:51368 <
>> http://192.168.1.10:51368/>
>> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> http://192.168.1.10:6342/>
>> > 192.168.1.10:51372 <http://192.168.1.10:51372/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> http://192.168.1.10:6342/>
>> > 192.168.1.10:51378 <http://192.168.1.10:51378/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> http://192.168.1.10:6342/>
>> > 192.168.1.10:51368 <http://192.168.1.10:51368/>
>> >
>> > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> http://192.168.1.10:6342/>
>> > 192.168.1.10:51380 <http://192.168.1.10:51380/>
>> > >
>> > > Executing it on the other non primary nodes, just returns absolutely
>> > nothing.
>> > >
>> > > Netstat show the following on each server:
>> > >
>> > > [root@nifiHost1 conf]# netstat -tulpn
>> > > Active Internet connections (only servers)
>> > > Proto Recv-Q Send-Q Local Address Foreign Address
>> > State PID/Program name
>> > > tcp 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> > 0.0.0.0:* LISTEN 10352/java
>> > >
>> > > [root@nifiHost2 conf]# netstat -tulpn
>> > > Active Internet connections (only servers)
>> > > Proto Recv-Q Send-Q Local Address Foreign Address
>> > State PID/Program name
>> > > tcp 0 0 192.168.1.11:6342 <http://192.168.1.11:6342/>
>> > 0.0.0.0:* LISTEN 31562/java
>> > >
>> > > [root@nifiHost3 conf]# netstat -tulpn
>> > > Active Internet connections (only servers)
>> > > Proto Recv-Q Send-Q Local Address Foreign Address
>> > State PID/Program name
>> > > tcp 0 0 192.168.1.12:6342 <http://192.168.1.12:6342/>
>> > 0.0.0.0:* LISTEN 31685/java
>> > >
>> > > And here is what our load balancing properties look like:
>> > >
>> > > # cluster load balancing properties #
>> > > nifi.cluster.load.balance.host=nifiHost1.contoso.com <
>> http://nifihost1.contoso.com/>
>> >
>> > > nifi.cluster.load.balance.address=0.0.0.0
>> > > nifi.cluster.load.balance.port=6342
>> > > nifi.cluster.load.balance.connections.per.node=4
>> > > nifi.cluster.load.balance.max.thread.count=8
>> > > nifi.cluster.load.balance.comms.timeout=30 sec
>> > >
>> > > When running Nifi in version 1.12.1 on the exact same setup in the
>> exact
>> > same environment, load balancing is working absolutely fine.
>> > > There was a time when load balancing even worked in version 1.13.2.
>> > But I'm not able to reproduce this and it just stopped
>> > > working one day after some restart, without changing any property or
>> > whatsoever.
>> > >
>> > > If any more information would be helpful please let me know and I'll
>> > try to provide it as fast as possible.
>> > >
>> > >
>> > >
>> > > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
>> > >
>> > > * Unbegrenzt Speicherplatz
>> > > * Eigenes Online-Büro
>> > > * 24h besten Mailempfang
>> > > * Spamschutz, Adressbuch
>> > >
>> > >
>> > >
>> > >
>> > > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
>> >
>> > >
>> > > * Unbegrenzt Speicherplatz
>> > > * Eigenes Online-Büro
>> > > * 24h besten Mailempfang
>> > > * Spamschutz, Adressbuch
>> > >
>> >
>> >
>>
>>
>>
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: Re: Re: No Load Balancing since 1.13.2
Posted by "Jens M. Kofoed" <jm...@gmail.com>.
Hi Axel
In version 1.13.2 I had to change the nifi.cluster.load.balance.host to
nifi.cluster.load.balance.address and I used the IP address to get it to
work.
In version 1.14.0 they have fixed the bug so now is actually use the
nifi.cluster.load.balance.host property so I have to change back again in
my property. But It still doesn't work if I use the hostname. I still have
to use the IP address.
I hope it is working for you.
Kind Regards
Jens M. Kofoed
Den tor. 29. jul. 2021 kl. 11.08 skrev Jens M. Kofoed <
jmkofoed.ube@gmail.com>:
> Hmm... I can't remember :-( sorry
>
> My configuration for version 1.13.2 is like this:
> # cluster node properties (only configure for cluster nodes) #
> nifi.cluster.is.node=true
> nifi.cluster.node.address=nifi-node01.domaine.com
> nifi.cluster.node.protocol.port=9443
> nifi.cluster.node.protocol.threads=10
> nifi.cluster.node.protocol.max.threads=50
> nifi.cluster.node.event.history.size=25
> nifi.cluster.node.connection.timeout=5 sec
> nifi.cluster.node.read.timeout=5 sec
> nifi.cluster.node.max.concurrent.requests=100
> nifi.cluster.firewall.file=
> nifi.cluster.flow.election.max.wait.time=5 mins
> nifi.cluster.flow.election.max.candidates=3
>
> # cluster load balancing properties #
> nifi.cluster.load.balance.address=192.168.1.11
> nifi.cluster.load.balance.port=6111
> nifi.cluster.load.balance.connections.per.node=4
> nifi.cluster.load.balance.max.thread.count=8
> nifi.cluster.load.balance.comms.timeout=30 sec
>
> So I defined "nifi.cluster.node.address" with the hostname and not an ip
> adress and the "nifi.cluster.load.balance.address" with the ip address of
> the server.
> And triple check the configuration at all servers :-)
>
> Kind Regards
> Jens M. Kofoed
>
>
> Den tor. 29. jul. 2021 kl. 10.11 skrev Axel Schwarz <Axelkopter@emailn.de
> >:
>
>> Hey Jens,
>>
>> in Issue Nifi-8643 you wrote the last comment with the exactly same
>> behaviour as we're experiencing now. 2 of 3 nodes were load balancing.
>> How did you get the third node to participate in load balancing? An
>> update to 1.14.0 does not change anything for us.
>>
>>
>> https://issues.apache.org/jira/browse/NIFI-8643?focusedCommentId=17361418&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17361418
>>
>>
>> --- Ursprüngliche Nachricht ---
>> Von: "Jens M. Kofoed" <jm...@gmail.com>
>> Datum: 28.07.2021 12:07:50
>> An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>> Betreff: Re: Re: No Load Balancing since 1.13.2
>>
>> > hi
>> >
>> > I can see that you have configured
>> nifi.cluster.load.balance.address=0.0.0.0
>> >
>> > Have your tried to set the correct ip adress?
>> > node1: nifi.cluster.load.balance.address=192.168.1.10
>> > node2: nifi.cluster.load.balance.address=192.168.1.11
>> > node3: nifi.cluster.load.balance.address=192.168.1.12
>> >
>> > regards
>> > Jens M. Kofoed
>> >
>> > Den ons. 28. jul. 2021 kl. 11.17 skrev Axel Schwarz <
>> Axelkopter@emailn.de>:
>> >
>> >
>> > > Just tried Java 11. But still does not work. Nothing changed. :(
>> > >
>> > > --- Ursprüngliche Nachricht ---
>> > > Von: Jorge Machado <jo...@me.com>
>> > > Datum: 27.07.2021 13:08:55
>> > > An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>> >
>> > > Betreff: Re: No Load Balancing since 1.13.2
>> > >
>> > > > Did you tried java 11 ? I have a client running a similar setup
>> > to yours
>> > > > but with a lower nigh version and it works fine. Maybe it is worth
>> > to try
>> > > > it.
>> > > >
>> > > >
>> > > > > On 27. Jul 2021, at 12:42, Axel Schwarz <Ax...@emailn.de>
>> >
>> > > > wrote:
>> > > > >
>> > > > > I did indeed, but I updated from u161 to u291, as this was
>> > the newest
>> > > > version at that time, because I thought it could help.
>> > > > > So the issue started under u161. But I just saw that u301
>> > is out. I
>> > > > will try this as well.
>> > > > > --- Ursprüngliche Nachricht ---
>> > > > > Von: Pierre Villard <pi...@gmail.com>
>> > > > > Datum: 27.07.2021 10:18:38
>> > > > > An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>> >
>> > > >
>> > > > > Betreff: Re: No Load Balancing since 1.13.2
>> > > > >
>> > > > > Hi,
>> > > > >
>> > > > > I believe the minor u291 is known to have issues (for some
>> > of its early
>> > > > builds). Did you upgrade the Java version recently?
>> > > > >
>> > > > > Thanks,
>> > > > > Pierre
>> > > > >
>> > > > > Le mar. 27 juil. 2021 à 08:07, Axel Schwarz <Axelkopter@emailn.de
>> >
>> > > > <ma...@emailn.de>> a écrit :
>> > > > > Dear Community,
>> > > > >
>> > > > > we're running a secured 3 node Nifi Cluster on Java 8_u291
>> > and Debian
>> > > > 7 and experiencing
>> > > > > problems with load balancing since version 1.13.2.
>> > > > >
>> > > > > I'm fully aware of Issue Nifi-8643 and tested alot around
>> > this, but
>> > > > gotta say, that this
>> > > > > is not our problem. Mainly because the balance port never
>> > binds to
>> > > localhost,
>> > > > but also because I
>> > > > > implemented all workarounds under version 1.13.2 and even
>> > tried version
>> > > > 1.14.0 by now,
>> > > > > but load blancing still does not work.
>> > > > > What we experience is best described as "the primary
>> > node balances
>> > > > with itself"...
>> > > > >
>> > > > > So what it does is, opening the balancing connections to its
>> > own IP
>> > > > instead of the IPs
>> > > > > of the other two nodes. And the other two nodes don't open
>> > balancing
>> > > > connections at all.
>> > > > >
>> > > > > When executing "ss | grep 6342" on the primary node,
>> > this
>> > > > is what it looks like:
>> > > > >
>> > > > > [root@nifiHost1 conf]# ss | grep 6342
>> > > > > tcp ESTAB 0 0 192.168.1.10:51380 <
>> > > http://192.168.1.10:51380/>
>> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:51376 <
>> > > http://192.168.1.10:51376/>
>> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:51378 <
>> > > http://192.168.1.10:51378/>
>> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:51370 <
>> > > http://192.168.1.10:51370/>
>> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:51372 <
>> > > http://192.168.1.10:51372/>
>> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> > > http://192.168.1.10:6342/>
>> > > > 192.168.1.10:51376 <http://192.168.1.10:51376/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:51374 <
>> > > http://192.168.1.10:51374/>
>> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> > > http://192.168.1.10:6342/>
>> > > > 192.168.1.10:51374 <http://192.168.1.10:51374/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:51366 <
>> > > http://192.168.1.10:51366/>
>> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> > > http://192.168.1.10:6342/>
>> > > > 192.168.1.10:51370 <http://192.168.1.10:51370/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> > > http://192.168.1.10:6342/>
>> > > > 192.168.1.10:51366 <http://192.168.1.10:51366/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:51368 <
>> > > http://192.168.1.10:51368/>
>> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> > > http://192.168.1.10:6342/>
>> > > > 192.168.1.10:51372 <http://192.168.1.10:51372/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> > > http://192.168.1.10:6342/>
>> > > > 192.168.1.10:51378 <http://192.168.1.10:51378/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> > > http://192.168.1.10:6342/>
>> > > > 192.168.1.10:51368 <http://192.168.1.10:51368/>
>> >
>> > > >
>> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
>> > > http://192.168.1.10:6342/>
>> > > > 192.168.1.10:51380 <http://192.168.1.10:51380/>
>> >
>> > > > >
>> > > > > Executing it on the other non primary nodes, just returns
>> > absolutely
>> > > > nothing.
>> > > > >
>> > > > > Netstat show the following on each server:
>> > > > >
>> > > > > [root@nifiHost1 conf]# netstat -tulpn
>> > > > > Active Internet connections (only servers)
>> > > > > Proto Recv-Q Send-Q Local Address Foreign Address
>> >
>> > > > State PID/Program name
>> > > > > tcp 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
>> >
>> > > > 0.0.0.0:* LISTEN 10352/java
>> > > > >
>> > > > > [root@nifiHost2 conf]# netstat -tulpn
>> > > > > Active Internet connections (only servers)
>> > > > > Proto Recv-Q Send-Q Local Address Foreign Address
>> >
>> > > > State PID/Program name
>> > > > > tcp 0 0 192.168.1.11:6342 <http://192.168.1.11:6342/>
>> >
>> > > > 0.0.0.0:* LISTEN 31562/java
>> > > > >
>> > > > > [root@nifiHost3 conf]# netstat -tulpn
>> > > > > Active Internet connections (only servers)
>> > > > > Proto Recv-Q Send-Q Local Address Foreign Address
>> >
>> > > > State PID/Program name
>> > > > > tcp 0 0 192.168.1.12:6342 <http://192.168.1.12:6342/>
>> >
>> > > > 0.0.0.0:* LISTEN 31685/java
>> > > > >
>> > > > > And here is what our load balancing properties look like:
>> >
>> > > > >
>> > > > > # cluster load balancing properties #
>> > > > > nifi.cluster.load.balance.host=nifiHost1.contoso.com <
>> >
>> > > http://nifihost1.contoso.com/>
>> > > >
>> > > > > nifi.cluster.load.balance.address=0.0.0.0
>> > > > > nifi.cluster.load.balance.port=6342
>> > > > > nifi.cluster.load.balance.connections.per.node=4
>> > > > > nifi.cluster.load.balance.max.thread.count=8
>> > > > > nifi.cluster.load.balance.comms.timeout=30 sec
>> > > > >
>> > > > > When running Nifi in version 1.12.1 on the exact same setup
>> > in the
>> > > exact
>> > > > same environment, load balancing is working absolutely fine.
>> > > > > There was a time when load balancing even worked in version
>> > 1.13.2.
>> > > > But I'm not able to reproduce this and it just stopped
>> > > > > working one day after some restart, without changing any property
>> > or
>> > > > whatsoever.
>> > > > >
>> > > > > If any more information would be helpful please let me know
>> > and I'll
>> > > > try to provide it as fast as possible.
>> > > > >
>> > > > >
>> > > > >
>> > > > > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
>> >
>> > > > >
>> > > > > * Unbegrenzt Speicherplatz
>> > > > > * Eigenes Online-Büro
>> > > > > * 24h besten Mailempfang
>> > > > > * Spamschutz, Adressbuch
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
>> >
>> > > >
>> > > > >
>> > > > > * Unbegrenzt Speicherplatz
>> > > > > * Eigenes Online-Büro
>> > > > > * 24h besten Mailempfang
>> > > > > * Spamschutz, Adressbuch
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> >
>>
>>
>>
Re: Re: Re: No Load Balancing since 1.13.2
Posted by "Jens M. Kofoed" <jm...@gmail.com>.
Hmm... I can't remember :-( sorry
My configuration for version 1.13.2 is like this:
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=nifi-node01.domaine.com
nifi.cluster.node.protocol.port=9443
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.flow.election.max.candidates=3
# cluster load balancing properties #
nifi.cluster.load.balance.address=192.168.1.11
nifi.cluster.load.balance.port=6111
nifi.cluster.load.balance.connections.per.node=4
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec
So I defined "nifi.cluster.node.address" with the hostname and not an ip
adress and the "nifi.cluster.load.balance.address" with the ip address of
the server.
And triple check the configuration at all servers :-)
Kind Regards
Jens M. Kofoed
Den tor. 29. jul. 2021 kl. 10.11 skrev Axel Schwarz <Ax...@emailn.de>:
> Hey Jens,
>
> in Issue Nifi-8643 you wrote the last comment with the exactly same
> behaviour as we're experiencing now. 2 of 3 nodes were load balancing.
> How did you get the third node to participate in load balancing? An update
> to 1.14.0 does not change anything for us.
>
>
> https://issues.apache.org/jira/browse/NIFI-8643?focusedCommentId=17361418&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17361418
>
>
> --- Ursprüngliche Nachricht ---
> Von: "Jens M. Kofoed" <jm...@gmail.com>
> Datum: 28.07.2021 12:07:50
> An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
> Betreff: Re: Re: No Load Balancing since 1.13.2
>
> > hi
> >
> > I can see that you have configured
> nifi.cluster.load.balance.address=0.0.0.0
> >
> > Have your tried to set the correct ip adress?
> > node1: nifi.cluster.load.balance.address=192.168.1.10
> > node2: nifi.cluster.load.balance.address=192.168.1.11
> > node3: nifi.cluster.load.balance.address=192.168.1.12
> >
> > regards
> > Jens M. Kofoed
> >
> > Den ons. 28. jul. 2021 kl. 11.17 skrev Axel Schwarz <
> Axelkopter@emailn.de>:
> >
> >
> > > Just tried Java 11. But still does not work. Nothing changed. :(
> > >
> > > --- Ursprüngliche Nachricht ---
> > > Von: Jorge Machado <jo...@me.com>
> > > Datum: 27.07.2021 13:08:55
> > > An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
> >
> > > Betreff: Re: No Load Balancing since 1.13.2
> > >
> > > > Did you tried java 11 ? I have a client running a similar setup
> > to yours
> > > > but with a lower nigh version and it works fine. Maybe it is worth
> > to try
> > > > it.
> > > >
> > > >
> > > > > On 27. Jul 2021, at 12:42, Axel Schwarz <Ax...@emailn.de>
> >
> > > > wrote:
> > > > >
> > > > > I did indeed, but I updated from u161 to u291, as this was
> > the newest
> > > > version at that time, because I thought it could help.
> > > > > So the issue started under u161. But I just saw that u301
> > is out. I
> > > > will try this as well.
> > > > > --- Ursprüngliche Nachricht ---
> > > > > Von: Pierre Villard <pi...@gmail.com>
> > > > > Datum: 27.07.2021 10:18:38
> > > > > An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
> >
> > > >
> > > > > Betreff: Re: No Load Balancing since 1.13.2
> > > > >
> > > > > Hi,
> > > > >
> > > > > I believe the minor u291 is known to have issues (for some
> > of its early
> > > > builds). Did you upgrade the Java version recently?
> > > > >
> > > > > Thanks,
> > > > > Pierre
> > > > >
> > > > > Le mar. 27 juil. 2021 à 08:07, Axel Schwarz <Axelkopter@emailn.de
> >
> > > > <ma...@emailn.de>> a écrit :
> > > > > Dear Community,
> > > > >
> > > > > we're running a secured 3 node Nifi Cluster on Java 8_u291
> > and Debian
> > > > 7 and experiencing
> > > > > problems with load balancing since version 1.13.2.
> > > > >
> > > > > I'm fully aware of Issue Nifi-8643 and tested alot around
> > this, but
> > > > gotta say, that this
> > > > > is not our problem. Mainly because the balance port never
> > binds to
> > > localhost,
> > > > but also because I
> > > > > implemented all workarounds under version 1.13.2 and even
> > tried version
> > > > 1.14.0 by now,
> > > > > but load blancing still does not work.
> > > > > What we experience is best described as "the primary
> > node balances
> > > > with itself"...
> > > > >
> > > > > So what it does is, opening the balancing connections to its
> > own IP
> > > > instead of the IPs
> > > > > of the other two nodes. And the other two nodes don't open
> > balancing
> > > > connections at all.
> > > > >
> > > > > When executing "ss | grep 6342" on the primary node,
> > this
> > > > is what it looks like:
> > > > >
> > > > > [root@nifiHost1 conf]# ss | grep 6342
> > > > > tcp ESTAB 0 0 192.168.1.10:51380 <
> > > http://192.168.1.10:51380/>
> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:51376 <
> > > http://192.168.1.10:51376/>
> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:51378 <
> > > http://192.168.1.10:51378/>
> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:51370 <
> > > http://192.168.1.10:51370/>
> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:51372 <
> > > http://192.168.1.10:51372/>
> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > > http://192.168.1.10:6342/>
> > > > 192.168.1.10:51376 <http://192.168.1.10:51376/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:51374 <
> > > http://192.168.1.10:51374/>
> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > > http://192.168.1.10:6342/>
> > > > 192.168.1.10:51374 <http://192.168.1.10:51374/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:51366 <
> > > http://192.168.1.10:51366/>
> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > > http://192.168.1.10:6342/>
> > > > 192.168.1.10:51370 <http://192.168.1.10:51370/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > > http://192.168.1.10:6342/>
> > > > 192.168.1.10:51366 <http://192.168.1.10:51366/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:51368 <
> > > http://192.168.1.10:51368/>
> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > > http://192.168.1.10:6342/>
> > > > 192.168.1.10:51372 <http://192.168.1.10:51372/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > > http://192.168.1.10:6342/>
> > > > 192.168.1.10:51378 <http://192.168.1.10:51378/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > > http://192.168.1.10:6342/>
> > > > 192.168.1.10:51368 <http://192.168.1.10:51368/>
> >
> > > >
> > > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > > http://192.168.1.10:6342/>
> > > > 192.168.1.10:51380 <http://192.168.1.10:51380/>
> >
> > > > >
> > > > > Executing it on the other non primary nodes, just returns
> > absolutely
> > > > nothing.
> > > > >
> > > > > Netstat show the following on each server:
> > > > >
> > > > > [root@nifiHost1 conf]# netstat -tulpn
> > > > > Active Internet connections (only servers)
> > > > > Proto Recv-Q Send-Q Local Address Foreign Address
> >
> > > > State PID/Program name
> > > > > tcp 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > > 0.0.0.0:* LISTEN 10352/java
> > > > >
> > > > > [root@nifiHost2 conf]# netstat -tulpn
> > > > > Active Internet connections (only servers)
> > > > > Proto Recv-Q Send-Q Local Address Foreign Address
> >
> > > > State PID/Program name
> > > > > tcp 0 0 192.168.1.11:6342 <http://192.168.1.11:6342/>
> >
> > > > 0.0.0.0:* LISTEN 31562/java
> > > > >
> > > > > [root@nifiHost3 conf]# netstat -tulpn
> > > > > Active Internet connections (only servers)
> > > > > Proto Recv-Q Send-Q Local Address Foreign Address
> >
> > > > State PID/Program name
> > > > > tcp 0 0 192.168.1.12:6342 <http://192.168.1.12:6342/>
> >
> > > > 0.0.0.0:* LISTEN 31685/java
> > > > >
> > > > > And here is what our load balancing properties look like:
> >
> > > > >
> > > > > # cluster load balancing properties #
> > > > > nifi.cluster.load.balance.host=nifiHost1.contoso.com <
> >
> > > http://nifihost1.contoso.com/>
> > > >
> > > > > nifi.cluster.load.balance.address=0.0.0.0
> > > > > nifi.cluster.load.balance.port=6342
> > > > > nifi.cluster.load.balance.connections.per.node=4
> > > > > nifi.cluster.load.balance.max.thread.count=8
> > > > > nifi.cluster.load.balance.comms.timeout=30 sec
> > > > >
> > > > > When running Nifi in version 1.12.1 on the exact same setup
> > in the
> > > exact
> > > > same environment, load balancing is working absolutely fine.
> > > > > There was a time when load balancing even worked in version
> > 1.13.2.
> > > > But I'm not able to reproduce this and it just stopped
> > > > > working one day after some restart, without changing any property
> > or
> > > > whatsoever.
> > > > >
> > > > > If any more information would be helpful please let me know
> > and I'll
> > > > try to provide it as fast as possible.
> > > > >
> > > > >
> > > > >
> > > > > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
> >
> > > > >
> > > > > * Unbegrenzt Speicherplatz
> > > > > * Eigenes Online-Büro
> > > > > * 24h besten Mailempfang
> > > > > * Spamschutz, Adressbuch
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
> >
> > > >
> > > > >
> > > > > * Unbegrenzt Speicherplatz
> > > > > * Eigenes Online-Büro
> > > > > * 24h besten Mailempfang
> > > > > * Spamschutz, Adressbuch
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> >
>
>
>
Re: Re: Re: No Load Balancing since 1.13.2
Posted by Axel Schwarz <Ax...@emailn.de>.
Hey Jens,
in Issue Nifi-8643 you wrote the last comment with the exactly same behaviour as we're experiencing now. 2 of 3 nodes were load balancing.
How did you get the third node to participate in load balancing? An update to 1.14.0 does not change anything for us.
https://issues.apache.org/jira/browse/NIFI-8643?focusedCommentId=17361418&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17361418
--- Ursprüngliche Nachricht ---
Von: "Jens M. Kofoed" <jm...@gmail.com>
Datum: 28.07.2021 12:07:50
An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
Betreff: Re: Re: No Load Balancing since 1.13.2
> hi
>
> I can see that you have configured nifi.cluster.load.balance.address=0.0.0.0
>
> Have your tried to set the correct ip adress?
> node1: nifi.cluster.load.balance.address=192.168.1.10
> node2: nifi.cluster.load.balance.address=192.168.1.11
> node3: nifi.cluster.load.balance.address=192.168.1.12
>
> regards
> Jens M. Kofoed
>
> Den ons. 28. jul. 2021 kl. 11.17 skrev Axel Schwarz <Ax...@emailn.de>:
>
>
> > Just tried Java 11. But still does not work. Nothing changed. :(
> >
> > --- Ursprüngliche Nachricht ---
> > Von: Jorge Machado <jo...@me.com>
> > Datum: 27.07.2021 13:08:55
> > An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>
> > Betreff: Re: No Load Balancing since 1.13.2
> >
> > > Did you tried java 11 ? I have a client running a similar setup
> to yours
> > > but with a lower nigh version and it works fine. Maybe it is worth
> to try
> > > it.
> > >
> > >
> > > > On 27. Jul 2021, at 12:42, Axel Schwarz <Ax...@emailn.de>
>
> > > wrote:
> > > >
> > > > I did indeed, but I updated from u161 to u291, as this was
> the newest
> > > version at that time, because I thought it could help.
> > > > So the issue started under u161. But I just saw that u301
> is out. I
> > > will try this as well.
> > > > --- Ursprüngliche Nachricht ---
> > > > Von: Pierre Villard <pi...@gmail.com>
> > > > Datum: 27.07.2021 10:18:38
> > > > An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>
> > >
> > > > Betreff: Re: No Load Balancing since 1.13.2
> > > >
> > > > Hi,
> > > >
> > > > I believe the minor u291 is known to have issues (for some
> of its early
> > > builds). Did you upgrade the Java version recently?
> > > >
> > > > Thanks,
> > > > Pierre
> > > >
> > > > Le mar. 27 juil. 2021 à 08:07, Axel Schwarz <Axelkopter@emailn.de
>
> > > <ma...@emailn.de>> a écrit :
> > > > Dear Community,
> > > >
> > > > we're running a secured 3 node Nifi Cluster on Java 8_u291
> and Debian
> > > 7 and experiencing
> > > > problems with load balancing since version 1.13.2.
> > > >
> > > > I'm fully aware of Issue Nifi-8643 and tested alot around
> this, but
> > > gotta say, that this
> > > > is not our problem. Mainly because the balance port never
> binds to
> > localhost,
> > > but also because I
> > > > implemented all workarounds under version 1.13.2 and even
> tried version
> > > 1.14.0 by now,
> > > > but load blancing still does not work.
> > > > What we experience is best described as "the primary
> node balances
> > > with itself"...
> > > >
> > > > So what it does is, opening the balancing connections to its
> own IP
> > > instead of the IPs
> > > > of the other two nodes. And the other two nodes don't open
> balancing
> > > connections at all.
> > > >
> > > > When executing "ss | grep 6342" on the primary node,
> this
> > > is what it looks like:
> > > >
> > > > [root@nifiHost1 conf]# ss | grep 6342
> > > > tcp ESTAB 0 0 192.168.1.10:51380 <
> > http://192.168.1.10:51380/>
> > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:51376 <
> > http://192.168.1.10:51376/>
> > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:51378 <
> > http://192.168.1.10:51378/>
> > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:51370 <
> > http://192.168.1.10:51370/>
> > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:51372 <
> > http://192.168.1.10:51372/>
> > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > http://192.168.1.10:6342/>
> > > 192.168.1.10:51376 <http://192.168.1.10:51376/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:51374 <
> > http://192.168.1.10:51374/>
> > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > http://192.168.1.10:6342/>
> > > 192.168.1.10:51374 <http://192.168.1.10:51374/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:51366 <
> > http://192.168.1.10:51366/>
> > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > http://192.168.1.10:6342/>
> > > 192.168.1.10:51370 <http://192.168.1.10:51370/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > http://192.168.1.10:6342/>
> > > 192.168.1.10:51366 <http://192.168.1.10:51366/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:51368 <
> > http://192.168.1.10:51368/>
> > > 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > http://192.168.1.10:6342/>
> > > 192.168.1.10:51372 <http://192.168.1.10:51372/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > http://192.168.1.10:6342/>
> > > 192.168.1.10:51378 <http://192.168.1.10:51378/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > http://192.168.1.10:6342/>
> > > 192.168.1.10:51368 <http://192.168.1.10:51368/>
>
> > >
> > > > tcp ESTAB 0 0 192.168.1.10:6342 <
> > http://192.168.1.10:6342/>
> > > 192.168.1.10:51380 <http://192.168.1.10:51380/>
>
> > > >
> > > > Executing it on the other non primary nodes, just returns
> absolutely
> > > nothing.
> > > >
> > > > Netstat show the following on each server:
> > > >
> > > > [root@nifiHost1 conf]# netstat -tulpn
> > > > Active Internet connections (only servers)
> > > > Proto Recv-Q Send-Q Local Address Foreign Address
>
> > > State PID/Program name
> > > > tcp 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > > 0.0.0.0:* LISTEN 10352/java
> > > >
> > > > [root@nifiHost2 conf]# netstat -tulpn
> > > > Active Internet connections (only servers)
> > > > Proto Recv-Q Send-Q Local Address Foreign Address
>
> > > State PID/Program name
> > > > tcp 0 0 192.168.1.11:6342 <http://192.168.1.11:6342/>
>
> > > 0.0.0.0:* LISTEN 31562/java
> > > >
> > > > [root@nifiHost3 conf]# netstat -tulpn
> > > > Active Internet connections (only servers)
> > > > Proto Recv-Q Send-Q Local Address Foreign Address
>
> > > State PID/Program name
> > > > tcp 0 0 192.168.1.12:6342 <http://192.168.1.12:6342/>
>
> > > 0.0.0.0:* LISTEN 31685/java
> > > >
> > > > And here is what our load balancing properties look like:
>
> > > >
> > > > # cluster load balancing properties #
> > > > nifi.cluster.load.balance.host=nifiHost1.contoso.com <
>
> > http://nifihost1.contoso.com/>
> > >
> > > > nifi.cluster.load.balance.address=0.0.0.0
> > > > nifi.cluster.load.balance.port=6342
> > > > nifi.cluster.load.balance.connections.per.node=4
> > > > nifi.cluster.load.balance.max.thread.count=8
> > > > nifi.cluster.load.balance.comms.timeout=30 sec
> > > >
> > > > When running Nifi in version 1.12.1 on the exact same setup
> in the
> > exact
> > > same environment, load balancing is working absolutely fine.
> > > > There was a time when load balancing even worked in version
> 1.13.2.
> > > But I'm not able to reproduce this and it just stopped
> > > > working one day after some restart, without changing any property
> or
> > > whatsoever.
> > > >
> > > > If any more information would be helpful please let me know
> and I'll
> > > try to provide it as fast as possible.
> > > >
> > > >
> > > >
> > > > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
>
> > > >
> > > > * Unbegrenzt Speicherplatz
> > > > * Eigenes Online-Büro
> > > > * 24h besten Mailempfang
> > > > * Spamschutz, Adressbuch
> > > >
> > > >
> > > >
> > > >
> > > > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
>
> > >
> > > >
> > > > * Unbegrenzt Speicherplatz
> > > > * Eigenes Online-Büro
> > > > * 24h besten Mailempfang
> > > > * Spamschutz, Adressbuch
> > > >
> > >
> > >
> >
> >
> >
>
Re: Re: No Load Balancing since 1.13.2
Posted by "Jens M. Kofoed" <jm...@gmail.com>.
hi
I can see that you have configured nifi.cluster.load.balance.address=0.0.0.0
Have your tried to set the correct ip adress?
node1: nifi.cluster.load.balance.address=192.168.1.10
node2: nifi.cluster.load.balance.address=192.168.1.11
node3: nifi.cluster.load.balance.address=192.168.1.12
regards
Jens M. Kofoed
Den ons. 28. jul. 2021 kl. 11.17 skrev Axel Schwarz <Ax...@emailn.de>:
> Just tried Java 11. But still does not work. Nothing changed. :(
>
> --- Ursprüngliche Nachricht ---
> Von: Jorge Machado <jo...@me.com>
> Datum: 27.07.2021 13:08:55
> An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
> Betreff: Re: No Load Balancing since 1.13.2
>
> > Did you tried java 11 ? I have a client running a similar setup to yours
> > but with a lower nigh version and it works fine. Maybe it is worth to try
> > it.
> >
> >
> > > On 27. Jul 2021, at 12:42, Axel Schwarz <Ax...@emailn.de>
> > wrote:
> > >
> > > I did indeed, but I updated from u161 to u291, as this was the newest
> > version at that time, because I thought it could help.
> > > So the issue started under u161. But I just saw that u301 is out. I
> > will try this as well.
> > > --- Ursprüngliche Nachricht ---
> > > Von: Pierre Villard <pi...@gmail.com>
> > > Datum: 27.07.2021 10:18:38
> > > An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
> >
> > > Betreff: Re: No Load Balancing since 1.13.2
> > >
> > > Hi,
> > >
> > > I believe the minor u291 is known to have issues (for some of its early
> > builds). Did you upgrade the Java version recently?
> > >
> > > Thanks,
> > > Pierre
> > >
> > > Le mar. 27 juil. 2021 à 08:07, Axel Schwarz <Axelkopter@emailn.de
> > <ma...@emailn.de>> a écrit :
> > > Dear Community,
> > >
> > > we're running a secured 3 node Nifi Cluster on Java 8_u291 and Debian
> > 7 and experiencing
> > > problems with load balancing since version 1.13.2.
> > >
> > > I'm fully aware of Issue Nifi-8643 and tested alot around this, but
> > gotta say, that this
> > > is not our problem. Mainly because the balance port never binds to
> localhost,
> > but also because I
> > > implemented all workarounds under version 1.13.2 and even tried version
> > 1.14.0 by now,
> > > but load blancing still does not work.
> > > What we experience is best described as "the primary node balances
> > with itself"...
> > >
> > > So what it does is, opening the balancing connections to its own IP
> > instead of the IPs
> > > of the other two nodes. And the other two nodes don't open balancing
> > connections at all.
> > >
> > > When executing "ss | grep 6342" on the primary node, this
> > is what it looks like:
> > >
> > > [root@nifiHost1 conf]# ss | grep 6342
> > > tcp ESTAB 0 0 192.168.1.10:51380 <
> http://192.168.1.10:51380/>
> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:51376 <
> http://192.168.1.10:51376/>
> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:51378 <
> http://192.168.1.10:51378/>
> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:51370 <
> http://192.168.1.10:51370/>
> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:51372 <
> http://192.168.1.10:51372/>
> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:6342 <
> http://192.168.1.10:6342/>
> > 192.168.1.10:51376 <http://192.168.1.10:51376/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:51374 <
> http://192.168.1.10:51374/>
> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:6342 <
> http://192.168.1.10:6342/>
> > 192.168.1.10:51374 <http://192.168.1.10:51374/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:51366 <
> http://192.168.1.10:51366/>
> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:6342 <
> http://192.168.1.10:6342/>
> > 192.168.1.10:51370 <http://192.168.1.10:51370/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:6342 <
> http://192.168.1.10:6342/>
> > 192.168.1.10:51366 <http://192.168.1.10:51366/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:51368 <
> http://192.168.1.10:51368/>
> > 192.168.1.10:6342 <http://192.168.1.10:6342/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:6342 <
> http://192.168.1.10:6342/>
> > 192.168.1.10:51372 <http://192.168.1.10:51372/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:6342 <
> http://192.168.1.10:6342/>
> > 192.168.1.10:51378 <http://192.168.1.10:51378/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:6342 <
> http://192.168.1.10:6342/>
> > 192.168.1.10:51368 <http://192.168.1.10:51368/>
> >
> > > tcp ESTAB 0 0 192.168.1.10:6342 <
> http://192.168.1.10:6342/>
> > 192.168.1.10:51380 <http://192.168.1.10:51380/>
> > >
> > > Executing it on the other non primary nodes, just returns absolutely
> > nothing.
> > >
> > > Netstat show the following on each server:
> > >
> > > [root@nifiHost1 conf]# netstat -tulpn
> > > Active Internet connections (only servers)
> > > Proto Recv-Q Send-Q Local Address Foreign Address
> > State PID/Program name
> > > tcp 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
> > 0.0.0.0:* LISTEN 10352/java
> > >
> > > [root@nifiHost2 conf]# netstat -tulpn
> > > Active Internet connections (only servers)
> > > Proto Recv-Q Send-Q Local Address Foreign Address
> > State PID/Program name
> > > tcp 0 0 192.168.1.11:6342 <http://192.168.1.11:6342/>
> > 0.0.0.0:* LISTEN 31562/java
> > >
> > > [root@nifiHost3 conf]# netstat -tulpn
> > > Active Internet connections (only servers)
> > > Proto Recv-Q Send-Q Local Address Foreign Address
> > State PID/Program name
> > > tcp 0 0 192.168.1.12:6342 <http://192.168.1.12:6342/>
> > 0.0.0.0:* LISTEN 31685/java
> > >
> > > And here is what our load balancing properties look like:
> > >
> > > # cluster load balancing properties #
> > > nifi.cluster.load.balance.host=nifiHost1.contoso.com <
> http://nifihost1.contoso.com/>
> >
> > > nifi.cluster.load.balance.address=0.0.0.0
> > > nifi.cluster.load.balance.port=6342
> > > nifi.cluster.load.balance.connections.per.node=4
> > > nifi.cluster.load.balance.max.thread.count=8
> > > nifi.cluster.load.balance.comms.timeout=30 sec
> > >
> > > When running Nifi in version 1.12.1 on the exact same setup in the
> exact
> > same environment, load balancing is working absolutely fine.
> > > There was a time when load balancing even worked in version 1.13.2.
> > But I'm not able to reproduce this and it just stopped
> > > working one day after some restart, without changing any property or
> > whatsoever.
> > >
> > > If any more information would be helpful please let me know and I'll
> > try to provide it as fast as possible.
> > >
> > >
> > >
> > > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
> > >
> > > * Unbegrenzt Speicherplatz
> > > * Eigenes Online-Büro
> > > * 24h besten Mailempfang
> > > * Spamschutz, Adressbuch
> > >
> > >
> > >
> > >
> > > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
> >
> > >
> > > * Unbegrenzt Speicherplatz
> > > * Eigenes Online-Büro
> > > * 24h besten Mailempfang
> > > * Spamschutz, Adressbuch
> > >
> >
> >
>
>
>
Re: Re: No Load Balancing since 1.13.2
Posted by Axel Schwarz <Ax...@emailn.de>.
Just tried Java 11. But still does not work. Nothing changed. :(
--- Ursprüngliche Nachricht ---
Von: Jorge Machado <jo...@me.com>
Datum: 27.07.2021 13:08:55
An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
Betreff: Re: No Load Balancing since 1.13.2
> Did you tried java 11 ? I have a client running a similar setup to yours
> but with a lower nigh version and it works fine. Maybe it is worth to try
> it.
>
>
> > On 27. Jul 2021, at 12:42, Axel Schwarz <Ax...@emailn.de>
> wrote:
> >
> > I did indeed, but I updated from u161 to u291, as this was the newest
> version at that time, because I thought it could help.
> > So the issue started under u161. But I just saw that u301 is out. I
> will try this as well.
> > --- Ursprüngliche Nachricht ---
> > Von: Pierre Villard <pi...@gmail.com>
> > Datum: 27.07.2021 10:18:38
> > An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>
> > Betreff: Re: No Load Balancing since 1.13.2
> >
> > Hi,
> >
> > I believe the minor u291 is known to have issues (for some of its early
> builds). Did you upgrade the Java version recently?
> >
> > Thanks,
> > Pierre
> >
> > Le mar. 27 juil. 2021 à 08:07, Axel Schwarz <Axelkopter@emailn.de
> <ma...@emailn.de>> a écrit :
> > Dear Community,
> >
> > we're running a secured 3 node Nifi Cluster on Java 8_u291 and Debian
> 7 and experiencing
> > problems with load balancing since version 1.13.2.
> >
> > I'm fully aware of Issue Nifi-8643 and tested alot around this, but
> gotta say, that this
> > is not our problem. Mainly because the balance port never binds to localhost,
> but also because I
> > implemented all workarounds under version 1.13.2 and even tried version
> 1.14.0 by now,
> > but load blancing still does not work.
> > What we experience is best described as "the primary node balances
> with itself"...
> >
> > So what it does is, opening the balancing connections to its own IP
> instead of the IPs
> > of the other two nodes. And the other two nodes don't open balancing
> connections at all.
> >
> > When executing "ss | grep 6342" on the primary node, this
> is what it looks like:
> >
> > [root@nifiHost1 conf]# ss | grep 6342
> > tcp ESTAB 0 0 192.168.1.10:51380 <http://192.168.1.10:51380/>
> 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > tcp ESTAB 0 0 192.168.1.10:51376 <http://192.168.1.10:51376/>
> 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > tcp ESTAB 0 0 192.168.1.10:51378 <http://192.168.1.10:51378/>
> 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > tcp ESTAB 0 0 192.168.1.10:51370 <http://192.168.1.10:51370/>
> 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > tcp ESTAB 0 0 192.168.1.10:51372 <http://192.168.1.10:51372/>
> 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
> 192.168.1.10:51376 <http://192.168.1.10:51376/>
>
> > tcp ESTAB 0 0 192.168.1.10:51374 <http://192.168.1.10:51374/>
> 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
> 192.168.1.10:51374 <http://192.168.1.10:51374/>
>
> > tcp ESTAB 0 0 192.168.1.10:51366 <http://192.168.1.10:51366/>
> 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
> 192.168.1.10:51370 <http://192.168.1.10:51370/>
>
> > tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
> 192.168.1.10:51366 <http://192.168.1.10:51366/>
>
> > tcp ESTAB 0 0 192.168.1.10:51368 <http://192.168.1.10:51368/>
> 192.168.1.10:6342 <http://192.168.1.10:6342/>
>
> > tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
> 192.168.1.10:51372 <http://192.168.1.10:51372/>
>
> > tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
> 192.168.1.10:51378 <http://192.168.1.10:51378/>
>
> > tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
> 192.168.1.10:51368 <http://192.168.1.10:51368/>
>
> > tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
> 192.168.1.10:51380 <http://192.168.1.10:51380/>
> >
> > Executing it on the other non primary nodes, just returns absolutely
> nothing.
> >
> > Netstat show the following on each server:
> >
> > [root@nifiHost1 conf]# netstat -tulpn
> > Active Internet connections (only servers)
> > Proto Recv-Q Send-Q Local Address Foreign Address
> State PID/Program name
> > tcp 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/>
> 0.0.0.0:* LISTEN 10352/java
> >
> > [root@nifiHost2 conf]# netstat -tulpn
> > Active Internet connections (only servers)
> > Proto Recv-Q Send-Q Local Address Foreign Address
> State PID/Program name
> > tcp 0 0 192.168.1.11:6342 <http://192.168.1.11:6342/>
> 0.0.0.0:* LISTEN 31562/java
> >
> > [root@nifiHost3 conf]# netstat -tulpn
> > Active Internet connections (only servers)
> > Proto Recv-Q Send-Q Local Address Foreign Address
> State PID/Program name
> > tcp 0 0 192.168.1.12:6342 <http://192.168.1.12:6342/>
> 0.0.0.0:* LISTEN 31685/java
> >
> > And here is what our load balancing properties look like:
> >
> > # cluster load balancing properties #
> > nifi.cluster.load.balance.host=nifiHost1.contoso.com <http://nifihost1.contoso.com/>
>
> > nifi.cluster.load.balance.address=0.0.0.0
> > nifi.cluster.load.balance.port=6342
> > nifi.cluster.load.balance.connections.per.node=4
> > nifi.cluster.load.balance.max.thread.count=8
> > nifi.cluster.load.balance.comms.timeout=30 sec
> >
> > When running Nifi in version 1.12.1 on the exact same setup in the exact
> same environment, load balancing is working absolutely fine.
> > There was a time when load balancing even worked in version 1.13.2.
> But I'm not able to reproduce this and it just stopped
> > working one day after some restart, without changing any property or
> whatsoever.
> >
> > If any more information would be helpful please let me know and I'll
> try to provide it as fast as possible.
> >
> >
> >
> > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
> >
> > * Unbegrenzt Speicherplatz
> > * Eigenes Online-Büro
> > * 24h besten Mailempfang
> > * Spamschutz, Adressbuch
> >
> >
> >
> >
> > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
>
> >
> > * Unbegrenzt Speicherplatz
> > * Eigenes Online-Büro
> > * 24h besten Mailempfang
> > * Spamschutz, Adressbuch
> >
>
>
Re: No Load Balancing since 1.13.2
Posted by Jorge Machado <jo...@me.com>.
Did you tried java 11 ? I have a client running a similar setup to yours but with a lower nigh version and it works fine. Maybe it is worth to try it.
> On 27. Jul 2021, at 12:42, Axel Schwarz <Ax...@emailn.de> wrote:
>
> I did indeed, but I updated from u161 to u291, as this was the newest version at that time, because I thought it could help.
> So the issue started under u161. But I just saw that u301 is out. I will try this as well.
> --- Ursprüngliche Nachricht ---
> Von: Pierre Villard <pi...@gmail.com>
> Datum: 27.07.2021 10:18:38
> An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
> Betreff: Re: No Load Balancing since 1.13.2
>
> Hi,
>
> I believe the minor u291 is known to have issues (for some of its early builds). Did you upgrade the Java version recently?
>
> Thanks,
> Pierre
>
> Le mar. 27 juil. 2021 à 08:07, Axel Schwarz <Axelkopter@emailn.de <ma...@emailn.de>> a écrit :
> Dear Community,
>
> we're running a secured 3 node Nifi Cluster on Java 8_u291 and Debian 7 and experiencing
> problems with load balancing since version 1.13.2.
>
> I'm fully aware of Issue Nifi-8643 and tested alot around this, but gotta say, that this
> is not our problem. Mainly because the balance port never binds to localhost, but also because I
> implemented all workarounds under version 1.13.2 and even tried version 1.14.0 by now,
> but load blancing still does not work.
> What we experience is best described as "the primary node balances with itself"...
>
> So what it does is, opening the balancing connections to its own IP instead of the IPs
> of the other two nodes. And the other two nodes don't open balancing connections at all.
>
> When executing "ss | grep 6342" on the primary node, this is what it looks like:
>
> [root@nifiHost1 conf]# ss | grep 6342
> tcp ESTAB 0 0 192.168.1.10:51380 <http://192.168.1.10:51380/> 192.168.1.10:6342 <http://192.168.1.10:6342/>
> tcp ESTAB 0 0 192.168.1.10:51376 <http://192.168.1.10:51376/> 192.168.1.10:6342 <http://192.168.1.10:6342/>
> tcp ESTAB 0 0 192.168.1.10:51378 <http://192.168.1.10:51378/> 192.168.1.10:6342 <http://192.168.1.10:6342/>
> tcp ESTAB 0 0 192.168.1.10:51370 <http://192.168.1.10:51370/> 192.168.1.10:6342 <http://192.168.1.10:6342/>
> tcp ESTAB 0 0 192.168.1.10:51372 <http://192.168.1.10:51372/> 192.168.1.10:6342 <http://192.168.1.10:6342/>
> tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/> 192.168.1.10:51376 <http://192.168.1.10:51376/>
> tcp ESTAB 0 0 192.168.1.10:51374 <http://192.168.1.10:51374/> 192.168.1.10:6342 <http://192.168.1.10:6342/>
> tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/> 192.168.1.10:51374 <http://192.168.1.10:51374/>
> tcp ESTAB 0 0 192.168.1.10:51366 <http://192.168.1.10:51366/> 192.168.1.10:6342 <http://192.168.1.10:6342/>
> tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/> 192.168.1.10:51370 <http://192.168.1.10:51370/>
> tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/> 192.168.1.10:51366 <http://192.168.1.10:51366/>
> tcp ESTAB 0 0 192.168.1.10:51368 <http://192.168.1.10:51368/> 192.168.1.10:6342 <http://192.168.1.10:6342/>
> tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/> 192.168.1.10:51372 <http://192.168.1.10:51372/>
> tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/> 192.168.1.10:51378 <http://192.168.1.10:51378/>
> tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/> 192.168.1.10:51368 <http://192.168.1.10:51368/>
> tcp ESTAB 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/> 192.168.1.10:51380 <http://192.168.1.10:51380/>
>
> Executing it on the other non primary nodes, just returns absolutely nothing.
>
> Netstat show the following on each server:
>
> [root@nifiHost1 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/> 0.0.0.0:* LISTEN 10352/java
>
> [root@nifiHost2 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp 0 0 192.168.1.11:6342 <http://192.168.1.11:6342/> 0.0.0.0:* LISTEN 31562/java
>
> [root@nifiHost3 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp 0 0 192.168.1.12:6342 <http://192.168.1.12:6342/> 0.0.0.0:* LISTEN 31685/java
>
> And here is what our load balancing properties look like:
>
> # cluster load balancing properties #
> nifi.cluster.load.balance.host=nifiHost1.contoso.com <http://nifihost1.contoso.com/>
> nifi.cluster.load.balance.address=0.0.0.0
> nifi.cluster.load.balance.port=6342
> nifi.cluster.load.balance.connections.per.node=4
> nifi.cluster.load.balance.max.thread.count=8
> nifi.cluster.load.balance.comms.timeout=30 sec
>
> When running Nifi in version 1.12.1 on the exact same setup in the exact same environment, load balancing is working absolutely fine.
> There was a time when load balancing even worked in version 1.13.2. But I'm not able to reproduce this and it just stopped
> working one day after some restart, without changing any property or whatsoever.
>
> If any more information would be helpful please let me know and I'll try to provide it as fast as possible.
>
>
>
> Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
>
> * Unbegrenzt Speicherplatz
> * Eigenes Online-Büro
> * 24h besten Mailempfang
> * Spamschutz, Adressbuch
>
>
>
>
> Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
>
> * Unbegrenzt Speicherplatz
> * Eigenes Online-Büro
> * 24h besten Mailempfang
> * Spamschutz, Adressbuch
>
Re: Re: No Load Balancing since 1.13.2
Posted by Axel Schwarz <Ax...@emailn.de>.
We're using Round Robin Strategy.
The exact scenario is a SFTPList-Processor, which is running "Primary Node only", and the following connection should load balance using Round Robin. But right at the next Processor (UpdateAttribute) it's clearly to see, via Provenance or Status History, that there is no load balancing at all. All FlowFiles are still processed by the primary node at that point.
--- Ursprüngliche Nachricht ---
Von: Mark Payne <ma...@hotmail.com>
Datum: 27.07.2021 17:08:06
An: "users@nifi.apache.org" <us...@nifi.apache.org>, Axel Schwarz <Ax...@emailn.de>
Betreff: Re: No Load Balancing since 1.13.2
> Axel,
>
> Which Load Balancing Strategy are you using?
>
> I.e., if you right-click on a Connection and configure, what’s configured
> in the Settings tab?
>
> Thanks
> -Mark
>
>
> > On Jul 27, 2021, at 7:05 AM, Axel Schwarz <Ax...@emailn.de>
> wrote:
> >
> > Just tried Java 8_u301, but it didn't change anything. Still the same
> behaviour.
> >
> > --- Ursprüngliche Nachricht ---
> > Von: "Axel Schwarz" <Ax...@emailn.de>
> > Datum: 27.07.2021 12:42:42
> > An: users@nifi.apache.org
> > Betreff: Re: Re: No Load Balancing since 1.13.2
> >
> >> I did indeed, but I updated from u161 to u291, as this was the
> newest version
> >> at that time, because I thought it could help. So the issue started
> under
> >> u161. But I just saw that u301 is out. I will try this as well.
>
> >> --- Ursprüngliche Nachricht ---
> >> Von: Pierre Villard <pi...@gmail.com>
> >> Datum: 27.07.2021 10:18:38
> >> An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>
> >> Betreff: Re: No Load Balancing since 1.13.2
> >>
> >> Hi,
> >> I believe the minor u291 is known to have issues (for some of its
> early
> >> builds). Did you upgrade the Java versionrecently?
> >> Thanks, Pierre
> >> Le mar. 27 juil. 2021 à 08:07, Axel Schwarz < Axelkopter@emailn.de
> > a
> >> écrit :
> >> Dear Community,
> >>
> >> we're running a secured 3 node Nifi Cluster on Java 8_u291 and Debian
> 7 and
> >> experiencing
> >> problems with load balancing since version 1.13.2.
> >>
> >> I'm fully aware of Issue Nifi-8643 and tested alot around this,
> but gotta
> >> say, that this
> >> is not our problem. Mainly because the balance port never binds
> to localhost,
> >> but also because I
> >> implemented all workarounds under version 1.13.2 and even tried
> version 1.14.0
> >> by now,
> >> but load blancingstill does not work.
> >> What we experience is best described as "the primary node balances
> with itself"...
> >>
> >>
> >> So what it does is, opening the balancing connections to its own
> IP instead
> >> of the IPs
> >> of the other two nodes. And the other two nodes don't open balancing
> connections
> >> at all.
> >>
> >> When executing "ss | grep 6342" on the primary node, this
> is what it looks
> >> like:
> >>
> >> [root@nifiHost1conf]# ss | grep 6342
> >> tcp ESTAB 0 0 192.168.1.10:51380
> 192.168.1.10:6342
> >>
> >> tcp ESTAB 0 0 192.168.1.10:51376
> 192.168.1.10:6342
> >>
> >> tcp ESTAB 0 0 192.168.1.10:51378
> 192.168.1.10:6342
> >>
> >> tcp ESTAB 0 0 192.168.1.10:51370
> 192.168.1.10:6342
> >>
> >> tcp ESTAB 0 0 192.168.1.10:51372
> 192.168.1.10:6342
> >>
> >> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51376
> >>
> >> tcp ESTAB 0 0 192.168.1.10:51374
> 192.168.1.10:6342
> >>
> >> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51374
> >>
> >> tcp ESTAB 0 0 192.168.1.10:51366
> 192.168.1.10:6342
> >>
> >> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51370
> >>
> >> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51366
> >>
> >> tcp ESTAB 0 0 192.168.1.10:51368
> 192.168.1.10:6342
> >>
> >> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51372
> >>
> >> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51378
> >>
> >> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51368
> >>
> >> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51380
> >>
> >>
> >> Executing it on the other non primarynodes, just returns absolutely
> nothing.
> >>
> >>
> >> Netstat show the following on each server:
> >>
> >> [root@nifiHost1 conf]# netstat -tulpn
> >> Active Internet connections (only servers)
> >> Proto Recv-Q Send-Q Local Address Foreign Address
> State
> >> PID/Program name
> >> tcp 0 0 192.168.1.10:6342 0.0.0.0:*
>
> >> LISTEN 10352/java
> >>
> >> [root@nifiHost2 conf]# netstat -tulpn
> >> Active Internet connections (only servers)
> >> Proto Recv-Q Send-Q Local Address Foreign Address
> State
> >> PID/Program name
> >> tcp 0 0 192.168.1.11:6342 0.0.0.0:*
>
> >> LISTEN 31562/java
> >>
> >> [root@nifiHost3 conf]# netstat -tulpn
> >> Active Internet connections (only servers)
> >> Proto Recv-Q Send-Q Local Address Foreign Address
> State
> >> PID/Program name
> >> tcp 0 0 192.168.1.12:6342 0.0.0.0:*
> LISTEN
> >> 31685/java
> >>
> >> And here is what our load balancing properties look like:
> >>
> >> # cluster load balancing properties #
> >> nifi.cluster.load.balance.host= nifiHost1.contoso.com
> >> nifi.cluster.load.balance.address=0.0.0.0
> >> nifi.cluster.load.balance.port=6342
> >> nifi.cluster.load.balance.connections.per.node=4
> >> nifi.cluster.load.balance.max.thread.count=8
> >> nifi.cluster.load.balance.comms.timeout=30 sec
> >>
> >> When running Nifi in version 1.12.1 on the exact same setup in the
> exact
> >> same environment, load balancing is working absolutely fine.
> >> There was a time when load balancing even worked in version 1.13.2.
> But I'm
> >> not able to reproduce this and it just stopped
> >> working one day after some restart, without changing any property
> or whatsoever.
> >>
> >>
> >> If any more information would be helpfulplease let me know and I'll
> try to
> >> provide it as fastas possible.
> >>
> >>
> >>
> >> Versendet mit Emailn.de - Freemail
> >>
> >> * Unbegrenzt Speicherplatz
> >> * Eigenes Online-Büro
> >> * 24h besten Mailempfang
> >> * Spamschutz, Adressbuch
> >>
> >>
> >>
> >>
> >> Versendet mit Emailn.de - Freemail
> >>
> >> * Unbegrenzt Speicherplatz
> >> * Eigenes Online-Büro
> >> * 24h besten Mailempfang
> >> * Spamschutz, Adressbuch
> >>
> >>
> >
> >
>
Re: No Load Balancing since 1.13.2
Posted by Mark Payne <ma...@hotmail.com>.
Axel,
Which Load Balancing Strategy are you using?
I.e., if you right-click on a Connection and configure, what’s configured in the Settings tab?
Thanks
-Mark
> On Jul 27, 2021, at 7:05 AM, Axel Schwarz <Ax...@emailn.de> wrote:
>
> Just tried Java 8_u301, but it didn't change anything. Still the same behaviour.
>
> --- Ursprüngliche Nachricht ---
> Von: "Axel Schwarz" <Ax...@emailn.de>
> Datum: 27.07.2021 12:42:42
> An: users@nifi.apache.org
> Betreff: Re: Re: No Load Balancing since 1.13.2
>
>> I did indeed, but I updated from u161 to u291, as this was the newest version
>> at that time, because I thought it could help. So the issue started under
>> u161. But I just saw that u301 is out. I will try this as well.
>> --- Ursprüngliche Nachricht ---
>> Von: Pierre Villard <pi...@gmail.com>
>> Datum: 27.07.2021 10:18:38
>> An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
>> Betreff: Re: No Load Balancing since 1.13.2
>>
>> Hi,
>> I believe the minor u291 is known to have issues (for some of its early
>> builds). Did you upgrade the Java versionrecently?
>> Thanks, Pierre
>> Le mar. 27 juil. 2021 à 08:07, Axel Schwarz < Axelkopter@emailn.de > a
>> écrit :
>> Dear Community,
>>
>> we're running a secured 3 node Nifi Cluster on Java 8_u291 and Debian 7 and
>> experiencing
>> problems with load balancing since version 1.13.2.
>>
>> I'm fully aware of Issue Nifi-8643 and tested alot around this, but gotta
>> say, that this
>> is not our problem. Mainly because the balance port never binds to localhost,
>> but also because I
>> implemented all workarounds under version 1.13.2 and even tried version 1.14.0
>> by now,
>> but load blancingstill does not work.
>> What we experience is best described as "the primary node balances with itself"...
>>
>>
>> So what it does is, opening the balancing connections to its own IP instead
>> of the IPs
>> of the other two nodes. And the other two nodes don't open balancing connections
>> at all.
>>
>> When executing "ss | grep 6342" on the primary node, this is what it looks
>> like:
>>
>> [root@nifiHost1conf]# ss | grep 6342
>> tcp ESTAB 0 0 192.168.1.10:51380 192.168.1.10:6342
>>
>> tcp ESTAB 0 0 192.168.1.10:51376 192.168.1.10:6342
>>
>> tcp ESTAB 0 0 192.168.1.10:51378 192.168.1.10:6342
>>
>> tcp ESTAB 0 0 192.168.1.10:51370 192.168.1.10:6342
>>
>> tcp ESTAB 0 0 192.168.1.10:51372 192.168.1.10:6342
>>
>> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51376
>>
>> tcp ESTAB 0 0 192.168.1.10:51374 192.168.1.10:6342
>>
>> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51374
>>
>> tcp ESTAB 0 0 192.168.1.10:51366 192.168.1.10:6342
>>
>> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51370
>>
>> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51366
>>
>> tcp ESTAB 0 0 192.168.1.10:51368 192.168.1.10:6342
>>
>> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51372
>>
>> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51378
>>
>> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51368
>>
>> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51380
>>
>>
>> Executing it on the other non primarynodes, just returns absolutely nothing.
>>
>>
>> Netstat show the following on each server:
>>
>> [root@nifiHost1 conf]# netstat -tulpn
>> Active Internet connections (only servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State
>> PID/Program name
>> tcp 0 0 192.168.1.10:6342 0.0.0.0:*
>> LISTEN 10352/java
>>
>> [root@nifiHost2 conf]# netstat -tulpn
>> Active Internet connections (only servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State
>> PID/Program name
>> tcp 0 0 192.168.1.11:6342 0.0.0.0:*
>> LISTEN 31562/java
>>
>> [root@nifiHost3 conf]# netstat -tulpn
>> Active Internet connections (only servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State
>> PID/Program name
>> tcp 0 0 192.168.1.12:6342 0.0.0.0:* LISTEN
>> 31685/java
>>
>> And here is what our load balancing properties look like:
>>
>> # cluster load balancing properties #
>> nifi.cluster.load.balance.host= nifiHost1.contoso.com
>> nifi.cluster.load.balance.address=0.0.0.0
>> nifi.cluster.load.balance.port=6342
>> nifi.cluster.load.balance.connections.per.node=4
>> nifi.cluster.load.balance.max.thread.count=8
>> nifi.cluster.load.balance.comms.timeout=30 sec
>>
>> When running Nifi in version 1.12.1 on the exact same setup in the exact
>> same environment, load balancing is working absolutely fine.
>> There was a time when load balancing even worked in version 1.13.2. But I'm
>> not able to reproduce this and it just stopped
>> working one day after some restart, without changing any property or whatsoever.
>>
>>
>> If any more information would be helpfulplease let me know and I'll try to
>> provide it as fastas possible.
>>
>>
>>
>> Versendet mit Emailn.de - Freemail
>>
>> * Unbegrenzt Speicherplatz
>> * Eigenes Online-Büro
>> * 24h besten Mailempfang
>> * Spamschutz, Adressbuch
>>
>>
>>
>>
>> Versendet mit Emailn.de - Freemail
>>
>> * Unbegrenzt Speicherplatz
>> * Eigenes Online-Büro
>> * 24h besten Mailempfang
>> * Spamschutz, Adressbuch
>>
>>
>
>
Re: Re: Re: No Load Balancing since 1.13.2
Posted by Axel Schwarz <Ax...@emailn.de>.
Just tried Java 8_u301, but it didn't change anything. Still the same behaviour.
--- Ursprüngliche Nachricht ---
Von: "Axel Schwarz" <Ax...@emailn.de>
Datum: 27.07.2021 12:42:42
An: users@nifi.apache.org
Betreff: Re: Re: No Load Balancing since 1.13.2
> I did indeed, but I updated from u161 to u291, as this was the newest version
> at that time, because I thought it could help. So the issue started under
> u161. But I just saw that u301 is out. I will try this as well.
>--- Ursprüngliche Nachricht ---
> Von: Pierre Villard <pi...@gmail.com>
> Datum: 27.07.2021 10:18:38
> An: users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
> Betreff: Re: No Load Balancing since 1.13.2
>
> Hi,
> I believe the minor u291 is known to have issues (for some of its early
> builds). Did you upgrade the Java versionrecently?
> Thanks, Pierre
> Le mar. 27 juil. 2021 à 08:07, Axel Schwarz < Axelkopter@emailn.de > a
> écrit :
> Dear Community,
>
> we're running a secured 3 node Nifi Cluster on Java 8_u291 and Debian 7 and
> experiencing
> problems with load balancing since version 1.13.2.
>
> I'm fully aware of Issue Nifi-8643 and tested alot around this, but gotta
> say, that this
> is not our problem. Mainly because the balance port never binds to localhost,
> but also because I
> implemented all workarounds under version 1.13.2 and even tried version 1.14.0
> by now,
> but load blancingstill does not work.
> What we experience is best described as "the primary node balances with itself"...
>
>
> So what it does is, opening the balancing connections to its own IP instead
> of the IPs
> of the other two nodes. And the other two nodes don't open balancing connections
> at all.
>
> When executing "ss | grep 6342" on the primary node, this is what it looks
> like:
>
> [root@nifiHost1conf]# ss | grep 6342
> tcp ESTAB 0 0 192.168.1.10:51380 192.168.1.10:6342
>
> tcp ESTAB 0 0 192.168.1.10:51376 192.168.1.10:6342
>
> tcp ESTAB 0 0 192.168.1.10:51378 192.168.1.10:6342
>
> tcp ESTAB 0 0 192.168.1.10:51370 192.168.1.10:6342
>
> tcp ESTAB 0 0 192.168.1.10:51372 192.168.1.10:6342
>
> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51376
>
> tcp ESTAB 0 0 192.168.1.10:51374 192.168.1.10:6342
>
> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51374
>
> tcp ESTAB 0 0 192.168.1.10:51366 192.168.1.10:6342
>
> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51370
>
> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51366
>
> tcp ESTAB 0 0 192.168.1.10:51368 192.168.1.10:6342
>
> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51372
>
> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51378
>
> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51368
>
> tcp ESTAB 0 0 192.168.1.10:6342 192.168.1.10:51380
>
>
> Executing it on the other non primarynodes, just returns absolutely nothing.
>
>
> Netstat show the following on each server:
>
> [root@nifiHost1 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> PID/Program name
> tcp 0 0 192.168.1.10:6342 0.0.0.0:*
> LISTEN 10352/java
>
> [root@nifiHost2 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> PID/Program name
> tcp 0 0 192.168.1.11:6342 0.0.0.0:*
> LISTEN 31562/java
>
> [root@nifiHost3 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> PID/Program name
> tcp 0 0 192.168.1.12:6342 0.0.0.0:* LISTEN
> 31685/java
>
> And here is what our load balancing properties look like:
>
> # cluster load balancing properties #
> nifi.cluster.load.balance.host= nifiHost1.contoso.com
> nifi.cluster.load.balance.address=0.0.0.0
> nifi.cluster.load.balance.port=6342
> nifi.cluster.load.balance.connections.per.node=4
> nifi.cluster.load.balance.max.thread.count=8
> nifi.cluster.load.balance.comms.timeout=30 sec
>
> When running Nifi in version 1.12.1 on the exact same setup in the exact
> same environment, load balancing is working absolutely fine.
> There was a time when load balancing even worked in version 1.13.2. But I'm
> not able to reproduce this and it just stopped
> working one day after some restart, without changing any property or whatsoever.
>
>
> If any more information would be helpfulplease let me know and I'll try to
> provide it as fastas possible.
>
>
>
> Versendet mit Emailn.de - Freemail
>
> * Unbegrenzt Speicherplatz
> * Eigenes Online-Büro
> * 24h besten Mailempfang
> * Spamschutz, Adressbuch
>
>
>
>
> Versendet mit Emailn.de - Freemail
>
> * Unbegrenzt Speicherplatz
> * Eigenes Online-Büro
> * 24h besten Mailempfang
> * Spamschutz, Adressbuch
>
>
Re: Re: No Load Balancing since 1.13.2
Posted by Axel Schwarz <Ax...@emailn.de>.
I did indeed, but I updated from u161 to u291, as this was the newest version
at that time, because I thought it could help.
So the issue started under u161. But I just saw that u301 is out. I will try
this as well.
> \--- Ursprüngliche Nachricht ---
> **Von:** Pierre Villard <pi...@gmail.com>
> **Datum:** 27.07.2021 10:18:38
> **An:** users@nifi.apache.org, Axel Schwarz <Ax...@emailn.de>
> **Betreff:** Re: No Load Balancing since 1.13.2
>
>
>
> Hi,
>
>
>
>
> I believe the minor u291 is known to have issues (for some of its early
builds). Did you upgrade the Java version recently?
>
>
>
>
> Thanks,
>
> Pierre
>
>
>
>
> Le mar. 27 juil. 2021 à 08:07, Axel Schwarz
<[Axelkopter@emailn.de](mailto:Axelkopter@emailn.de)> a écrit :
>
>
>> Dear Community,
>
> we're running a secured 3 node Nifi Cluster on Java 8_u291 and Debian 7 and
experiencing
> problems with load balancing since version 1.13.2.
>
> I'm fully aware of Issue Nifi-8643 and tested alot around this, but gotta
say, that this
> is not our problem. Mainly because the balance port never binds to
localhost, but also because I
> implemented all workarounds under version 1.13.2 and even tried version
1.14.0 by now,
> but load blancing still does not work.
> What we experience is best described as "the primary node balances with
itself"...
>
> So what it does is, opening the balancing connections to its own IP instead
of the IPs
> of the other two nodes. And the other two nodes don't open balancing
connections at all.
>
> When executing "ss | grep 6342" on the primary node, this is what it looks
like:
>
> [root@nifiHost1 conf]# ss | grep 6342
> tcp ESTAB 0 0
[192.168.1.10:51380](http://192.168.1.10:51380)
[192.168.1.10:6342](http://192.168.1.10:6342)
> tcp ESTAB 0 0
[192.168.1.10:51376](http://192.168.1.10:51376)
[192.168.1.10:6342](http://192.168.1.10:6342)
> tcp ESTAB 0 0
[192.168.1.10:51378](http://192.168.1.10:51378)
[192.168.1.10:6342](http://192.168.1.10:6342)
> tcp ESTAB 0 0
[192.168.1.10:51370](http://192.168.1.10:51370)
[192.168.1.10:6342](http://192.168.1.10:6342)
> tcp ESTAB 0 0
[192.168.1.10:51372](http://192.168.1.10:51372)
[192.168.1.10:6342](http://192.168.1.10:6342)
> tcp ESTAB 0 0
[192.168.1.10:6342](http://192.168.1.10:6342)
[192.168.1.10:51376](http://192.168.1.10:51376)
> tcp ESTAB 0 0
[192.168.1.10:51374](http://192.168.1.10:51374)
[192.168.1.10:6342](http://192.168.1.10:6342)
> tcp ESTAB 0 0
[192.168.1.10:6342](http://192.168.1.10:6342)
[192.168.1.10:51374](http://192.168.1.10:51374)
> tcp ESTAB 0 0
[192.168.1.10:51366](http://192.168.1.10:51366)
[192.168.1.10:6342](http://192.168.1.10:6342)
> tcp ESTAB 0 0
[192.168.1.10:6342](http://192.168.1.10:6342)
[192.168.1.10:51370](http://192.168.1.10:51370)
> tcp ESTAB 0 0
[192.168.1.10:6342](http://192.168.1.10:6342)
[192.168.1.10:51366](http://192.168.1.10:51366)
> tcp ESTAB 0 0
[192.168.1.10:51368](http://192.168.1.10:51368)
[192.168.1.10:6342](http://192.168.1.10:6342)
> tcp ESTAB 0 0
[192.168.1.10:6342](http://192.168.1.10:6342)
[192.168.1.10:51372](http://192.168.1.10:51372)
> tcp ESTAB 0 0
[192.168.1.10:6342](http://192.168.1.10:6342)
[192.168.1.10:51378](http://192.168.1.10:51378)
> tcp ESTAB 0 0
[192.168.1.10:6342](http://192.168.1.10:6342)
[192.168.1.10:51368](http://192.168.1.10:51368)
> tcp ESTAB 0 0
[192.168.1.10:6342](http://192.168.1.10:6342)
[192.168.1.10:51380](http://192.168.1.10:51380)
>
> Executing it on the other non primary nodes, just returns absolutely
nothing.
>
> Netstat show the following on each server:
>
> [root@nifiHost1 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
> tcp 0 0 [192.168.1.10:6342](http://192.168.1.10:6342)
0.0.0.0:* LISTEN 10352/java
>
> [root@nifiHost2 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
> tcp 0 0 [192.168.1.11:6342](http://192.168.1.11:6342)
0.0.0.0:* LISTEN 31562/java
>
> [root@nifiHost3 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
> tcp 0 0 [192.168.1.12:6342](http://192.168.1.12:6342)
0.0.0.0:* LISTEN 31685/java
>
> And here is what our load balancing properties look like:
>
> # cluster load balancing properties #
>
nifi.cluster.load.balance.host=[nifiHost1.contoso.com](http://nifiHost1.contoso.com)
> nifi.cluster.load.balance.address=0.0.0.0
> nifi.cluster.load.balance.port=6342
> nifi.cluster.load.balance.connections.per.node=4
> nifi.cluster.load.balance.max.thread.count=8
> nifi.cluster.load.balance.comms.timeout=30 sec
>
> When running Nifi in version 1.12.1 on the exact same setup in the exact
same environment, load balancing is working absolutely fine.
> There was a time when load balancing even worked in version 1.13.2. But I'm
not able to reproduce this and it just stopped
> working one day after some restart, without changing any property or
whatsoever.
>
> If any more information would be helpful please let me know and I'll try to
provide it as fast as possible.
>
>
>
>>
>> * * *
>>
>>
> Versendet mit [Emailn.de](https://www.emailn.de/) \- Freemail
>
> * Unbegrenzt Speicherplatz
> * Eigenes Online-Büro
> * 24h besten Mailempfang
> * Spamschutz, Adressbuch
>
>
* * *
Versendet mit [Emailn.de](https://www.emailn.de/) \- Freemail
* Unbegrenzt Speicherplatz
* Eigenes Online-Büro
* 24h besten Mailempfang
* Spamschutz, Adressbuch
Re: No Load Balancing since 1.13.2
Posted by Pierre Villard <pi...@gmail.com>.
Hi,
I believe the minor u291 is known to have issues (for some of its early
builds). Did you upgrade the Java version recently?
Thanks,
Pierre
Le mar. 27 juil. 2021 à 08:07, Axel Schwarz <Ax...@emailn.de> a écrit :
> Dear Community,
>
> we're running a secured 3 node Nifi Cluster on Java 8_u291 and Debian 7
> and experiencing
> problems with load balancing since version 1.13.2.
>
> I'm fully aware of Issue Nifi-8643 and tested alot around this, but gotta
> say, that this
> is not our problem. Mainly because the balance port never binds to
> localhost, but also because I
> implemented all workarounds under version 1.13.2 and even tried version
> 1.14.0 by now,
> but load blancing still does not work.
> What we experience is best described as "the primary node balances with
> itself"...
>
> So what it does is, opening the balancing connections to its own IP
> instead of the IPs
> of the other two nodes. And the other two nodes don't open balancing
> connections at all.
>
> When executing "ss | grep 6342" on the primary node, this is what it looks
> like:
>
> [root@nifiHost1 conf]# ss | grep 6342
> tcp ESTAB 0 0 192.168.1.10:51380
> 192.168.1.10:6342
> tcp ESTAB 0 0 192.168.1.10:51376
> 192.168.1.10:6342
> tcp ESTAB 0 0 192.168.1.10:51378
> 192.168.1.10:6342
> tcp ESTAB 0 0 192.168.1.10:51370
> 192.168.1.10:6342
> tcp ESTAB 0 0 192.168.1.10:51372
> 192.168.1.10:6342
> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51376
> tcp ESTAB 0 0 192.168.1.10:51374
> 192.168.1.10:6342
> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51374
> tcp ESTAB 0 0 192.168.1.10:51366
> 192.168.1.10:6342
> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51370
> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51366
> tcp ESTAB 0 0 192.168.1.10:51368
> 192.168.1.10:6342
> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51372
> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51378
> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51368
> tcp ESTAB 0 0 192.168.1.10:6342
> 192.168.1.10:51380
>
> Executing it on the other non primary nodes, just returns absolutely
> nothing.
>
> Netstat show the following on each server:
>
> [root@nifiHost1 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address
> State PID/Program name
> tcp 0 0 192.168.1.10:6342 0.0.0.0:*
> LISTEN 10352/java
>
> [root@nifiHost2 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address
> State PID/Program name
> tcp 0 0 192.168.1.11:6342 0.0.0.0:*
> LISTEN 31562/java
>
> [root@nifiHost3 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address
> State PID/Program name
> tcp 0 0 192.168.1.12:6342 0.0.0.0:*
> LISTEN 31685/java
>
> And here is what our load balancing properties look like:
>
> # cluster load balancing properties #
> nifi.cluster.load.balance.host=nifiHost1.contoso.com
> nifi.cluster.load.balance.address=0.0.0.0
> nifi.cluster.load.balance.port=6342
> nifi.cluster.load.balance.connections.per.node=4
> nifi.cluster.load.balance.max.thread.count=8
> nifi.cluster.load.balance.comms.timeout=30 sec
>
> When running Nifi in version 1.12.1 on the exact same setup in the exact
> same environment, load balancing is working absolutely fine.
> There was a time when load balancing even worked in version 1.13.2. But
> I'm not able to reproduce this and it just stopped
> working one day after some restart, without changing any property or
> whatsoever.
>
> If any more information would be helpful please let me know and I'll try
> to provide it as fast as possible.
>
>
> ------------------------------
>
> Versendet mit Emailn.de <https://www.emailn.de/> - Freemail
>
> * Unbegrenzt Speicherplatz
> * Eigenes Online-Büro
> * 24h besten Mailempfang
> * Spamschutz, Adressbuch
>
>