You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Pat White <pa...@verizonmedia.com> on 2021/02/08 21:46:53 UTC
After upgrade to 1.11.4, flowController fails to start due to invalid
clusterCoordinator port 0
Hi Folks,
Appreciate any debugging help on a very odd error, after upgrading a Nifi
cluster from 1.6.0 to 1.11.4, flowController fails to start due to:
Caused by: org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'flowService': FactoryBean threw exception on
object creation; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'flowController' defined in class path resource
[nifi-context.xml]: Cannot resolve reference to bean 'clusterCoordinator'
while setting bean property 'clusterCoordinator'; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'clusterCoordinator': FactoryBean threw exception on object
creation; nested exception is java.lang.IllegalArgumentException: Port must
be inclusively in the range [1, 65535]. Port given: 0
The error trace is very similar to the example Andy described in NIFI-6336,
the issue there i believe is not specifying
'nifi.cluster.node.protocol.port' in 'nifi.properties', however my conf has
that set, 'nifi.cluster.node.protocol.port=50233' and should be using
50233 instead of '0'.
Cluster had been running fine previously and as far as i can tell, Nifi and
ZK confs and settings are all ok. Also compared to another cluster that had
been upgraded with no issues, and is running 1.11.4 just fine.
Increased debug logging but without success so far. Am looking at the right
property association?
patw
Re: [E] After upgrade to 1.11.4, flowController fails to start due to
invalid clusterCoordinator port 0
Posted by Pierre Villard <pi...@gmail.com>.
Thanks for following up and providing the explanation!
Le sam. 13 févr. 2021 à 18:46, Pat White <pa...@verizonmedia.com> a
écrit :
> Hi Folks, found the reason for my issue, following up just FYI.
>
> My local state snapshot, on the updated 1.11.4 nodes, had bad values for
> load balancer host and port, so the snapshot had entries like:
>
> "loadBalanceAddress":null,"loadBalancePort":0,
>
> This caused a deadlock where the flow and cluster coordinators had to rely
> on existing state values, and therefore couldn't instantiate to apply
> correct values, even with these settings explicitly set in nifi.properties.
> This condition was actually discussed in NIFI-6336, regarding the port
> exclusion checks added in 1.11.x, that was very helpful in knowing where to
> look, to track this down.
>
> Solution was simple enough, delete the state snapshot and allow it to be
> recreated, this allowed my 1.11.4 nodes to start successfully. After they
> came up and entered cluster, verified all nodes correctly updated state
> info with expected defaults, even without needing to set the
> loadbalancer property values.
>
> Not exactly sure how i got the new nodes in this state, my debugging tests
> had stomped on history and forensic info, most likely had a 1.6 node still
> running while trying to bring up a 1.11.4 node, causing the state collison.
>
> Thanks again for folks providing info and feedback!
>
>
>
>
>
>
>
>
> On Wed, Feb 10, 2021 at 1:57 PM Pat White <pa...@verizonmedia.com>
> wrote:
>
>> Thank you Jorge, appreciate the help and feedback!
>>
>>
>>
>>
>> On Wed, Feb 10, 2021 at 1:32 PM Jorge Machado <jo...@me.com> wrote:
>>
>>> For cluster mode check the configs that are on xml files. I had similar
>>> issues when I did not define the values. Letting them empty makes issues.
>>> Best regards
>>> Jorge CEO of Datamesh GmbH (www.dmesh.io
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.dmesh.io&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=RKlnNMaarXTZQkVsOrIOoDLw3dqxyFhzhGlxUBU6zZw&m=npGWjOdDNQ2GxtRow04hsP6HFXXaYoTlgRE9P687qZg&s=tO80ez8je-2Sju3jiq0FrmtsQUjIQv_qrjXXnJ2jrjc&e=>
>>> )
>>>
>>> On 9. Feb 2021, at 02:19, Pat White <pa...@verizonmedia.com> wrote:
>>>
>>> Thanks very much for the feedback Joe, much appreciated. Checking as you
>>> suggested, nothing yet but that's got to be it, some config issue on my
>>> part, that's messing up property parsing.
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Feb 8, 2021 at 5:15 PM Joe Witt <jo...@gmail.com> wrote:
>>>
>>>> PatW
>>>>
>>>> I'd triple-check to ensure there are no weird/special/unexpected
>>>> characters in your nifi.properties file. These are often not obvious in
>>>> default text views so you might need to explicitly set some view to expose
>>>> them.
>>>>
>>>> Yeah this is certainly not a great user experience - we give you just
>>>> enough to have an idea but leave plenty to the imagination here.
>>>>
>>>> I suppose the good news is we know it is a port.
>>>>
>>>> Check lines in/around
>>>> nifi.remote.input.socket.port=
>>>> nifi.web.http.port=8080
>>>> nifi.web.https.port=
>>>> nifi.cluster.node.protocol.port=
>>>> nifi.cluster.load.balance.port=6342
>>>>
>>>> Thanks
>>>>
>>>> On Mon, Feb 8, 2021 at 2:47 PM Pat White <pa...@verizonmedia.com>
>>>> wrote:
>>>>
>>>>> Hi Folks,
>>>>>
>>>>> Appreciate any debugging help on a very odd error, after upgrading a
>>>>> Nifi cluster from 1.6.0 to 1.11.4, flowController fails to start due to:
>>>>>
>>>>> Caused by: org.springframework.beans.factory.BeanCreationException:
>>>>> Error creating bean with name 'flowService': FactoryBean threw exception on
>>>>> object creation; nested exception is
>>>>> org.springframework.beans.factory.BeanCreationException: Error creating
>>>>> bean with name 'flowController' defined in class path resource
>>>>> [nifi-context.xml]: Cannot resolve reference to bean 'clusterCoordinator'
>>>>> while setting bean property 'clusterCoordinator'; nested exception is
>>>>> org.springframework.beans.factory.BeanCreationException: Error creating
>>>>> bean with name 'clusterCoordinator': FactoryBean threw exception on object
>>>>> creation; nested exception is java.lang.IllegalArgumentException: Port must
>>>>> be inclusively in the range [1, 65535]. Port given: 0
>>>>>
>>>>>
>>>>> The error trace is very similar to the example Andy described in
>>>>> NIFI-6336, the issue there i believe is not specifying
>>>>> 'nifi.cluster.node.protocol.port' in 'nifi.properties', however my conf has
>>>>> that set, 'nifi.cluster.node.protocol.port=50233' and should be using
>>>>> 50233 instead of '0'.
>>>>>
>>>>> Cluster had been running fine previously and as far as i can tell,
>>>>> Nifi and ZK confs and settings are all ok. Also compared to another cluster
>>>>> that had been upgraded with no issues, and is running 1.11.4 just fine.
>>>>>
>>>>> Increased debug logging but without success so far. Am looking at the
>>>>> right property association?
>>>>>
>>>>> patw
>>>>>
>>>>>
>>>>>
>>>
Re: [E] After upgrade to 1.11.4, flowController fails to start due to
invalid clusterCoordinator port 0
Posted by Pat White <pa...@verizonmedia.com>.
Hi Folks, found the reason for my issue, following up just FYI.
My local state snapshot, on the updated 1.11.4 nodes, had bad values for
load balancer host and port, so the snapshot had entries like:
"loadBalanceAddress":null,"loadBalancePort":0,
This caused a deadlock where the flow and cluster coordinators had to rely
on existing state values, and therefore couldn't instantiate to apply
correct values, even with these settings explicitly set in nifi.properties.
This condition was actually discussed in NIFI-6336, regarding the port
exclusion checks added in 1.11.x, that was very helpful in knowing where to
look, to track this down.
Solution was simple enough, delete the state snapshot and allow it to be
recreated, this allowed my 1.11.4 nodes to start successfully. After they
came up and entered cluster, verified all nodes correctly updated state
info with expected defaults, even without needing to set the
loadbalancer property values.
Not exactly sure how i got the new nodes in this state, my debugging tests
had stomped on history and forensic info, most likely had a 1.6 node still
running while trying to bring up a 1.11.4 node, causing the state collison.
Thanks again for folks providing info and feedback!
On Wed, Feb 10, 2021 at 1:57 PM Pat White <pa...@verizonmedia.com> wrote:
> Thank you Jorge, appreciate the help and feedback!
>
>
>
>
> On Wed, Feb 10, 2021 at 1:32 PM Jorge Machado <jo...@me.com> wrote:
>
>> For cluster mode check the configs that are on xml files. I had similar
>> issues when I did not define the values. Letting them empty makes issues.
>> Best regards
>> Jorge CEO of Datamesh GmbH (www.dmesh.io
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.dmesh.io&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=RKlnNMaarXTZQkVsOrIOoDLw3dqxyFhzhGlxUBU6zZw&m=npGWjOdDNQ2GxtRow04hsP6HFXXaYoTlgRE9P687qZg&s=tO80ez8je-2Sju3jiq0FrmtsQUjIQv_qrjXXnJ2jrjc&e=>
>> )
>>
>> On 9. Feb 2021, at 02:19, Pat White <pa...@verizonmedia.com> wrote:
>>
>> Thanks very much for the feedback Joe, much appreciated. Checking as you
>> suggested, nothing yet but that's got to be it, some config issue on my
>> part, that's messing up property parsing.
>>
>>
>>
>>
>>
>> On Mon, Feb 8, 2021 at 5:15 PM Joe Witt <jo...@gmail.com> wrote:
>>
>>> PatW
>>>
>>> I'd triple-check to ensure there are no weird/special/unexpected
>>> characters in your nifi.properties file. These are often not obvious in
>>> default text views so you might need to explicitly set some view to expose
>>> them.
>>>
>>> Yeah this is certainly not a great user experience - we give you just
>>> enough to have an idea but leave plenty to the imagination here.
>>>
>>> I suppose the good news is we know it is a port.
>>>
>>> Check lines in/around
>>> nifi.remote.input.socket.port=
>>> nifi.web.http.port=8080
>>> nifi.web.https.port=
>>> nifi.cluster.node.protocol.port=
>>> nifi.cluster.load.balance.port=6342
>>>
>>> Thanks
>>>
>>> On Mon, Feb 8, 2021 at 2:47 PM Pat White <pa...@verizonmedia.com>
>>> wrote:
>>>
>>>> Hi Folks,
>>>>
>>>> Appreciate any debugging help on a very odd error, after upgrading a
>>>> Nifi cluster from 1.6.0 to 1.11.4, flowController fails to start due to:
>>>>
>>>> Caused by: org.springframework.beans.factory.BeanCreationException:
>>>> Error creating bean with name 'flowService': FactoryBean threw exception on
>>>> object creation; nested exception is
>>>> org.springframework.beans.factory.BeanCreationException: Error creating
>>>> bean with name 'flowController' defined in class path resource
>>>> [nifi-context.xml]: Cannot resolve reference to bean 'clusterCoordinator'
>>>> while setting bean property 'clusterCoordinator'; nested exception is
>>>> org.springframework.beans.factory.BeanCreationException: Error creating
>>>> bean with name 'clusterCoordinator': FactoryBean threw exception on object
>>>> creation; nested exception is java.lang.IllegalArgumentException: Port must
>>>> be inclusively in the range [1, 65535]. Port given: 0
>>>>
>>>>
>>>> The error trace is very similar to the example Andy described in
>>>> NIFI-6336, the issue there i believe is not specifying
>>>> 'nifi.cluster.node.protocol.port' in 'nifi.properties', however my conf has
>>>> that set, 'nifi.cluster.node.protocol.port=50233' and should be using
>>>> 50233 instead of '0'.
>>>>
>>>> Cluster had been running fine previously and as far as i can tell, Nifi
>>>> and ZK confs and settings are all ok. Also compared to another cluster that
>>>> had been upgraded with no issues, and is running 1.11.4 just fine.
>>>>
>>>> Increased debug logging but without success so far. Am looking at the
>>>> right property association?
>>>>
>>>> patw
>>>>
>>>>
>>>>
>>
Re: [E] After upgrade to 1.11.4, flowController fails to start due to
invalid clusterCoordinator port 0
Posted by Pat White <pa...@verizonmedia.com>.
Thank you Jorge, appreciate the help and feedback!
On Wed, Feb 10, 2021 at 1:32 PM Jorge Machado <jo...@me.com> wrote:
> For cluster mode check the configs that are on xml files. I had similar
> issues when I did not define the values. Letting them empty makes issues.
> Best regards
> Jorge CEO of Datamesh GmbH (www.dmesh.io
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.dmesh.io&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=RKlnNMaarXTZQkVsOrIOoDLw3dqxyFhzhGlxUBU6zZw&m=npGWjOdDNQ2GxtRow04hsP6HFXXaYoTlgRE9P687qZg&s=tO80ez8je-2Sju3jiq0FrmtsQUjIQv_qrjXXnJ2jrjc&e=>
> )
>
> On 9. Feb 2021, at 02:19, Pat White <pa...@verizonmedia.com> wrote:
>
> Thanks very much for the feedback Joe, much appreciated. Checking as you
> suggested, nothing yet but that's got to be it, some config issue on my
> part, that's messing up property parsing.
>
>
>
>
>
> On Mon, Feb 8, 2021 at 5:15 PM Joe Witt <jo...@gmail.com> wrote:
>
>> PatW
>>
>> I'd triple-check to ensure there are no weird/special/unexpected
>> characters in your nifi.properties file. These are often not obvious in
>> default text views so you might need to explicitly set some view to expose
>> them.
>>
>> Yeah this is certainly not a great user experience - we give you just
>> enough to have an idea but leave plenty to the imagination here.
>>
>> I suppose the good news is we know it is a port.
>>
>> Check lines in/around
>> nifi.remote.input.socket.port=
>> nifi.web.http.port=8080
>> nifi.web.https.port=
>> nifi.cluster.node.protocol.port=
>> nifi.cluster.load.balance.port=6342
>>
>> Thanks
>>
>> On Mon, Feb 8, 2021 at 2:47 PM Pat White <pa...@verizonmedia.com>
>> wrote:
>>
>>> Hi Folks,
>>>
>>> Appreciate any debugging help on a very odd error, after upgrading a
>>> Nifi cluster from 1.6.0 to 1.11.4, flowController fails to start due to:
>>>
>>> Caused by: org.springframework.beans.factory.BeanCreationException:
>>> Error creating bean with name 'flowService': FactoryBean threw exception on
>>> object creation; nested exception is
>>> org.springframework.beans.factory.BeanCreationException: Error creating
>>> bean with name 'flowController' defined in class path resource
>>> [nifi-context.xml]: Cannot resolve reference to bean 'clusterCoordinator'
>>> while setting bean property 'clusterCoordinator'; nested exception is
>>> org.springframework.beans.factory.BeanCreationException: Error creating
>>> bean with name 'clusterCoordinator': FactoryBean threw exception on object
>>> creation; nested exception is java.lang.IllegalArgumentException: Port must
>>> be inclusively in the range [1, 65535]. Port given: 0
>>>
>>>
>>> The error trace is very similar to the example Andy described in
>>> NIFI-6336, the issue there i believe is not specifying
>>> 'nifi.cluster.node.protocol.port' in 'nifi.properties', however my conf has
>>> that set, 'nifi.cluster.node.protocol.port=50233' and should be using
>>> 50233 instead of '0'.
>>>
>>> Cluster had been running fine previously and as far as i can tell, Nifi
>>> and ZK confs and settings are all ok. Also compared to another cluster that
>>> had been upgraded with no issues, and is running 1.11.4 just fine.
>>>
>>> Increased debug logging but without success so far. Am looking at the
>>> right property association?
>>>
>>> patw
>>>
>>>
>>>
>
Re: [E] After upgrade to 1.11.4, flowController fails to start due to
invalid clusterCoordinator port 0
Posted by Jorge Machado <jo...@me.com>.
For cluster mode check the configs that are on xml files. I had similar issues when I did not define the values. Letting them empty makes issues.
Best regards
Jorge CEO of Datamesh GmbH (www.dmesh.io)
> On 9. Feb 2021, at 02:19, Pat White <pa...@verizonmedia.com> wrote:
>
> Thanks very much for the feedback Joe, much appreciated. Checking as you suggested, nothing yet but that's got to be it, some config issue on my part, that's messing up property parsing.
>
>
>
>
>
> On Mon, Feb 8, 2021 at 5:15 PM Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
> PatW
>
> I'd triple-check to ensure there are no weird/special/unexpected characters in your nifi.properties file. These are often not obvious in default text views so you might need to explicitly set some view to expose them.
>
> Yeah this is certainly not a great user experience - we give you just enough to have an idea but leave plenty to the imagination here.
>
> I suppose the good news is we know it is a port.
>
> Check lines in/around
> nifi.remote.input.socket.port=
> nifi.web.http.port=8080
> nifi.web.https.port=
> nifi.cluster.node.protocol.port=
> nifi.cluster.load.balance.port=6342
>
> Thanks
>
> On Mon, Feb 8, 2021 at 2:47 PM Pat White <patwhite@verizonmedia.com <ma...@verizonmedia.com>> wrote:
> Hi Folks,
>
> Appreciate any debugging help on a very odd error, after upgrading a Nifi cluster from 1.6.0 to 1.11.4, flowController fails to start due to:
>
> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController' defined in class path resource [nifi-context.xml]: Cannot resolve reference to bean 'clusterCoordinator' while setting bean property 'clusterCoordinator'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinator': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: Port must be inclusively in the range [1, 65535]. Port given: 0
>
>
> The error trace is very similar to the example Andy described in NIFI-6336, the issue there i believe is not specifying 'nifi.cluster.node.protocol.port' in 'nifi.properties', however my conf has that set, 'nifi.cluster.node.protocol.port=50233' and should be using 50233 instead of '0'.
>
> Cluster had been running fine previously and as far as i can tell, Nifi and ZK confs and settings are all ok. Also compared to another cluster that had been upgraded with no issues, and is running 1.11.4 just fine.
>
> Increased debug logging but without success so far. Am looking at the right property association?
>
> patw
>
>
Re: [E] Re: After upgrade to 1.11.4, flowController fails to start
due to invalid clusterCoordinator port 0
Posted by Pat White <pa...@verizonmedia.com>.
Thanks very much for the feedback Joe, much appreciated. Checking as you
suggested, nothing yet but that's got to be it, some config issue on my
part, that's messing up property parsing.
On Mon, Feb 8, 2021 at 5:15 PM Joe Witt <jo...@gmail.com> wrote:
> PatW
>
> I'd triple-check to ensure there are no weird/special/unexpected
> characters in your nifi.properties file. These are often not obvious in
> default text views so you might need to explicitly set some view to expose
> them.
>
> Yeah this is certainly not a great user experience - we give you just
> enough to have an idea but leave plenty to the imagination here.
>
> I suppose the good news is we know it is a port.
>
> Check lines in/around
> nifi.remote.input.socket.port=
> nifi.web.http.port=8080
> nifi.web.https.port=
> nifi.cluster.node.protocol.port=
> nifi.cluster.load.balance.port=6342
>
> Thanks
>
> On Mon, Feb 8, 2021 at 2:47 PM Pat White <pa...@verizonmedia.com>
> wrote:
>
>> Hi Folks,
>>
>> Appreciate any debugging help on a very odd error, after upgrading a Nifi
>> cluster from 1.6.0 to 1.11.4, flowController fails to start due to:
>>
>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>> creating bean with name 'flowService': FactoryBean threw exception on
>> object creation; nested exception is
>> org.springframework.beans.factory.BeanCreationException: Error creating
>> bean with name 'flowController' defined in class path resource
>> [nifi-context.xml]: Cannot resolve reference to bean 'clusterCoordinator'
>> while setting bean property 'clusterCoordinator'; nested exception is
>> org.springframework.beans.factory.BeanCreationException: Error creating
>> bean with name 'clusterCoordinator': FactoryBean threw exception on object
>> creation; nested exception is java.lang.IllegalArgumentException: Port must
>> be inclusively in the range [1, 65535]. Port given: 0
>>
>>
>> The error trace is very similar to the example Andy described in
>> NIFI-6336, the issue there i believe is not specifying
>> 'nifi.cluster.node.protocol.port' in 'nifi.properties', however my conf has
>> that set, 'nifi.cluster.node.protocol.port=50233' and should be using
>> 50233 instead of '0'.
>>
>> Cluster had been running fine previously and as far as i can tell, Nifi
>> and ZK confs and settings are all ok. Also compared to another cluster that
>> had been upgraded with no issues, and is running 1.11.4 just fine.
>>
>> Increased debug logging but without success so far. Am looking at the
>> right property association?
>>
>> patw
>>
>>
>>
Re: After upgrade to 1.11.4, flowController fails to start due to
invalid clusterCoordinator port 0
Posted by Joe Witt <jo...@gmail.com>.
PatW
I'd triple-check to ensure there are no weird/special/unexpected characters
in your nifi.properties file. These are often not obvious in default text
views so you might need to explicitly set some view to expose them.
Yeah this is certainly not a great user experience - we give you just
enough to have an idea but leave plenty to the imagination here.
I suppose the good news is we know it is a port.
Check lines in/around
nifi.remote.input.socket.port=
nifi.web.http.port=8080
nifi.web.https.port=
nifi.cluster.node.protocol.port=
nifi.cluster.load.balance.port=6342
Thanks
On Mon, Feb 8, 2021 at 2:47 PM Pat White <pa...@verizonmedia.com> wrote:
> Hi Folks,
>
> Appreciate any debugging help on a very odd error, after upgrading a Nifi
> cluster from 1.6.0 to 1.11.4, flowController fails to start due to:
>
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'flowService': FactoryBean threw exception on
> object creation; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'flowController' defined in class path resource
> [nifi-context.xml]: Cannot resolve reference to bean 'clusterCoordinator'
> while setting bean property 'clusterCoordinator'; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'clusterCoordinator': FactoryBean threw exception on object
> creation; nested exception is java.lang.IllegalArgumentException: Port must
> be inclusively in the range [1, 65535]. Port given: 0
>
>
> The error trace is very similar to the example Andy described in
> NIFI-6336, the issue there i believe is not specifying
> 'nifi.cluster.node.protocol.port' in 'nifi.properties', however my conf has
> that set, 'nifi.cluster.node.protocol.port=50233' and should be using
> 50233 instead of '0'.
>
> Cluster had been running fine previously and as far as i can tell, Nifi
> and ZK confs and settings are all ok. Also compared to another cluster that
> had been upgraded with no issues, and is running 1.11.4 just fine.
>
> Increased debug logging but without success so far. Am looking at the
> right property association?
>
> patw
>
>
>