RE: Issues in using Remote process group in a nifi cluster (nifi 1.4.0)

[mo...@open-insights.co.in]

Hi,

 

I have tried using site-to-site two local nifi instances, still I’m facing the same issue. I feel I’ve incorrectly configured the nifi.

Could you please share the sample config.

 

From: mohit.jain@open-insights.co.in [mailto:mohit.jain@open-insights.co.in] 
Sent: 28 February 2018 21:22
To: users@nifi.apache.org
Subject: RE: Issues in using Remote process group in a nifi cluster (nifi 1.4.0)

 

Nifi repositories are located in the installed Nifi directory only.  I am getting this issue only in this case, i.e, while using the remote process group. It is working absolutely fine without it. 

 

From: Bryan Bende [mailto:bbende@gmail.com] 
Sent: 28 February 2018 21:14
To: users@nifi.apache.org <ma...@nifi.apache.org> 
Subject: Re: Issues in using Remote process group in a nifi cluster (nifi 1.4.0)

 

Have you changed where NiFi's repositories are located?

By default they will be under the root of NiFi's install directory.

Your screenshots are showing that you are likely out of disk space on the the disk where the repositories are located.

 

 

On Wed, Feb 28, 2018 at 10:37 AM, <mohit.jain@open-insights.co.in <ma...@open-insights.co.in> > wrote:

Sharing the Site-to-Site property for reference :-

 

Node-1 :- 

10.0.0.42 – dn1

10.0.0.203 – dn2

 

# Site to Site properties

nifi.remote.input.host=10.0.0.42

nifi.remote.input.secure=false

nifi.remote.input.socket.port=9090

nifi.remote.input.http.enabled=true

nifi.remote.input.http.transaction.ttl=30 sec

 

 

Node-2 :- 

 

 

# Site to Site properties

nifi.remote.input.host=10.0.0.203

nifi.remote.input.secure=false

nifi.remote.input.socket.port=9090

nifi.remote.input.http.enabled=true

nifi.remote.input.http.transaction.ttl=30 sec

 

It is running on EC2 with one edge node and two datanode.

 

 

From: mohit.jain@open-insights.co.in <ma...@open-insights.co.in>  [mailto:mohit.jain@open-insights.co.in <ma...@open-insights.co.in> ] 
Sent: 28 February 2018 21:02
To: users@nifi.apache.org <ma...@nifi.apache.org> 
Subject: RE: Issues in using Remote process group in a nifi cluster (nifi 1.4.0)

 

Hi,

 

Generate flowfile is scheduled to run on the primary node only. There is no additional information in the logs. It also throws some  exception on the GenerateFlowFile if it not stopped for sometime.

 



 

I have shared the logs in the attachment.

 

Regards,

Mohit

 

From: Matt Burgess [mailto:mattyb149@apache.org] 
Sent: 28 February 2018 20:40
To: users@nifi.apache.org <ma...@nifi.apache.org> 
Subject: Re: Issues in using Remote process group in a nifi cluster (nifi 1.4.0)

 

Mohit,

 

Is your GenerateFlowFile configured to run on the Primary Node Only? If not, each node in the cluster will be generating flow files, and they will be re-distributed via the RPGs to the cluster. Also can you share your nifi-app.log, or the relevant stack trace that accompanies the error bulletin(s) you're seeing. NullPointerExceptions usually indicate bugs (at least in error handling, even if it occurs due to misconfiguration).

 

Thanks,

Matt

 

On Wed, Feb 28, 2018 at 10:04 AM, <mohit.jain@open-insights.co.in <ma...@open-insights.co.in> > wrote:

Hi, 

 

I am trying to use nifi cluster to distribute the flow. I am facing a lot of issues to achieve it.  I have created a sample flow consisting of GenerateFlowFile, Remote Process Group, and a LogAttribute processor. 

 

The flowfile generated by the GenerateFlowFile is still in the queue, but rip(remote input port) keeps on generating the flowfiles at a bizarre rate. For example, for 1 flowfile, it creates 35. This is happening for every flow(tried it using List/Fetch paradigm as well). It also throws the following bulletins.

 

 



 

 

I am getting the following error. Please let me know if I’m doing anything wrong.

 

Thanks,

Mohit