You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nifi.apache.org by Rahul Dahiya <ra...@impetus.co.in> on 2016/08/03 06:05:15 UTC

NiFi List/Fetch SFTP processors making 100 plus connections

Hi Team,


We are using NiFi for downloading files from a remote machine. We are using fetch and list sftp processors to download the files. We are able to download the files from the remote machine but we see 100+ connections on the remote machine from the NiFi cluster. We are using NiFi version -0.6.1 .

Can you help us in finding out why so many connections are being made and how can we limit the number of connections to the remote machine ? . Any help is appreciated . Thanks in advance.


Regards ,

Rahul

________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.

Re: NiFi List/Fetch SFTP processors making 100 plus connections

Posted by Bryan Bende <bb...@gmail.com>.

Hi Kislay,

If you have a 3 node cluster and you setup ListSFTP -> FetchSFTP, it is
actually running this on all 3 nodes and likely fetching the same files
three times.
I'm not sure that would equate to 100s of connections, but it would
definitely be more connections then needed.

Typically the way you use the List + Fetch processors in a cluster is to
schedule the List processor to run on primary node only, and connect it to
a Remote Process Group.
The Remote Process Group points back to the same cluster and connects to an
Input Port port, this is essentially distributing all of the filenames
equally across your three nodes.
The Input Port then connects to FetchFile so all three nodes fetch in
parallel, but fetch different files.

Some resources that show how this work are here:
https://community.hortonworks.com/storage/attachments/1989-03-pulling-with-redistribution.png
http://www.slideshare.net/BryanBende/building-data-pipelines-for-solr-with-apache-nifi/31?src=clipshare

Let us know if this helps reduce the connections.

Thanks,

Bryan


On Wed, Aug 3, 2016 at 12:38 PM, Kislay Kumar <ki...@impetus.co.in>
wrote:

> Hi Bryan
>
> We have 3 node nifi cluster.
>
> And we are trying to connect to one sftp server.
>
> Flow is listsftp then fetchsftp.
>
> But when we check no. Of connections it shows 100+.
>
> Regards
> Kislay
>
> Sent from my iPhone
>
> On 03-Aug-2016, at 8:11 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> Hi Rahul,
>
> A couple of questions to better understand the scenario...
>
> How many nodes in your NiFi cluster?
>
> Are you running ListSFPT on the primary node and using site-to-site to
> redistribute to the other nodes?
>
> Have you increased the concurrent tasks on any of the FetchSFPT processors?
>
> Thanks,
>
> Bryan
>
> On Wed, Aug 3, 2016 at 2:05 AM, Rahul Dahiya <ra...@impetus.co.in>
> wrote:
>
>> Hi Team,
>>
>>
>> We are using NiFi for downloading files from a remote machine. We are
>> using fetch and list sftp processors to download the files. We are able to
>> download the files from the remote machine but we see 100+ connections on
>> the remote machine from the NiFi cluster. We are using NiFi version -0.6.1 .
>>
>> Can you help us in finding out why so many connections are being made and
>> how can we limit the number of connections to the remote machine ? . Any
>> help is appreciated . Thanks in advance.
>>
>>
>> Regards ,
>>
>> Rahul
>>
>> ________________________________
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>

Re: NiFi List/Fetch SFTP processors making 100 plus connections

Posted by Kislay Kumar <ki...@impetus.co.in>.

Hi Bryan

We have 3 node nifi cluster.

And we are trying to connect to one sftp server.

Flow is listsftp then fetchsftp.

But when we check no. Of connections it shows 100+.

Regards
Kislay

Sent from my iPhone

On 03-Aug-2016, at 8:11 PM, Bryan Bende <bb...@gmail.com>> wrote:

Hi Rahul,

A couple of questions to better understand the scenario...

How many nodes in your NiFi cluster?

Are you running ListSFPT on the primary node and using site-to-site to redistribute to the other nodes?

Have you increased the concurrent tasks on any of the FetchSFPT processors?

Thanks,

Bryan

On Wed, Aug 3, 2016 at 2:05 AM, Rahul Dahiya <ra...@impetus.co.in>> wrote:
Hi Team,

We are using NiFi for downloading files from a remote machine. We are using fetch and list sftp processors to download the files. We are able to download the files from the remote machine but we see 100+ connections on the remote machine from the NiFi cluster. We are using NiFi version -0.6.1 .

Can you help us in finding out why so many connections are being made and how can we limit the number of connections to the remote machine ? . Any help is appreciated . Thanks in advance.

Regards ,

Rahul

________________________________

NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.

________________________________

Re: NiFi List/Fetch SFTP processors making 100 plus connections

Posted by Bryan Bende <bb...@gmail.com>.

Hi Rahul,

A couple of questions to better understand the scenario...

How many nodes in your NiFi cluster?

Are you running ListSFPT on the primary node and using site-to-site to
redistribute to the other nodes?

Have you increased the concurrent tasks on any of the FetchSFPT processors?

Thanks,

Bryan

On Wed, Aug 3, 2016 at 2:05 AM, Rahul Dahiya <ra...@impetus.co.in>
wrote:

> Hi Team,
>
>
> We are using NiFi for downloading files from a remote machine. We are
> using fetch and list sftp processors to download the files. We are able to
> download the files from the remote machine but we see 100+ connections on
> the remote machine from the NiFi cluster. We are using NiFi version -0.6.1 .
>
> Can you help us in finding out why so many connections are being made and
> how can we limit the number of connections to the remote machine ? . Any
> help is appreciated . Thanks in advance.
>
>
> Regards ,
>
> Rahul
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>