HDFS -> NiFi -> NiFi -> HDFS

[Michael Dyer <mi...@trapezoid.com>]

I have a set of files to move between two HDFS systems and I'm considering
the use of a NiFi remote processing group.

However, I'm having a hard time visualizing the physical data flow -
specifically with regards to the content repository and provenance.  What
happens when the HDFS files exceed the physical space available to the
content repository?

Thanks!

Michael

Re: HDFS -> NiFi -> NiFi -> HDFS

[Mark Payne <ma...@hotmail.com>]

We should also mention that, in addition to warning you, you can configure your flow
to apply backpressure [1] so that if you are not able to send to the downstream system,
you can stop pulling data from HDFS until you are able to pull again. Then the flow will
resume.

Thanks
-Mark

[1] http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#settings <http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#settings>


> On Dec 10, 2015, at 6:07 PM, Corey Flowers <cf...@onyxpoint.com> wrote:
> 
> Nifi will only keep the files in its repository until they are
> delivered to your second hdfs system. So basically nifi repositories
> are only temporary storage until the data has been processed and
> delivered. The only thing you would have to worry about is if the
> system can not be delivered and the data backs up in your system. But
> even then there are mechanisms that you can use to warn you before
> anything gets out of hand.
> 
> Sent from my iPhone
> 
>> On Dec 10, 2015, at 4:31 PM, Michael Dyer <mi...@trapezoid.com> wrote:
>> 
>> I have a set of files to move between two HDFS systems and I'm considering the use of a NiFi remote processing group.
>> 
>> However, I'm having a hard time visualizing the physical data flow - specifically with regards to the content repository and provenance.  What happens when the HDFS files exceed the physical space available to the content repository?
>> 
>> Thanks!
>> 
>> Michael
>> 
>> 
>> 

Re: HDFS -> NiFi -> NiFi -> HDFS

[Corey Flowers <cf...@onyxpoint.com>]

Nifi will only keep the files in its repository until they are
delivered to your second hdfs system. So basically nifi repositories
are only temporary storage until the data has been processed and
delivered. The only thing you would have to worry about is if the
system can not be delivered and the data backs up in your system. But
even then there are mechanisms that you can use to warn you before
anything gets out of hand.

Sent from my iPhone

> On Dec 10, 2015, at 4:31 PM, Michael Dyer <mi...@trapezoid.com> wrote:
>
> I have a set of files to move between two HDFS systems and I'm considering the use of a NiFi remote processing group.
>
> However, I'm having a hard time visualizing the physical data flow - specifically with regards to the content repository and provenance.  What happens when the HDFS files exceed the physical space available to the content repository?
>
> Thanks!
>
> Michael
>
>
>