GetHDFS NiFi expression language

[Michael Parco <33...@cua.edu>]

Has there been thoughts/discussion for thoughts on the GetHDFS processor
supporting the NiFi expression language for file retrieval? Similar to how
PutHDFS can utilize the NiFi expression lanaguage to dynamically generate
directories, I see the benefit for the GetHDFS processor to do the same in
order to retrieve files based on say a time (YYYY;/MM/DD/hh) directory
structure.

~

Re: GetHDFS NiFi expression language

[Michael Parco <33...@cua.edu>]

I believe the list/fetch processors could work, it is just going to take a
step in between to route the specific directory I am looking for. For
example, I want to pull back all files from the previous hour (by previous
hour I am referencing my directory structure '/in/data/yyyy/MM/dd/HH/'), so
if I put a RouteOnAttribute inbetween the List/Fetch I should be able to
match for the specific directory I need... working on this now with the
expression language to match
/in/data/${now():toNumber():minus(3600000):format('yyyy/MM/dd/HH')}

On Mon, Feb 8, 2016 at 11:24 AM, Bryan Bende <bb...@gmail.com> wrote:

> Hello,
>
> There is another processor, FetchHDFS, which supports expression language
> for the file to fetch.
> It is intended to be used with ListHDFS to feed the results of the listing
> into FetchHDFS.
>
> Do the List/Fetch processors work for your use-case?
>
> -Bryan
>
>
> On Mon, Feb 8, 2016 at 8:23 AM, Michael Parco <33...@cua.edu> wrote:
>
> > Has there been thoughts/discussion for thoughts on the GetHDFS processor
> > supporting the NiFi expression language for file retrieval? Similar to
> how
> > PutHDFS can utilize the NiFi expression lanaguage to dynamically generate
> > directories, I see the benefit for the GetHDFS processor to do the same
> in
> > order to retrieve files based on say a time (YYYY;/MM/DD/hh) directory
> > structure.
> >
> > ~
> >
>

Re: GetHDFS NiFi expression language

[Bryan Bende <bb...@gmail.com>]

Hello,

There is another processor, FetchHDFS, which supports expression language
for the file to fetch.
It is intended to be used with ListHDFS to feed the results of the listing
into FetchHDFS.

Do the List/Fetch processors work for your use-case?

-Bryan


On Mon, Feb 8, 2016 at 8:23 AM, Michael Parco <33...@cua.edu> wrote:

> Has there been thoughts/discussion for thoughts on the GetHDFS processor
> supporting the NiFi expression language for file retrieval? Similar to how
> PutHDFS can utilize the NiFi expression lanaguage to dynamically generate
> directories, I see the benefit for the GetHDFS processor to do the same in
> order to retrieve files based on say a time (YYYY;/MM/DD/hh) directory
> structure.
>
> ~
>