How to pass custom attribute to a ListFile and putHDFS

[Vdsa <vi...@persistent.co.in>]

I have a custom processor which sets a source and destination directories. I
need to pass these attributes as a directory to ListFile and PutHDFS. I have
added attribute in session like 

flowFile = session.putAttribute(flowFile, "source dir", source); 

and I am trying to access it using expression language like Input Directory
= ${'source dir'} in processor properties.

I get directory empty. How can i get directory value for listFile and
PutHdfs ??



--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/How-to-pass-custom-attribute-to-a-ListFile-and-putHDFS-tp15558.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: How to pass custom attribute to a ListFile and putHDFS

[Vdsa <vi...@persistent.co.in>]

Thanks a lot for the detailed explanation, I agree with all your points.
Just wanted to know how to define custom attributes. Thanks in advance. 



--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/How-to-pass-custom-attribute-to-a-ListFile-and-putHDFS-tp15558p15560.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: How to pass custom attribute to a ListFile and putHDFS

[Andy LoPresto <al...@gmail.com>]

Hi,

I would try a couple things (not at my machine right now to double check the spaces option)

* ensure that the custom processor is calling session.commit() to persist the attribute value. You can also examine the flowfiles in the queue between your custom processor and the following processor to view all attributes (or just modified ones)
* try using an attribute name without spaces. This means you won't need to reference it with quotes in the expression language. 
* ListFile does not accept input, so you can't route any flowfiles to it. Therefore, no non-standard attributes are available when it executes. You would need to define a custom property somewhere in order to reference a dynamic value in ListFile, but I don't see how that would work -- ListFile is triggered only by itself, not receiving a signal somewhere else. 
* You can raise a Jira to request dynamic properties for ListFile but I honestly don't see it being implemented -- perhaps I am missing something but it doesn't make sense to me to allow that. Expression Language is supported to allow things like inserting the date to monitor a new directory each day, not to allow incoming flowfiles (again, not supported by this processor) to determine the directory on the fly. 



Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Apr 25, 2017, at 20:32, Vdsa <vi...@persistent.co.in> wrote:
> 
> I have a custom processor which sets a source and destination directories. I
> need to pass these attributes as a directory to ListFile and PutHDFS. I have
> added attribute in session like 
> 
> flowFile = session.putAttribute(flowFile, "source dir", source); 
> 
> and I am trying to access it using expression language like Input Directory
> = ${'source dir'} in processor properties.
> 
> I get directory empty. How can i get directory value for listFile and
> PutHdfs ??
> 
> 
> 
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/How-to-pass-custom-attribute-to-a-ListFile-and-putHDFS-tp15558.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.