Using Regular Expression/Pattern matching

[pokayoke <ar...@intuit.com>]

Is it possible to use patter matching in ListS3 prefix property. 

Basically I need to get a file with the latest timestamp in its name toward
the end. So for that I am hoping to use something like *_YYYY-MMDD* format
in the List S3 prefix property, but its not working. Any help or advise
appreciated.



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Using Regular Expression/Pattern matching

[Kevin Doran <kd...@gmail.com>]

Hi,

I believe the prefix property is treated literally as a prefix. i.e. think "starts with". S3 object keys are usually treated as a file path would be on a traditional file system, so following that convention the prefix property is allowing you to specify a "directory" to list in the S3 bucket.

If you have ownership of the process that is populating the S3 bucket, one approach I have seen is to write objects with date/time in the keys in a format such as:

application/logs/YYYY/MM/DD/hh/application.log

If you have no control over how objects are being written to the S3 bucket, you could use ListS3 to pull all flow files and then use each flowfile's filename attribute or s3.lastmodified attribute to do filtering later in your flow. 

Hope this helps!

Cheers,
Kevin


On 11/14/17, 16:29, "pokayoke" <ar...@intuit.com> wrote:

    Is it possible to use patter matching in ListS3 prefix property. 
    
    Basically I need to get a file with the latest timestamp in its name toward
    the end. So for that I am hoping to use something like *_YYYY-MMDD* format
    in the List S3 prefix property, but its not working. Any help or advise
    appreciated.
    
    
    
    --
    Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/