You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Shawn Weeks <sw...@weeksconsulting.us> on 2018/02/19 20:10:38 UTC
Routing Files Based on Dynamic Pattern Matching
Hi, I'm looking for some ideas on how to handle a workflow I'm developing. I have NiFi monitoring a drop off location where files are delivered. Files from this drop off location need to be routed to various target directories based on a series of file name patterns. Currently I have a custom processor I developed that queries a database table comparing the incoming file name against known file name patterns stored as regular expressions and attaches those attributes to the flow file. I feel like there is a better way to do this but I'm still fairly new to NiFi.
Thanks
Shawn Weeks
RE: Routing Files Based on Dynamic Pattern Matching
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
The issue with RouteOnAttribute is that the Regular Expression must be known when the NiFi Workflow is built. There are hundreds of unique patterns and each match must go to a different location in the target system. It also needs to be updatable without changing the NiFi workflow.
Thanks
Shawn
From: Juan Sequeiros <he...@gmail.com>
Sent: Monday, February 19, 2018 4:14 PM
To: users@nifi.apache.org
Subject: Re: Routing Files Based on Dynamic Pattern Matching
You could also RouteOnAttribute [1] doing the same regular expression checks using NIFI expression language [2].
For instance, I can create a route for "DATA A" if my attribute "filename" matches a regular expression.
${filename:find'SomeJavaRegEx')}
If you can control your workflow maybe bring it into NIFI using Rest and giving it headers which one of them can be "NifiData:MyData"
Then depending what "Listener" you are using you can grab that header and make routing decisions of it with RouteOnAtrrbiute"
[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.RouteOnAttribute/index.html
[2] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
On Mon, Feb 19, 2018 at 4:48 PM Andrew Grande <ap...@gmail.com>> wrote:
Maybe take a look at one of the ScanContent/Attribute processors? It allows to map in a reloadable external file.
Or, better yet, one of the LookupService variants, which is more generic.
HTH,
Andrew
On Mon, Feb 19, 2018, 3:10 PM Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Hi, I’m looking for some ideas on how to handle a workflow I’m developing. I have NiFi monitoring a drop off location where files are delivered. Files from this drop off location need to be routed to various target directories based on a series of file name patterns. Currently I have a custom processor I developed that queries a database table comparing the incoming file name against known file name patterns stored as regular expressions and attaches those attributes to the flow file. I feel like there is a better way to do this but I’m still fairly new to NiFi.
Thanks
Shawn Weeks
Re: Routing Files Based on Dynamic Pattern Matching
Posted by Juan Sequeiros <he...@gmail.com>.
You could also RouteOnAttribute [1] doing the same regular
expression checks using NIFI expression language [2].
For instance, I can create a route for "DATA A" if my attribute "filename"
matches a regular expression.
${filename:find'SomeJavaRegEx')}
If you can control your workflow maybe bring it into NIFI using Rest and
giving it headers which one of them can be "NifiData:MyData"
Then depending what "Listener" you are using you can grab that header and
make routing decisions of it with RouteOnAtrrbiute"
[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.RouteOnAttribute/index.html
[2]
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
On Mon, Feb 19, 2018 at 4:48 PM Andrew Grande <ap...@gmail.com> wrote:
> Maybe take a look at one of the ScanContent/Attribute processors? It
> allows to map in a reloadable external file.
>
> Or, better yet, one of the LookupService variants, which is more generic.
>
> HTH,
> Andrew
>
>
> On Mon, Feb 19, 2018, 3:10 PM Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
>> Hi, I’m looking for some ideas on how to handle a workflow I’m
>> developing. I have NiFi monitoring a drop off location where files are
>> delivered. Files from this drop off location need to be routed to various
>> target directories based on a series of file name patterns. Currently I
>> have a custom processor I developed that queries a database table comparing
>> the incoming file name against known file name patterns stored as regular
>> expressions and attaches those attributes to the flow file. I feel like
>> there is a better way to do this but I’m still fairly new to NiFi.
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>
Re: Routing Files Based on Dynamic Pattern Matching
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
I’ve been looking at doing that but the current scripted lookup service isn’t supported by the lookup attribute processor. I had been hoping I wouldn’t have to create my own nar.
Shawn
On Feb 20, 2018, at 5:40 AM, Andrew Grande <ap...@gmail.com>> wrote:
I think this is exactly what a Lookup Service was designed to do. You are free to implement any logic of yours behind the scenes.
Andrew
On Mon, Feb 19, 2018, 5:12 PM Shawn Weeks <sw...@weeksconsulting.us>> wrote:
The problem I see with the Scan Processors is they don’t return anything other than matched or not matched. And Lookup Services assume you have a specific key you’re trying to match. I need to look for regular expression matches by comparing against a list of regular expressions not specific values. Let’s say I have a file named 123_blah_test_blah.csv, I might have a regular expression defined that says if the file name matches .*blah_test_blah(?!_somethingelse).* then it should goto directory /tmp/blah_test
Thanks
Shawn
From: Andrew Grande <ap...@gmail.com>>
Sent: Monday, February 19, 2018 3:48 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Routing Files Based on Dynamic Pattern Matching
Maybe take a look at one of the ScanContent/Attribute processors? It allows to map in a reloadable external file.
Or, better yet, one of the LookupService variants, which is more generic.
HTH,
Andrew
On Mon, Feb 19, 2018, 3:10 PM Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Hi, I’m looking for some ideas on how to handle a workflow I’m developing. I have NiFi monitoring a drop off location where files are delivered. Files from this drop off location need to be routed to various target directories based on a series of file name patterns. Currently I have a custom processor I developed that queries a database table comparing the incoming file name against known file name patterns stored as regular expressions and attaches those attributes to the flow file. I feel like there is a better way to do this but I’m still fairly new to NiFi.
Thanks
Shawn Weeks
Re: Routing Files Based on Dynamic Pattern Matching
Posted by Andrew Grande <ap...@gmail.com>.
I think this is exactly what a Lookup Service was designed to do. You are
free to implement any logic of yours behind the scenes.
Andrew
On Mon, Feb 19, 2018, 5:12 PM Shawn Weeks <sw...@weeksconsulting.us> wrote:
> The problem I see with the Scan Processors is they don’t return anything
> other than matched or not matched. And Lookup Services assume you have a
> specific key you’re trying to match. I need to look for regular expression
> matches by comparing against a list of regular expressions not specific
> values. Let’s say I have a file named 123_blah_test_blah.csv, I might have
> a regular expression defined that says if the file name matches
> .*blah_test_blah(?!_somethingelse).* then it should goto directory
> /tmp/blah_test
>
>
>
> Thanks
>
> Shawn
>
>
>
> *From:* Andrew Grande <ap...@gmail.com>
> *Sent:* Monday, February 19, 2018 3:48 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Routing Files Based on Dynamic Pattern Matching
>
>
>
> Maybe take a look at one of the ScanContent/Attribute processors? It
> allows to map in a reloadable external file.
>
>
>
> Or, better yet, one of the LookupService variants, which is more generic.
>
>
>
> HTH,
>
> Andrew
>
> On Mon, Feb 19, 2018, 3:10 PM Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> Hi, I’m looking for some ideas on how to handle a workflow I’m developing.
> I have NiFi monitoring a drop off location where files are delivered. Files
> from this drop off location need to be routed to various target directories
> based on a series of file name patterns. Currently I have a custom
> processor I developed that queries a database table comparing the incoming
> file name against known file name patterns stored as regular expressions
> and attaches those attributes to the flow file. I feel like there is a
> better way to do this but I’m still fairly new to NiFi.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
RE: Routing Files Based on Dynamic Pattern Matching
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
The problem I see with the Scan Processors is they don’t return anything other than matched or not matched. And Lookup Services assume you have a specific key you’re trying to match. I need to look for regular expression matches by comparing against a list of regular expressions not specific values. Let’s say I have a file named 123_blah_test_blah.csv, I might have a regular expression defined that says if the file name matches .*blah_test_blah(?!_somethingelse).* then it should goto directory /tmp/blah_test
Thanks
Shawn
From: Andrew Grande <ap...@gmail.com>
Sent: Monday, February 19, 2018 3:48 PM
To: users@nifi.apache.org
Subject: Re: Routing Files Based on Dynamic Pattern Matching
Maybe take a look at one of the ScanContent/Attribute processors? It allows to map in a reloadable external file.
Or, better yet, one of the LookupService variants, which is more generic.
HTH,
Andrew
On Mon, Feb 19, 2018, 3:10 PM Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Hi, I’m looking for some ideas on how to handle a workflow I’m developing. I have NiFi monitoring a drop off location where files are delivered. Files from this drop off location need to be routed to various target directories based on a series of file name patterns. Currently I have a custom processor I developed that queries a database table comparing the incoming file name against known file name patterns stored as regular expressions and attaches those attributes to the flow file. I feel like there is a better way to do this but I’m still fairly new to NiFi.
Thanks
Shawn Weeks
Re: Routing Files Based on Dynamic Pattern Matching
Posted by Andrew Grande <ap...@gmail.com>.
Maybe take a look at one of the ScanContent/Attribute processors? It allows
to map in a reloadable external file.
Or, better yet, one of the LookupService variants, which is more generic.
HTH,
Andrew
On Mon, Feb 19, 2018, 3:10 PM Shawn Weeks <sw...@weeksconsulting.us> wrote:
> Hi, I’m looking for some ideas on how to handle a workflow I’m developing.
> I have NiFi monitoring a drop off location where files are delivered. Files
> from this drop off location need to be routed to various target directories
> based on a series of file name patterns. Currently I have a custom
> processor I developed that queries a database table comparing the incoming
> file name against known file name patterns stored as regular expressions
> and attaches those attributes to the flow file. I feel like there is a
> better way to do this but I’m still fairly new to NiFi.
>
>
>
> Thanks
>
> Shawn Weeks
>