You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Ashraf Hasson <AH...@WindMobile.ca> on 2016/04/15 16:40:09 UTC
datetime argument to the GetSFTP processor
Hi there,
I've been trying to use the GetSFTP processor to fetch files that have a 'yyyymmdd' part but the only option I have is to use Java Regular Expression, not sure how to specify today or yesterday's date dynamically.
I thought an alternative is to use a locally generated file that contains the datetime string, use a GetFile + UpdateAttribute to pass in a custom attribute to the GetSFTP processor that way, but sounded a bit cumbersome tbh.
Any thoughts/hints please?
Thanks,
Ashraf
Re: datetime argument to the GetSFTP processor
Posted by Joe Witt <jo...@gmail.com>.
Perfect! You got it.
On Thu, Apr 21, 2016 at 1:11 PM, Ashraf Hasson <AH...@windmobile.ca> wrote:
> Hi there,
>
> Okay, so I should use RouteOnAttribute I think.
>
> Here's the source filename: test.file.20160420-015931.csv
>
> I've configured the RouteOnAttribute to route the `filename` to success when the following is matched:
>
> property: filename == value: ${filename:substringAfter('test.file.'):startsWith( ${now():toNumber():minus(86400000):format('yyyymmdd')} )}
>
> Maybe this is not the right way but I'd highly appreciate some help.
>
> Thanks,
>
> ________________________________________
> From: Ashraf Hasson
> Sent: April 21, 2016 12:50 PM
> To: users@nifi.apache.org
> Subject: Re: datetime argument to the GetSFTP processor
>
> Hi James,
>
> Thanks for your reply.
> The source has multiple files generated per day and I'm interested in one of those, so filtering is required I guess.
> I don't know how to filter based on the filename, was trying to pipe things like this: ListSFTP -> UpdateAttribute -> FetchSFTP
> but I'm sure there's a better way to filter. Can you help by giving me a pointer or two please?
>
> I also tried to pipe directly from ListSFTP -> FetchSFTP and used ${now():toNumber():minus(86400000):format('yyyymmdd')}-*.csv but then this doesn't match the remote file guessing because FetchSFTP doesn't do globing.
>
> Thanks,
> ________________________________________
> From: Joe Witt <jo...@gmail.com>
> Sent: April 15, 2016 1:35 PM
> To: users@nifi.apache.org
> Subject: Re: datetime argument to the GetSFTP processor
>
> Ahh - great idea James!
>
> On Fri, Apr 15, 2016 at 1:34 PM, James Wing <jv...@gmail.com> wrote:
>> Have you looked at a combination of ListSFTP -> FetchSFTP? I believe
>> ListSFTP has built-in support for tracking recent files, and it might
>> satisfy your use case. If not, you can certainly filter the listed files by
>> the "filename" attribute before calling FetchSFTP.
>>
>> On Fri, Apr 15, 2016 at 7:40 AM, Ashraf Hasson <AH...@windmobile.ca>
>> wrote:
>>>
>>> Hi there,
>>>
>>>
>>> I've been trying to use the GetSFTP processor to fetch files that have a
>>> 'yyyymmdd' part but the only option I have is to use Java Regular
>>> Expression, not sure how to specify today or yesterday's date dynamically.
>>>
>>>
>>> I thought an alternative is to use a locally generated file that contains
>>> the datetime string, use a GetFile + UpdateAttribute to pass in a custom
>>> attribute to the GetSFTP processor that way, but sounded a bit cumbersome
>>> tbh.
>>>
>>>
>>> Any thoughts/hints please?
>>>
>>>
>>> Thanks,
>>>
>>> Ashraf
>>
>>
Re: datetime argument to the GetSFTP processor
Posted by Ashraf Hasson <AH...@WindMobile.ca>.
Thanks Joe and Pierre,
Yep, that was the way to go indeed and thanks for your confirmation, time and replies [😊]
I've moved matching the static part to the ListSFTP regex with File Filter Regex = test.file.\d{8}-\d{6}.csv
and matched with Pierre's expression to simplify the route condition. Further down in GetSFTP, I had to specify ${path}/${filename} to fetch the file.
Best regards,
________________________________
From: Pierre Villard <pi...@gmail.com>
Sent: April 21, 2016 1:18 PM
To: users@nifi.apache.org
Subject: Re: datetime argument to the GetSFTP processor
Hi Ashraf,
I believe you are right, RouteOnAttribute is certainly what you should use.
Regarding the expression, it sounds good to me. Depending of the filename format characteristics, may be you can directly use
${filename:contains( ${now():toNumber():minus(86400000):format('yyyymmdd')} )}
2016-04-21 19:11 GMT+02:00 Ashraf Hasson <AH...@windmobile.ca>>:
Hi there,
Okay, so I should use RouteOnAttribute I think.
Here's the source filename: test.file.20160420-015931.csv
I've configured the RouteOnAttribute to route the `filename` to success when the following is matched:
property: filename == value: ${filename:substringAfter('test.file.'):startsWith( ${now():toNumber():minus(86400000):format('yyyymmdd')} )}
Maybe this is not the right way but I'd highly appreciate some help.
Thanks,
________________________________________
From: Ashraf Hasson
Sent: April 21, 2016 12:50 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: datetime argument to the GetSFTP processor
Hi James,
Thanks for your reply.
The source has multiple files generated per day and I'm interested in one of those, so filtering is required I guess.
I don't know how to filter based on the filename, was trying to pipe things like this: ListSFTP -> UpdateAttribute -> FetchSFTP
but I'm sure there's a better way to filter. Can you help by giving me a pointer or two please?
I also tried to pipe directly from ListSFTP -> FetchSFTP and used ${now():toNumber():minus(86400000):format('yyyymmdd')}-*.csv but then this doesn't match the remote file guessing because FetchSFTP doesn't do globing.
Thanks,
________________________________________
From: Joe Witt <jo...@gmail.com>>
Sent: April 15, 2016 1:35 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: datetime argument to the GetSFTP processor
Ahh - great idea James!
On Fri, Apr 15, 2016 at 1:34 PM, James Wing <jv...@gmail.com>> wrote:
> Have you looked at a combination of ListSFTP -> FetchSFTP? I believe
> ListSFTP has built-in support for tracking recent files, and it might
> satisfy your use case. If not, you can certainly filter the listed files by
> the "filename" attribute before calling FetchSFTP.
>
> On Fri, Apr 15, 2016 at 7:40 AM, Ashraf Hasson <AH...@windmobile.ca>>
> wrote:
>>
>> Hi there,
>>
>>
>> I've been trying to use the GetSFTP processor to fetch files that have a
>> 'yyyymmdd' part but the only option I have is to use Java Regular
>> Expression, not sure how to specify today or yesterday's date dynamically.
>>
>>
>> I thought an alternative is to use a locally generated file that contains
>> the datetime string, use a GetFile + UpdateAttribute to pass in a custom
>> attribute to the GetSFTP processor that way, but sounded a bit cumbersome
>> tbh.
>>
>>
>> Any thoughts/hints please?
>>
>>
>> Thanks,
>>
>> Ashraf
>
>
Re: datetime argument to the GetSFTP processor
Posted by Pierre Villard <pi...@gmail.com>.
Hi Ashraf,
I believe you are right, RouteOnAttribute is certainly what you should use.
Regarding the expression, it sounds good to me. Depending of the filename
format characteristics, may be you can directly use
${filename:contains( ${now():toNumber():minus(86400000):format('yyyymmdd')}
)}
2016-04-21 19:11 GMT+02:00 Ashraf Hasson <AH...@windmobile.ca>:
> Hi there,
>
> Okay, so I should use RouteOnAttribute I think.
>
> Here's the source filename: test.file.20160420-015931.csv
>
> I've configured the RouteOnAttribute to route the `filename` to success
> when the following is matched:
>
> property: filename == value:
> ${filename:substringAfter('test.file.'):startsWith(
> ${now():toNumber():minus(86400000):format('yyyymmdd')} )}
>
> Maybe this is not the right way but I'd highly appreciate some help.
>
> Thanks,
>
> ________________________________________
> From: Ashraf Hasson
> Sent: April 21, 2016 12:50 PM
> To: users@nifi.apache.org
> Subject: Re: datetime argument to the GetSFTP processor
>
> Hi James,
>
> Thanks for your reply.
> The source has multiple files generated per day and I'm interested in one
> of those, so filtering is required I guess.
> I don't know how to filter based on the filename, was trying to pipe
> things like this: ListSFTP -> UpdateAttribute -> FetchSFTP
> but I'm sure there's a better way to filter. Can you help by giving me a
> pointer or two please?
>
> I also tried to pipe directly from ListSFTP -> FetchSFTP and used
> ${now():toNumber():minus(86400000):format('yyyymmdd')}-*.csv but then this
> doesn't match the remote file guessing because FetchSFTP doesn't do globing.
>
> Thanks,
> ________________________________________
> From: Joe Witt <jo...@gmail.com>
> Sent: April 15, 2016 1:35 PM
> To: users@nifi.apache.org
> Subject: Re: datetime argument to the GetSFTP processor
>
> Ahh - great idea James!
>
> On Fri, Apr 15, 2016 at 1:34 PM, James Wing <jv...@gmail.com> wrote:
> > Have you looked at a combination of ListSFTP -> FetchSFTP? I believe
> > ListSFTP has built-in support for tracking recent files, and it might
> > satisfy your use case. If not, you can certainly filter the listed
> files by
> > the "filename" attribute before calling FetchSFTP.
> >
> > On Fri, Apr 15, 2016 at 7:40 AM, Ashraf Hasson <AH...@windmobile.ca>
> > wrote:
> >>
> >> Hi there,
> >>
> >>
> >> I've been trying to use the GetSFTP processor to fetch files that have a
> >> 'yyyymmdd' part but the only option I have is to use Java Regular
> >> Expression, not sure how to specify today or yesterday's date
> dynamically.
> >>
> >>
> >> I thought an alternative is to use a locally generated file that
> contains
> >> the datetime string, use a GetFile + UpdateAttribute to pass in a custom
> >> attribute to the GetSFTP processor that way, but sounded a bit
> cumbersome
> >> tbh.
> >>
> >>
> >> Any thoughts/hints please?
> >>
> >>
> >> Thanks,
> >>
> >> Ashraf
> >
> >
>
Re: datetime argument to the GetSFTP processor
Posted by Ashraf Hasson <AH...@WindMobile.ca>.
Hi there,
Okay, so I should use RouteOnAttribute I think.
Here's the source filename: test.file.20160420-015931.csv
I've configured the RouteOnAttribute to route the `filename` to success when the following is matched:
property: filename == value: ${filename:substringAfter('test.file.'):startsWith( ${now():toNumber():minus(86400000):format('yyyymmdd')} )}
Maybe this is not the right way but I'd highly appreciate some help.
Thanks,
________________________________________
From: Ashraf Hasson
Sent: April 21, 2016 12:50 PM
To: users@nifi.apache.org
Subject: Re: datetime argument to the GetSFTP processor
Hi James,
Thanks for your reply.
The source has multiple files generated per day and I'm interested in one of those, so filtering is required I guess.
I don't know how to filter based on the filename, was trying to pipe things like this: ListSFTP -> UpdateAttribute -> FetchSFTP
but I'm sure there's a better way to filter. Can you help by giving me a pointer or two please?
I also tried to pipe directly from ListSFTP -> FetchSFTP and used ${now():toNumber():minus(86400000):format('yyyymmdd')}-*.csv but then this doesn't match the remote file guessing because FetchSFTP doesn't do globing.
Thanks,
________________________________________
From: Joe Witt <jo...@gmail.com>
Sent: April 15, 2016 1:35 PM
To: users@nifi.apache.org
Subject: Re: datetime argument to the GetSFTP processor
Ahh - great idea James!
On Fri, Apr 15, 2016 at 1:34 PM, James Wing <jv...@gmail.com> wrote:
> Have you looked at a combination of ListSFTP -> FetchSFTP? I believe
> ListSFTP has built-in support for tracking recent files, and it might
> satisfy your use case. If not, you can certainly filter the listed files by
> the "filename" attribute before calling FetchSFTP.
>
> On Fri, Apr 15, 2016 at 7:40 AM, Ashraf Hasson <AH...@windmobile.ca>
> wrote:
>>
>> Hi there,
>>
>>
>> I've been trying to use the GetSFTP processor to fetch files that have a
>> 'yyyymmdd' part but the only option I have is to use Java Regular
>> Expression, not sure how to specify today or yesterday's date dynamically.
>>
>>
>> I thought an alternative is to use a locally generated file that contains
>> the datetime string, use a GetFile + UpdateAttribute to pass in a custom
>> attribute to the GetSFTP processor that way, but sounded a bit cumbersome
>> tbh.
>>
>>
>> Any thoughts/hints please?
>>
>>
>> Thanks,
>>
>> Ashraf
>
>
Re: datetime argument to the GetSFTP processor
Posted by Joe Witt <jo...@gmail.com>.
Ashraf,
The flow would be
ListSFTP -> RouteOnAttribute -> FetchSFTP
In RouteOnAttribute you'd put a filename filter in place to detect
filenames of interest to you. Route things that match to FetchSFTP
and things that do not match you can terminate or do whatever you
need.
RouteOnAttribute is pretty flexible and should definitely do what you need here.
Thanks
Joe
On Thu, Apr 21, 2016 at 12:50 PM, Ashraf Hasson <AH...@windmobile.ca> wrote:
> Hi James,
>
> Thanks for your reply.
> The source has multiple files generated per day and I'm interested in one of those, so filtering is required I guess.
> I don't know how to filter based on the filename, was trying to pipe things like this: ListSFTP -> UpdateAttribute -> FetchSFTP
> but I'm sure there's a better way to filter. Can you help by giving me a pointer or two please?
>
> I also tried to pipe directly from ListSFTP -> FetchSFTP and used ${now():toNumber():minus(86400000):format('yyyymmdd')}-*.csv but then this doesn't match the remote file guessing because FetchSFTP doesn't do globing.
>
> Thanks,
> ________________________________________
> From: Joe Witt <jo...@gmail.com>
> Sent: April 15, 2016 1:35 PM
> To: users@nifi.apache.org
> Subject: Re: datetime argument to the GetSFTP processor
>
> Ahh - great idea James!
>
> On Fri, Apr 15, 2016 at 1:34 PM, James Wing <jv...@gmail.com> wrote:
>> Have you looked at a combination of ListSFTP -> FetchSFTP? I believe
>> ListSFTP has built-in support for tracking recent files, and it might
>> satisfy your use case. If not, you can certainly filter the listed files by
>> the "filename" attribute before calling FetchSFTP.
>>
>> On Fri, Apr 15, 2016 at 7:40 AM, Ashraf Hasson <AH...@windmobile.ca>
>> wrote:
>>>
>>> Hi there,
>>>
>>>
>>> I've been trying to use the GetSFTP processor to fetch files that have a
>>> 'yyyymmdd' part but the only option I have is to use Java Regular
>>> Expression, not sure how to specify today or yesterday's date dynamically.
>>>
>>>
>>> I thought an alternative is to use a locally generated file that contains
>>> the datetime string, use a GetFile + UpdateAttribute to pass in a custom
>>> attribute to the GetSFTP processor that way, but sounded a bit cumbersome
>>> tbh.
>>>
>>>
>>> Any thoughts/hints please?
>>>
>>>
>>> Thanks,
>>>
>>> Ashraf
>>
>>
Re: datetime argument to the GetSFTP processor
Posted by Ashraf Hasson <AH...@WindMobile.ca>.
Hi James,
Thanks for your reply.
The source has multiple files generated per day and I'm interested in one of those, so filtering is required I guess.
I don't know how to filter based on the filename, was trying to pipe things like this: ListSFTP -> UpdateAttribute -> FetchSFTP
but I'm sure there's a better way to filter. Can you help by giving me a pointer or two please?
I also tried to pipe directly from ListSFTP -> FetchSFTP and used ${now():toNumber():minus(86400000):format('yyyymmdd')}-*.csv but then this doesn't match the remote file guessing because FetchSFTP doesn't do globing.
Thanks,
________________________________________
From: Joe Witt <jo...@gmail.com>
Sent: April 15, 2016 1:35 PM
To: users@nifi.apache.org
Subject: Re: datetime argument to the GetSFTP processor
Ahh - great idea James!
On Fri, Apr 15, 2016 at 1:34 PM, James Wing <jv...@gmail.com> wrote:
> Have you looked at a combination of ListSFTP -> FetchSFTP? I believe
> ListSFTP has built-in support for tracking recent files, and it might
> satisfy your use case. If not, you can certainly filter the listed files by
> the "filename" attribute before calling FetchSFTP.
>
> On Fri, Apr 15, 2016 at 7:40 AM, Ashraf Hasson <AH...@windmobile.ca>
> wrote:
>>
>> Hi there,
>>
>>
>> I've been trying to use the GetSFTP processor to fetch files that have a
>> 'yyyymmdd' part but the only option I have is to use Java Regular
>> Expression, not sure how to specify today or yesterday's date dynamically.
>>
>>
>> I thought an alternative is to use a locally generated file that contains
>> the datetime string, use a GetFile + UpdateAttribute to pass in a custom
>> attribute to the GetSFTP processor that way, but sounded a bit cumbersome
>> tbh.
>>
>>
>> Any thoughts/hints please?
>>
>>
>> Thanks,
>>
>> Ashraf
>
>
Re: datetime argument to the GetSFTP processor
Posted by Joe Witt <jo...@gmail.com>.
Ahh - great idea James!
On Fri, Apr 15, 2016 at 1:34 PM, James Wing <jv...@gmail.com> wrote:
> Have you looked at a combination of ListSFTP -> FetchSFTP? I believe
> ListSFTP has built-in support for tracking recent files, and it might
> satisfy your use case. If not, you can certainly filter the listed files by
> the "filename" attribute before calling FetchSFTP.
>
> On Fri, Apr 15, 2016 at 7:40 AM, Ashraf Hasson <AH...@windmobile.ca>
> wrote:
>>
>> Hi there,
>>
>>
>> I've been trying to use the GetSFTP processor to fetch files that have a
>> 'yyyymmdd' part but the only option I have is to use Java Regular
>> Expression, not sure how to specify today or yesterday's date dynamically.
>>
>>
>> I thought an alternative is to use a locally generated file that contains
>> the datetime string, use a GetFile + UpdateAttribute to pass in a custom
>> attribute to the GetSFTP processor that way, but sounded a bit cumbersome
>> tbh.
>>
>>
>> Any thoughts/hints please?
>>
>>
>> Thanks,
>>
>> Ashraf
>
>
Re: datetime argument to the GetSFTP processor
Posted by James Wing <jv...@gmail.com>.
Have you looked at a combination of ListSFTP -> FetchSFTP? I believe
ListSFTP has built-in support for tracking recent files, and it might
satisfy your use case. If not, you can certainly filter the listed files
by the "filename" attribute before calling FetchSFTP.
On Fri, Apr 15, 2016 at 7:40 AM, Ashraf Hasson <AH...@windmobile.ca>
wrote:
> Hi there,
>
>
> I've been trying to use the GetSFTP processor to fetch files that have a
> 'yyyymmdd' part but the only option I have is to use Java Regular
> Expression, not sure how to specify today or yesterday's date dynamically.
>
>
> I thought an alternative is to use a locally generated file that contains
> the datetime string, use a GetFile + UpdateAttribute to pass in a custom
> attribute to the GetSFTP processor that way, but sounded a bit cumbersome
> tbh.
>
>
> Any thoughts/hints please?
>
>
> Thanks,
>
> Ashraf
>
>