You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Stephen Greszczyszyn <sg...@gmail.com> on 2019/05/28 15:44:16 UTC
Debugging NiFi Flows in Docker - problem with ListFile?
Hi there,
I'm running NiFi in docker. I'm trying to enable flow debugs and also to
determine if NiFi can actually see in my data directory path. I'm trying
to use a simple recursive list file processor to find the files based on a
regular expression "^c.r".
I can docker exec into the container and can see that the nifi container
can "see" into my filepath OK:
nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r'
cdr_AP2-HKG-CCP-UCM_12_201905281539_35251
cdr_AP2-HKG-CCP-UCM_06_201905281539_297892
cmr_AP2-HKG-CCP-UCM_06_201905281539_297892
cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435
cdr_AP2-HKG-CCP-UCM_11_201905281539_98483
However, I can't get the ListFile processor to work and find those files.
Any ideas on debugging strategies?
Thanks.
Re: Debugging NiFi Flows in Docker - problem with ListFile?
Posted by Stephen Greszczyszyn <sg...@gmail.com>.
OK, it seems like NiFi was actually trying to find files in the directory,
although there were way too many and it was running out of memory... I
guess I need to run something to archive the old logs and start with only
the new ones.
Thanks for your help Mark!
On Tue, 28 May 2019 at 21:33, Stephen Greszczyszyn <sg...@gmail.com>
wrote:
> OK, thanks, this us useful information, I did also test the filenames
> with a regex parser, but didn't know NiFi was different. I really don't
> need an expression anyway as I want to get all the files. I'm wondering if
> the problem is that I have too many files in the directory, so I'll just
> set up a test directory and no regex and see what happens.
>
>
> On Tue, 28 May 2019 at 20:21, Mark Payne <ma...@hotmail.com> wrote:
>
>> Stephen,
>>
>> Egrep evaluates regex'es very differently than NiFi does. With egrep, any
>> line that contains text matching the regex
>> will be output. With NiFi, however, the entire line of text must match
>> the regex exactly in order to be selected. So '^c.r' will
>> not match but '^c.r.*' will.
>>
>> Thanks
>> -Mark
>>
>> On May 28, 2019, at 3:15 PM, Stephen Greszczyszyn <sg...@gmail.com>
>> wrote:
>>
>> I don't think that is the problem as I have checked the regex and can
>> find the files (this is from inside the container so I also can read the
>> directory):
>>
>> nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r'
>> cdr_AP2-HKG-CCP-UCM_12_201905281539_35251
>> cdr_AP2-HKG-CCP-UCM_06_201905281539_297892
>> cmr_AP2-HKG-CCP-UCM_06_201905281539_297892
>> cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435
>> cdr_AP2-HKG-CCP-UCM_11_201905281539_98483
>>
>> On Tue, 28 May 2019 at 17:02, Mark Payne <ma...@hotmail.com> wrote:
>>
>>> Stephen,
>>>
>>> The regex that you have there in interpreted by NiFi as "the filename
>>> must consist of 3 characters, beginning with a c and ending with an r".
>>> What you want
>>> is "the filename must begin with a c, followed by any character,
>>> followed by an r, followed by anything else." So you'd want "c.r.*" I do
>>> believe.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> > On May 28, 2019, at 11:44 AM, Stephen Greszczyszyn <sg...@gmail.com>
>>> wrote:
>>> >
>>> > Hi there,
>>> >
>>> > I'm running NiFi in docker. I'm trying to enable flow debugs and also
>>> to determine if NiFi can actually see in my data directory path. I'm
>>> trying to use a simple recursive list file processor to find the files
>>> based on a regular expression "^c.r".
>>> >
>>> > I can docker exec into the container and can see that the nifi
>>> container can "see" into my filepath OK:
>>> >
>>> > nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r'
>>> > cdr_AP2-HKG-CCP-UCM_12_201905281539_35251
>>> > cdr_AP2-HKG-CCP-UCM_06_201905281539_297892
>>> > cmr_AP2-HKG-CCP-UCM_06_201905281539_297892
>>> > cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435
>>> > cdr_AP2-HKG-CCP-UCM_11_201905281539_98483
>>> >
>>> > However, I can't get the ListFile processor to work and find those
>>> files. Any ideas on debugging strategies?
>>> >
>>> > Thanks.
>>>
>>>
>>
Re: Debugging NiFi Flows in Docker - problem with ListFile?
Posted by Stephen Greszczyszyn <sg...@gmail.com>.
OK, thanks, this us useful information, I did also test the filenames with
a regex parser, but didn't know NiFi was different. I really don't need an
expression anyway as I want to get all the files. I'm wondering if the
problem is that I have too many files in the directory, so I'll just set up
a test directory and no regex and see what happens.
On Tue, 28 May 2019 at 20:21, Mark Payne <ma...@hotmail.com> wrote:
> Stephen,
>
> Egrep evaluates regex'es very differently than NiFi does. With egrep, any
> line that contains text matching the regex
> will be output. With NiFi, however, the entire line of text must match the
> regex exactly in order to be selected. So '^c.r' will
> not match but '^c.r.*' will.
>
> Thanks
> -Mark
>
> On May 28, 2019, at 3:15 PM, Stephen Greszczyszyn <sg...@gmail.com>
> wrote:
>
> I don't think that is the problem as I have checked the regex and can find
> the files (this is from inside the container so I also can read the
> directory):
>
> nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r'
> cdr_AP2-HKG-CCP-UCM_12_201905281539_35251
> cdr_AP2-HKG-CCP-UCM_06_201905281539_297892
> cmr_AP2-HKG-CCP-UCM_06_201905281539_297892
> cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435
> cdr_AP2-HKG-CCP-UCM_11_201905281539_98483
>
> On Tue, 28 May 2019 at 17:02, Mark Payne <ma...@hotmail.com> wrote:
>
>> Stephen,
>>
>> The regex that you have there in interpreted by NiFi as "the filename
>> must consist of 3 characters, beginning with a c and ending with an r".
>> What you want
>> is "the filename must begin with a c, followed by any character, followed
>> by an r, followed by anything else." So you'd want "c.r.*" I do believe.
>>
>> Thanks
>> -Mark
>>
>>
>> > On May 28, 2019, at 11:44 AM, Stephen Greszczyszyn <sg...@gmail.com>
>> wrote:
>> >
>> > Hi there,
>> >
>> > I'm running NiFi in docker. I'm trying to enable flow debugs and also
>> to determine if NiFi can actually see in my data directory path. I'm
>> trying to use a simple recursive list file processor to find the files
>> based on a regular expression "^c.r".
>> >
>> > I can docker exec into the container and can see that the nifi
>> container can "see" into my filepath OK:
>> >
>> > nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r'
>> > cdr_AP2-HKG-CCP-UCM_12_201905281539_35251
>> > cdr_AP2-HKG-CCP-UCM_06_201905281539_297892
>> > cmr_AP2-HKG-CCP-UCM_06_201905281539_297892
>> > cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435
>> > cdr_AP2-HKG-CCP-UCM_11_201905281539_98483
>> >
>> > However, I can't get the ListFile processor to work and find those
>> files. Any ideas on debugging strategies?
>> >
>> > Thanks.
>>
>>
>
Re: Debugging NiFi Flows in Docker - problem with ListFile?
Posted by Mark Payne <ma...@hotmail.com>.
Stephen,
Egrep evaluates regex'es very differently than NiFi does. With egrep, any line that contains text matching the regex
will be output. With NiFi, however, the entire line of text must match the regex exactly in order to be selected. So '^c.r' will
not match but '^c.r.*' will.
Thanks
-Mark
On May 28, 2019, at 3:15 PM, Stephen Greszczyszyn <sg...@gmail.com>> wrote:
I don't think that is the problem as I have checked the regex and can find the files (this is from inside the container so I also can read the directory):
nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r'
cdr_AP2-HKG-CCP-UCM_12_201905281539_35251
cdr_AP2-HKG-CCP-UCM_06_201905281539_297892
cmr_AP2-HKG-CCP-UCM_06_201905281539_297892
cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435
cdr_AP2-HKG-CCP-UCM_11_201905281539_98483
On Tue, 28 May 2019 at 17:02, Mark Payne <ma...@hotmail.com>> wrote:
Stephen,
The regex that you have there in interpreted by NiFi as "the filename must consist of 3 characters, beginning with a c and ending with an r". What you want
is "the filename must begin with a c, followed by any character, followed by an r, followed by anything else." So you'd want "c.r.*" I do believe.
Thanks
-Mark
> On May 28, 2019, at 11:44 AM, Stephen Greszczyszyn <sg...@gmail.com>> wrote:
>
> Hi there,
>
> I'm running NiFi in docker. I'm trying to enable flow debugs and also to determine if NiFi can actually see in my data directory path. I'm trying to use a simple recursive list file processor to find the files based on a regular expression "^c.r".
>
> I can docker exec into the container and can see that the nifi container can "see" into my filepath OK:
>
> nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r'
> cdr_AP2-HKG-CCP-UCM_12_201905281539_35251
> cdr_AP2-HKG-CCP-UCM_06_201905281539_297892
> cmr_AP2-HKG-CCP-UCM_06_201905281539_297892
> cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435
> cdr_AP2-HKG-CCP-UCM_11_201905281539_98483
>
> However, I can't get the ListFile processor to work and find those files. Any ideas on debugging strategies?
>
> Thanks.
Re: Debugging NiFi Flows in Docker - problem with ListFile?
Posted by Stephen Greszczyszyn <sg...@gmail.com>.
I don't think that is the problem as I have checked the regex and can find
the files (this is from inside the container so I also can read the
directory):
nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r'
cdr_AP2-HKG-CCP-UCM_12_201905281539_35251
cdr_AP2-HKG-CCP-UCM_06_201905281539_297892
cmr_AP2-HKG-CCP-UCM_06_201905281539_297892
cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435
cdr_AP2-HKG-CCP-UCM_11_201905281539_98483
On Tue, 28 May 2019 at 17:02, Mark Payne <ma...@hotmail.com> wrote:
> Stephen,
>
> The regex that you have there in interpreted by NiFi as "the filename must
> consist of 3 characters, beginning with a c and ending with an r". What you
> want
> is "the filename must begin with a c, followed by any character, followed
> by an r, followed by anything else." So you'd want "c.r.*" I do believe.
>
> Thanks
> -Mark
>
>
> > On May 28, 2019, at 11:44 AM, Stephen Greszczyszyn <sg...@gmail.com>
> wrote:
> >
> > Hi there,
> >
> > I'm running NiFi in docker. I'm trying to enable flow debugs and also
> to determine if NiFi can actually see in my data directory path. I'm
> trying to use a simple recursive list file processor to find the files
> based on a regular expression "^c.r".
> >
> > I can docker exec into the container and can see that the nifi container
> can "see" into my filepath OK:
> >
> > nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r'
> > cdr_AP2-HKG-CCP-UCM_12_201905281539_35251
> > cdr_AP2-HKG-CCP-UCM_06_201905281539_297892
> > cmr_AP2-HKG-CCP-UCM_06_201905281539_297892
> > cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435
> > cdr_AP2-HKG-CCP-UCM_11_201905281539_98483
> >
> > However, I can't get the ListFile processor to work and find those
> files. Any ideas on debugging strategies?
> >
> > Thanks.
>
>
Re: Debugging NiFi Flows in Docker - problem with ListFile?
Posted by Mark Payne <ma...@hotmail.com>.
Stephen,
The regex that you have there in interpreted by NiFi as "the filename must consist of 3 characters, beginning with a c and ending with an r". What you want
is "the filename must begin with a c, followed by any character, followed by an r, followed by anything else." So you'd want "c.r.*" I do believe.
Thanks
-Mark
> On May 28, 2019, at 11:44 AM, Stephen Greszczyszyn <sg...@gmail.com> wrote:
>
> Hi there,
>
> I'm running NiFi in docker. I'm trying to enable flow debugs and also to determine if NiFi can actually see in my data directory path. I'm trying to use a simple recursive list file processor to find the files based on a regular expression "^c.r".
>
> I can docker exec into the container and can see that the nifi container can "see" into my filepath OK:
>
> nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r'
> cdr_AP2-HKG-CCP-UCM_12_201905281539_35251
> cdr_AP2-HKG-CCP-UCM_06_201905281539_297892
> cmr_AP2-HKG-CCP-UCM_06_201905281539_297892
> cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435
> cdr_AP2-HKG-CCP-UCM_11_201905281539_98483
>
> However, I can't get the ListFile processor to work and find those files. Any ideas on debugging strategies?
>
> Thanks.