[jira] [Commented] (NIFI-9577) GetFile and ListFile list no files if Path Filter is not empty

["Mark Payne (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-9577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477269#comment-17477269 ] 

Mark Payne commented on NIFI-9577:
----------------------------------

I tried this locally. But rather than writing to a directory name based on the current minute I stopped at the current hour. I had no problem with it running this way. When it was configured to use directory names that are down to the minute, it did miss a lot of files. Which makes sense, because GenerateFlowFile runs. Then 30+ seconds later the GetFile / ListFile runs. In the meantime the minute of the hour has changed, resulting in the ListFile/GetFile looking in a different directory than PutFile wrote to.

This approach is always going to give you problems, as there's no guarantee of when the timestamp will change between invocations of ListFile. Rather, you will probably want to perform the listing at the level of {{tmp/prod/cltest/}} and then just list files for all dates/times

> GetFile and ListFile list no files if Path Filter is not empty
> --------------------------------------------------------------
>
>                 Key: NIFI-9577
>                 URL: https://issues.apache.org/jira/browse/NIFI-9577
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 1.15.2
>            Reporter: Christoph Langheld
>            Priority: Major
>         Attachments: GetFile-ListFile-Problem.xml, workflow.png
>
>
> Hello,
> under special circumstances GetFile and ListFile do not process files.
> To reproduce the problem I attached a template: [^GetFile-ListFile-Problem.xml]
> The template generates every minute (0 0/1 * ? * MON-SAT) a flow file of 10B and stores it locally in a *new* ( ! ) folder: tmp/prod/cltest/${now():format('yyyyMMdd-HHmm')}/gaz/artikel.
> GetFile and ListFile run about 30 seconds after GenerateFlowFile processor(35 0/1 * ? * MON-SAT). For both processors the input directory is: tmp/prod/cltest/${now():format('yyyyMMdd-HHmm')}/.
> Path Filter is set to: ((anzg)|(gaz))\/artikel
> The first time when the processors are running the result set of each processor contains 1 file.
> After the first run the result will always be empty. There occurs no error. It's only empty.
> !workflow.png|width=813,height=449!
> We use NiFi versions 1.11.4 and 1.15.2. Both have the same problem.
> Could you please have a look on it.
> Thank you and regards
> Christoph



--
This message was sent by Atlassian Jira
(v8.20.1#820001)