[jira] [Commented] (MINIFICPP-788) GetFile multithreading: directory listing should be single-threaded

["Mr TheSegfault (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MINIFICPP-788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800847#comment-16800847 ] 

Mr TheSegfault commented on MINIFICPP-788:
------------------------------------------

 

Directory can use flow file attributes to evaluate the directory, so there may be one to many directories to evaluate. If the listing is empty the listing will be populated then the threads will work on that list in batches – Since this is a bug, what bug do you purport it causes? 

> GetFile multithreading: directory listing should be single-threaded
> -------------------------------------------------------------------
>
>                 Key: MINIFICPP-788
>                 URL: https://issues.apache.org/jira/browse/MINIFICPP-788
>             Project: NiFi MiNiFi C++
>          Issue Type: Bug
>            Reporter: Arpad Boda
>            Assignee: Arpad Boda
>            Priority: Minor
>
> While working on MINIFICPP-786 I found that multithreading might be ineffective in some edgecase.
> While multiple consumers are fine (they create flow files from the listed files), producer should be single-threaded. Listing the same folder in parallel threads seems to make few sense.
> This can only occur in case listing takes longer than scheduling/configured time and it adds no files to the queue. This shouldn't occur frequently, but I can imagine edge-cases, like using getfile on a folder with a lot of files, while age/size/regex properties filter most of those.
> In my opnion simply skip listing in case listing is already being executed would be a slight improvement in such cases.
> Set prio to minor as it's really an edgecase.
> [~phrocker] what's your opinion?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)