You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flume.apache.org by "Israel Ekpo (JIRA)" <ji...@apache.org> on 2013/04/10 16:16:16 UTC

[jira] [Updated] (FLUME-1983) files in the “spooling” directory only can transfer to sink one by one.

     [ https://issues.apache.org/jira/browse/FLUME-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Israel Ekpo updated FLUME-1983:
-------------------------------

      Priority: Minor  (was: Major)
    Issue Type: Question  (was: Bug)

This is more of a question to the development group.

Nevertheless, it's a very interesting question.

Be that as it may, I would recommend that you resolve the issue and send an email with this question to the development mailing list for Flume instead.

The email address is dev@flume.apache.org.

Without looking into the implementation in details, I believe there could be a good a reason why it was originally designed this way.

There are cases where adding additional user threads does not really improve performance significantly due to hardware, operating systems and threading library implementation strategies. Worker threads could end up competing and blocking one another when system calls to the file system are made.

The benefits and risks must be meticulously analyzed so that we don't end up causing issues for a majority of the use cases deploying this specific source. 
                
> files in the “spooling” directory only can transfer to sink one by one.
> -----------------------------------------------------------------------
>
>                 Key: FLUME-1983
>                 URL: https://issues.apache.org/jira/browse/FLUME-1983
>             Project: Flume
>          Issue Type: Question
>            Reporter: zhouqingfa
>            Priority: Minor
>
>     why only can the files in the “spooling” directory transfer to sink one by one? In case there files are very big and growthing real-time,then other files can not transfer to sink.why thread pool only has one thread instead of multiple?Could you consider "Executors.newCachedThreadPool()" instead of "Executors.newSingleThreadScheduledExecutor()"? 
> Look forward to your reply

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira