[jira] [Updated] (FLUME-1767) HDFS sink performance should parallelize HDFS operations

["Israel Ekpo (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLUME-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Israel Ekpo updated FLUME-1767:
-------------------------------

    Issue Type: Improvement  (was: Bug)
    
> HDFS sink performance should parallelize HDFS operations
> --------------------------------------------------------
>
>                 Key: FLUME-1767
>                 URL: https://issues.apache.org/jira/browse/FLUME-1767
>             Project: Flume
>          Issue Type: Improvement
>            Reporter: Hari Shreedharan
>            Assignee: Hari Shreedharan
>
> It looks like the HDFS sink's process method calls the append method, which in turn calls the callWithTimeout method which then waits till the HDFS operation is completed before starting the next one. We could parallelize this to improve performance a lot. Since all the methods in BucketWriter are synchronized only one bucket would still be updated at any point in time, so writing multiple events at the same time would essentially affect only different buckets.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira