You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flume.apache.org by "zhenzhao wang (JIRA)" <ji...@apache.org> on 2018/08/15 04:05:00 UTC
[jira] [Comment Edited] (FLUME-3268) Introducing micro batch
processing to HDFSEventSink
[ https://issues.apache.org/jira/browse/FLUME-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580590#comment-16580590 ]
zhenzhao wang edited comment on FLUME-3268 at 8/15/18 4:04 AM:
---------------------------------------------------------------
[~fszabo] hdfs.batchSize is used for flush frequency. HdfsEventSink will call append rpc for each event now . What we are going to do is batching events for HDFS append. There're some problem with current pull request while I try to refactor the code, will let you known it's good for reviewing.
was (Author: wzzdreamer):
[~fszabo] hdfs.batchSize is used for flush frequency. It will call append rpc for each event . What we are going to do is batching events for HDFS append. There're some problem with current pull request while I try to refactor the code, will let you known it's good for reviewing.
> Introducing micro batch processing to HDFSEventSink
> ---------------------------------------------------
>
> Key: FLUME-3268
> URL: https://issues.apache.org/jira/browse/FLUME-3268
> Project: Flume
> Issue Type: New Feature
> Reporter: zhenzhao wang
> Priority: Major
> Attachments: FLUME-3268-0.patch
>
>
> In our test with HDFSEvent sink, we found that we could increase the draining speed of HDFSSink up to 4x by introducing micro batch processing. With the micro batch processing feature, we will batch the events written to HDFS instead of one by one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org