You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flume.apache.org by Marina <pp...@yahoo.com> on 2015/04/15 22:15:40 UTC

ElasticSearch workers thread number?

One more question on Flume ElasticSearchSink. I see that for HDFS sink, you can control the number of worker threads using this parameter: hdfs.threadsPoolSize. However, I do not see any such parameter for ElasticSearchSink.... Does it mean that ElasticSsearchSink will span only one thread to push events form Flume to ES? If I have rates of a few hundred events per second (say, up to 3K) - this will become a performance bottleneck.
thanks!Marina

Re: ElasticSearch workers thread number?

Posted by Hari Shreedharan <hs...@cloudera.com>.

You can add multiple ES sinks to read from the same channel to parallelize writes. 




Thanks, Hari

On Wed, Apr 15, 2015 at 1:17 PM, Marina <pp...@yahoo.com> wrote:

> One more question on Flume ElasticSearchSink. I see that for HDFS sink, you can control the number of worker threads using this parameter: hdfs.threadsPoolSize. However, I do not see any such parameter for ElasticSearchSink.... Does it mean that ElasticSsearchSink will span only one thread to push events form Flume to ES? If I have rates of a few hundred events per second (say, up to 3K) - this will become a performance bottleneck.
> thanks!Marina
>