You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by /pig五餐肉 <76...@qq.com> on 2014/07/11 04:29:07 UTC
Storm Drpc Performance
Hi All
I am using to storm drpc topology to do data sorting and filterting.
the input stream is the data string sequence.
everything is fine when i use drpcclient class to send string to out drpc server in a single thread.the latency is about 65ms.
but when i send data in mutiple threads (about 200 threads) at the same time, the latency is high. some threads are more than 1000ms.
I have also try the offical demo BasicDRPCTestTopology.
the issue is the same.
env: 11 machines(7g ram,8 cores)
is there a way to address this issue ?
回复： Storm Drpc Performance
Posted by /pig五餐肉 <76...@qq.com>.
Hi Derek
Thanks. I set drpc.worker.threads to 256 and drpc.queue.size to 512 before testing.
------------------ 原始邮件 ------------------
发件人: "Derek Dagit";<de...@yahoo-inc.com>;
发送时间: 2014年7月11日(星期五) 晚上9:33
收件人: "user"<us...@storm.incubator.apache.org>;
主题: Re: Storm Drpc Performance
drpc.worker.threads defaults to 64.
Maybe try increasing this?
--
Derek
On 7/10/14, 21:29, /pig五餐肉 wrote:
> Hi All
>
>
> I am using to storm drpc topology to do data sorting and filterting.
> the input stream is the data string sequence.
> everything is fine when i use drpcclient class to send string to out drpc server in a single thread.the latency is about 65ms.
> but when i send data in mutiple threads (about 200 threads) at the same time, the latency is high. some threads are more than 1000ms.
>
>
> I have also try the offical demo BasicDRPCTestTopology.
> the issue is the same.
>
>
> env: 11 machines(7g ram,8 cores)
>
>
> is there a way to address this issue ?
>
Re: Storm Drpc Performance
Posted by Derek Dagit <de...@yahoo-inc.com>.
drpc.worker.threads defaults to 64.
Maybe try increasing this?
--
Derek
On 7/10/14, 21:29, /pig五餐肉 wrote:
> Hi All
>
>
> I am using to storm drpc topology to do data sorting and filterting.
> the input stream is the data string sequence.
> everything is fine when i use drpcclient class to send string to out drpc server in a single thread.the latency is about 65ms.
> but when i send data in mutiple threads (about 200 threads) at the same time, the latency is high. some threads are more than 1000ms.
>
>
> I have also try the offical demo BasicDRPCTestTopology.
> the issue is the same.
>
>
> env: 11 machines(7g ram,8 cores)
>
>
> is there a way to address this issue ?
>
回复： Storm Drpc Performance
Posted by /pig五餐肉 <76...@qq.com>.
Thanks,Kiddon.
I try it. However, it seems that there is no improvement and the latency is higher than the original class of DrpcClient.
------------------ 原始邮件 ------------------
发件人: "Kidong Lee";<my...@gmail.com>;
发送时间: 2014年7月11日(星期五) 中午1:45
收件人: "user"<us...@storm.incubator.apache.org>;
主题: Re: Storm Drpc Performance
This could help:
https://github.com/mykidong/storm-finagle-drpc-client
2014-07-11 11:29 GMT+09:00 /pig五餐肉 <76...@qq.com>:
Hi All
I am using to storm drpc topology to do data sorting and filterting.
the input stream is the data string sequence.
everything is fine when i use drpcclient class to send string to out drpc server in a single thread.the latency is about 65ms.
but when i send data in mutiple threads (about 200 threads) at the same time, the latency is high. some threads are more than 1000ms.
I have also try the offical demo BasicDRPCTestTopology.
the issue is the same.
env: 11 machines(7g ram,8 cores)
is there a way to address this issue ?
Re: Storm Drpc Performance
Posted by Kidong Lee <my...@gmail.com>.
This could help:
https://github.com/mykidong/storm-finagle-drpc-client
2014-07-11 11:29 GMT+09:00 /pig五餐肉 <76...@qq.com>:
> Hi All
>
> I am using to storm drpc topology to do data sorting and filterting.
> the input stream is the data string sequence.
> everything is fine when i use drpcclient class to send string to out drpc
> server in a single thread.the latency is about 65ms.
> but when i send data in mutiple threads (about 200 threads) at the same
> time, the latency is high. some threads are more than 1000ms.
>
> I have also try the offical demo BasicDRPCTestTopology.
> the issue is the same.
>
> env: 11 machines(7g ram,8 cores)
>
> is there a way to address this issue ?
>