You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Andrew Lee <li...@gmail.com> on 2015/03/26 08:45:23 UTC
look for help about Streaming benchmark and how to have a stress
testing in Storm
Hi, everyone.
I'm a researcher in the distributed system field. Now i'm going to find the
bottlenecks of Storm and Spark Streaming. I'd like to know the popular
benchmarks to test the thoughout and latency of Storm or Spark Streaming.
If someone has experiences or professional knowledge, i'm happy for your
advice.
Another question is how to have a stress testing in Storm. I create a
topology that has a spout (emit sentence) and a bolt that discards the
messages recevied. And the number of spout tasks is 28, of bolts is 28 too.
Of course, the physical resource of servers can satisfy the scenario. In
the running, i observerd the number of emitting sentences is just 500 per
minute per task. I'm so amazed because it can be up to 243,783 messages in
using netty for transferring messages.
I didn't do any advanced configuration. I have no idea how to let the storm
fully using the server resource and then the bottleneck can be found.
Thank you!
Andrew Lee
Re: look for help about Streaming benchmark and how to have a stress
testing in Storm
Posted by Andrew Lee <li...@gmail.com>.
That's very helpful! Thank you very much!
Andrew Lee
2015-03-26 22:38 GMT+08:00 Grant Overby (groverby) <gr...@cisco.com>:
> I found the following link useful in a similar situation. It doesn’t
> sound like you’re using trident, but I expect at least some of the content
> will still be relevant.
>
> https://gist.github.com/mrflip/5958028
>
> *Grant Overby*
> Software Engineer
> Cisco.com <http://www.cisco.com/>
> groverby@cisco.com
> Mobile: *865 724 4910*
>
>
>
> Think before you print.
>
> This email may contain confidential and privileged material for the sole
> use of the intended recipient. Any review, use, distribution or disclosure
> by others is strictly prohibited. If you are not the intended recipient (or
> authorized to receive for the recipient), please contact the sender by
> reply email and delete all copies of this message.
>
> Please click here
> <http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for
> Company Registration Information.
>
>
>
>
> From: Andrew Lee <li...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Thursday, March 26, 2015 at 3:45 AM
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: look for help about Streaming benchmark and how to have a stress
> testing in Storm
>
> Hi, everyone.
>
> I'm a researcher in the distributed system field. Now i'm going to find
> the bottlenecks of Storm and Spark Streaming. I'd like to know the popular
> benchmarks to test the thoughout and latency of Storm or Spark Streaming.
> If someone has experiences or professional knowledge, i'm happy for your
> advice.
>
> Another question is how to have a stress testing in Storm. I create a
> topology that has a spout (emit sentence) and a bolt that discards the
> messages recevied. And the number of spout tasks is 28, of bolts is 28 too.
> Of course, the physical resource of servers can satisfy the scenario. In
> the running, i observerd the number of emitting sentences is just 500 per
> minute per task. I'm so amazed because it can be up to 243,783 messages in
> using netty for transferring messages.
> I didn't do any advanced configuration. I have no idea how to let the
> storm fully using the server resource and then the bottleneck can be found.
>
> Thank you!
>
> Andrew Lee
>
--
*Andrew Lee*
from ICT of Chinese Academy of Sciences
Life For Joy
微笑着 追逐每一份阳光
Re: look for help about Streaming benchmark and how to have a
stress testing in Storm
Posted by "Grant Overby (groverby)" <gr...@cisco.com>.
I found the following link useful in a similar situation. It doesn’t sound like you’re using trident, but I expect at least some of the content will still be relevant.
https://gist.github.com/mrflip/5958028
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<ma...@cisco.com>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.
From: Andrew Lee <li...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Thursday, March 26, 2015 at 3:45 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: look for help about Streaming benchmark and how to have a stress testing in Storm
Hi, everyone.
I'm a researcher in the distributed system field. Now i'm going to find the bottlenecks of Storm and Spark Streaming. I'd like to know the popular benchmarks to test the thoughout and latency of Storm or Spark Streaming. If someone has experiences or professional knowledge, i'm happy for your advice.
Another question is how to have a stress testing in Storm. I create a topology that has a spout (emit sentence) and a bolt that discards the messages recevied. And the number of spout tasks is 28, of bolts is 28 too. Of course, the physical resource of servers can satisfy the scenario. In the running, i observerd the number of emitting sentences is just 500 per minute per task. I'm so amazed because it can be up to 243,783 messages in using netty for transferring messages.
I didn't do any advanced configuration. I have no idea how to let the storm fully using the server resource and then the bottleneck can be found.
Thank you!
Andrew Lee