You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Denis DEBARBIEUX <dd...@norsys.fr> on 2015/05/28 09:46:52 UTC
Throughput : local mode is faster than cluster mode
Dear all,
I measured the performance of my topology written in pure storm (no
trident). I compared it in local mode and then in cluster. The two
experiements were done on the same single server (deployed with docker
in cluster configuration). The result was that *the local mode is 20
times faster than in cluster configuration.*
I am surprised about it. What could I do to improve the performance on
the cluster?
Thanks for your help.
D. Debarbieux
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
http://www.avast.com
Re: Throughput : local mode is faster than cluster mode
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Dear all,
I investiguated a little more. I guess that the partitionBy primitive
impacts a lot my performance since my data are not well balanced (70% of
my tuples are assigned to a single task). It looks like the cluster mode
takes many time to root the data.
Denis
Le 28/05/2015 21:28, Jeffery Maass a écrit :
> Denis:
>
> I would avail myself of Nathan Leung's offer to help. His statement
> was that a factor of 20 is much too great a difference. He asked for
> more details.
>
> Thank you for your time!
>
> +++++++++++++++++++++
> Jeff Maass <ma...@gmail.com>
> linkedin.com/in/jeffmaass <http://linkedin.com/in/jeffmaass>
> stackoverflow.com/users/373418/maassql
> <http://stackoverflow.com/users/373418/maassql>
> +++++++++++++++++++++
>
>
> On Thu, May 28, 2015 at 10:51 AM, Denis DEBARBIEUX
> <ddebarbieux@norsys.fr <ma...@norsys.fr>> wrote:
>
> Thanks for your help.
>
> Denis
>
>
> Le 28/05/2015 15:15, Jeffery Maass a écrit :
>> Did you know that a horse can race a human and the human will win
>> -- in the first several yards? Spread out the race longer and
>> the horse will always win.
>>
>> The tests being run aren't covering enough situations. Storm was
>> designed to take business code and scale it out in order to
>> handle massive amounts of data. Try increasing your records per
>> minute by a thousand fold and see how local mode functions versus
>> cluster mode and business code spread out over multiple nodes.
>>
>> Thank you for your time!
>>
>> +++++++++++++++++++++
>> Jeff Maass <ma...@gmail.com>
>> linkedin.com/in/jeffmaass <http://linkedin.com/in/jeffmaass>
>> stackoverflow.com/users/373418/maassql
>> <http://stackoverflow.com/users/373418/maassql>
>> +++++++++++++++++++++
>>
>>
>> On Thu, May 28, 2015 at 2:46 AM, Denis DEBARBIEUX
>> <ddebarbieux@norsys.fr <ma...@norsys.fr>> wrote:
>>
>> Dear all,
>>
>> I measured the performance of my topology written in pure
>> storm (no trident). I compared it in local mode and then in
>> cluster. The two experiements were done on the same single
>> server (deployed with docker in cluster configuration). The
>> result was that *the local mode is 20 times faster than in
>> cluster configuration.*
>>
>> I am surprised about it. What could I do to improve the
>> performance on the cluster?
>>
>> Thanks for your help.
>>
>> D. Debarbieux
>>
>>
>> ------------------------------------------------------------------------
>> <http://www.avast.com/>
>>
>> L'absence de virus dans ce courrier électronique a été
>> vérifiée par le logiciel antivirus Avast.
>> www.avast.com <http://www.avast.com/>
>>
>>
>>
>
>
>
> ------------------------------------------------------------------------
> <http://www.avast.com/>
>
> L'absence de virus dans ce courrier électronique a été vérifiée
> par le logiciel antivirus Avast.
> www.avast.com <http://www.avast.com/>
>
>
>
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
http://www.avast.com
Re: Throughput : local mode is faster than cluster mode
Posted by Jeffery Maass <ma...@gmail.com>.
Denis:
I would avail myself of Nathan Leung's offer to help. His statement was
that a factor of 20 is much too great a difference. He asked for more
details.
Thank you for your time!
+++++++++++++++++++++
Jeff Maass <ma...@gmail.com>
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+++++++++++++++++++++
On Thu, May 28, 2015 at 10:51 AM, Denis DEBARBIEUX <dd...@norsys.fr>
wrote:
> Thanks for your help.
>
> Denis
>
>
> Le 28/05/2015 15:15, Jeffery Maass a écrit :
>
> Did you know that a horse can race a human and the human will win -- in
> the first several yards? Spread out the race longer and the horse will
> always win.
>
> The tests being run aren't covering enough situations. Storm was designed
> to take business code and scale it out in order to handle massive amounts
> of data. Try increasing your records per minute by a thousand fold and see
> how local mode functions versus cluster mode and business code spread out
> over multiple nodes.
>
> Thank you for your time!
>
> +++++++++++++++++++++
> Jeff Maass <ma...@gmail.com>
> linkedin.com/in/jeffmaass
> stackoverflow.com/users/373418/maassql
> +++++++++++++++++++++
>
>
> On Thu, May 28, 2015 at 2:46 AM, Denis DEBARBIEUX <dd...@norsys.fr>
> wrote:
>
>> Dear all,
>>
>> I measured the performance of my topology written in pure storm (no
>> trident). I compared it in local mode and then in cluster. The two
>> experiements were done on the same single server (deployed with docker in
>> cluster configuration). The result was that *the local mode is 20 times
>> faster than in cluster configuration.*
>>
>> I am surprised about it. What could I do to improve the performance on
>> the cluster?
>>
>> Thanks for your help.
>>
>> D. Debarbieux
>>
>>
>> ------------------------------
>> <http://www.avast.com/>
>>
>> L'absence de virus dans ce courrier électronique a été vérifiée par le
>> logiciel antivirus Avast.
>> www.avast.com
>>
>>
>
>
>
> ------------------------------
> <http://www.avast.com/>
>
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> www.avast.com
>
>
Re: Throughput : local mode is faster than cluster mode
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Thanks for your help.
Denis
Le 28/05/2015 15:15, Jeffery Maass a écrit :
> Did you know that a horse can race a human and the human will win --
> in the first several yards? Spread out the race longer and the horse
> will always win.
>
> The tests being run aren't covering enough situations. Storm was
> designed to take business code and scale it out in order to handle
> massive amounts of data. Try increasing your records per minute by a
> thousand fold and see how local mode functions versus cluster mode and
> business code spread out over multiple nodes.
>
> Thank you for your time!
>
> +++++++++++++++++++++
> Jeff Maass <ma...@gmail.com>
> linkedin.com/in/jeffmaass <http://linkedin.com/in/jeffmaass>
> stackoverflow.com/users/373418/maassql
> <http://stackoverflow.com/users/373418/maassql>
> +++++++++++++++++++++
>
>
> On Thu, May 28, 2015 at 2:46 AM, Denis DEBARBIEUX
> <ddebarbieux@norsys.fr <ma...@norsys.fr>> wrote:
>
> Dear all,
>
> I measured the performance of my topology written in pure storm
> (no trident). I compared it in local mode and then in cluster. The
> two experiements were done on the same single server (deployed
> with docker in cluster configuration). The result was that *the
> local mode is 20 times faster than in cluster configuration.*
>
> I am surprised about it. What could I do to improve the
> performance on the cluster?
>
> Thanks for your help.
>
> D. Debarbieux
>
>
> ------------------------------------------------------------------------
> <http://www.avast.com/>
>
> L'absence de virus dans ce courrier électronique a été vérifiée
> par le logiciel antivirus Avast.
> www.avast.com <http://www.avast.com/>
>
>
>
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
http://www.avast.com
Re: Throughput : local mode is faster than cluster mode
Posted by Jeffery Maass <ma...@gmail.com>.
Did you know that a horse can race a human and the human will win -- in the
first several yards? Spread out the race longer and the horse will always
win.
The tests being run aren't covering enough situations. Storm was designed
to take business code and scale it out in order to handle massive amounts
of data. Try increasing your records per minute by a thousand fold and see
how local mode functions versus cluster mode and business code spread out
over multiple nodes.
Thank you for your time!
+++++++++++++++++++++
Jeff Maass <ma...@gmail.com>
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+++++++++++++++++++++
On Thu, May 28, 2015 at 2:46 AM, Denis DEBARBIEUX <dd...@norsys.fr>
wrote:
> Dear all,
>
> I measured the performance of my topology written in pure storm (no
> trident). I compared it in local mode and then in cluster. The two
> experiements were done on the same single server (deployed with docker in
> cluster configuration). The result was that *the local mode is 20 times
> faster than in cluster configuration.*
>
> I am surprised about it. What could I do to improve the performance on the
> cluster?
>
> Thanks for your help.
>
> D. Debarbieux
>
>
> ------------------------------
> <http://www.avast.com/>
>
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> www.avast.com
>
>
Re: Throughput : local mode is faster than cluster mode
Posted by Nathan Leung <nc...@gmail.com>.
It's hard to help without any details about what you did. I have not seen
local mode outperform cluster mode by this factor, especially if it's all
on one server, so more information would help determine what is going on.