You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Klausen Schaefersinho <kl...@gmail.com> on 2014/05/12 09:49:26 UTC
Interesting Comparison
Hi,
I found some interesting comparison of IBM Stream and Storm:
https://www.ibmdw.net/streamsdev/2014/04/22/streams-apache-storm/
It also includes an interesting comparison between ZeroMQ and the Netty
Performance.
Cheers,
Klaus
Re: Interesting Comparison
Posted by Corey Nolet <cj...@gmail.com>.
Interesting that the paper was written by IBM people defending an IBM
product. Not saying that it's biased or anything...
Nathan, I agree that the windowing is better served as a layer on top.
Personally, I appreciate that Storm deals with clustering, distributed
state, fault-tolerance, and threading so that all I have to think about is
processing the tuples received in a bolt (and I don't even have to worry
about my algorithms being thread safe). This is not the case in InfoSphere
Streams.
I also agree that the windowing is better served as an abstraction layer on
top of a generic streams processing platform. I also appreciate this about
Storm- I'm not limited to a single language/API for processing my streams.
Myself and former coworkers of mine were integrating java with InfoSphere
streams a couple years ago and actually found it to be faster in many cases
than the C++ counterparts. I will say InfoSphere's windowing abstractions
are very well thought out. In fact, I've been working on trying to provide
a similar solution here: https://github.com/calrissian/flowmix largely
based on their design. It's still largely experimental but it is holding up
well on the clusters on which it is deployed. As Nathan put it, It's been
holding up well enough for my use cases.
On Mon, May 12, 2014 at 11:28 AM, Nathan Leung <nc...@gmail.com> wrote:
> a couple thoughts
>
> 1) IBM streams is certainly more mature, as it's been in development for a
> longer amount of time and storm is not even at release 1.0 yet. Though I
> am not familiar with SPL, It would also make sense that it's faster to
> implement as it is a higher level abstraction.
>
> 2) Operator fusion will allow more efficiency in passing data between
> steps in your flow, as localOrShuffleGrouping will still need to go over
> disruptor whereas operator fusion from what I understand basically passes
> the pointer directly. As fast as disruptor is (I've seen benchmarks of
> millions of messages passed / s), it won't be directly passing data to the
> next step (cost: a few instructions). The downside of this is your flow
> always needs to be created and compiled before you can execute it.
> Something like a rebalance will require a recompile of your stream.
> Building a topology dynamically (which is possible in storm, but not a
> feature that is really exposed out of the box) is possible in storm, but
> not in IBM streams.
>
> 3) they took 1 month to optimize storm but I suspect some of this work was
> unnecessary. Python? For a benchmark? Also, uniform message distribution
> by size feels like a premature optimization. I can understand that they
> would want to explore all avenues to account for a performance difference,
> but in many (most?) practical cases this would not be necessary. I can
> sympathize on other points. Tuning the message buffers of storm requires
> pretty specific understanding of the system. Also if you run out of heap
> and/or have to tune GC, then... yeah. Not fun. This would be true for any
> java app though.
>
> 4) I'm not sure they really took language differences seriously enough.
> I've written certain algorithms in Java that (based on similar algorithms
> that I implemented separately in C++) I would suspect are close to an order
> of magnitude slower just because I ran them in Java. While I haven't dug
> into this deeply (for example by using an identical algorithm for both Java
> and C++), consider a HashMap indexed by a primitive type. In Java, these
> are separate objects stored in an array of references. In C++ these are
> stored sequentially in an array. C++ allows direct key access in the array
> (as opposed to going through the reference), and is also potentially much
> friendlier with the cache. Just because the JVM is healthy does not mean
> it's going to perform like C++ for all applications. I suppose you could
> then argue that for best performance Storm is more or less limited to the
> JVM, but I choose not to consider that point here for brevity. Note this
> is not to say that it's impossible to write fast code in Java (see
> previously mentioned disruptor). I would just argue that it's a good bit
> harder.
>
> 5) I'm not sure I buy their argument that application logic costs are
> unlikely to mask the differences in framework performance. This depends
> very heavily on your application. If you're hitting external data sources
> a lot (e.g. memcache or database) then that will certainly mask a good
> portion of the difference. Maybe part of this argument is a C++ vs Java
> difference, in which case I'm somewhat more inclined to agree.
>
> 6) From a business perspective, the question changes from "is it faster?"
> to "what does it cost to support the throughput that we need?" which is a
> very different question. In many cases storm performs well enough.
>
>
> On Mon, May 12, 2014 at 9:02 AM, John Welcher <jp...@gmail.com> wrote:
>
>> Hi
>>
>> Streams also cost 40,000 US while Storm is free.
>>
>> John
>>
>>
>> On Mon, May 12, 2014 at 3:49 AM, Klausen Schaefersinho <
>> klaus.schaefers@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I found some interesting comparison of IBM Stream and Storm:
>>>
>>> https://www.ibmdw.net/streamsdev/2014/04/22/streams-apache-storm/
>>>
>>> It also includes an interesting comparison between ZeroMQ and the Netty
>>> Performance.
>>>
>>>
>>> Cheers,
>>>
>>> Klaus
>>>
>>
>>
>
Re: Interesting Comparison
Posted by Nathan Leung <nc...@gmail.com>.
a couple thoughts
1) IBM streams is certainly more mature, as it's been in development for a
longer amount of time and storm is not even at release 1.0 yet. Though I
am not familiar with SPL, It would also make sense that it's faster to
implement as it is a higher level abstraction.
2) Operator fusion will allow more efficiency in passing data between steps
in your flow, as localOrShuffleGrouping will still need to go over
disruptor whereas operator fusion from what I understand basically passes
the pointer directly. As fast as disruptor is (I've seen benchmarks of
millions of messages passed / s), it won't be directly passing data to the
next step (cost: a few instructions). The downside of this is your flow
always needs to be created and compiled before you can execute it.
Something like a rebalance will require a recompile of your stream.
Building a topology dynamically (which is possible in storm, but not a
feature that is really exposed out of the box) is possible in storm, but
not in IBM streams.
3) they took 1 month to optimize storm but I suspect some of this work was
unnecessary. Python? For a benchmark? Also, uniform message distribution
by size feels like a premature optimization. I can understand that they
would want to explore all avenues to account for a performance difference,
but in many (most?) practical cases this would not be necessary. I can
sympathize on other points. Tuning the message buffers of storm requires
pretty specific understanding of the system. Also if you run out of heap
and/or have to tune GC, then... yeah. Not fun. This would be true for any
java app though.
4) I'm not sure they really took language differences seriously enough.
I've written certain algorithms in Java that (based on similar algorithms
that I implemented separately in C++) I would suspect are close to an order
of magnitude slower just because I ran them in Java. While I haven't dug
into this deeply (for example by using an identical algorithm for both Java
and C++), consider a HashMap indexed by a primitive type. In Java, these
are separate objects stored in an array of references. In C++ these are
stored sequentially in an array. C++ allows direct key access in the array
(as opposed to going through the reference), and is also potentially much
friendlier with the cache. Just because the JVM is healthy does not mean
it's going to perform like C++ for all applications. I suppose you could
then argue that for best performance Storm is more or less limited to the
JVM, but I choose not to consider that point here for brevity. Note this
is not to say that it's impossible to write fast code in Java (see
previously mentioned disruptor). I would just argue that it's a good bit
harder.
5) I'm not sure I buy their argument that application logic costs are
unlikely to mask the differences in framework performance. This depends
very heavily on your application. If you're hitting external data sources
a lot (e.g. memcache or database) then that will certainly mask a good
portion of the difference. Maybe part of this argument is a C++ vs Java
difference, in which case I'm somewhat more inclined to agree.
6) From a business perspective, the question changes from "is it faster?"
to "what does it cost to support the throughput that we need?" which is a
very different question. In many cases storm performs well enough.
On Mon, May 12, 2014 at 9:02 AM, John Welcher <jp...@gmail.com> wrote:
> Hi
>
> Streams also cost 40,000 US while Storm is free.
>
> John
>
>
> On Mon, May 12, 2014 at 3:49 AM, Klausen Schaefersinho <
> klaus.schaefers@gmail.com> wrote:
>
>> Hi,
>>
>> I found some interesting comparison of IBM Stream and Storm:
>>
>> https://www.ibmdw.net/streamsdev/2014/04/22/streams-apache-storm/
>>
>> It also includes an interesting comparison between ZeroMQ and the Netty
>> Performance.
>>
>>
>> Cheers,
>>
>> Klaus
>>
>
>
Re: Interesting Comparison
Posted by Ted Dunning <te...@gmail.com>.
Regardless of what IBM does, there is clearly a lot that Storm is not doing
for performance. This is largely by design for simplicity. For instance,
storm could use reflection and byte code engineering to merge bolts while
still allowing rearrangement and rebalancing. Likewise, when throughput
gets high enough that some batching is worthwhile, technologies like
empirical schema, column orientation, bit weaving and zero serialization
formats[1,2] could all be used to move records. Together, these would give
several orders of magnitude speedup. The fact that these have not yet been
worth looking at for Storm users indicate that the virtues and challenges
of Storm probably live elsewhere.
The right thing to do is stick to Storm's knitting and make it the best
that it can be within its own self-concept. No need to ignore other
offerings, but considering of alternatives should be limited to finding
ways to improve Storm.
As I see it, the thing driving Storm adoption is reasonably flexible
parallel execution of streams with a very simple programming and fail-over
model. Performance is not a major issue for most needs since a million
tuples per second is larger than what most people need and that is pretty
easy to achieve with Storm.
[1] See Apache Drill for empirical schema, on-the fly code generation and
columnar zero serialization format
[2] See h2o for magical compression of columnar data with resulting monster
speed
On Mon, May 12, 2014 at 12:09 PM, Klausen Schaefersinho <
klaus.schaefers@gmail.com> wrote:
> Hi,
>
> my guess is that 40k are per CPU or so... for sure not for an entire
> cluster.
>
>
> On Mon, May 12, 2014 at 4:46 PM, Marc Vaillant <va...@animetrics.com>wrote:
>
>> To play devil's advocate, if you believe the stream performance gains,
>> then the 40k will likely pay for itself in needing to deploy a fraction
>> of the resources for the same throughput.
>>
>> On Mon, May 12, 2014 at 09:02:53AM -0400, John Welcher wrote:
>> > Hi
>> >
>> > Streams also cost 40,000 US while Storm is free.
>> >
>> > John
>> >
>> >
>> > On Mon, May 12, 2014 at 3:49 AM, Klausen Schaefersinho <
>> > klaus.schaefers@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I found some interesting comparison of IBM Stream and Storm:
>> >
>> > https://www.ibmdw.net/streamsdev/2014/04/22/streams-apache-storm/
>> >
>> > It also includes an interesting comparison between ZeroMQ and the
>> Netty
>> > Performance.
>> >
>> >
>> > Cheers,
>> >
>> > Klaus
>> >
>> >
>>
>
>
Re: Interesting Comparison
Posted by Klausen Schaefersinho <kl...@gmail.com>.
Hi,
my guess is that 40k are per CPU or so... for sure not for an entire
cluster.
On Mon, May 12, 2014 at 4:46 PM, Marc Vaillant <va...@animetrics.com>wrote:
> To play devil's advocate, if you believe the stream performance gains,
> then the 40k will likely pay for itself in needing to deploy a fraction
> of the resources for the same throughput.
>
> On Mon, May 12, 2014 at 09:02:53AM -0400, John Welcher wrote:
> > Hi
> >
> > Streams also cost 40,000 US while Storm is free.
> >
> > John
> >
> >
> > On Mon, May 12, 2014 at 3:49 AM, Klausen Schaefersinho <
> > klaus.schaefers@gmail.com> wrote:
> >
> > Hi,
> >
> > I found some interesting comparison of IBM Stream and Storm:
> >
> > https://www.ibmdw.net/streamsdev/2014/04/22/streams-apache-storm/
> >
> > It also includes an interesting comparison between ZeroMQ and the
> Netty
> > Performance.
> >
> >
> > Cheers,
> >
> > Klaus
> >
> >
>
Re: Interesting Comparison
Posted by Corey Nolet <cj...@gmail.com>.
It's hard to make a comparison without knowing exactly how their tests were
written. Especially being from the company of the product being recognized
as "superior". Storm is still young in the larger community and I certainly
think there's place for it to grow.
On Mon, May 12, 2014 at 9:03 PM, Jon Logan <jm...@buffalo.edu> wrote:
> The claims are certainly interesting...I haven't looked through it super
> detailed, but I would definitely keep in mind who is making the claims.
> Looking at it briefly, it looks like something is really wrong, looking at
> their scaling graphs. Without further information, I think it's hard to
> properly analyze their results, especially coming from a competing vendor.
>
>
> I don't know where this 40k figure comes from...coming from IBM's own
> cost-analysis paper, the pricing is more like starting at 500k, and easily
> going 1mil+.
>
> http://public.dhe.ibm.com/common/ssi/ecm/en/ime14024usen/IME14024USEN.PDF
>
>
> It would be interesting if they posted their source code, to see if
> they're doing something silly, or if anyone could rectify their performance
> issues. Otherwise, I think it's fair to assume this is potentially a
> "novice" versus "first-party supported expert" comparisons of
> implementations.
>
>
>
> On Mon, May 12, 2014 at 2:49 PM, Ted Dunning <te...@gmail.com>wrote:
>
>>
>> Anybody who has ever only paid 40K$ to IBM for anything should deserve a
>> prize. That is just the entry fee.
>>
>>
>>
>>
>> On Mon, May 12, 2014 at 7:46 AM, Marc Vaillant <va...@animetrics.com>wrote:
>>
>>> To play devil's advocate, if you believe the stream performance gains,
>>> then the 40k will likely pay for itself in needing to deploy a fraction
>>> of the resources for the same throughput.
>>>
>>> On Mon, May 12, 2014 at 09:02:53AM -0400, John Welcher wrote:
>>> > Hi
>>> >
>>> > Streams also cost 40,000 US while Storm is free.
>>> >
>>> > John
>>> >
>>> >
>>> > On Mon, May 12, 2014 at 3:49 AM, Klausen Schaefersinho <
>>> > klaus.schaefers@gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I found some interesting comparison of IBM Stream and Storm:
>>> >
>>> > https://www.ibmdw.net/streamsdev/2014/04/22/streams-apache-storm/
>>> >
>>> > It also includes an interesting comparison between ZeroMQ and the
>>> Netty
>>> > Performance.
>>> >
>>> >
>>> > Cheers,
>>> >
>>> > Klaus
>>> >
>>> >
>>>
>>
>>
>
Re: Interesting Comparison
Posted by Jon Logan <jm...@buffalo.edu>.
The claims are certainly interesting...I haven't looked through it super
detailed, but I would definitely keep in mind who is making the claims.
Looking at it briefly, it looks like something is really wrong, looking at
their scaling graphs. Without further information, I think it's hard to
properly analyze their results, especially coming from a competing vendor.
I don't know where this 40k figure comes from...coming from IBM's own
cost-analysis paper, the pricing is more like starting at 500k, and easily
going 1mil+.
http://public.dhe.ibm.com/common/ssi/ecm/en/ime14024usen/IME14024USEN.PDF
It would be interesting if they posted their source code, to see if they're
doing something silly, or if anyone could rectify their performance issues.
Otherwise, I think it's fair to assume this is potentially a "novice"
versus "first-party supported expert" comparisons of implementations.
On Mon, May 12, 2014 at 2:49 PM, Ted Dunning <te...@gmail.com> wrote:
>
> Anybody who has ever only paid 40K$ to IBM for anything should deserve a
> prize. That is just the entry fee.
>
>
>
>
> On Mon, May 12, 2014 at 7:46 AM, Marc Vaillant <va...@animetrics.com>wrote:
>
>> To play devil's advocate, if you believe the stream performance gains,
>> then the 40k will likely pay for itself in needing to deploy a fraction
>> of the resources for the same throughput.
>>
>> On Mon, May 12, 2014 at 09:02:53AM -0400, John Welcher wrote:
>> > Hi
>> >
>> > Streams also cost 40,000 US while Storm is free.
>> >
>> > John
>> >
>> >
>> > On Mon, May 12, 2014 at 3:49 AM, Klausen Schaefersinho <
>> > klaus.schaefers@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I found some interesting comparison of IBM Stream and Storm:
>> >
>> > https://www.ibmdw.net/streamsdev/2014/04/22/streams-apache-storm/
>> >
>> > It also includes an interesting comparison between ZeroMQ and the
>> Netty
>> > Performance.
>> >
>> >
>> > Cheers,
>> >
>> > Klaus
>> >
>> >
>>
>
>
Re: Interesting Comparison
Posted by Ted Dunning <te...@gmail.com>.
Anybody who has ever only paid 40K$ to IBM for anything should deserve a
prize. That is just the entry fee.
On Mon, May 12, 2014 at 7:46 AM, Marc Vaillant <va...@animetrics.com>wrote:
> To play devil's advocate, if you believe the stream performance gains,
> then the 40k will likely pay for itself in needing to deploy a fraction
> of the resources for the same throughput.
>
> On Mon, May 12, 2014 at 09:02:53AM -0400, John Welcher wrote:
> > Hi
> >
> > Streams also cost 40,000 US while Storm is free.
> >
> > John
> >
> >
> > On Mon, May 12, 2014 at 3:49 AM, Klausen Schaefersinho <
> > klaus.schaefers@gmail.com> wrote:
> >
> > Hi,
> >
> > I found some interesting comparison of IBM Stream and Storm:
> >
> > https://www.ibmdw.net/streamsdev/2014/04/22/streams-apache-storm/
> >
> > It also includes an interesting comparison between ZeroMQ and the
> Netty
> > Performance.
> >
> >
> > Cheers,
> >
> > Klaus
> >
> >
>
Re: Interesting Comparison
Posted by Marc Vaillant <va...@animetrics.com>.
To play devil's advocate, if you believe the stream performance gains,
then the 40k will likely pay for itself in needing to deploy a fraction
of the resources for the same throughput.
On Mon, May 12, 2014 at 09:02:53AM -0400, John Welcher wrote:
> Hi
>
> Streams also cost 40,000 US while Storm is free.
>
> John
>
>
> On Mon, May 12, 2014 at 3:49 AM, Klausen Schaefersinho <
> klaus.schaefers@gmail.com> wrote:
>
> Hi,
>
> I found some interesting comparison of IBM Stream and Storm:
>
> https://www.ibmdw.net/streamsdev/2014/04/22/streams-apache-storm/
>
> It also includes an interesting comparison between ZeroMQ and the Netty
> Performance.
>
>
> Cheers,
>
> Klaus
>
>
Re: Interesting Comparison
Posted by John Welcher <jp...@gmail.com>.
Hi
Streams also cost 40,000 US while Storm is free.
John
On Mon, May 12, 2014 at 3:49 AM, Klausen Schaefersinho <
klaus.schaefers@gmail.com> wrote:
> Hi,
>
> I found some interesting comparison of IBM Stream and Storm:
>
> https://www.ibmdw.net/streamsdev/2014/04/22/streams-apache-storm/
>
> It also includes an interesting comparison between ZeroMQ and the Netty
> Performance.
>
>
> Cheers,
>
> Klaus
>