You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Eliza <el...@ChinaBuckets.com> on 2019/08/21 06:35:08 UTC
Questions for platform to choose
Hello,
We have all of spark, flink, storm, kafka installed.
For realtime streaming calculation, which one is the best above?
Like other big players, the logs in our stack are huge.
Thanks.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Questions for platform to choose
Posted by Eliza <el...@ChinaBuckets.com>.
Also I found a excellent article for comparision.
https://www.linkedin.com/pulse/spark-streaming-vs-flink-storm-kafka-streams-samza-choose-prakash
regards.
on 2019/8/21 16:53, Eliza wrote:
> Hi,
>
> on 2019/8/21 16:44, Aziret Satybaldiev wrote:
>> In my experience, Kafka + Spark streaming + (perhaps HBase if you want
>> to store the metrics) is so far the best combo.
>> Not only because the technology is mature, but also because there are
>> a lot of examples available on the web and books.
>> They typically should cover most of what you would probably need and
>> you can easily build something more complex on top of that.
>> However, if you can, I would advise building several prototypes and
>> then choosing the best stack. At least that's how we got where we are.
>>
>
> Someone other told me that we should consider to use Flink for realtime
> streaming and Kafka for message queues, and Spark streaming is weak, and
> never consider Storm.
>
> I will try to test them in my cases.
>
> Thanks & regards.
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Questions for platform to choose
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi,
What is the definition of real time here?
The engineering definition of real time is roughly fast enough to be
interactive. However, I put a stronger definition. In real time application
or data, there is no such thing as an answer which is supposed to be late
and correct. The timeliness is part of the application. If we get the right
answer too slowly it becomes useless or wrong. We also need to be aware of
latency trades off with throughput. So it all depends what do you want to
do with all these artifacts for your needs.
Also within a larger architecture often latency is dictated by the lowest
denominator which often does not adhere to our definition of low latency.
For example, Kafka as widely deployed today in Big Data Architecture is
micro-batch. A moderate-latency message queue that is Kafka plus low
latency processor equals a moderate-latency architecture. Hence, the low
latency architecture must be treated within that context.
Have a look at this article of mine:
https://www.linkedin.com/pulse/real-time-processing-trade-data-kafka-flume-spark-talebzadeh-ph-d-/
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Wed, 21 Aug 2019 at 09:53, Eliza <el...@chinabuckets.com> wrote:
> Hi,
>
> on 2019/8/21 16:44, Aziret Satybaldiev wrote:
> > In my experience, Kafka + Spark streaming + (perhaps HBase if you want
> > to store the metrics) is so far the best combo.
> > Not only because the technology is mature, but also because there are a
> > lot of examples available on the web and books.
> > They typically should cover most of what you would probably need and you
> > can easily build something more complex on top of that.
> > However, if you can, I would advise building several prototypes and then
> > choosing the best stack. At least that's how we got where we are.
> >
>
> Someone other told me that we should consider to use Flink for realtime
> streaming and Kafka for message queues, and Spark streaming is weak, and
> never consider Storm.
>
> I will try to test them in my cases.
>
> Thanks & regards.
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: Questions for platform to choose
Posted by Eliza <el...@ChinaBuckets.com>.
Hi,
on 2019/8/21 16:44, Aziret Satybaldiev wrote:
> In my experience, Kafka + Spark streaming + (perhaps HBase if you want
> to store the metrics) is so far the best combo.
> Not only because the technology is mature, but also because there are a
> lot of examples available on the web and books.
> They typically should cover most of what you would probably need and you
> can easily build something more complex on top of that.
> However, if you can, I would advise building several prototypes and then
> choosing the best stack. At least that's how we got where we are.
>
Someone other told me that we should consider to use Flink for realtime
streaming and Kafka for message queues, and Spark streaming is weak, and
never consider Storm.
I will try to test them in my cases.
Thanks & regards.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Questions for platform to choose
Posted by Aziret Satybaldiev <sa...@gmail.com>.
Hi Eliza,
In my experience, Kafka + Spark streaming + (perhaps HBase if you want to
store the metrics) is so far the best combo.
Not only because the technology is mature, but also because there are a lot
of examples available on the web and books.
They typically should cover most of what you would probably need and you
can easily build something more complex on top of that.
However, if you can, I would advise building several prototypes and then
choosing the best stack. At least that's how we got where we are.
Kind regards,
Aziret
On Wed, 21 Aug 2019 at 08:43, Eliza <el...@chinabuckets.com> wrote:
> Hello,
>
> We have all of spark, flink, storm, kafka installed.
> For realtime streaming calculation, which one is the best above?
> Like other big players, the logs in our stack are huge.
>
> Thanks.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
--
Kind regards,
Aziret Satybaldiev
Re: Questions for platform to choose
Posted by Liam Clarke-Hutchinson <li...@steelsky.co.nz>.
Hi Eliza,
As I mentioned to you in the Kafka mailing list when you asked this there,
there are pros and cons to all of the technologies you've mentioned, and
you really need to sit down and try each solution to see what suits your
needs best.
Kind regards,
Liam Clarke
On Wed, Aug 21, 2019 at 9:46 PM Magnus Nilsson <ma...@kth.se> wrote:
> Well, you are posting on the Spark mailing list. Though for streaming I'd
> recommend Flink over Spark any day of the week. Flink was written as a
> streaming platform from the beginning quickly aligning the API with the
> theoretical framework of Google's Dataflow whitepaper. It's awesome for
> streaming. Spark not so much so far. Might become better, though the inital
> use case for Spark wasn't streaming, they might overcome that or not. I'd
> still go with Flink for streaming.
>
> If you need cross platform support you can take a look at Beam. Beam has
> Dataflow, Spark and Flink runners among others.
>
> Regards,
>
> Magnus
>
> On Wed, Aug 21, 2019 at 8:43 AM Eliza <el...@chinabuckets.com> wrote:
>
>> Hello,
>>
>> We have all of spark, flink, storm, kafka installed.
>> For realtime streaming calculation, which one is the best above?
>> Like other big players, the logs in our stack are huge.
>>
>> Thanks.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>
Re: Questions for platform to choose
Posted by Magnus Nilsson <ma...@kth.se>.
Well, you are posting on the Spark mailing list. Though for streaming I'd
recommend Flink over Spark any day of the week. Flink was written as a
streaming platform from the beginning quickly aligning the API with the
theoretical framework of Google's Dataflow whitepaper. It's awesome for
streaming. Spark not so much so far. Might become better, though the inital
use case for Spark wasn't streaming, they might overcome that or not. I'd
still go with Flink for streaming.
If you need cross platform support you can take a look at Beam. Beam has
Dataflow, Spark and Flink runners among others.
Regards,
Magnus
On Wed, Aug 21, 2019 at 8:43 AM Eliza <el...@chinabuckets.com> wrote:
> Hello,
>
> We have all of spark, flink, storm, kafka installed.
> For realtime streaming calculation, which one is the best above?
> Like other big players, the logs in our stack are huge.
>
> Thanks.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>