You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by salemi <al...@udo.edu> on 2014/08/14 23:27:22 UTC
spark streaming - lamda architecture
Hi,
How would you implement the batch layer of lamda architecture with
spark/spark streaming?
Thanks,
Ali
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark streaming - lamda architecture
Posted by Sean Owen <so...@cloudera.com>.
You may be interested in https://github.com/OryxProject/oryx which is
at heart exactly "lambda architecture on Spark Streaming". With ML
pipelines on top. The architecture diagram and a peek at the code may
give you a good example of how this could be implemented. I choose to
view the batch layer as just a long-period streaming job on Spark
Streaming, and implement the speed layer as a short-period streaming
job. Summingbird is a good example too although it uses Storm and
MapReduce, and is architected specifically for simple aggregations. I
am not sure it "generalizes" but you may not need anything complex.
On Thu, Aug 14, 2014 at 10:27 PM, salemi <al...@udo.edu> wrote:
> Hi,
>
> How would you implement the batch layer of lamda architecture with
> spark/spark streaming?
>
> Thanks,
> Ali
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: spark streaming - lamda architecture
Posted by "Shao, Saisai" <sa...@intel.com>.
Hi Ali,
Maybe you can take a look at twitter's Summingbird project (https://github.com/twitter/summingbird), which is currently one of the few open source choices of lambda Architecture. There's a undergoing sub-project called summingbird-spark, that might be the one you wanted, might this can help you.
Thanks
Jerry
-----Original Message-----
From: salemi [mailto:alireza.salemi@udo.edu]
Sent: Friday, August 15, 2014 11:25 AM
To: user@spark.incubator.apache.org
Subject: Re: spark streaming - lamda architecture
below is what is what I understand under lambda architecture. The batch layer provides the historical data and the speed layer provides the real-time view!
All data entering the system is dispatched to both the batch layer and the speed layer for processing.
The batch layer has two functions:
(i) managing the master dataset (an immutable, append-only set of raw data), and
(ii) to pre-compute the batch views.
The speed layer compensates for the high latency of updates to the serving layer and deals with recent data only.
The serving layer indexes the batch views so that they can be queried in low-latency, ad-hoc way.
Any incoming query can be answered by merging results from batch views and real-time views.
In my system I have events coming in from Kafka sources and currently we need to process 10,000 messages per second and write them out to hdfs and make them available to be queried by a serving layer.
What would be your suggestion to architecturally solve this issue? How many solution with which would approx. be needed for the proposed architecture.
Thanks,
Ali
Tathagata Das wrote
> Can you be a bit more specific about what you mean by lambda architecture?
>
>
> On Thu, Aug 14, 2014 at 2:27 PM, salemi <
> alireza.salemi@
> > wrote:
>
>> Hi,
>>
>> How would you implement the batch layer of lamda architecture with
>> spark/spark streaming?
>>
>> Thanks,
>> Ali
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-l
>> amda-architecture-tp12142.html Sent from the Apache Spark User List
>> mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
> user-unsubscribe@.apache
>> For additional commands, e-mail:
> user-help@.apache
>>
>>
Tathagata Das wrote
> Can you be a bit more specific about what you mean by lambda architecture?
>
>
> On Thu, Aug 14, 2014 at 2:27 PM, salemi <
> alireza.salemi@
> > wrote:
>
>> Hi,
>>
>> How would you implement the batch layer of lamda architecture with
>> spark/spark streaming?
>>
>> Thanks,
>> Ali
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-l
>> amda-architecture-tp12142.html Sent from the Apache Spark User List
>> mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
> user-unsubscribe@.apache
>> For additional commands, e-mail:
> user-help@.apache
>>
>>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142p12163.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark streaming - lamda architecture
Posted by Michael Hausenblas <mi...@gmail.com>.
>>> How would you implement the batch layer of lamda architecture with
>>> spark/spark streaming?
I assume you’re familiar with resources such as https://speakerdeck.com/mhausenblas/lambda-architecture-with-apache-spark and are after more detailed advices?
Cheers,
Michael
--
Michael Hausenblas
Ireland, Europe
http://mhausenblas.info/
On 15 Aug 2014, at 05:25, salemi <al...@udo.edu> wrote:
> below is what is what I understand under lambda architecture. The batch layer
> provides the historical data and the speed layer provides the real-time
> view!
>
> All data entering the system is dispatched to both the batch layer and the
> speed layer for processing.
> The batch layer has two functions:
> (i) managing the master dataset (an immutable, append-only set of raw data),
> and
> (ii) to pre-compute the batch views.
>
> The speed layer compensates for the high latency of updates to the serving
> layer and deals with recent data only.
>
> The serving layer indexes the batch views so that they can be queried in
> low-latency, ad-hoc way.
>
> Any incoming query can be answered by merging results from batch views and
> real-time views.
>
> In my system I have events coming in from Kafka sources and currently we
> need to process 10,000 messages per second and write them out to hdfs and
> make them available to be queried by a serving layer.
>
> What would be your suggestion to architecturally solve this issue? How many
> solution with which would approx. be needed for the proposed architecture.
>
> Thanks,
> Ali
>
>
> Tathagata Das wrote
>> Can you be a bit more specific about what you mean by lambda architecture?
>>
>>
>> On Thu, Aug 14, 2014 at 2:27 PM, salemi <
>
>> alireza.salemi@
>
>> > wrote:
>>
>>> Hi,
>>>
>>> How would you implement the batch layer of lamda architecture with
>>> spark/spark streaming?
>>>
>>> Thanks,
>>> Ali
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>
>> user-unsubscribe@.apache
>
>>> For additional commands, e-mail:
>
>> user-help@.apache
>
>>>
>>>
>
>
> Tathagata Das wrote
>> Can you be a bit more specific about what you mean by lambda architecture?
>>
>>
>> On Thu, Aug 14, 2014 at 2:27 PM, salemi <
>
>> alireza.salemi@
>
>> > wrote:
>>
>>> Hi,
>>>
>>> How would you implement the batch layer of lamda architecture with
>>> spark/spark streaming?
>>>
>>> Thanks,
>>> Ali
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>
>> user-unsubscribe@.apache
>
>>> For additional commands, e-mail:
>
>> user-help@.apache
>
>>>
>>>
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142p12163.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark streaming - lamda architecture
Posted by Jörn Franke <jo...@gmail.com>.
Hi,
Maybe this helps you. For the "speed" layer I think something like complex
event processing as it is - to some extent - supported by Spark Streaming
can make sense. You process the events as they come in. You store them
afterwards. The Spark Streaming web page gives a nice example: trend
analysis. You detect from a lot of incoming events the current trend with
Spark Streaming and compare them to historical trends (stored in Spark,
which I would compare here with a serving layer). You can use this
information to see if customers are more happy (or more angry) with your
new products than usual.
I hope it helps. Your use case sounds very technical, it would be
interesting if you could share the business use case.
Best regards,
Jörn
On Fri, Aug 15, 2014 at 5:25 AM, salemi <al...@udo.edu> wrote:
> below is what is what I understand under lambda architecture. The batch
> layer
> provides the historical data and the speed layer provides the real-time
> view!
>
> All data entering the system is dispatched to both the batch layer and the
> speed layer for processing.
> The batch layer has two functions:
> (i) managing the master dataset (an immutable, append-only set of raw
> data),
> and
> (ii) to pre-compute the batch views.
>
> The speed layer compensates for the high latency of updates to the serving
> layer and deals with recent data only.
>
> The serving layer indexes the batch views so that they can be queried in
> low-latency, ad-hoc way.
>
> Any incoming query can be answered by merging results from batch views and
> real-time views.
>
> In my system I have events coming in from Kafka sources and currently we
> need to process 10,000 messages per second and write them out to hdfs and
> make them available to be queried by a serving layer.
>
> What would be your suggestion to architecturally solve this issue? How many
> solution with which would approx. be needed for the proposed architecture.
>
> Thanks,
> Ali
>
>
> Tathagata Das wrote
> > Can you be a bit more specific about what you mean by lambda
> architecture?
> >
> >
> > On Thu, Aug 14, 2014 at 2:27 PM, salemi <
>
> > alireza.salemi@
>
> > > wrote:
> >
> >> Hi,
> >>
> >> How would you implement the batch layer of lamda architecture with
> >> spark/spark streaming?
> >>
> >> Thanks,
> >> Ali
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html
> >> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail:
>
> > user-unsubscribe@.apache
>
> >> For additional commands, e-mail:
>
> > user-help@.apache
>
> >>
> >>
>
>
> Tathagata Das wrote
> > Can you be a bit more specific about what you mean by lambda
> architecture?
> >
> >
> > On Thu, Aug 14, 2014 at 2:27 PM, salemi <
>
> > alireza.salemi@
>
> > > wrote:
> >
> >> Hi,
> >>
> >> How would you implement the batch layer of lamda architecture with
> >> spark/spark streaming?
> >>
> >> Thanks,
> >> Ali
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html
> >> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail:
>
> > user-unsubscribe@.apache
>
> >> For additional commands, e-mail:
>
> > user-help@.apache
>
> >>
> >>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142p12163.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: spark streaming - lamda architecture
Posted by salemi <al...@udo.edu>.
below is what is what I understand under lambda architecture. The batch layer
provides the historical data and the speed layer provides the real-time
view!
All data entering the system is dispatched to both the batch layer and the
speed layer for processing.
The batch layer has two functions:
(i) managing the master dataset (an immutable, append-only set of raw data),
and
(ii) to pre-compute the batch views.
The speed layer compensates for the high latency of updates to the serving
layer and deals with recent data only.
The serving layer indexes the batch views so that they can be queried in
low-latency, ad-hoc way.
Any incoming query can be answered by merging results from batch views and
real-time views.
In my system I have events coming in from Kafka sources and currently we
need to process 10,000 messages per second and write them out to hdfs and
make them available to be queried by a serving layer.
What would be your suggestion to architecturally solve this issue? How many
solution with which would approx. be needed for the proposed architecture.
Thanks,
Ali
Tathagata Das wrote
> Can you be a bit more specific about what you mean by lambda architecture?
>
>
> On Thu, Aug 14, 2014 at 2:27 PM, salemi <
> alireza.salemi@
> > wrote:
>
>> Hi,
>>
>> How would you implement the batch layer of lamda architecture with
>> spark/spark streaming?
>>
>> Thanks,
>> Ali
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
> user-unsubscribe@.apache
>> For additional commands, e-mail:
> user-help@.apache
>>
>>
Tathagata Das wrote
> Can you be a bit more specific about what you mean by lambda architecture?
>
>
> On Thu, Aug 14, 2014 at 2:27 PM, salemi <
> alireza.salemi@
> > wrote:
>
>> Hi,
>>
>> How would you implement the batch layer of lamda architecture with
>> spark/spark streaming?
>>
>> Thanks,
>> Ali
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
> user-unsubscribe@.apache
>> For additional commands, e-mail:
> user-help@.apache
>>
>>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142p12163.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark streaming - lamda architecture
Posted by Tathagata Das <ta...@gmail.com>.
Can you be a bit more specific about what you mean by lambda architecture?
On Thu, Aug 14, 2014 at 2:27 PM, salemi <al...@udo.edu> wrote:
> Hi,
>
> How would you implement the batch layer of lamda architecture with
> spark/spark streaming?
>
> Thanks,
> Ali
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-lamda-architecture-tp12142.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>