You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ashok Kumar <as...@yahoo.com.INVALID> on 2016/08/28 17:04:29 UTC
Design patterns involving Spark
Hi,
There are design patterns that use Spark extensively. I am new to this area so I would appreciate if someone explains where Spark fits in especially within faster or streaming use case.
What are the best practices involving Spark. Is it always best to deploy it for processing engine,
For example when we have a pattern
Input Data -> Data in Motion -> Processing -> Storage
Where does Spark best fit in.
Thanking you
Re: Design patterns involving Spark
Posted by Harish Butani <rh...@gmail.com>.
BTW, we now support OLAP functionality natively in spark w/o the need for
Druid, through our Spark native BI platform(SNAP):
https://www.linkedin.com/pulse/integrated-business-intelligence-big-data-stacks-harish-butani
- we provide SQL commands to: create star schema, create olap index, and
insert into olap index. So you can be up and running very quickly in a
Spark env.
- Query Acceleration is provided through an OLAP Index FileFormat and Query
Optimizer extensions(just like spark-druid-olap).
- We have also posted details on a BI Benchmark
<https://www.linkedin.com/pulse/integrated-business-intelligence-big-data-stacks-harish-butani>
to quantify
query acceleration and cost.
- haven't looked at integration with Spark Streaming yet, but since we have
a FileFormat should be possible to integrate. Please ping me if this is of
interest.
regards,
Harish.
On Mon, Aug 29, 2016 at 7:19 PM, Chanh Le <gi...@gmail.com> wrote:
> Hi everyone,
>
> Seems a lot people using Druid for realtime Dashboard.
> I’m just wondering of using Druid for main storage engine because Druid
> can store the raw data and can integrate with Spark also (theoretical).
> In that case do we need to store 2 separate storage Druid (store segment
> in HDFS) and HDFS?.
> BTW did anyone try this one https://github.com/
> SparklineData/spark-druid-olap?
>
>
> Regards,
> Chanh
>
>
> On Aug 30, 2016, at 3:23 AM, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
> Thanks Bhaarat and everyone.
>
> This is an updated version of the same diagram
>
> <LambdaArchitecture.png>
>
> The frequency of Recent data is defined by the Windows length in Spark
> Streaming. It can vary between 0.5 seconds to an hour. ( Don't think we can
> move any Spark granularity below 0.5 seconds in anger. For some
> applications like Credit card transactions and fraud detection. Data is
> stored real time by Spark in Hbase tables. Hbase tables will be on HDFS as
> well. The same Spark Streaming will write asynchronously to HDFS Hive
> tables.
> One school of thought is never write to Hive from Spark, write straight
> to Hbase and then read Hbase tables into Hive periodically?
>
> Now the third component in this layer is Serving Layer that can combine
> data from the current (Hbase) and the historical (Hive tables) to give the
> user visual analytics. Now that visual analytics can be Real time dashboard
> on top of Serving Layer. That Serving layer could be an in-memory NoSQL
> offering or Data from Hbase (Red Box) combined with Hive tables.
>
> I am not aware of any industrial strength Real time Dashboard. The idea
> is that one uses such dashboard in real time. Dashboard in this sense
> meaning a general purpose API to data store of some type like on Serving
> layer to provide visual analytics real time on demand, combining real time
> data and aggregate views. As usual the devil in the detail.
>
>
>
> Let me know your thoughts. Anyway this is first cut pattern.
>
>
>
> Dr Mich Talebzadeh
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
> http://talebzadehmich.wordpress.com
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 29 August 2016 at 18:53, Bhaarat Sharma <bh...@gmail.com> wrote:
>
>> Hi Mich
>>
>> This is really helpful. I'm trying to wrap my head around the last
>> diagram you shared (the one with kafka). In this diagram spark streaming is
>> pushing data to HDFS and NoSql. However, I'm confused by the "Real Time
>> Queries, Dashboards" annotation. Based on this diagram, will real time
>> queries be running on Spark or HBase?
>>
>> PS: My intention was not to steer the conversation away from what Ashok
>> asked but I found the diagrams shared by Mich very insightful.
>>
>> On Sun, Aug 28, 2016 at 7:18 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> In terms of positioning, Spark is really the first Big Data platform to
>>> integrate batch, streaming and interactive computations in a unified
>>> framework. What this boils down to is the fact that whichever way one look
>>> at it there is somewhere that Spark can make a contribution to. In general,
>>> there are few design patterns common to Big Data
>>>
>>>
>>>
>>> - *ETL & Batch*
>>>
>>> The first one is the most common one with Established tools like Sqoop,
>>> Talend for ETL and HDFS for storage of some kind. Spark can be used as the
>>> execution engine for Hive at the storage level which actually makes it
>>> a true vendor independent (BTW, Impala and Tez and LLAP) are offered by
>>> vendors) processing engine. Personally I use Spark at ETL layer by
>>> extracting data from sources through plug ins (JDBC and others) and storing
>>> in on HDFS in some kind
>>>
>>>
>>>
>>> - *Batch, real time plus Analytics*
>>>
>>> In this pattern you have data coming in real time and you want to query
>>> them real time through real time dashboard. HDFS is not ideal for updating
>>> data in real time and neither for random access of data. Source could be
>>> all sorts of Web Servers and need Flume Agent with Flume. At the storage
>>> layer we are probably looking at something like Hbase. The crucial point
>>> being that saved data needs to be ready for queries immediately The
>>> dashboards requires Hbase APIs. The Analytics can be done through Hive
>>> again running on Spark engine. Again note here that we ideally should
>>> process batch and real time separately.
>>>
>>>
>>>
>>> - *Real time / Streaming*
>>>
>>> This is most relevant to Spark as we are moving to near real time. Where
>>> Spark excels. We need to capture the incoming events (logs, sensor data,
>>> pricing, emails) through interfaces like Kafka, Message Queues etc. Need
>>> to process these events with minimum latency. Again Spark is a very good
>>> candidate here with its Spark Streaming and micro-batching capabilities.
>>> There are others like Storm, Flink etc. that are event based but you don’t
>>> hear much. Again for streaming architecture you need to sync data in real
>>> time using something like Hbase, Cassandra (?) and others as real time
>>> store or forever storage HDFS or Hive etc.
>>>
>>>
>>> In general there is also *Lambda Architecture* that is
>>> designed for streaming analytics. The streaming data ends up in both batch
>>> layer and speed layer. Batch layer is used to answer batch queries. On the
>>> other hand speed later is used ti handle fast/real time queries. This model
>>> is really cool as Spark Streaming can feed both the batch layer and
>>> the speed layer.
>>>
>>>
>>> At a high level this looks like this, from
>>> http://lambda-architecture.net/
>>>
>>> <image.png>
>>>
>>>
>>>
>>>
>>>
>>> My favourite would be something like below with Spark playing a major
>>> role
>>>
>>>
>>> <LambdaArchitecture.png>
>>>
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 28 August 2016 at 19:43, Sivakumaran S <si...@me.com> wrote:
>>>
>>>> Spark best fits for processing. But depending on the use case, you
>>>> could expand the scope of Spark to moving data using the native connectors.
>>>> The only that Spark is not, is Storage. Connectors are available for most
>>>> storage options though.
>>>>
>>>> Regards,
>>>>
>>>> Sivakumaran S
>>>>
>>>>
>>>>
>>>> On 28-Aug-2016, at 6:04 PM, Ashok Kumar <ashok34668@yahoo.com.INVALID
>>>> <as...@yahoo.com.invalid>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> There are design patterns that use Spark extensively. I am new to this
>>>> area so I would appreciate if someone explains where Spark fits in
>>>> especially within faster or streaming use case.
>>>>
>>>> What are the best practices involving Spark. Is it always best to
>>>> deploy it for processing engine,
>>>>
>>>> For example when we have a pattern
>>>>
>>>> Input Data -> Data in Motion -> Processing -> Storage
>>>>
>>>> Where does Spark best fit in.
>>>>
>>>> Thanking you
>>>>
>>>>
>>>
>>
>
>
Re: Design patterns involving Spark
Posted by Todd Nist <ts...@gmail.com>.
Have not tried this, but looks quite useful if one is using Druid:
https://github.com/implydata/pivot - An interactive data exploration UI
for Druid
On Tue, Aug 30, 2016 at 4:10 AM, Alonso Isidoro Roman <al...@gmail.com>
wrote:
> Thanks Mitch, i will check it.
>
> Cheers
>
>
> Alonso Isidoro Roman
> [image: https://]about.me/alonso.isidoro.roman
>
> <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links>
>
> 2016-08-30 9:52 GMT+02:00 Mich Talebzadeh <mi...@gmail.com>:
>
>> You can use Hbase for building real time dashboards
>>
>> Check this link
>> <https://www.sigmoid.com/integrating-spark-kafka-hbase-to-power-a-real-time-dashboard/>
>>
>> HTH
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 30 August 2016 at 08:33, Alonso Isidoro Roman <al...@gmail.com>
>> wrote:
>>
>>> HBase for real time queries? HBase was designed with the batch in mind.
>>> Impala should be a best choice, but i do not know what Druid can do....
>>>
>>>
>>> Cheers
>>>
>>> Alonso Isidoro Roman
>>> [image: https://]about.me/alonso.isidoro.roman
>>>
>>> <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links>
>>>
>>> 2016-08-30 8:56 GMT+02:00 Mich Talebzadeh <mi...@gmail.com>:
>>>
>>>> Hi Chanh,
>>>>
>>>> Druid sounds like a good choice.
>>>>
>>>> But again the point being is that what else Druid brings on top of
>>>> Hbase.
>>>>
>>>> Unless one decides to use Druid for both historical data and real time
>>>> data in place of Hbase!
>>>>
>>>> It is easier to write API against Druid that Hbase? You still want a UI
>>>> dashboard?
>>>>
>>>> Cheers
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>> On 30 August 2016 at 03:19, Chanh Le <gi...@gmail.com> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Seems a lot people using Druid for realtime Dashboard.
>>>>> I’m just wondering of using Druid for main storage engine because
>>>>> Druid can store the raw data and can integrate with Spark also
>>>>> (theoretical).
>>>>> In that case do we need to store 2 separate storage Druid (store
>>>>> segment in HDFS) and HDFS?.
>>>>> BTW did anyone try this one https://github.com/Sparkli
>>>>> neData/spark-druid-olap?
>>>>>
>>>>>
>>>>> Regards,
>>>>> Chanh
>>>>>
>>>>>
>>>>> On Aug 30, 2016, at 3:23 AM, Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>> Thanks Bhaarat and everyone.
>>>>>
>>>>> This is an updated version of the same diagram
>>>>>
>>>>> <LambdaArchitecture.png>
>>>>>
>>>>> The frequency of Recent data is defined by the Windows length in Spark
>>>>> Streaming. It can vary between 0.5 seconds to an hour. ( Don't think we can
>>>>> move any Spark granularity below 0.5 seconds in anger. For some
>>>>> applications like Credit card transactions and fraud detection. Data is
>>>>> stored real time by Spark in Hbase tables. Hbase tables will be on HDFS as
>>>>> well. The same Spark Streaming will write asynchronously to HDFS Hive
>>>>> tables.
>>>>> One school of thought is never write to Hive from Spark, write
>>>>> straight to Hbase and then read Hbase tables into Hive periodically?
>>>>>
>>>>> Now the third component in this layer is Serving Layer that can
>>>>> combine data from the current (Hbase) and the historical (Hive tables) to
>>>>> give the user visual analytics. Now that visual analytics can be Real time
>>>>> dashboard on top of Serving Layer. That Serving layer could be an in-memory
>>>>> NoSQL offering or Data from Hbase (Red Box) combined with Hive tables.
>>>>>
>>>>> I am not aware of any industrial strength Real time Dashboard. The
>>>>> idea is that one uses such dashboard in real time. Dashboard in this sense
>>>>> meaning a general purpose API to data store of some type like on Serving
>>>>> layer to provide visual analytics real time on demand, combining real time
>>>>> data and aggregate views. As usual the devil in the detail.
>>>>>
>>>>>
>>>>>
>>>>> Let me know your thoughts. Anyway this is first cut pattern.
>>>>>
>>>>>
>>>>>
>>>>> Dr Mich Talebzadeh
>>>>>
>>>>>
>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>
>>>>>
>>>>> http://talebzadehmich.wordpress.com
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>> arising from such loss, damage or destruction.
>>>>>
>>>>>
>>>>>
>>>>> On 29 August 2016 at 18:53, Bhaarat Sharma <bh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Mich
>>>>>>
>>>>>> This is really helpful. I'm trying to wrap my head around the last
>>>>>> diagram you shared (the one with kafka). In this diagram spark streaming is
>>>>>> pushing data to HDFS and NoSql. However, I'm confused by the "Real Time
>>>>>> Queries, Dashboards" annotation. Based on this diagram, will real time
>>>>>> queries be running on Spark or HBase?
>>>>>>
>>>>>> PS: My intention was not to steer the conversation away from what
>>>>>> Ashok asked but I found the diagrams shared by Mich very insightful.
>>>>>>
>>>>>> On Sun, Aug 28, 2016 at 7:18 PM, Mich Talebzadeh <
>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> In terms of positioning, Spark is really the first Big Data platform
>>>>>>> to integrate batch, streaming and interactive computations in a unified
>>>>>>> framework. What this boils down to is the fact that whichever way one look
>>>>>>> at it there is somewhere that Spark can make a contribution to. In general,
>>>>>>> there are few design patterns common to Big Data
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> - *ETL & Batch*
>>>>>>>
>>>>>>> The first one is the most common one with Established tools like
>>>>>>> Sqoop, Talend for ETL and HDFS for storage of some kind. Spark can be used
>>>>>>> as the execution engine for Hive at the storage level which actually
>>>>>>> makes it a true vendor independent (BTW, Impala and Tez and LLAP) are
>>>>>>> offered by vendors) processing engine. Personally I use Spark at ETL layer
>>>>>>> by extracting data from sources through plug ins (JDBC and others) and
>>>>>>> storing in on HDFS in some kind
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> - *Batch, real time plus Analytics*
>>>>>>>
>>>>>>> In this pattern you have data coming in real time and you want to
>>>>>>> query them real time through real time dashboard. HDFS is not ideal for
>>>>>>> updating data in real time and neither for random access of data. Source
>>>>>>> could be all sorts of Web Servers and need Flume Agent with Flume. At the
>>>>>>> storage layer we are probably looking at something like Hbase. The crucial
>>>>>>> point being that saved data needs to be ready for queries immediately The
>>>>>>> dashboards requires Hbase APIs. The Analytics can be done through Hive
>>>>>>> again running on Spark engine. Again note here that we ideally should
>>>>>>> process batch and real time separately.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> - *Real time / Streaming*
>>>>>>>
>>>>>>> This is most relevant to Spark as we are moving to near real time.
>>>>>>> Where Spark excels. We need to capture the incoming events (logs, sensor
>>>>>>> data, pricing, emails) through interfaces like Kafka, Message Queues etc.
>>>>>>> Need to process these events with minimum latency. Again Spark is
>>>>>>> a very good candidate here with its Spark Streaming and micro-batching
>>>>>>> capabilities. There are others like Storm, Flink etc. that are event based
>>>>>>> but you don’t hear much. Again for streaming architecture you need to sync
>>>>>>> data in real time using something like Hbase, Cassandra (?) and others as
>>>>>>> real time store or forever storage HDFS or Hive etc.
>>>>>>>
>>>>>>>
>>>>>>> In general there is also *Lambda Architecture* that is
>>>>>>> designed for streaming analytics. The streaming data ends up in both batch
>>>>>>> layer and speed layer. Batch layer is used to answer batch queries. On the
>>>>>>> other hand speed later is used ti handle fast/real time queries. This model
>>>>>>> is really cool as Spark Streaming can feed both the batch layer and
>>>>>>> the speed layer.
>>>>>>>
>>>>>>>
>>>>>>> At a high level this looks like this, from
>>>>>>> http://lambda-architecture.net/
>>>>>>>
>>>>>>> <image.png>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> My favourite would be something like below with Spark playing a
>>>>>>> major role
>>>>>>>
>>>>>>>
>>>>>>> <LambdaArchitecture.png>
>>>>>>>
>>>>>>>
>>>>>>> HTH
>>>>>>>
>>>>>>> Dr Mich Talebzadeh
>>>>>>>
>>>>>>>
>>>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>>>
>>>>>>>
>>>>>>> http://talebzadehmich.wordpress.com
>>>>>>>
>>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>>>>>> for any loss, damage or destruction of data or any other property which may
>>>>>>> arise from relying on this email's technical content is explicitly
>>>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>>>> arising from such loss, damage or destruction.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 28 August 2016 at 19:43, Sivakumaran S <si...@me.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Spark best fits for processing. But depending on the use case, you
>>>>>>>> could expand the scope of Spark to moving data using the native connectors.
>>>>>>>> The only that Spark is not, is Storage. Connectors are available for most
>>>>>>>> storage options though.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Sivakumaran S
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 28-Aug-2016, at 6:04 PM, Ashok Kumar <
>>>>>>>> ashok34668@yahoo.com.INVALID <as...@yahoo.com.invalid>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> There are design patterns that use Spark extensively. I am new to
>>>>>>>> this area so I would appreciate if someone explains where Spark fits in
>>>>>>>> especially within faster or streaming use case.
>>>>>>>>
>>>>>>>> What are the best practices involving Spark. Is it always best to
>>>>>>>> deploy it for processing engine,
>>>>>>>>
>>>>>>>> For example when we have a pattern
>>>>>>>>
>>>>>>>> Input Data -> Data in Motion -> Processing -> Storage
>>>>>>>>
>>>>>>>> Where does Spark best fit in.
>>>>>>>>
>>>>>>>> Thanking you
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Design patterns involving Spark
Posted by Alonso Isidoro Roman <al...@gmail.com>.
Thanks Mitch, i will check it.
Cheers
Alonso Isidoro Roman
[image: https://]about.me/alonso.isidoro.roman
<https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links>
2016-08-30 9:52 GMT+02:00 Mich Talebzadeh <mi...@gmail.com>:
> You can use Hbase for building real time dashboards
>
> Check this link
> <https://www.sigmoid.com/integrating-spark-kafka-hbase-to-power-a-real-time-dashboard/>
>
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 30 August 2016 at 08:33, Alonso Isidoro Roman <al...@gmail.com>
> wrote:
>
>> HBase for real time queries? HBase was designed with the batch in mind.
>> Impala should be a best choice, but i do not know what Druid can do....
>>
>>
>> Cheers
>>
>> Alonso Isidoro Roman
>> [image: https://]about.me/alonso.isidoro.roman
>>
>> <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links>
>>
>> 2016-08-30 8:56 GMT+02:00 Mich Talebzadeh <mi...@gmail.com>:
>>
>>> Hi Chanh,
>>>
>>> Druid sounds like a good choice.
>>>
>>> But again the point being is that what else Druid brings on top of
>>> Hbase.
>>>
>>> Unless one decides to use Druid for both historical data and real time
>>> data in place of Hbase!
>>>
>>> It is easier to write API against Druid that Hbase? You still want a UI
>>> dashboard?
>>>
>>> Cheers
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 30 August 2016 at 03:19, Chanh Le <gi...@gmail.com> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Seems a lot people using Druid for realtime Dashboard.
>>>> I’m just wondering of using Druid for main storage engine because Druid
>>>> can store the raw data and can integrate with Spark also (theoretical).
>>>> In that case do we need to store 2 separate storage Druid (store
>>>> segment in HDFS) and HDFS?.
>>>> BTW did anyone try this one https://github.com/Sparkli
>>>> neData/spark-druid-olap?
>>>>
>>>>
>>>> Regards,
>>>> Chanh
>>>>
>>>>
>>>> On Aug 30, 2016, at 3:23 AM, Mich Talebzadeh <mi...@gmail.com>
>>>> wrote:
>>>>
>>>> Thanks Bhaarat and everyone.
>>>>
>>>> This is an updated version of the same diagram
>>>>
>>>> <LambdaArchitecture.png>
>>>>
>>>> The frequency of Recent data is defined by the Windows length in Spark
>>>> Streaming. It can vary between 0.5 seconds to an hour. ( Don't think we can
>>>> move any Spark granularity below 0.5 seconds in anger. For some
>>>> applications like Credit card transactions and fraud detection. Data is
>>>> stored real time by Spark in Hbase tables. Hbase tables will be on HDFS as
>>>> well. The same Spark Streaming will write asynchronously to HDFS Hive
>>>> tables.
>>>> One school of thought is never write to Hive from Spark, write
>>>> straight to Hbase and then read Hbase tables into Hive periodically?
>>>>
>>>> Now the third component in this layer is Serving Layer that can combine
>>>> data from the current (Hbase) and the historical (Hive tables) to give the
>>>> user visual analytics. Now that visual analytics can be Real time dashboard
>>>> on top of Serving Layer. That Serving layer could be an in-memory NoSQL
>>>> offering or Data from Hbase (Red Box) combined with Hive tables.
>>>>
>>>> I am not aware of any industrial strength Real time Dashboard. The
>>>> idea is that one uses such dashboard in real time. Dashboard in this sense
>>>> meaning a general purpose API to data store of some type like on Serving
>>>> layer to provide visual analytics real time on demand, combining real time
>>>> data and aggregate views. As usual the devil in the detail.
>>>>
>>>>
>>>>
>>>> Let me know your thoughts. Anyway this is first cut pattern.
>>>>
>>>>
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>> On 29 August 2016 at 18:53, Bhaarat Sharma <bh...@gmail.com> wrote:
>>>>
>>>>> Hi Mich
>>>>>
>>>>> This is really helpful. I'm trying to wrap my head around the last
>>>>> diagram you shared (the one with kafka). In this diagram spark streaming is
>>>>> pushing data to HDFS and NoSql. However, I'm confused by the "Real Time
>>>>> Queries, Dashboards" annotation. Based on this diagram, will real time
>>>>> queries be running on Spark or HBase?
>>>>>
>>>>> PS: My intention was not to steer the conversation away from what
>>>>> Ashok asked but I found the diagrams shared by Mich very insightful.
>>>>>
>>>>> On Sun, Aug 28, 2016 at 7:18 PM, Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> In terms of positioning, Spark is really the first Big Data platform
>>>>>> to integrate batch, streaming and interactive computations in a unified
>>>>>> framework. What this boils down to is the fact that whichever way one look
>>>>>> at it there is somewhere that Spark can make a contribution to. In general,
>>>>>> there are few design patterns common to Big Data
>>>>>>
>>>>>>
>>>>>>
>>>>>> - *ETL & Batch*
>>>>>>
>>>>>> The first one is the most common one with Established tools like
>>>>>> Sqoop, Talend for ETL and HDFS for storage of some kind. Spark can be used
>>>>>> as the execution engine for Hive at the storage level which actually
>>>>>> makes it a true vendor independent (BTW, Impala and Tez and LLAP) are
>>>>>> offered by vendors) processing engine. Personally I use Spark at ETL layer
>>>>>> by extracting data from sources through plug ins (JDBC and others) and
>>>>>> storing in on HDFS in some kind
>>>>>>
>>>>>>
>>>>>>
>>>>>> - *Batch, real time plus Analytics*
>>>>>>
>>>>>> In this pattern you have data coming in real time and you want to
>>>>>> query them real time through real time dashboard. HDFS is not ideal for
>>>>>> updating data in real time and neither for random access of data. Source
>>>>>> could be all sorts of Web Servers and need Flume Agent with Flume. At the
>>>>>> storage layer we are probably looking at something like Hbase. The crucial
>>>>>> point being that saved data needs to be ready for queries immediately The
>>>>>> dashboards requires Hbase APIs. The Analytics can be done through Hive
>>>>>> again running on Spark engine. Again note here that we ideally should
>>>>>> process batch and real time separately.
>>>>>>
>>>>>>
>>>>>>
>>>>>> - *Real time / Streaming*
>>>>>>
>>>>>> This is most relevant to Spark as we are moving to near real time.
>>>>>> Where Spark excels. We need to capture the incoming events (logs, sensor
>>>>>> data, pricing, emails) through interfaces like Kafka, Message Queues etc.
>>>>>> Need to process these events with minimum latency. Again Spark is a
>>>>>> very good candidate here with its Spark Streaming and micro-batching
>>>>>> capabilities. There are others like Storm, Flink etc. that are event based
>>>>>> but you don’t hear much. Again for streaming architecture you need to sync
>>>>>> data in real time using something like Hbase, Cassandra (?) and others as
>>>>>> real time store or forever storage HDFS or Hive etc.
>>>>>>
>>>>>>
>>>>>> In general there is also *Lambda Architecture* that is
>>>>>> designed for streaming analytics. The streaming data ends up in both batch
>>>>>> layer and speed layer. Batch layer is used to answer batch queries. On the
>>>>>> other hand speed later is used ti handle fast/real time queries. This model
>>>>>> is really cool as Spark Streaming can feed both the batch layer and
>>>>>> the speed layer.
>>>>>>
>>>>>>
>>>>>> At a high level this looks like this, from
>>>>>> http://lambda-architecture.net/
>>>>>>
>>>>>> <image.png>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> My favourite would be something like below with Spark playing a major
>>>>>> role
>>>>>>
>>>>>>
>>>>>> <LambdaArchitecture.png>
>>>>>>
>>>>>>
>>>>>> HTH
>>>>>>
>>>>>> Dr Mich Talebzadeh
>>>>>>
>>>>>>
>>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>>
>>>>>>
>>>>>> http://talebzadehmich.wordpress.com
>>>>>>
>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>>>>> for any loss, damage or destruction of data or any other property which may
>>>>>> arise from relying on this email's technical content is explicitly
>>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>>> arising from such loss, damage or destruction.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 28 August 2016 at 19:43, Sivakumaran S <si...@me.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Spark best fits for processing. But depending on the use case, you
>>>>>>> could expand the scope of Spark to moving data using the native connectors.
>>>>>>> The only that Spark is not, is Storage. Connectors are available for most
>>>>>>> storage options though.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Sivakumaran S
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 28-Aug-2016, at 6:04 PM, Ashok Kumar <
>>>>>>> ashok34668@yahoo.com.INVALID <as...@yahoo.com.invalid>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> There are design patterns that use Spark extensively. I am new to
>>>>>>> this area so I would appreciate if someone explains where Spark fits in
>>>>>>> especially within faster or streaming use case.
>>>>>>>
>>>>>>> What are the best practices involving Spark. Is it always best to
>>>>>>> deploy it for processing engine,
>>>>>>>
>>>>>>> For example when we have a pattern
>>>>>>>
>>>>>>> Input Data -> Data in Motion -> Processing -> Storage
>>>>>>>
>>>>>>> Where does Spark best fit in.
>>>>>>>
>>>>>>> Thanking you
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Design patterns involving Spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
You can use Hbase for building real time dashboards
Check this link
<https://www.sigmoid.com/integrating-spark-kafka-hbase-to-power-a-real-time-dashboard/>
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 30 August 2016 at 08:33, Alonso Isidoro Roman <al...@gmail.com> wrote:
> HBase for real time queries? HBase was designed with the batch in mind.
> Impala should be a best choice, but i do not know what Druid can do....
>
>
> Cheers
>
> Alonso Isidoro Roman
> [image: https://]about.me/alonso.isidoro.roman
>
> <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links>
>
> 2016-08-30 8:56 GMT+02:00 Mich Talebzadeh <mi...@gmail.com>:
>
>> Hi Chanh,
>>
>> Druid sounds like a good choice.
>>
>> But again the point being is that what else Druid brings on top of Hbase.
>>
>> Unless one decides to use Druid for both historical data and real time
>> data in place of Hbase!
>>
>> It is easier to write API against Druid that Hbase? You still want a UI
>> dashboard?
>>
>> Cheers
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 30 August 2016 at 03:19, Chanh Le <gi...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> Seems a lot people using Druid for realtime Dashboard.
>>> I’m just wondering of using Druid for main storage engine because Druid
>>> can store the raw data and can integrate with Spark also (theoretical).
>>> In that case do we need to store 2 separate storage Druid (store segment
>>> in HDFS) and HDFS?.
>>> BTW did anyone try this one https://github.com/Sparkli
>>> neData/spark-druid-olap?
>>>
>>>
>>> Regards,
>>> Chanh
>>>
>>>
>>> On Aug 30, 2016, at 3:23 AM, Mich Talebzadeh <mi...@gmail.com>
>>> wrote:
>>>
>>> Thanks Bhaarat and everyone.
>>>
>>> This is an updated version of the same diagram
>>>
>>> <LambdaArchitecture.png>
>>>
>>> The frequency of Recent data is defined by the Windows length in Spark
>>> Streaming. It can vary between 0.5 seconds to an hour. ( Don't think we can
>>> move any Spark granularity below 0.5 seconds in anger. For some
>>> applications like Credit card transactions and fraud detection. Data is
>>> stored real time by Spark in Hbase tables. Hbase tables will be on HDFS as
>>> well. The same Spark Streaming will write asynchronously to HDFS Hive
>>> tables.
>>> One school of thought is never write to Hive from Spark, write straight
>>> to Hbase and then read Hbase tables into Hive periodically?
>>>
>>> Now the third component in this layer is Serving Layer that can combine
>>> data from the current (Hbase) and the historical (Hive tables) to give the
>>> user visual analytics. Now that visual analytics can be Real time dashboard
>>> on top of Serving Layer. That Serving layer could be an in-memory NoSQL
>>> offering or Data from Hbase (Red Box) combined with Hive tables.
>>>
>>> I am not aware of any industrial strength Real time Dashboard. The idea
>>> is that one uses such dashboard in real time. Dashboard in this sense
>>> meaning a general purpose API to data store of some type like on Serving
>>> layer to provide visual analytics real time on demand, combining real time
>>> data and aggregate views. As usual the devil in the detail.
>>>
>>>
>>>
>>> Let me know your thoughts. Anyway this is first cut pattern.
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 29 August 2016 at 18:53, Bhaarat Sharma <bh...@gmail.com> wrote:
>>>
>>>> Hi Mich
>>>>
>>>> This is really helpful. I'm trying to wrap my head around the last
>>>> diagram you shared (the one with kafka). In this diagram spark streaming is
>>>> pushing data to HDFS and NoSql. However, I'm confused by the "Real Time
>>>> Queries, Dashboards" annotation. Based on this diagram, will real time
>>>> queries be running on Spark or HBase?
>>>>
>>>> PS: My intention was not to steer the conversation away from what Ashok
>>>> asked but I found the diagrams shared by Mich very insightful.
>>>>
>>>> On Sun, Aug 28, 2016 at 7:18 PM, Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> In terms of positioning, Spark is really the first Big Data platform
>>>>> to integrate batch, streaming and interactive computations in a unified
>>>>> framework. What this boils down to is the fact that whichever way one look
>>>>> at it there is somewhere that Spark can make a contribution to. In general,
>>>>> there are few design patterns common to Big Data
>>>>>
>>>>>
>>>>>
>>>>> - *ETL & Batch*
>>>>>
>>>>> The first one is the most common one with Established tools like
>>>>> Sqoop, Talend for ETL and HDFS for storage of some kind. Spark can be used
>>>>> as the execution engine for Hive at the storage level which actually
>>>>> makes it a true vendor independent (BTW, Impala and Tez and LLAP) are
>>>>> offered by vendors) processing engine. Personally I use Spark at ETL layer
>>>>> by extracting data from sources through plug ins (JDBC and others) and
>>>>> storing in on HDFS in some kind
>>>>>
>>>>>
>>>>>
>>>>> - *Batch, real time plus Analytics*
>>>>>
>>>>> In this pattern you have data coming in real time and you want to
>>>>> query them real time through real time dashboard. HDFS is not ideal for
>>>>> updating data in real time and neither for random access of data. Source
>>>>> could be all sorts of Web Servers and need Flume Agent with Flume. At the
>>>>> storage layer we are probably looking at something like Hbase. The crucial
>>>>> point being that saved data needs to be ready for queries immediately The
>>>>> dashboards requires Hbase APIs. The Analytics can be done through Hive
>>>>> again running on Spark engine. Again note here that we ideally should
>>>>> process batch and real time separately.
>>>>>
>>>>>
>>>>>
>>>>> - *Real time / Streaming*
>>>>>
>>>>> This is most relevant to Spark as we are moving to near real time.
>>>>> Where Spark excels. We need to capture the incoming events (logs, sensor
>>>>> data, pricing, emails) through interfaces like Kafka, Message Queues etc.
>>>>> Need to process these events with minimum latency. Again Spark is a
>>>>> very good candidate here with its Spark Streaming and micro-batching
>>>>> capabilities. There are others like Storm, Flink etc. that are event based
>>>>> but you don’t hear much. Again for streaming architecture you need to sync
>>>>> data in real time using something like Hbase, Cassandra (?) and others as
>>>>> real time store or forever storage HDFS or Hive etc.
>>>>>
>>>>>
>>>>> In general there is also *Lambda Architecture* that is
>>>>> designed for streaming analytics. The streaming data ends up in both batch
>>>>> layer and speed layer. Batch layer is used to answer batch queries. On the
>>>>> other hand speed later is used ti handle fast/real time queries. This model
>>>>> is really cool as Spark Streaming can feed both the batch layer and
>>>>> the speed layer.
>>>>>
>>>>>
>>>>> At a high level this looks like this, from
>>>>> http://lambda-architecture.net/
>>>>>
>>>>> <image.png>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> My favourite would be something like below with Spark playing a major
>>>>> role
>>>>>
>>>>>
>>>>> <LambdaArchitecture.png>
>>>>>
>>>>>
>>>>> HTH
>>>>>
>>>>> Dr Mich Talebzadeh
>>>>>
>>>>>
>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>
>>>>>
>>>>> http://talebzadehmich.wordpress.com
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>> arising from such loss, damage or destruction.
>>>>>
>>>>>
>>>>>
>>>>> On 28 August 2016 at 19:43, Sivakumaran S <si...@me.com> wrote:
>>>>>
>>>>>> Spark best fits for processing. But depending on the use case, you
>>>>>> could expand the scope of Spark to moving data using the native connectors.
>>>>>> The only that Spark is not, is Storage. Connectors are available for most
>>>>>> storage options though.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Sivakumaran S
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 28-Aug-2016, at 6:04 PM, Ashok Kumar <ashok34668@yahoo.com.INVALID
>>>>>> <as...@yahoo.com.invalid>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> There are design patterns that use Spark extensively. I am new to
>>>>>> this area so I would appreciate if someone explains where Spark fits in
>>>>>> especially within faster or streaming use case.
>>>>>>
>>>>>> What are the best practices involving Spark. Is it always best to
>>>>>> deploy it for processing engine,
>>>>>>
>>>>>> For example when we have a pattern
>>>>>>
>>>>>> Input Data -> Data in Motion -> Processing -> Storage
>>>>>>
>>>>>> Where does Spark best fit in.
>>>>>>
>>>>>> Thanking you
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
Re: Design patterns involving Spark
Posted by Alonso Isidoro Roman <al...@gmail.com>.
HBase for real time queries? HBase was designed with the batch in mind.
Impala should be a best choice, but i do not know what Druid can do....
Cheers
Alonso Isidoro Roman
[image: https://]about.me/alonso.isidoro.roman
<https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links>
2016-08-30 8:56 GMT+02:00 Mich Talebzadeh <mi...@gmail.com>:
> Hi Chanh,
>
> Druid sounds like a good choice.
>
> But again the point being is that what else Druid brings on top of Hbase.
>
> Unless one decides to use Druid for both historical data and real time
> data in place of Hbase!
>
> It is easier to write API against Druid that Hbase? You still want a UI
> dashboard?
>
> Cheers
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 30 August 2016 at 03:19, Chanh Le <gi...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Seems a lot people using Druid for realtime Dashboard.
>> I’m just wondering of using Druid for main storage engine because Druid
>> can store the raw data and can integrate with Spark also (theoretical).
>> In that case do we need to store 2 separate storage Druid (store segment
>> in HDFS) and HDFS?.
>> BTW did anyone try this one https://github.com/Sparkli
>> neData/spark-druid-olap?
>>
>>
>> Regards,
>> Chanh
>>
>>
>> On Aug 30, 2016, at 3:23 AM, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>> Thanks Bhaarat and everyone.
>>
>> This is an updated version of the same diagram
>>
>> <LambdaArchitecture.png>
>>
>> The frequency of Recent data is defined by the Windows length in Spark
>> Streaming. It can vary between 0.5 seconds to an hour. ( Don't think we can
>> move any Spark granularity below 0.5 seconds in anger. For some
>> applications like Credit card transactions and fraud detection. Data is
>> stored real time by Spark in Hbase tables. Hbase tables will be on HDFS as
>> well. The same Spark Streaming will write asynchronously to HDFS Hive
>> tables.
>> One school of thought is never write to Hive from Spark, write straight
>> to Hbase and then read Hbase tables into Hive periodically?
>>
>> Now the third component in this layer is Serving Layer that can combine
>> data from the current (Hbase) and the historical (Hive tables) to give the
>> user visual analytics. Now that visual analytics can be Real time dashboard
>> on top of Serving Layer. That Serving layer could be an in-memory NoSQL
>> offering or Data from Hbase (Red Box) combined with Hive tables.
>>
>> I am not aware of any industrial strength Real time Dashboard. The idea
>> is that one uses such dashboard in real time. Dashboard in this sense
>> meaning a general purpose API to data store of some type like on Serving
>> layer to provide visual analytics real time on demand, combining real time
>> data and aggregate views. As usual the devil in the detail.
>>
>>
>>
>> Let me know your thoughts. Anyway this is first cut pattern.
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 29 August 2016 at 18:53, Bhaarat Sharma <bh...@gmail.com> wrote:
>>
>>> Hi Mich
>>>
>>> This is really helpful. I'm trying to wrap my head around the last
>>> diagram you shared (the one with kafka). In this diagram spark streaming is
>>> pushing data to HDFS and NoSql. However, I'm confused by the "Real Time
>>> Queries, Dashboards" annotation. Based on this diagram, will real time
>>> queries be running on Spark or HBase?
>>>
>>> PS: My intention was not to steer the conversation away from what Ashok
>>> asked but I found the diagrams shared by Mich very insightful.
>>>
>>> On Sun, Aug 28, 2016 at 7:18 PM, Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> In terms of positioning, Spark is really the first Big Data platform to
>>>> integrate batch, streaming and interactive computations in a unified
>>>> framework. What this boils down to is the fact that whichever way one look
>>>> at it there is somewhere that Spark can make a contribution to. In general,
>>>> there are few design patterns common to Big Data
>>>>
>>>>
>>>>
>>>> - *ETL & Batch*
>>>>
>>>> The first one is the most common one with Established tools like Sqoop,
>>>> Talend for ETL and HDFS for storage of some kind. Spark can be used as the
>>>> execution engine for Hive at the storage level which actually makes
>>>> it a true vendor independent (BTW, Impala and Tez and LLAP) are offered by
>>>> vendors) processing engine. Personally I use Spark at ETL layer by
>>>> extracting data from sources through plug ins (JDBC and others) and storing
>>>> in on HDFS in some kind
>>>>
>>>>
>>>>
>>>> - *Batch, real time plus Analytics*
>>>>
>>>> In this pattern you have data coming in real time and you want to query
>>>> them real time through real time dashboard. HDFS is not ideal for updating
>>>> data in real time and neither for random access of data. Source could be
>>>> all sorts of Web Servers and need Flume Agent with Flume. At the storage
>>>> layer we are probably looking at something like Hbase. The crucial point
>>>> being that saved data needs to be ready for queries immediately The
>>>> dashboards requires Hbase APIs. The Analytics can be done through Hive
>>>> again running on Spark engine. Again note here that we ideally should
>>>> process batch and real time separately.
>>>>
>>>>
>>>>
>>>> - *Real time / Streaming*
>>>>
>>>> This is most relevant to Spark as we are moving to near real time.
>>>> Where Spark excels. We need to capture the incoming events (logs, sensor
>>>> data, pricing, emails) through interfaces like Kafka, Message Queues etc.
>>>> Need to process these events with minimum latency. Again Spark is a
>>>> very good candidate here with its Spark Streaming and micro-batching
>>>> capabilities. There are others like Storm, Flink etc. that are event based
>>>> but you don’t hear much. Again for streaming architecture you need to sync
>>>> data in real time using something like Hbase, Cassandra (?) and others as
>>>> real time store or forever storage HDFS or Hive etc.
>>>>
>>>>
>>>> In general there is also *Lambda Architecture* that is
>>>> designed for streaming analytics. The streaming data ends up in both batch
>>>> layer and speed layer. Batch layer is used to answer batch queries. On the
>>>> other hand speed later is used ti handle fast/real time queries. This model
>>>> is really cool as Spark Streaming can feed both the batch layer and
>>>> the speed layer.
>>>>
>>>>
>>>> At a high level this looks like this, from
>>>> http://lambda-architecture.net/
>>>>
>>>> <image.png>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> My favourite would be something like below with Spark playing a major
>>>> role
>>>>
>>>>
>>>> <LambdaArchitecture.png>
>>>>
>>>>
>>>> HTH
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>> On 28 August 2016 at 19:43, Sivakumaran S <si...@me.com> wrote:
>>>>
>>>>> Spark best fits for processing. But depending on the use case, you
>>>>> could expand the scope of Spark to moving data using the native connectors.
>>>>> The only that Spark is not, is Storage. Connectors are available for most
>>>>> storage options though.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Sivakumaran S
>>>>>
>>>>>
>>>>>
>>>>> On 28-Aug-2016, at 6:04 PM, Ashok Kumar <ashok34668@yahoo.com.INVALID
>>>>> <as...@yahoo.com.invalid>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> There are design patterns that use Spark extensively. I am new to this
>>>>> area so I would appreciate if someone explains where Spark fits in
>>>>> especially within faster or streaming use case.
>>>>>
>>>>> What are the best practices involving Spark. Is it always best to
>>>>> deploy it for processing engine,
>>>>>
>>>>> For example when we have a pattern
>>>>>
>>>>> Input Data -> Data in Motion -> Processing -> Storage
>>>>>
>>>>> Where does Spark best fit in.
>>>>>
>>>>> Thanking you
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
Re: Design patterns involving Spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Chanh,
Druid sounds like a good choice.
But again the point being is that what else Druid brings on top of Hbase.
Unless one decides to use Druid for both historical data and real time data
in place of Hbase!
It is easier to write API against Druid that Hbase? You still want a UI
dashboard?
Cheers
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 30 August 2016 at 03:19, Chanh Le <gi...@gmail.com> wrote:
> Hi everyone,
>
> Seems a lot people using Druid for realtime Dashboard.
> I’m just wondering of using Druid for main storage engine because Druid
> can store the raw data and can integrate with Spark also (theoretical).
> In that case do we need to store 2 separate storage Druid (store segment
> in HDFS) and HDFS?.
> BTW did anyone try this one https://github.com/
> SparklineData/spark-druid-olap?
>
>
> Regards,
> Chanh
>
>
> On Aug 30, 2016, at 3:23 AM, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
> Thanks Bhaarat and everyone.
>
> This is an updated version of the same diagram
>
> <LambdaArchitecture.png>
>
> The frequency of Recent data is defined by the Windows length in Spark
> Streaming. It can vary between 0.5 seconds to an hour. ( Don't think we can
> move any Spark granularity below 0.5 seconds in anger. For some
> applications like Credit card transactions and fraud detection. Data is
> stored real time by Spark in Hbase tables. Hbase tables will be on HDFS as
> well. The same Spark Streaming will write asynchronously to HDFS Hive
> tables.
> One school of thought is never write to Hive from Spark, write straight
> to Hbase and then read Hbase tables into Hive periodically?
>
> Now the third component in this layer is Serving Layer that can combine
> data from the current (Hbase) and the historical (Hive tables) to give the
> user visual analytics. Now that visual analytics can be Real time dashboard
> on top of Serving Layer. That Serving layer could be an in-memory NoSQL
> offering or Data from Hbase (Red Box) combined with Hive tables.
>
> I am not aware of any industrial strength Real time Dashboard. The idea
> is that one uses such dashboard in real time. Dashboard in this sense
> meaning a general purpose API to data store of some type like on Serving
> layer to provide visual analytics real time on demand, combining real time
> data and aggregate views. As usual the devil in the detail.
>
>
>
> Let me know your thoughts. Anyway this is first cut pattern.
>
>
>
> Dr Mich Talebzadeh
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
> http://talebzadehmich.wordpress.com
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 29 August 2016 at 18:53, Bhaarat Sharma <bh...@gmail.com> wrote:
>
>> Hi Mich
>>
>> This is really helpful. I'm trying to wrap my head around the last
>> diagram you shared (the one with kafka). In this diagram spark streaming is
>> pushing data to HDFS and NoSql. However, I'm confused by the "Real Time
>> Queries, Dashboards" annotation. Based on this diagram, will real time
>> queries be running on Spark or HBase?
>>
>> PS: My intention was not to steer the conversation away from what Ashok
>> asked but I found the diagrams shared by Mich very insightful.
>>
>> On Sun, Aug 28, 2016 at 7:18 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> In terms of positioning, Spark is really the first Big Data platform to
>>> integrate batch, streaming and interactive computations in a unified
>>> framework. What this boils down to is the fact that whichever way one look
>>> at it there is somewhere that Spark can make a contribution to. In general,
>>> there are few design patterns common to Big Data
>>>
>>>
>>>
>>> - *ETL & Batch*
>>>
>>> The first one is the most common one with Established tools like Sqoop,
>>> Talend for ETL and HDFS for storage of some kind. Spark can be used as the
>>> execution engine for Hive at the storage level which actually makes it
>>> a true vendor independent (BTW, Impala and Tez and LLAP) are offered by
>>> vendors) processing engine. Personally I use Spark at ETL layer by
>>> extracting data from sources through plug ins (JDBC and others) and storing
>>> in on HDFS in some kind
>>>
>>>
>>>
>>> - *Batch, real time plus Analytics*
>>>
>>> In this pattern you have data coming in real time and you want to query
>>> them real time through real time dashboard. HDFS is not ideal for updating
>>> data in real time and neither for random access of data. Source could be
>>> all sorts of Web Servers and need Flume Agent with Flume. At the storage
>>> layer we are probably looking at something like Hbase. The crucial point
>>> being that saved data needs to be ready for queries immediately The
>>> dashboards requires Hbase APIs. The Analytics can be done through Hive
>>> again running on Spark engine. Again note here that we ideally should
>>> process batch and real time separately.
>>>
>>>
>>>
>>> - *Real time / Streaming*
>>>
>>> This is most relevant to Spark as we are moving to near real time. Where
>>> Spark excels. We need to capture the incoming events (logs, sensor data,
>>> pricing, emails) through interfaces like Kafka, Message Queues etc. Need
>>> to process these events with minimum latency. Again Spark is a very good
>>> candidate here with its Spark Streaming and micro-batching capabilities.
>>> There are others like Storm, Flink etc. that are event based but you don’t
>>> hear much. Again for streaming architecture you need to sync data in real
>>> time using something like Hbase, Cassandra (?) and others as real time
>>> store or forever storage HDFS or Hive etc.
>>>
>>>
>>> In general there is also *Lambda Architecture* that is
>>> designed for streaming analytics. The streaming data ends up in both batch
>>> layer and speed layer. Batch layer is used to answer batch queries. On the
>>> other hand speed later is used ti handle fast/real time queries. This model
>>> is really cool as Spark Streaming can feed both the batch layer and
>>> the speed layer.
>>>
>>>
>>> At a high level this looks like this, from
>>> http://lambda-architecture.net/
>>>
>>> <image.png>
>>>
>>>
>>>
>>>
>>>
>>> My favourite would be something like below with Spark playing a major
>>> role
>>>
>>>
>>> <LambdaArchitecture.png>
>>>
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 28 August 2016 at 19:43, Sivakumaran S <si...@me.com> wrote:
>>>
>>>> Spark best fits for processing. But depending on the use case, you
>>>> could expand the scope of Spark to moving data using the native connectors.
>>>> The only that Spark is not, is Storage. Connectors are available for most
>>>> storage options though.
>>>>
>>>> Regards,
>>>>
>>>> Sivakumaran S
>>>>
>>>>
>>>>
>>>> On 28-Aug-2016, at 6:04 PM, Ashok Kumar <ashok34668@yahoo.com.INVALID
>>>> <as...@yahoo.com.invalid>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> There are design patterns that use Spark extensively. I am new to this
>>>> area so I would appreciate if someone explains where Spark fits in
>>>> especially within faster or streaming use case.
>>>>
>>>> What are the best practices involving Spark. Is it always best to
>>>> deploy it for processing engine,
>>>>
>>>> For example when we have a pattern
>>>>
>>>> Input Data -> Data in Motion -> Processing -> Storage
>>>>
>>>> Where does Spark best fit in.
>>>>
>>>> Thanking you
>>>>
>>>>
>>>
>>
>
>
Re: Design patterns involving Spark
Posted by Chanh Le <gi...@gmail.com>.
Hi everyone,
Seems a lot people using Druid for realtime Dashboard.
I’m just wondering of using Druid for main storage engine because Druid can store the raw data and can integrate with Spark also (theoretical).
In that case do we need to store 2 separate storage Druid (store segment in HDFS) and HDFS?.
BTW did anyone try this one https://github.com/SparklineData/spark-druid-olap <https://github.com/SparklineData/spark-druid-olap>?
Regards,
Chanh
> On Aug 30, 2016, at 3:23 AM, Mich Talebzadeh <mi...@gmail.com> wrote:
>
> Thanks Bhaarat and everyone.
>
> This is an updated version of the same diagram
>
> <LambdaArchitecture.png>
>
> The frequency of Recent data is defined by the Windows length in Spark Streaming. It can vary between 0.5 seconds to an hour. ( Don't think we can move any Spark granularity below 0.5 seconds in anger. For some applications like Credit card transactions and fraud detection. Data is stored real time by Spark in Hbase tables. Hbase tables will be on HDFS as well. The same Spark Streaming will write asynchronously to HDFS Hive tables.
> One school of thought is never write to Hive from Spark, write straight to Hbase and then read Hbase tables into Hive periodically?
>
> Now the third component in this layer is Serving Layer that can combine data from the current (Hbase) and the historical (Hive tables) to give the user visual analytics. Now that visual analytics can be Real time dashboard on top of Serving Layer. That Serving layer could be an in-memory NoSQL offering or Data from Hbase (Red Box) combined with Hive tables.
>
> I am not aware of any industrial strength Real time Dashboard. The idea is that one uses such dashboard in real time. Dashboard in this sense meaning a general purpose API to data store of some type like on Serving layer to provide visual analytics real time on demand, combining real time data and aggregate views. As usual the devil in the detail.
>
>
>
> Let me know your thoughts. Anyway this is first cut pattern.
>
>
>
> Dr Mich Talebzadeh
>
> LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
>
>
> On 29 August 2016 at 18:53, Bhaarat Sharma <bhaarat.s@gmail.com <ma...@gmail.com>> wrote:
> Hi Mich
>
> This is really helpful. I'm trying to wrap my head around the last diagram you shared (the one with kafka). In this diagram spark streaming is pushing data to HDFS and NoSql. However, I'm confused by the "Real Time Queries, Dashboards" annotation. Based on this diagram, will real time queries be running on Spark or HBase?
>
> PS: My intention was not to steer the conversation away from what Ashok asked but I found the diagrams shared by Mich very insightful.
>
> On Sun, Aug 28, 2016 at 7:18 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> In terms of positioning, Spark is really the first Big Data platform to integrate batch, streaming and interactive computations in a unified framework. What this boils down to is the fact that whichever way one look at it there is somewhere that Spark can make a contribution to. In general, there are few design patterns common to Big Data
>
> ETL & Batch
> The first one is the most common one with Established tools like Sqoop, Talend for ETL and HDFS for storage of some kind. Spark can be used as the execution engine for Hive at the storage level which actually makes it a true vendor independent (BTW, Impala and Tez and LLAP) are offered by vendors) processing engine. Personally I use Spark at ETL layer by extracting data from sources through plug ins (JDBC and others) and storing in on HDFS in some kind
>
> Batch, real time plus Analytics
> In this pattern you have data coming in real time and you want to query them real time through real time dashboard. HDFS is not ideal for updating data in real time and neither for random access of data. Source could be all sorts of Web Servers and need Flume Agent with Flume. At the storage layer we are probably looking at something like Hbase. The crucial point being that saved data needs to be ready for queries immediately The dashboards requires Hbase APIs. The Analytics can be done through Hive again running on Spark engine. Again note here that we ideally should process batch and real time separately.
>
> Real time / Streaming
> This is most relevant to Spark as we are moving to near real time. Where Spark excels. We need to capture the incoming events (logs, sensor data, pricing, emails) through interfaces like Kafka, Message Queues etc. Need to process these events with minimum latency. Again Spark is a very good candidate here with its Spark Streaming and micro-batching capabilities. There are others like Storm, Flink etc. that are event based but you don’t hear much. Again for streaming architecture you need to sync data in real time using something like Hbase, Cassandra (?) and others as real time store or forever storage HDFS or Hive etc.
>
> In general there is also Lambda Architecture that is designed for streaming analytics. The streaming data ends up in both batch layer and speed layer. Batch layer is used to answer batch queries. On the other hand speed later is used ti handle fast/real time queries. This model is really cool as Spark Streaming can feed both the batch layer and the speed layer.
>
> At a high level this looks like this, from http://lambda-architecture.net/ <http://lambda-architecture.net/>
>
> <image.png>
>
>
>
>
>
> My favourite would be something like below with Spark playing a major role
>
>
> <LambdaArchitecture.png>
>
>
> HTH
>
> Dr Mich Talebzadeh
>
> LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
>
>
> On 28 August 2016 at 19:43, Sivakumaran S <siva.kumaran@me.com <ma...@me.com>> wrote:
> Spark best fits for processing. But depending on the use case, you could expand the scope of Spark to moving data using the native connectors. The only that Spark is not, is Storage. Connectors are available for most storage options though.
>
> Regards,
>
> Sivakumaran S
>
>
>
> On 28-Aug-2016, at 6:04 PM, Ashok Kumar <ashok34668@yahoo.com.INVALID <ma...@yahoo.com.invalid>> wrote:
>
>> Hi,
>>
>> There are design patterns that use Spark extensively. I am new to this area so I would appreciate if someone explains where Spark fits in especially within faster or streaming use case.
>>
>> What are the best practices involving Spark. Is it always best to deploy it for processing engine,
>>
>> For example when we have a pattern
>>
>> Input Data -> Data in Motion -> Processing -> Storage
>>
>> Where does Spark best fit in.
>>
>> Thanking you
>
>
>
Re: Design patterns involving Spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Bhaarat and everyone.
This is an updated version of the same diagram
The frequency of Recent data is defined by the Windows length in Spark
Streaming. It can vary between 0.5 seconds to an hour. ( Don't think we can
move any Spark granularity below 0.5 seconds in anger. For some
applications like Credit card transactions and fraud detection. Data is
stored real time by Spark in Hbase tables. Hbase tables will be on HDFS as
well. The same Spark Streaming will write asynchronously to HDFS Hive
tables.
One school of thought is never write to Hive from Spark, write straight to
Hbase and then read Hbase tables into Hive periodically?
Now the third component in this layer is Serving Layer that can combine
data from the current (Hbase) and the historical (Hive tables) to give the
user visual analytics. Now that visual analytics can be Real time dashboard
on top of Serving Layer. That Serving layer could be an in-memory NoSQL
offering or Data from Hbase (Red Box) combined with Hive tables.
I am not aware of any industrial strength Real time Dashboard. The idea is
that one uses such dashboard in real time. Dashboard in this sense meaning
a general purpose API to data store of some type like on Serving layer to
provide visual analytics real time on demand, combining real time data and
aggregate views. As usual the devil in the detail.
Let me know your thoughts. Anyway this is first cut pattern.
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 29 August 2016 at 18:53, Bhaarat Sharma <bh...@gmail.com> wrote:
> Hi Mich
>
> This is really helpful. I'm trying to wrap my head around the last diagram
> you shared (the one with kafka). In this diagram spark streaming is pushing
> data to HDFS and NoSql. However, I'm confused by the "Real Time Queries,
> Dashboards" annotation. Based on this diagram, will real time queries be
> running on Spark or HBase?
>
> PS: My intention was not to steer the conversation away from what Ashok
> asked but I found the diagrams shared by Mich very insightful.
>
> On Sun, Aug 28, 2016 at 7:18 PM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>> Hi,
>>
>> In terms of positioning, Spark is really the first Big Data platform to
>> integrate batch, streaming and interactive computations in a unified
>> framework. What this boils down to is the fact that whichever way one look
>> at it there is somewhere that Spark can make a contribution to. In general,
>> there are few design patterns common to Big Data
>>
>>
>>
>> -
>>
>> *ETL & Batch*
>>
>> The first one is the most common one with Established tools like Sqoop,
>> Talend for ETL and HDFS for storage of some kind. Spark can be used as the
>> execution engine for Hive at the storage level which actually makes it
>> a true vendor independent (BTW, Impala and Tez and LLAP) are offered by
>> vendors) processing engine. Personally I use Spark at ETL layer by
>> extracting data from sources through plug ins (JDBC and others) and storing
>> in on HDFS in some kind
>>
>>
>>
>> -
>>
>> *Batch, real time plus Analytics*
>>
>> In this pattern you have data coming in real time and you want to query
>> them real time through real time dashboard. HDFS is not ideal for updating
>> data in real time and neither for random access of data. Source could be
>> all sorts of Web Servers and need Flume Agent with Flume. At the storage
>> layer we are probably looking at something like Hbase. The crucial point
>> being that saved data needs to be ready for queries immediately The
>> dashboards requires Hbase APIs. The Analytics can be done through Hive
>> again running on Spark engine. Again note here that we ideally should
>> process batch and real time separately.
>>
>>
>>
>> -
>>
>> *Real time / Streaming*
>>
>> This is most relevant to Spark as we are moving to near real time. Where
>> Spark excels. We need to capture the incoming events (logs, sensor data,
>> pricing, emails) through interfaces like Kafka, Message Queues etc. Need
>> to process these events with minimum latency. Again Spark is a very good
>> candidate here with its Spark Streaming and micro-batching capabilities.
>> There are others like Storm, Flink etc. that are event based but you don’t
>> hear much. Again for streaming architecture you need to sync data in real
>> time using something like Hbase, Cassandra (?) and others as real time
>> store or forever storage HDFS or Hive etc.
>>
>>
>>
>> In general there is also *Lambda Architecture* that is
>> designed for streaming analytics. The streaming data ends up in both batch
>> layer and speed layer. Batch layer is used to answer batch queries. On the
>> other hand speed later is used ti handle fast/real time queries. This model
>> is really cool as Spark Streaming can feed both the batch layer and
>> the speed layer.
>>
>>
>>
>> At a high level this looks like this, from http://lambda-architecture.net
>> /
>>
>>
>> [image: Inline images 2]
>>
>>
>>
>>
>>
>>
>> My favourite would be something like below with Spark playing a major role
>>
>>
>>
>>
>>
>>
>>
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 28 August 2016 at 19:43, Sivakumaran S <si...@me.com> wrote:
>>
>>> Spark best fits for processing. But depending on the use case, you could
>>> expand the scope of Spark to moving data using the native connectors. The
>>> only that Spark is not, is Storage. Connectors are available for most
>>> storage options though.
>>>
>>> Regards,
>>>
>>> Sivakumaran S
>>>
>>>
>>>
>>> On 28-Aug-2016, at 6:04 PM, Ashok Kumar <ashok34668@yahoo.com.INVALID
>>> <as...@yahoo.com.invalid>> wrote:
>>>
>>> Hi,
>>>
>>> There are design patterns that use Spark extensively. I am new to this
>>> area so I would appreciate if someone explains where Spark fits in
>>> especially within faster or streaming use case.
>>>
>>> What are the best practices involving Spark. Is it always best to deploy
>>> it for processing engine,
>>>
>>> For example when we have a pattern
>>>
>>> Input Data -> Data in Motion -> Processing -> Storage
>>>
>>> Where does Spark best fit in.
>>>
>>> Thanking you
>>>
>>>
>>
>
Re: Design patterns involving Spark
Posted by Bhaarat Sharma <bh...@gmail.com>.
Hi Mich
This is really helpful. I'm trying to wrap my head around the last diagram
you shared (the one with kafka). In this diagram spark streaming is pushing
data to HDFS and NoSql. However, I'm confused by the "Real Time Queries,
Dashboards" annotation. Based on this diagram, will real time queries be
running on Spark or HBase?
PS: My intention was not to steer the conversation away from what Ashok
asked but I found the diagrams shared by Mich very insightful.
On Sun, Aug 28, 2016 at 7:18 PM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Hi,
>
> In terms of positioning, Spark is really the first Big Data platform to
> integrate batch, streaming and interactive computations in a unified
> framework. What this boils down to is the fact that whichever way one look
> at it there is somewhere that Spark can make a contribution to. In general,
> there are few design patterns common to Big Data
>
>
>
> -
>
> *ETL & Batch*
>
> The first one is the most common one with Established tools like Sqoop,
> Talend for ETL and HDFS for storage of some kind. Spark can be used as the
> execution engine for Hive at the storage level which actually makes it a
> true vendor independent (BTW, Impala and Tez and LLAP) are offered by
> vendors) processing engine. Personally I use Spark at ETL layer by
> extracting data from sources through plug ins (JDBC and others) and storing
> in on HDFS in some kind
>
>
>
> -
>
> *Batch, real time plus Analytics*
>
> In this pattern you have data coming in real time and you want to query
> them real time through real time dashboard. HDFS is not ideal for updating
> data in real time and neither for random access of data. Source could be
> all sorts of Web Servers and need Flume Agent with Flume. At the storage
> layer we are probably looking at something like Hbase. The crucial point
> being that saved data needs to be ready for queries immediately The
> dashboards requires Hbase APIs. The Analytics can be done through Hive
> again running on Spark engine. Again note here that we ideally should
> process batch and real time separately.
>
>
>
> -
>
> *Real time / Streaming*
>
> This is most relevant to Spark as we are moving to near real time. Where
> Spark excels. We need to capture the incoming events (logs, sensor data,
> pricing, emails) through interfaces like Kafka, Message Queues etc. Need
> to process these events with minimum latency. Again Spark is a very good
> candidate here with its Spark Streaming and micro-batching capabilities.
> There are others like Storm, Flink etc. that are event based but you don’t
> hear much. Again for streaming architecture you need to sync data in real
> time using something like Hbase, Cassandra (?) and others as real time
> store or forever storage HDFS or Hive etc.
>
>
>
> In general there is also *Lambda Architecture* that is
> designed for streaming analytics. The streaming data ends up in both batch
> layer and speed layer. Batch layer is used to answer batch queries. On the
> other hand speed later is used ti handle fast/real time queries. This model
> is really cool as Spark Streaming can feed both the batch layer and
> the speed layer.
>
>
>
> At a high level this looks like this, from http://lambda-architecture.net/
>
>
> [image: Inline images 2]
>
>
>
>
>
>
> My favourite would be something like below with Spark playing a major role
>
>
>
>
>
>
>
>
> HTH
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 28 August 2016 at 19:43, Sivakumaran S <si...@me.com> wrote:
>
>> Spark best fits for processing. But depending on the use case, you could
>> expand the scope of Spark to moving data using the native connectors. The
>> only that Spark is not, is Storage. Connectors are available for most
>> storage options though.
>>
>> Regards,
>>
>> Sivakumaran S
>>
>>
>>
>> On 28-Aug-2016, at 6:04 PM, Ashok Kumar <ashok34668@yahoo.com.INVALID
>> <as...@yahoo.com.invalid>> wrote:
>>
>> Hi,
>>
>> There are design patterns that use Spark extensively. I am new to this
>> area so I would appreciate if someone explains where Spark fits in
>> especially within faster or streaming use case.
>>
>> What are the best practices involving Spark. Is it always best to deploy
>> it for processing engine,
>>
>> For example when we have a pattern
>>
>> Input Data -> Data in Motion -> Processing -> Storage
>>
>> Where does Spark best fit in.
>>
>> Thanking you
>>
>>
>
Re: Design patterns involving Spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
Interesting points Ayan.
What are the available real time dashboards to Spark? I don't believer
Zeppelin, QlikView or Tableau can b e considered as real time dashboards.
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 29 August 2016 at 14:02, ayan guha <gu...@gmail.com> wrote:
> In addition to what Mitch explained, Spark is the sole platform where you
> can extend batch semantics to real time semantics within same framework. I
> will suggest you to watch last spark summit videos,, esp Michael A's demo
> on structured streaming, which shows how powerful such easy transition
> between semantics can become.
> On 29 Aug 2016 22:20, "Ashok Kumar" <as...@yahoo.com.invalid> wrote:
>
>> Hi,
>>
>> Thank you for your explanations. Very fruitful.
>>
>> Warmest
>>
>>
>> On Monday, 29 August 2016, 0:18, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>
>> Hi,
>>
>> In terms of positioning, Spark is really the first Big Data platform to
>> integrate batch, streaming and interactive computations in a unified
>> framework. What this boils down to is the fact that whichever way one look
>> at it there is somewhere that Spark can make a contribution to. In general,
>> there are few design patterns common to Big Data
>>
>>
>> - *ETL & Batch*
>>
>> The first one is the most common one with Established tools like Sqoop,
>> Talend for ETL and HDFS for storage of some kind. Spark can be used as the
>> execution engine for Hive at the storage level which actually makes it
>> a true vendor independent (BTW, Impala and Tez and LLAP) are offered by
>> vendors) processing engine. Personally I use Spark at ETL layer by
>> extracting data from sources through plug ins (JDBC and others) and storing
>> in on HDFS in some kind
>>
>>
>> - *Batch, real time plus Analytics*
>>
>> In this pattern you have data coming in real time and you want to query
>> them real time through real time dashboard. HDFS is not ideal for updating
>> data in real time and neither for random access of data. Source could be
>> all sorts of Web Servers and need Flume Agent with Flume. At the storage
>> layer we are probably looking at something like Hbase. The crucial point
>> being that saved data needs to be ready for queries immediately The
>> dashboards requires Hbase APIs. The Analytics can be done through Hive
>> again running on Spark engine. Again note here that we ideally should
>> process batch and real time separately.
>>
>>
>> - *Real time / Streaming*
>>
>> This is most relevant to Spark as we are moving to near real time. Where
>> Spark excels. We need to capture the incoming events (logs, sensor data,
>> pricing, emails) through interfaces like Kafka, Message Queues etc. Need
>> to process these events with minimum latency. Again Spark is a very good
>> candidate here with its Spark Streaming and micro-batching capabilities.
>> There are others like Storm, Flink etc. that are event based but you don’t
>> hear much. Again for streaming architecture you need to sync data in real
>> time using something like Hbase, Cassandra (?) and others as real time
>> store or forever storage HDFS or Hive etc.
>>
>> In general there is also *Lambda Architecture* that is
>> designed for streaming analytics. The streaming data ends up in both batch
>> layer and speed layer. Batch layer is used to answer batch queries. On the
>> other hand speed later is used ti handle fast/real time queries. This model
>> is really cool as Spark Streaming can feed both the batch layer and
>> the speed layer.
>>
>> At a high level this looks like this, from http://lambda-architecture.net
>> /
>>
>> [image: Inline images 2]
>>
>>
>>
>>
>>
>> My favourite would be something like below with Spark playing a major role
>>
>>
>>
>>
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>> http://talebzadehmich.wordpress.com
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>> On 28 August 2016 at 19:43, Sivakumaran S <si...@me.com> wrote:
>>
>> Spark best fits for processing. But depending on the use case, you could
>> expand the scope of Spark to moving data using the native connectors. The
>> only that Spark is not, is Storage. Connectors are available for most
>> storage options though.
>>
>> Regards,
>>
>> Sivakumaran S
>>
>>
>>
>> On 28-Aug-2016, at 6:04 PM, Ashok Kumar <ashok34668@yahoo.com.INVALID
>> <as...@yahoo.com.invalid>> wrote:
>>
>> Hi,
>>
>> There are design patterns that use Spark extensively. I am new to this
>> area so I would appreciate if someone explains where Spark fits in
>> especially within faster or streaming use case.
>>
>> What are the best practices involving Spark. Is it always best to deploy
>> it for processing engine,
>>
>> For example when we have a pattern
>>
>> Input Data -> Data in Motion -> Processing -> Storage
>>
>> Where does Spark best fit in.
>>
>> Thanking you
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>
Re: Design patterns involving Spark
Posted by ayan guha <gu...@gmail.com>.
In addition to what Mitch explained, Spark is the sole platform where you
can extend batch semantics to real time semantics within same framework. I
will suggest you to watch last spark summit videos,, esp Michael A's demo
on structured streaming, which shows how powerful such easy transition
between semantics can become.
On 29 Aug 2016 22:20, "Ashok Kumar" <as...@yahoo.com.invalid> wrote:
> Hi,
>
> Thank you for your explanations. Very fruitful.
>
> Warmest
>
>
> On Monday, 29 August 2016, 0:18, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>
> Hi,
>
> In terms of positioning, Spark is really the first Big Data platform to
> integrate batch, streaming and interactive computations in a unified
> framework. What this boils down to is the fact that whichever way one look
> at it there is somewhere that Spark can make a contribution to. In general,
> there are few design patterns common to Big Data
>
>
> - *ETL & Batch*
>
> The first one is the most common one with Established tools like Sqoop,
> Talend for ETL and HDFS for storage of some kind. Spark can be used as the
> execution engine for Hive at the storage level which actually makes it a
> true vendor independent (BTW, Impala and Tez and LLAP) are offered by
> vendors) processing engine. Personally I use Spark at ETL layer by
> extracting data from sources through plug ins (JDBC and others) and storing
> in on HDFS in some kind
>
>
> - *Batch, real time plus Analytics*
>
> In this pattern you have data coming in real time and you want to query
> them real time through real time dashboard. HDFS is not ideal for updating
> data in real time and neither for random access of data. Source could be
> all sorts of Web Servers and need Flume Agent with Flume. At the storage
> layer we are probably looking at something like Hbase. The crucial point
> being that saved data needs to be ready for queries immediately The
> dashboards requires Hbase APIs. The Analytics can be done through Hive
> again running on Spark engine. Again note here that we ideally should
> process batch and real time separately.
>
>
> - *Real time / Streaming*
>
> This is most relevant to Spark as we are moving to near real time. Where
> Spark excels. We need to capture the incoming events (logs, sensor data,
> pricing, emails) through interfaces like Kafka, Message Queues etc. Need
> to process these events with minimum latency. Again Spark is a very good
> candidate here with its Spark Streaming and micro-batching capabilities.
> There are others like Storm, Flink etc. that are event based but you don’t
> hear much. Again for streaming architecture you need to sync data in real
> time using something like Hbase, Cassandra (?) and others as real time
> store or forever storage HDFS or Hive etc.
>
> In general there is also *Lambda Architecture* that is
> designed for streaming analytics. The streaming data ends up in both batch
> layer and speed layer. Batch layer is used to answer batch queries. On the
> other hand speed later is used ti handle fast/real time queries. This model
> is really cool as Spark Streaming can feed both the batch layer and
> the speed layer.
>
> At a high level this looks like this, from http://lambda-architecture.net/
>
> [image: Inline images 2]
>
>
>
>
>
> My favourite would be something like below with Spark playing a major role
>
>
>
>
>
> HTH
>
> Dr Mich Talebzadeh
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
> http://talebzadehmich.wordpress.com
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
> On 28 August 2016 at 19:43, Sivakumaran S <si...@me.com> wrote:
>
> Spark best fits for processing. But depending on the use case, you could
> expand the scope of Spark to moving data using the native connectors. The
> only that Spark is not, is Storage. Connectors are available for most
> storage options though.
>
> Regards,
>
> Sivakumaran S
>
>
>
> On 28-Aug-2016, at 6:04 PM, Ashok Kumar <ashok34668@yahoo.com.INVALID
> <as...@yahoo.com.invalid>> wrote:
>
> Hi,
>
> There are design patterns that use Spark extensively. I am new to this
> area so I would appreciate if someone explains where Spark fits in
> especially within faster or streaming use case.
>
> What are the best practices involving Spark. Is it always best to deploy
> it for processing engine,
>
> For example when we have a pattern
>
> Input Data -> Data in Motion -> Processing -> Storage
>
> Where does Spark best fit in.
>
> Thanking you
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
Re: Design patterns involving Spark
Posted by Ashok Kumar <as...@yahoo.com.INVALID>.
Hi,
Thank you for your explanations. Very fruitful.
Warmest
On Monday, 29 August 2016, 0:18, Mich Talebzadeh <mi...@gmail.com> wrote:
Hi,
In terms of positioning,Spark is really the first Big Data platform to integrate batch, streaming andinteractive computations in a unified framework. What this boils down to is thefact that whichever way one look at it there is somewhere that Spark can make acontribution to. In general, there are few design patterns common to Big Data
- ETL & Batch
Thefirst one is the most common one with Established tools like Sqoop, Talend forETL and HDFS for storage of some kind. Spark can be used as the executionengine for Hive at the storage level which actually makes it a true vendor independent(BTW, Impala and Tez and LLAP) are offered by vendors) processing engine.Personally I use Spark at ETL layer by extracting data from sources throughplug ins (JDBC and others) and storing in on HDFS in some kind
- Batch, real time plus Analytics
In thispattern you have data coming in real time and you want to query them real timethrough real time dashboard. HDFS is not ideal for updating data in real timeand neither for random access of data. Source could be all sorts of Web Serversand need Flume Agent with Flume. At the storage layer we are probably looking atsomething like Hbase. The crucial point being that saved data needs to be readyfor queries immediately The dashboards requires Hbase APIs. The Analytics canbe done through Hive again running on Spark engine. Again note here that weideally should process batch and real time separately.
- Real time / Streaming
This ismost relevant to Spark as we are moving to near real time. Where Spark excels.We need to capture the incoming events (logs, sensor data, pricing, emails) throughinterfaces like Kafka, Message Queues etc. Need to process these events with minimumlatency. Again Spark is a very good candidate here with its Spark Streaming andmicro-batching capabilities. There are others like Storm, Flink etc. that areevent based but you don’t hear much. Again for streaming architecture you needto sync data in real time using something like Hbase, Cassandra (?) and othersas real time store or forever storage HDFS or Hive etc. In general there is also Lambda Architecture that isdesigned for streaming analytics. The streaming data ends up in both batchlayer and speed layer. Batch layer is used to answer batch queries. On theother hand speed later is used ti handle fast/real time queries. This model isreally cool as Spark Streaming can feed both the batch layer and the speed layer. At a high level this lookslike this, from http://lambda-architecture.net/
My favourite would besomething like below with Spark playing a major role
HTH
Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destructionof data or any other property which may arise from relying on this email's technical content is explicitly disclaimed.The author will in no case be liable for any monetary damages arising from suchloss, damage or destruction.
On 28 August 2016 at 19:43, Sivakumaran S <si...@me.com> wrote:
Spark best fits for processing. But depending on the use case, you could expand the scope of Spark to moving data using the native connectors. The only that Spark is not, is Storage. Connectors are available for most storage options though.
Regards,
Sivakumaran S
On 28-Aug-2016, at 6:04 PM, Ashok Kumar <as...@yahoo.com.INVALID> wrote:
Hi,
There are design patterns that use Spark extensively. I am new to this area so I would appreciate if someone explains where Spark fits in especially within faster or streaming use case.
What are the best practices involving Spark. Is it always best to deploy it for processing engine,
For example when we have a pattern
Input Data -> Data in Motion -> Processing -> Storage
Where does Spark best fit in.
Thanking you
Re: Design patterns involving Spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi,
In terms of positioning, Spark is really the first Big Data platform to
integrate batch, streaming and interactive computations in a unified
framework. What this boils down to is the fact that whichever way one look
at it there is somewhere that Spark can make a contribution to. In general,
there are few design patterns common to Big Data
-
*ETL & Batch*
The first one is the most common one with Established tools like Sqoop,
Talend for ETL and HDFS for storage of some kind. Spark can be used as the
execution engine for Hive at the storage level which actually makes it a
true vendor independent (BTW, Impala and Tez and LLAP) are offered by
vendors) processing engine. Personally I use Spark at ETL layer by
extracting data from sources through plug ins (JDBC and others) and storing
in on HDFS in some kind
-
*Batch, real time plus Analytics*
In this pattern you have data coming in real time and you want to query
them real time through real time dashboard. HDFS is not ideal for updating
data in real time and neither for random access of data. Source could be
all sorts of Web Servers and need Flume Agent with Flume. At the storage
layer we are probably looking at something like Hbase. The crucial point
being that saved data needs to be ready for queries immediately The
dashboards requires Hbase APIs. The Analytics can be done through Hive
again running on Spark engine. Again note here that we ideally should
process batch and real time separately.
-
*Real time / Streaming*
This is most relevant to Spark as we are moving to near real time. Where
Spark excels. We need to capture the incoming events (logs, sensor data,
pricing, emails) through interfaces like Kafka, Message Queues etc. Need
to process these events with minimum latency. Again Spark is a very good
candidate here with its Spark Streaming and micro-batching capabilities.
There are others like Storm, Flink etc. that are event based but you don’t
hear much. Again for streaming architecture you need to sync data in real
time using something like Hbase, Cassandra (?) and others as real time
store or forever storage HDFS or Hive etc.
In general there is also *Lambda Architecture* that is designed
for streaming analytics. The streaming data ends up in both batch layer and
speed layer. Batch layer is used to answer batch queries. On the other hand
speed later is used ti handle fast/real time queries. This model is really
cool as Spark Streaming can feed both the batch layer and the speed layer.
At a high level this looks like this, from http://lambda-architecture.net/
[image: Inline images 2]
My favourite would be something like below with Spark playing a major role
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 28 August 2016 at 19:43, Sivakumaran S <si...@me.com> wrote:
> Spark best fits for processing. But depending on the use case, you could
> expand the scope of Spark to moving data using the native connectors. The
> only that Spark is not, is Storage. Connectors are available for most
> storage options though.
>
> Regards,
>
> Sivakumaran S
>
>
>
> On 28-Aug-2016, at 6:04 PM, Ashok Kumar <ashok34668@yahoo.com.INVALID
> <as...@yahoo.com.invalid>> wrote:
>
> Hi,
>
> There are design patterns that use Spark extensively. I am new to this
> area so I would appreciate if someone explains where Spark fits in
> especially within faster or streaming use case.
>
> What are the best practices involving Spark. Is it always best to deploy
> it for processing engine,
>
> For example when we have a pattern
>
> Input Data -> Data in Motion -> Processing -> Storage
>
> Where does Spark best fit in.
>
> Thanking you
>
>
Re: Design patterns involving Spark
Posted by Sivakumaran S <si...@me.com>.
Spark best fits for processing. But depending on the use case, you could expand the scope of Spark to moving data using the native connectors. The only that Spark is not, is Storage. Connectors are available for most storage options though.
Regards,
Sivakumaran S
> On 28-Aug-2016, at 6:04 PM, Ashok Kumar <as...@yahoo.com.INVALID> wrote:
>
> Hi,
>
> There are design patterns that use Spark extensively. I am new to this area so I would appreciate if someone explains where Spark fits in especially within faster or streaming use case.
>
> What are the best practices involving Spark. Is it always best to deploy it for processing engine,
>
> For example when we have a pattern
>
> Input Data -> Data in Motion -> Processing -> Storage
>
> Where does Spark best fit in.
>
> Thanking you