You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by anna stax <an...@gmail.com> on 2017/10/18 19:52:49 UTC
Spark streaming for CEP
Hello all,
Has anyone used spark streaming for CEP (Complex Event processing). Any
CEP libraries that works well with spark. I have a use case for CEP and
trying to see if spark streaming is a good fit.
Currently we have a data pipeline using Kafka, Spark streaming and
Cassandra for data ingestion and near real time dashboard.
Please share your experience.
Thanks much.
-Anna
Re: Spark streaming for CEP
Posted by anna stax <an...@gmail.com>.
Thanks very much Mich, Thomas and Stephan . I will look into it.
On Tue, Oct 24, 2017 at 8:02 PM, lucas.gary@gmail.com <lu...@gmail.com>
wrote:
> This looks really interesting, thanks for linking!
>
> Gary Lucas
>
> On 24 October 2017 at 15:06, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Great thanks Steve
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 24 October 2017 at 22:58, Stephen Boesch <ja...@gmail.com> wrote:
>>
>>> Hi Mich, the github link has a brief intro - including a link to the
>>> formal docs http://logisland.readthedocs.io/en/latest/index.html .
>>> They have an architectural overview, developer guide, tutorial, and pretty
>>> comprehensive api docs.
>>>
>>> 2017-10-24 13:31 GMT-07:00 Mich Talebzadeh <mi...@gmail.com>:
>>>
>>>> thanks Thomas.
>>>>
>>>> do you have a summary write-up for this tool please?
>>>>
>>>>
>>>> regards,
>>>>
>>>>
>>>>
>>>>
>>>> Thomas
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>> On 24 October 2017 at 13:53, Thomas Bailet <th...@hurence.com>
>>>> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> we (@ hurence) have released on open source middleware based on
>>>>> SparkStreaming over Kafka to do CEP and log mining, called *logisland*
>>>>> (https://github.com/Hurence/logisland/) it has been deployed into
>>>>> production for 2 years now and does a great job. You should have a look.
>>>>>
>>>>>
>>>>> bye
>>>>>
>>>>> Thomas Bailet
>>>>>
>>>>> CTO : hurence
>>>>>
>>>>> Le 18/10/17 à 22:05, Mich Talebzadeh a écrit :
>>>>>
>>>>> As you may be aware the granularity that Spark streaming has is
>>>>> micro-batching and that is limited to 0.5 second. So if you have continuous
>>>>> ingestion of data then Spark streaming may not be granular enough for CEP.
>>>>> You may consider other products.
>>>>>
>>>>> Worth looking at this old thread on mine "Spark support for Complex
>>>>> Event Processing (CEP)
>>>>>
>>>>> https://mail-archives.apache.org/mod_mbox/spark-user/201604.
>>>>> mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=LbQ@
>>>>> mail.gmail.com%3E
>>>>>
>>>>> HTH
>>>>>
>>>>>
>>>>> Dr Mich Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>
>>>>>
>>>>>
>>>>> http://talebzadehmich.wordpress.com
>>>>>
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>> arising from such loss, damage or destruction.
>>>>>
>>>>>
>>>>>
>>>>> On 18 October 2017 at 20:52, anna stax <an...@gmail.com> wrote:
>>>>>
>>>>>> Hello all,
>>>>>>
>>>>>> Has anyone used spark streaming for CEP (Complex Event processing).
>>>>>> Any CEP libraries that works well with spark. I have a use case for CEP and
>>>>>> trying to see if spark streaming is a good fit.
>>>>>>
>>>>>> Currently we have a data pipeline using Kafka, Spark streaming and
>>>>>> Cassandra for data ingestion and near real time dashboard.
>>>>>>
>>>>>> Please share your experience.
>>>>>> Thanks much.
>>>>>> -Anna
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Spark streaming for CEP
Posted by "lucas.gary@gmail.com" <lu...@gmail.com>.
This looks really interesting, thanks for linking!
Gary Lucas
On 24 October 2017 at 15:06, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Great thanks Steve
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 24 October 2017 at 22:58, Stephen Boesch <ja...@gmail.com> wrote:
>
>> Hi Mich, the github link has a brief intro - including a link to the
>> formal docs http://logisland.readthedocs.io/en/latest/index.html .
>> They have an architectural overview, developer guide, tutorial, and pretty
>> comprehensive api docs.
>>
>> 2017-10-24 13:31 GMT-07:00 Mich Talebzadeh <mi...@gmail.com>:
>>
>>> thanks Thomas.
>>>
>>> do you have a summary write-up for this tool please?
>>>
>>>
>>> regards,
>>>
>>>
>>>
>>>
>>> Thomas
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 24 October 2017 at 13:53, Thomas Bailet <th...@hurence.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> we (@ hurence) have released on open source middleware based on
>>>> SparkStreaming over Kafka to do CEP and log mining, called *logisland*
>>>> (https://github.com/Hurence/logisland/) it has been deployed into
>>>> production for 2 years now and does a great job. You should have a look.
>>>>
>>>>
>>>> bye
>>>>
>>>> Thomas Bailet
>>>>
>>>> CTO : hurence
>>>>
>>>> Le 18/10/17 à 22:05, Mich Talebzadeh a écrit :
>>>>
>>>> As you may be aware the granularity that Spark streaming has is
>>>> micro-batching and that is limited to 0.5 second. So if you have continuous
>>>> ingestion of data then Spark streaming may not be granular enough for CEP.
>>>> You may consider other products.
>>>>
>>>> Worth looking at this old thread on mine "Spark support for Complex
>>>> Event Processing (CEP)
>>>>
>>>> https://mail-archives.apache.org/mod_mbox/spark-user/201604.
>>>> mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=LbQ@
>>>> mail.gmail.com%3E
>>>>
>>>> HTH
>>>>
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>> On 18 October 2017 at 20:52, anna stax <an...@gmail.com> wrote:
>>>>
>>>>> Hello all,
>>>>>
>>>>> Has anyone used spark streaming for CEP (Complex Event processing).
>>>>> Any CEP libraries that works well with spark. I have a use case for CEP and
>>>>> trying to see if spark streaming is a good fit.
>>>>>
>>>>> Currently we have a data pipeline using Kafka, Spark streaming and
>>>>> Cassandra for data ingestion and near real time dashboard.
>>>>>
>>>>> Please share your experience.
>>>>> Thanks much.
>>>>> -Anna
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Spark streaming for CEP
Posted by Mich Talebzadeh <mi...@gmail.com>.
Great thanks Steve
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 24 October 2017 at 22:58, Stephen Boesch <ja...@gmail.com> wrote:
> Hi Mich, the github link has a brief intro - including a link to the
> formal docs http://logisland.readthedocs.io/en/latest/index.html . They
> have an architectural overview, developer guide, tutorial, and pretty
> comprehensive api docs.
>
> 2017-10-24 13:31 GMT-07:00 Mich Talebzadeh <mi...@gmail.com>:
>
>> thanks Thomas.
>>
>> do you have a summary write-up for this tool please?
>>
>>
>> regards,
>>
>>
>>
>>
>> Thomas
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 24 October 2017 at 13:53, Thomas Bailet <th...@hurence.com>
>> wrote:
>>
>>> Hi
>>>
>>> we (@ hurence) have released on open source middleware based on
>>> SparkStreaming over Kafka to do CEP and log mining, called *logisland*
>>> (https://github.com/Hurence/logisland/) it has been deployed into
>>> production for 2 years now and does a great job. You should have a look.
>>>
>>>
>>> bye
>>>
>>> Thomas Bailet
>>>
>>> CTO : hurence
>>>
>>> Le 18/10/17 à 22:05, Mich Talebzadeh a écrit :
>>>
>>> As you may be aware the granularity that Spark streaming has is
>>> micro-batching and that is limited to 0.5 second. So if you have continuous
>>> ingestion of data then Spark streaming may not be granular enough for CEP.
>>> You may consider other products.
>>>
>>> Worth looking at this old thread on mine "Spark support for Complex
>>> Event Processing (CEP)
>>>
>>> https://mail-archives.apache.org/mod_mbox/spark-user/201604.
>>> mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=LbQ@
>>> mail.gmail.com%3E
>>>
>>> HTH
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 18 October 2017 at 20:52, anna stax <an...@gmail.com> wrote:
>>>
>>>> Hello all,
>>>>
>>>> Has anyone used spark streaming for CEP (Complex Event processing).
>>>> Any CEP libraries that works well with spark. I have a use case for CEP and
>>>> trying to see if spark streaming is a good fit.
>>>>
>>>> Currently we have a data pipeline using Kafka, Spark streaming and
>>>> Cassandra for data ingestion and near real time dashboard.
>>>>
>>>> Please share your experience.
>>>> Thanks much.
>>>> -Anna
>>>>
>>>>
>>>>
>>>
>>>
>>
>
Re: Spark streaming for CEP
Posted by Stephen Boesch <ja...@gmail.com>.
Hi Mich, the github link has a brief intro - including a link to the formal
docs http://logisland.readthedocs.io/en/latest/index.html . They have an
architectural overview, developer guide, tutorial, and pretty comprehensive
api docs.
2017-10-24 13:31 GMT-07:00 Mich Talebzadeh <mi...@gmail.com>:
> thanks Thomas.
>
> do you have a summary write-up for this tool please?
>
>
> regards,
>
>
>
>
> Thomas
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 24 October 2017 at 13:53, Thomas Bailet <th...@hurence.com>
> wrote:
>
>> Hi
>>
>> we (@ hurence) have released on open source middleware based on
>> SparkStreaming over Kafka to do CEP and log mining, called *logisland* (
>> https://github.com/Hurence/logisland/) it has been deployed into
>> production for 2 years now and does a great job. You should have a look.
>>
>>
>> bye
>>
>> Thomas Bailet
>>
>> CTO : hurence
>>
>> Le 18/10/17 à 22:05, Mich Talebzadeh a écrit :
>>
>> As you may be aware the granularity that Spark streaming has is
>> micro-batching and that is limited to 0.5 second. So if you have continuous
>> ingestion of data then Spark streaming may not be granular enough for CEP.
>> You may consider other products.
>>
>> Worth looking at this old thread on mine "Spark support for Complex Event
>> Processing (CEP)
>>
>> https://mail-archives.apache.org/mod_mbox/spark-user/201604.
>> mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=LbQ@
>> mail.gmail.com%3E
>>
>> HTH
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 18 October 2017 at 20:52, anna stax <an...@gmail.com> wrote:
>>
>>> Hello all,
>>>
>>> Has anyone used spark streaming for CEP (Complex Event processing). Any
>>> CEP libraries that works well with spark. I have a use case for CEP and
>>> trying to see if spark streaming is a good fit.
>>>
>>> Currently we have a data pipeline using Kafka, Spark streaming and
>>> Cassandra for data ingestion and near real time dashboard.
>>>
>>> Please share your experience.
>>> Thanks much.
>>> -Anna
>>>
>>>
>>>
>>
>>
>
Re: Spark streaming for CEP
Posted by Mich Talebzadeh <mi...@gmail.com>.
thanks Thomas.
do you have a summary write-up for this tool please?
regards,
Thomas
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 24 October 2017 at 13:53, Thomas Bailet <th...@hurence.com>
wrote:
> Hi
>
> we (@ hurence) have released on open source middleware based on
> SparkStreaming over Kafka to do CEP and log mining, called *logisland* (
> https://github.com/Hurence/logisland/) it has been deployed into
> production for 2 years now and does a great job. You should have a look.
>
>
> bye
>
> Thomas Bailet
>
> CTO : hurence
>
> Le 18/10/17 à 22:05, Mich Talebzadeh a écrit :
>
> As you may be aware the granularity that Spark streaming has is
> micro-batching and that is limited to 0.5 second. So if you have continuous
> ingestion of data then Spark streaming may not be granular enough for CEP.
> You may consider other products.
>
> Worth looking at this old thread on mine "Spark support for Complex Event
> Processing (CEP)
>
> https://mail-archives.apache.org/mod_mbox/spark-user/201604.mbox/%
> 3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=LbQ@mail.gmail.com%3E
>
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 18 October 2017 at 20:52, anna stax <an...@gmail.com> wrote:
>
>> Hello all,
>>
>> Has anyone used spark streaming for CEP (Complex Event processing). Any
>> CEP libraries that works well with spark. I have a use case for CEP and
>> trying to see if spark streaming is a good fit.
>>
>> Currently we have a data pipeline using Kafka, Spark streaming and
>> Cassandra for data ingestion and near real time dashboard.
>>
>> Please share your experience.
>> Thanks much.
>> -Anna
>>
>>
>>
>
>
Re: Spark streaming for CEP
Posted by Thomas Bailet <th...@hurence.com>.
Hi
we (@ hurence) have released on open source middleware based on
SparkStreaming over Kafka to do CEP and log mining, called *logisland*
(https://github.com/Hurence/logisland/) it has been deployed into
production for 2 years now and does a great job. You should have a look.
bye
Thomas Bailet
CTO : hurence
Le 18/10/17 à 22:05, Mich Talebzadeh a écrit :
> As you may be aware the granularity that Spark streaming has is
> micro-batching and that is limited to 0.5 second. So if you have
> continuous ingestion of data then Spark streaming may not be granular
> enough for CEP. You may consider other products.
>
> Worth looking at this old thread on mine "Spark support for Complex
> Event Processing (CEP)
>
> https://mail-archives.apache.org/mod_mbox/spark-user/201604.mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=LbQ@mail.gmail.com%3E
>
> HTH
>
>
> Dr Mich Talebzadeh
>
> LinkedIn
> /https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw/
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk.Any and all responsibility for
> any loss, damage or destruction of data or any other property which
> may arise from relying on this email's technical content is explicitly
> disclaimed. The author will in no case be liable for any monetary
> damages arising from such loss, damage or destruction.
>
>
> On 18 October 2017 at 20:52, anna stax <annastax80@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hello all,
>
> Has anyone used spark streaming for CEP (Complex Event
> processing). Any CEP libraries that works well with spark. I have
> a use case for CEP and trying to see if spark streaming is a good
> fit.
>
> Currently we have a data pipeline using Kafka, Spark streaming and
> Cassandra for data ingestion and near real time dashboard.
>
> Please share your experience.
> Thanks much.
> -Anna
>
>
>
Re: Spark streaming for CEP
Posted by Mich Talebzadeh <mi...@gmail.com>.
As you may be aware the granularity that Spark streaming has is
micro-batching and that is limited to 0.5 second. So if you have continuous
ingestion of data then Spark streaming may not be granular enough for CEP.
You may consider other products.
Worth looking at this old thread on mine "Spark support for Complex Event
Processing (CEP)
https://mail-archives.apache.org/mod_mbox/spark-user/201604.mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=LbQ@mail.gmail.com%3E
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 18 October 2017 at 20:52, anna stax <an...@gmail.com> wrote:
> Hello all,
>
> Has anyone used spark streaming for CEP (Complex Event processing). Any
> CEP libraries that works well with spark. I have a use case for CEP and
> trying to see if spark streaming is a good fit.
>
> Currently we have a data pipeline using Kafka, Spark streaming and
> Cassandra for data ingestion and near real time dashboard.
>
> Please share your experience.
> Thanks much.
> -Anna
>
>
>