You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shyam P <sh...@gmail.com> on 2019/04/05 09:50:59 UTC
Is there any spark API function to handle a group of companies at
once in this scenario?
Hi ,
In my scenario I have few companies , for which I need to calculate few
stats like avg I need to be stored in Cassandra , for next set of records I
need to get previously calculated and over it i need to calculate
accumulated results ( i.e preset set of data + previously stored stats) and
stored it back to Cassandra.
what function/API of spark be used while calculating the above for a group
of companies?
Regards,
Shyam
Checking if cascading graph computation is possible in Spark
Posted by Basavaraj <ra...@gmail.com>.
Hi
Have two questions
#1
I am trying to process events in realtime, outcome of the processing has to find a node in the GraphX and update that node as well (in case if any anomaly or state change), If a node is updated, I have to update the related nodes as well, want to know if GraphX can help in this by providing some native support
#2
I want to do the above as a event driven way, without using the batches (i tried micro batches, but I realised that’s not what I want), i.e., for each arriving event or as soon as a event message come my stream, not by accumulating the event
I humbly welcome any pointers, constructive criticism
Regards
Basav
Re: Checking if cascading graph computation is possible in Spark
Posted by Jason Nerothin <ja...@gmail.com>.
*I guess I was focusing on this:*
#2
I want to do the above as a event driven way, *without using the batches*
(i tried micro batches, but I realised that’s not what I want), i.e., *for
each arriving event or as soon as a event message come my stream, not by
accumulating the event *
If you want to update your graph without pulling the older data back
through the entire DAG, it seems like you need to store the graph data
somewhere. So that's why I jumped to accumulators - the state would be
around from event to event, and not require a "reaggregation" for each
event.
Arbitrary stateful streaming has this ability "built in" - that is, the
engine stores your intermediate results in RAM and then the next event
picks up where the last one left off.
I've just implemented the arbitrary stateful streaming option... Couldn't
figure out a better way of avoiding the re-shuffle, so ended up keeping the
prior state in the engine.
I'm not using GraphX, but it seems like the approach should work
irrespective - there's an interface called GroupState that you hand off an
iterator for from call to call.
Do keep in mind that you have to think about out of order event arrivals...
Send me a message to my direct email and I'll provide a link to the
source... Not sure I'm fully grokking your entire use case...
On Fri, Apr 5, 2019 at 1:15 PM Basavaraj <ra...@gmail.com> wrote:
> I have checked broadcast of accumulated values, but not satellite stateful
> stabbing
>
> But, I am not sure how that helps here
>
> On Fri, 5 Apr 2019, 10:13 pm Jason Nerothin, <ja...@gmail.com>
> wrote:
>
>> Have you looked at Arbitrary Stateful Streaming and Broadcast
>> Accumulators?
>>
>> On Fri, Apr 5, 2019 at 10:55 AM Basavaraj <ra...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> Have two questions
>>>
>>> #1
>>> I am trying to process events in realtime, outcome of the processing has
>>> to find a node in the GraphX and update that node as well (in case if any
>>> anomaly or state change), If a node is updated, I have to update the
>>> related nodes as well, want to know if GraphX can help in this by providing
>>> some native support
>>>
>>> #2
>>> I want to do the above as a event driven way, without using the batches
>>> (i tried micro batches, but I realised that’s not what I want), i.e., for
>>> each arriving event or as soon as a event message come my stream, not by
>>> accumulating the event
>>>
>>> I humbly welcome any pointers, constructive criticism
>>>
>>> Regards
>>> Basav
>>> --------------------------------------------------------------------- To
>>> unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>
>>
>> --
>> Thanks,
>> Jason
>>
>
--
Thanks,
Jason
Re: Checking if cascading graph computation is possible in Spark
Posted by Basavaraj <ra...@gmail.com>.
I have checked broadcast of accumulated values, but not satellite stateful
stabbing
But, I am not sure how that helps here
On Fri, 5 Apr 2019, 10:13 pm Jason Nerothin, <ja...@gmail.com>
wrote:
> Have you looked at Arbitrary Stateful Streaming and Broadcast Accumulators?
>
> On Fri, Apr 5, 2019 at 10:55 AM Basavaraj <ra...@gmail.com> wrote:
>
>> Hi
>>
>> Have two questions
>>
>> #1
>> I am trying to process events in realtime, outcome of the processing has
>> to find a node in the GraphX and update that node as well (in case if any
>> anomaly or state change), If a node is updated, I have to update the
>> related nodes as well, want to know if GraphX can help in this by providing
>> some native support
>>
>> #2
>> I want to do the above as a event driven way, without using the batches
>> (i tried micro batches, but I realised that’s not what I want), i.e., for
>> each arriving event or as soon as a event message come my stream, not by
>> accumulating the event
>>
>> I humbly welcome any pointers, constructive criticism
>>
>> Regards
>> Basav
>> --------------------------------------------------------------------- To
>> unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
>
> --
> Thanks,
> Jason
>
Re: Checking if cascading graph computation is possible in Spark
Posted by Jason Nerothin <ja...@gmail.com>.
Have you looked at Arbitrary Stateful Streaming and Broadcast Accumulators?
On Fri, Apr 5, 2019 at 10:55 AM Basavaraj <ra...@gmail.com> wrote:
> Hi
>
> Have two questions
>
> #1
> I am trying to process events in realtime, outcome of the processing has
> to find a node in the GraphX and update that node as well (in case if any
> anomaly or state change), If a node is updated, I have to update the
> related nodes as well, want to know if GraphX can help in this by providing
> some native support
>
> #2
> I want to do the above as a event driven way, without using the batches (i
> tried micro batches, but I realised that’s not what I want), i.e., for each
> arriving event or as soon as a event message come my stream, not by
> accumulating the event
>
> I humbly welcome any pointers, constructive criticism
>
> Regards
> Basav
> --------------------------------------------------------------------- To
> unsubscribe e-mail: user-unsubscribe@spark.apache.org
--
Thanks,
Jason
Checking if cascading graph computation is possible in Spark
Posted by Basavaraj <ra...@gmail.com>.
Hi
Have two questions
#1
I am trying to process events in realtime, outcome of the processing has to
find a node in the GraphX and update that node as well (in case if any anomaly
or state change), If a node is updated, I have to update the related nodes as
well, want to know if GraphX can help in this by providing some native support
#2
I want to do the above as a event driven way, without using the batches (i
tried micro batches, but I realised that’s not what I want), i.e., for each
arriving event or as soon as a event message come my stream, not by
accumulating the event
I humbly welcome any pointers, constructive criticism
Regards
Basav
\--------------------------------------------------------------------- To
unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Is there any spark API function to handle a group of companies at
once in this scenario?
Posted by Shyam P <sh...@gmail.com>.
Hi Mich,
Sorry sorry for late reply.
Yes this flow is near except HBase and Flume. Why do we need flume when we
use spark streaming?
You did not address my base question yet.
Regards
Shyam
On Tue, 9 Apr 2019, 22:08 Mich Talebzadeh, <mi...@gmail.com>
wrote:
> Fine, how do you store these Kafka tropics? Are they loaded into HDFS via
> Kafka --> Flume --> HDFS?
>
> Look at this simple diagram below. Replace MongoDB with Cassandra. Is this
> what you are trying to do from an architecture point of view?
>
>
> [image: image.png]
>
> HTH,
>
> Nuch
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Mon, 8 Apr 2019 at 08:19, Shyam P <sh...@gmail.com> wrote:
>
>> Hi Mich,
>> thanks for your prompt reply.
>> I get few company financial data like profits and etc results .
>> I would get this company data through Kafka topics which is fed by an
>> rest service.
>> I am thinking of using spark-structured streaming.
>> Put them back in HIVE/C*.
>>
>> Regards,
>> Shyam
>>
>>
>>
>> On Sun, Apr 7, 2019 at 2:04 PM Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> Are these ticker prices for these companies like share value etc?
>>>
>>> How do you get this company data in Spark? Are you using Spark streaming
>>> to get the prices, then work out the stats (AVG, STDDEV etc) and put them
>>> back into DB?
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Fri, 5 Apr 2019 at 10:51, Shyam P <sh...@gmail.com> wrote:
>>>
>>>> Hi ,
>>>> In my scenario I have few companies , for which I need to calculate few
>>>> stats like avg I need to be stored in Cassandra , for next set of records I
>>>> need to get previously calculated and over it i need to calculate
>>>> accumulated results ( i.e preset set of data + previously stored stats) and
>>>> stored it back to Cassandra.
>>>>
>>>> what function/API of spark be used while calculating the above for a
>>>> group of companies?
>>>>
>>>>
>>>> Regards,
>>>> Shyam
>>>>
>>>
Re: Is there any spark API function to handle a group of companies at
once in this scenario?
Posted by Mich Talebzadeh <mi...@gmail.com>.
Fine, how do you store these Kafka tropics? Are they loaded into HDFS via
Kafka --> Flume --> HDFS?
Look at this simple diagram below. Replace MongoDB with Cassandra. Is this
what you are trying to do from an architecture point of view?
[image: image.png]
HTH,
Nuch
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Mon, 8 Apr 2019 at 08:19, Shyam P <sh...@gmail.com> wrote:
> Hi Mich,
> thanks for your prompt reply.
> I get few company financial data like profits and etc results .
> I would get this company data through Kafka topics which is fed by an rest
> service.
> I am thinking of using spark-structured streaming.
> Put them back in HIVE/C*.
>
> Regards,
> Shyam
>
>
>
> On Sun, Apr 7, 2019 at 2:04 PM Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Are these ticker prices for these companies like share value etc?
>>
>> How do you get this company data in Spark? Are you using Spark streaming
>> to get the prices, then work out the stats (AVG, STDDEV etc) and put them
>> back into DB?
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Fri, 5 Apr 2019 at 10:51, Shyam P <sh...@gmail.com> wrote:
>>
>>> Hi ,
>>> In my scenario I have few companies , for which I need to calculate few
>>> stats like avg I need to be stored in Cassandra , for next set of records I
>>> need to get previously calculated and over it i need to calculate
>>> accumulated results ( i.e preset set of data + previously stored stats) and
>>> stored it back to Cassandra.
>>>
>>> what function/API of spark be used while calculating the above for a
>>> group of companies?
>>>
>>>
>>> Regards,
>>> Shyam
>>>
>>
Re: Is there any spark API function to handle a group of companies at
once in this scenario?
Posted by Shyam P <sh...@gmail.com>.
Hi Mich,
thanks for your prompt reply.
I get few company financial data like profits and etc results .
I would get this company data through Kafka topics which is fed by an rest
service.
I am thinking of using spark-structured streaming.
Put them back in HIVE/C*.
Regards,
Shyam
On Sun, Apr 7, 2019 at 2:04 PM Mich Talebzadeh <mi...@gmail.com>
wrote:
> Are these ticker prices for these companies like share value etc?
>
> How do you get this company data in Spark? Are you using Spark streaming
> to get the prices, then work out the stats (AVG, STDDEV etc) and put them
> back into DB?
>
> HTH
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Fri, 5 Apr 2019 at 10:51, Shyam P <sh...@gmail.com> wrote:
>
>> Hi ,
>> In my scenario I have few companies , for which I need to calculate few
>> stats like avg I need to be stored in Cassandra , for next set of records I
>> need to get previously calculated and over it i need to calculate
>> accumulated results ( i.e preset set of data + previously stored stats) and
>> stored it back to Cassandra.
>>
>> what function/API of spark be used while calculating the above for a
>> group of companies?
>>
>>
>> Regards,
>> Shyam
>>
>
Re: Is there any spark API function to handle a group of companies at
once in this scenario?
Posted by Mich Talebzadeh <mi...@gmail.com>.
Are these ticker prices for these companies like share value etc?
How do you get this company data in Spark? Are you using Spark streaming to
get the prices, then work out the stats (AVG, STDDEV etc) and put them back
into DB?
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Fri, 5 Apr 2019 at 10:51, Shyam P <sh...@gmail.com> wrote:
> Hi ,
> In my scenario I have few companies , for which I need to calculate few
> stats like avg I need to be stored in Cassandra , for next set of records I
> need to get previously calculated and over it i need to calculate
> accumulated results ( i.e preset set of data + previously stored stats) and
> stored it back to Cassandra.
>
> what function/API of spark be used while calculating the above for a group
> of companies?
>
>
> Regards,
> Shyam
>