You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Joanne Contact <jo...@gmail.com> on 2015/08/18 10:57:15 UTC
global variable in spark streaming with no dependency on key
Hi Gurus,
Please help.
But please don't tell me to use updateStateByKey because I need a
global variable (something like the clock time) across the micro
batches but not depending on key. For my case, it is not acceptable to
maintain a state for each key since each key comes in different times.
Yes my global variable is related to time but cannot use machine
clock.
Any hint? Or is this lack of global variable by design?
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: global variable in spark streaming with no dependency on key
Posted by Joanne Contact <jo...@gmail.com>.
Thanks. I tried. The problem is I have to updateStatebyKey to maintain
other states related to keys.
Not sure where to pass this accumulator variable into updateStateBykey.
On Tue, Aug 18, 2015 at 2:17 AM, Hemant Bhanawat <he...@gmail.com> wrote:
> See if SparkContext.accumulator helps.
>
> On Tue, Aug 18, 2015 at 2:27 PM, Joanne Contact <jo...@gmail.com>
> wrote:
>>
>> Hi Gurus,
>>
>> Please help.
>>
>> But please don't tell me to use updateStateByKey because I need a
>> global variable (something like the clock time) across the micro
>> batches but not depending on key. For my case, it is not acceptable to
>> maintain a state for each key since each key comes in different times.
>> Yes my global variable is related to time but cannot use machine
>> clock.
>>
>> Any hint? Or is this lack of global variable by design?
>>
>> Thanks!
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: global variable in spark streaming with no dependency on key
Posted by Hemant Bhanawat <he...@gmail.com>.
See if SparkContext.accumulator helps.
On Tue, Aug 18, 2015 at 2:27 PM, Joanne Contact <jo...@gmail.com>
wrote:
> Hi Gurus,
>
> Please help.
>
> But please don't tell me to use updateStateByKey because I need a
> global variable (something like the clock time) across the micro
> batches but not depending on key. For my case, it is not acceptable to
> maintain a state for each key since each key comes in different times.
> Yes my global variable is related to time but cannot use machine
> clock.
>
> Any hint? Or is this lack of global variable by design?
>
> Thanks!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>