Is it possible to create a global variable for every executors of same bolt

[#ZHANG SHUHAO# <SZ...@e.ntu.edu.sg>]

Hi,

Basically, it's the standard word-count topology, 1 spout, 1 split, 1 count, and 1 sink.

Now, suppose I give the parallelism hint to each of them as 1:2:1:1. Hence, there are two executors for split are created. My questions is how can I maintain a global shared counter for both of the two threads of split.

Is it possible to do that in storm? I have tried some methods e.g., AtomicInteger, but later I realized that the use of serlization makes the contents of every thread instance of split isolated.

Is there any way to work-around of it?

Many thanks.

Re: Is it possible to create a global variable for every executors of same bolt

["Matthias J. Sax" <mj...@informatik.hu-berlin.de>]

I would not do it this way... If you don't provide a custom scheduler,
you don't know if both executors will be deployed to the same worker JVM
(actually, the changes are almost zero that this happens...).

(Furthermore, you need to do proper synchronization between both
executors accessing the same variable.)

In general, a global shared counter does not scale! I would use
something like this with high caution. If you really need it, you could
have an additional bolt (with dop=1), that consumers to local counts and
computes the global count. Just declare a second output stream in split
bolt and emit the local counts into it.

-Matthias

On 08/31/2015 04:51 PM, Denis DEBARBIEUX wrote:
> Hi,
> 
> Are the 2 executors on the same worker (ie in the same JVM)?
> 
> If yes, your question should be fix a static field.
> 
> If no, I have no simple idea.
> 
> Denis
> 
> Le 31/08/2015 16:28, #ZHANG SHUHAO# a écrit :
>>
>> Hi,
>>
>>  
>>
>> Basically, it’s the standard word-count topology, 1 spout, 1 split, 1
>> count, and 1 sink.
>>
>>  
>>
>> Now, suppose I give the parallelism hint to each of them as 1:2:1:1.
>> Hence, there are two executors for split are created. My questions is
>> how can I maintain _a global shared counter_ for both of the two
>> threads of split.
>>
>>  
>>
>> Is it possible to do that in storm? I have tried some methods e.g.,
>> AtomicInteger, but later I realized that the use of serlization makes
>> the contents of every thread instance of split isolated.
>>
>>  
>>
>> Is there any way to work-around of it?
>>
>>  
>>
>> Many thanks.
>>
>>  
>>
> 
> 
> 
> ------------------------------------------------------------------------
> Avast logo <https://www.avast.com/antivirus> 	
> 
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> www.avast.com <https://www.avast.com/antivirus>
> 
> 

Re: Is it possible to create a global variable for every executors of same bolt

[Denis DEBARBIEUX <dd...@norsys.fr>]

Hi,

Are the 2 executors on the same worker (ie in the same JVM)?

If yes, your question should be fix a static field.

If no, I have no simple idea.

Denis

Le 31/08/2015 16:28, #ZHANG SHUHAO# a écrit :
>
> Hi,
>
> Basically, it’s the standard word-count topology, 1 spout, 1 split, 1 
> count, and 1 sink.
>
> Now, suppose I give the parallelism hint to each of them as 1:2:1:1. 
> Hence, there are two executors for split are created. My questions is 
> how can I maintain _a global shared counter_ for both of the two 
> threads of split.
>
> Is it possible to do that in storm? I have tried some methods e.g., 
> AtomicInteger, but later I realized that the use of serlization makes 
> the contents of every thread instance of split isolated.
>
> Is there any way to work-around of it?
>
> Many thanks.
>



---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus