You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Huy Le Van <hu...@insight-centre.org> on 2015/05/06 02:17:17 UTC
Re: Shared static data between bolts
Could someone help me please?
On Thursday, Apr 30, 2015 at 4:27 a.m., Huy Le Van <hu...@insight-centre.org>, wrote:
Hi Michael,
If a bolt of type A on a JVM modifies the shared data, will another bolt of type B on another JVM see the changes? To be clear, I have a use-case where I need multiple bolts of multiple types to be able to read/write on the shared collection. You can think of it as a shared database. In this case, can I still use this solution or do I need to use Hazelcast/Redis/Memcached?
Best regards,
Huy, Le Van
On Thursday, Dec 4, 2014 at 8:53 p.m., Michael Rose <mi...@fullcontact.com>, wrote:
Yes, as long as they're in the same JVM. If you have 4 workers, that's 4 JVMs, and 4 copies of the data.
Storm will ensure that if you have 4 bolts of type A, and 8 of type B, that 1 of type A and 2 of type B exist in each JVM. Since the singleton's field is static, it'll be shared.
Michael Rose (@Xorlev)
Senior Platform Engineer, FullContact
michael@fullcontact.com
On Thu, Dec 4, 2014 at 1:34 PM, Eyal Golan <eg...@gmail.com> wrote:
Michael,
I understand your solution, however, if bolt of type A "wins" the initialization, will bolts of type B see the same instance?
Sent from my Android phone
On Dec 4, 2014 10:26 PM, "Eyal Golan" <eg...@gmail.com> wrote:
Thanks Michael and Ben !
Sent from my Android phone
On Dec 4, 2014 9:53 PM, "Flint, Ben" <bf...@popcap.com> wrote:
I’ve used Hazelcast for use cases like this with great success. It makes it relatively straightforward to implement a distributed data structure.
From: Eyal Golan <eg...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Thursday, December 4, 2014 at 11:40 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: Shared static data between bolts
Thanks!
All the answers give me good idea.
Here's why I need this list be sharable.
One bolt needs the list for computation.
Another bolt will periodically update the list.
In other words, the list is changeable.
I tried passing it in conf but there are some 3rd party libs in the wrapper.
Anyway, I also need to improve the topology. Separate bolt to two etc. That way, it would be easier to handle that cache.
One more question, I thought about giving hazelcast a try.
Create local node and then the bolts will have clients.
What do you think?
Sent from my Android phone
On Dec 4, 2014 9:14 PM, "Kushan Maskey" <ku...@mmillerassociates.com> wrote:
Hi Eyal,
I had the similar issue earlier. Static objects are not a good way of passing the variables around in Storm due to clustered nature of the Storm. You will see null for all the static variables.
This is how I implemented mine, for instance I wanted to pass around the server properties such and host name and username/password. I read the properties file and then created a map of these properties then set it into the Config object. Like below.
In topology:
Map<Stirng, Stirng> map = new HashMap();
map.put("hostname", "localhost");
map.put("username", "testuser");
map.put("password", "testpassord");
map.put("port", "1234");
Config config = new Config();
config.put("MY_MAP", map)
In Bolts:
public void prepare(Map stormConf, TopologyContext context) {
HashMap<Stirng, Stirng> map = stormConfig.get("MY_MAP");
}
This how i resolved my problem.Try that if that helps.
--
Kushan Maskey
817.403.7500
M. Miller & Associates
kushan.maskey@mmillerassociates.com
On Thu, Dec 4, 2014 at 1:01 PM, Eyal Golan
<eg...@gmail.com> wrote:
Hi all,
I am newbie with Storm and having some issues with static objects in the topology.
In the main class, where I set the topology, I also set a static object. It's actually a wrapper for a collection of strings.
This object needs to he shared between different bolts.
It's big so I want it to be only one instance.
The problem is that when a bolt tries to get this static instance: example - MyMain.Shared... It's null. Although I instantiated it in the main method.
I guess I'm doing something wrong.
Thanks for any assistance.
Eyal
Sent from my Android phone
Re: Shared static data between bolts
Posted by Huy Le Van <hu...@insight-centre.org>.
Hi,Thanks Supun and Devang. After a little research, I think I’ll give Redis 3.0 a try.
Best regards,
Huy, Le Van
On Wednesday, May 6, 2015 at 2:40 a.m., Devang Shah <de...@gmail.com>, wrote:
You can also use the power of Zookeeper to share data across JVM's, much like what storm and Kafka do.
Thanks and Regards,
Devang
On 6 May 2015 09:07, "Supun Kamburugamuva" <su...@gmail.com> wrote:
I think you will need something like redis.
Supun..
On May 5, 2015 8:17 PM, "Huy Le Van" <hu...@insight-centre.org> wrote:
Could someone help me please?
On Thursday, Apr 30, 2015 at 4:27 a.m., Huy Le Van <hu...@insight-centre.org>, wrote:
Hi Michael,
If a bolt of type A on a JVM modifies the shared data, will another bolt of type B on another JVM see the changes? To be clear, I have a use-case where I need multiple bolts of multiple types to be able to read/write on the shared collection. You can think of it as a shared database. In this case, can I still use this solution or do I need to use Hazelcast/Redis/Memcached?
Best regards,
Huy, Le Van
On Thursday, Dec 4, 2014 at 8:53 p.m., Michael Rose <mi...@fullcontact.com>, wrote:
Yes, as long as they're in the same JVM. If you have 4 workers, that's 4 JVMs, and 4 copies of the data.
Storm will ensure that if you have 4 bolts of type A, and 8 of type B, that 1 of type A and 2 of type B exist in each JVM. Since the singleton's field is static, it'll be shared.
Michael Rose (@Xorlev)
Senior Platform Engineer, FullContact
michael@fullcontact.com
On Thu, Dec 4, 2014 at 1:34 PM, Eyal Golan <eg...@gmail.com> wrote:
Michael,
I understand your solution, however, if bolt of type A "wins" the initialization, will bolts of type B see the same instance?
Sent from my Android phone
On Dec 4, 2014 10:26 PM, "Eyal Golan" <eg...@gmail.com> wrote:
Thanks Michael and Ben !
Sent from my Android phone
On Dec 4, 2014 9:53 PM, "Flint, Ben" <bf...@popcap.com> wrote:
I’ve used Hazelcast for use cases like this with great success. It makes it relatively straightforward to implement a distributed data structure.
From: Eyal Golan <eg...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Thursday, December 4, 2014 at 11:40 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: Shared static data between bolts
Thanks!
All the answers give me good idea.
Here's why I need this list be sharable.
One bolt needs the list for computation.
Another bolt will periodically update the list.
In other words, the list is changeable.
I tried passing it in conf but there are some 3rd party libs in the wrapper.
Anyway, I also need to improve the topology. Separate bolt to two etc. That way, it would be easier to handle that cache.
One more question, I thought about giving hazelcast a try.
Create local node and then the bolts will have clients.
What do you think?
Sent from my Android phone
On Dec 4, 2014 9:14 PM, "Kushan Maskey" <ku...@mmillerassociates.com> wrote:
Hi Eyal,
I had the similar issue earlier. Static objects are not a good way of passing the variables around in Storm due to clustered nature of the Storm. You will see null for all the static variables.
This is how I implemented mine, for instance I wanted to pass around the server properties such and host name and username/password. I read the properties file and then created a map of these properties then set it into the Config object. Like below.
In topology:
Map<Stirng, Stirng> map = new HashMap();
map.put("hostname", "localhost");
map.put("username", "testuser");
map.put("password", "testpassord");
map.put("port", "1234");
Config config = new Config();
config.put("MY_MAP", map)
In Bolts:
public void prepare(Map stormConf, TopologyContext context) {
HashMap<Stirng, Stirng> map = stormConfig.get("MY_MAP");
}
This how i resolved my problem.Try that if that helps.
--
Kushan Maskey
817.403.7500
M. Miller & Associates
kushan.maskey@mmillerassociates.com
On Thu, Dec 4, 2014 at 1:01 PM, Eyal Golan
<eg...@gmail.com> wrote:
Hi all,
I am newbie with Storm and having some issues with static objects in the topology.
In the main class, where I set the topology, I also set a static object. It's actually a wrapper for a collection of strings.
This object needs to he shared between different bolts.
It's big so I want it to be only one instance.
The problem is that when a bolt tries to get this static instance: example - MyMain.Shared... It's null. Although I instantiated it in the main method.
I guess I'm doing something wrong.
Thanks for any assistance.
Eyal
Sent from my Android phone
Re: Shared static data between bolts
Posted by Devang Shah <de...@gmail.com>.
You can also use the power of Zookeeper to share data across JVM's, much
like what storm and Kafka do.
Thanks and Regards,
Devang
On 6 May 2015 09:07, "Supun Kamburugamuva" <su...@gmail.com> wrote:
> I think you will need something like redis.
>
> Supun..
> On May 5, 2015 8:17 PM, "Huy Le Van" <hu...@insight-centre.org> wrote:
>
>> Could someone help me please?
>>
>> On Thursday, Apr 30, 2015 at 4:27 a.m., Huy Le Van <
>> huy.levan@insight-centre.org>, wrote:
>>
>>> Hi Michael,
>>>
>>> If a bolt of type A on a JVM modifies the shared data, will another bolt
>>> of type B on another JVM see the changes? To be clear, I have a use-case
>>> where I need multiple bolts of multiple types to be able to read/write on
>>> the shared collection. You can think of it as a shared database. In this
>>> case, can I still use this solution or do I need to use
>>> Hazelcast/Redis/Memcached?
>>>
>>> Best regards,
>>> Huy, Le Van
>>>
>>> On Thursday, Dec 4, 2014 at 8:53 p.m., Michael Rose <
>>> michael@fullcontact.com>, wrote:
>>>
>>>> Yes, as long as they're in the same JVM. If you have 4 workers, that's
>>>> 4 JVMs, and 4 copies of the data.
>>>>
>>>> Storm will ensure that if you have 4 bolts of type A, and 8 of type B,
>>>> that 1 of type A and 2 of type B exist in each JVM. Since the singleton's
>>>> field is static, it'll be shared.
>>>>
>>>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>>>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>>>> michael@fullcontact.com
>>>>
>>>> On Thu, Dec 4, 2014 at 1:34 PM, Eyal Golan <eg...@gmail.com> wrote:
>>>>
>>>>> Michael,
>>>>> I understand your solution, however, if bolt of type A "wins" the
>>>>> initialization, will bolts of type B see the same instance?
>>>>>
>>>>> Sent from my Android phone
>>>>> On Dec 4, 2014 10:26 PM, "Eyal Golan" <eg...@gmail.com> wrote:
>>>>>
>>>>>> Thanks Michael and Ben !
>>>>>>
>>>>>> Sent from my Android phone
>>>>>> On Dec 4, 2014 9:53 PM, "Flint, Ben" <bf...@popcap.com> wrote:
>>>>>>
>>>>>>> I’ve used Hazelcast for use cases like this with great success.
>>>>>>> It makes it relatively straightforward to implement a distributed data
>>>>>>> structure.
>>>>>>>
>>>>>>> From: Eyal Golan <eg...@gmail.com>
>>>>>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>> Date: Thursday, December 4, 2014 at 11:40 AM
>>>>>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>>> Subject: Re: Shared static data between bolts
>>>>>>>
>>>>>>> Thanks!
>>>>>>> All the answers give me good idea.
>>>>>>> Here's why I need this list be sharable.
>>>>>>> One bolt needs the list for computation.
>>>>>>> Another bolt will periodically update the list.
>>>>>>> In other words, the list is changeable.
>>>>>>>
>>>>>>> I tried passing it in conf but there are some 3rd party libs in the
>>>>>>> wrapper.
>>>>>>>
>>>>>>> Anyway, I also need to improve the topology. Separate bolt to two
>>>>>>> etc. That way, it would be easier to handle that cache.
>>>>>>>
>>>>>>> One more question, I thought about giving hazelcast a try.
>>>>>>> Create local node and then the bolts will have clients.
>>>>>>>
>>>>>>> What do you think?
>>>>>>>
>>>>>>> Sent from my Android phone
>>>>>>> On Dec 4, 2014 9:14 PM, "Kushan Maskey" <
>>>>>>> kushan.maskey@mmillerassociates.com> wrote:
>>>>>>>
>>>>>>>> Hi Eyal,
>>>>>>>>
>>>>>>>> I had the similar issue earlier. Static objects are not a good way
>>>>>>>> of passing the variables around in Storm due to clustered nature of the
>>>>>>>> Storm. You will see null for all the static variables.
>>>>>>>>
>>>>>>>> This is how I implemented mine, for instance I wanted to pass
>>>>>>>> around the server properties such and host name and username/password. I
>>>>>>>> read the properties file and then created a map of these properties then
>>>>>>>> set it into the Config object. Like below.
>>>>>>>>
>>>>>>>> *In topology:*
>>>>>>>>
>>>>>>>> Map<Stirng, Stirng> map = new HashMap();
>>>>>>>> map.put("hostname", "localhost");
>>>>>>>> map.put("username", "testuser");
>>>>>>>> map.put("password", "testpassord");
>>>>>>>> map.put("port", "1234");
>>>>>>>>
>>>>>>>> Config config = new Config();
>>>>>>>>
>>>>>>>> config.put("MY_MAP", map)
>>>>>>>> *In Bolts:*
>>>>>>>>
>>>>>>>> public void prepare(Map stormConf, TopologyContext context) {
>>>>>>>>
>>>>>>>> HashMap<Stirng, Stirng> map = stormConfig.get("MY_MAP");
>>>>>>>>
>>>>>>>> }
>>>>>>>> This how i resolved my problem.Try that if that helps.
>>>>>>>> --
>>>>>>>> Kushan Maskey
>>>>>>>> 817.403.7500
>>>>>>>> M. Miller & Associates <http://mmillerassociates.com/>
>>>>>>>> kushan.maskey@mmillerassociates.com
>>>>>>>>
>>>>>>>> On Thu, Dec 4, 2014 at 1:01 PM, Eyal Golan <eg...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>> I am newbie with Storm and having some issues with static objects
>>>>>>>>> in the topology.
>>>>>>>>> In the main class, where I set the topology, I also set a static
>>>>>>>>> object. It's actually a wrapper for a collection of strings.
>>>>>>>>> This object needs to he shared between different bolts.
>>>>>>>>> It's big so I want it to be only one instance.
>>>>>>>>>
>>>>>>>>> The problem is that when a bolt tries to get this static instance:
>>>>>>>>> example - MyMain.Shared... It's null. Although I instantiated it in the
>>>>>>>>> main method.
>>>>>>>>>
>>>>>>>>> I guess I'm doing something wrong.
>>>>>>>>>
>>>>>>>>> Thanks for any assistance.
>>>>>>>>>
>>>>>>>>> Eyal
>>>>>>>>>
>>>>>>>>> Sent from my Android phone
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>
Re: Shared static data between bolts
Posted by Supun Kamburugamuva <su...@gmail.com>.
I think you will need something like redis.
Supun..
On May 5, 2015 8:17 PM, "Huy Le Van" <hu...@insight-centre.org> wrote:
> Could someone help me please?
>
> On Thursday, Apr 30, 2015 at 4:27 a.m., Huy Le Van <
> huy.levan@insight-centre.org>, wrote:
>
>> Hi Michael,
>>
>> If a bolt of type A on a JVM modifies the shared data, will another bolt
>> of type B on another JVM see the changes? To be clear, I have a use-case
>> where I need multiple bolts of multiple types to be able to read/write on
>> the shared collection. You can think of it as a shared database. In this
>> case, can I still use this solution or do I need to use
>> Hazelcast/Redis/Memcached?
>>
>> Best regards,
>> Huy, Le Van
>>
>> On Thursday, Dec 4, 2014 at 8:53 p.m., Michael Rose <
>> michael@fullcontact.com>, wrote:
>>
>>> Yes, as long as they're in the same JVM. If you have 4 workers, that's 4
>>> JVMs, and 4 copies of the data.
>>>
>>> Storm will ensure that if you have 4 bolts of type A, and 8 of type B,
>>> that 1 of type A and 2 of type B exist in each JVM. Since the singleton's
>>> field is static, it'll be shared.
>>>
>>> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
>>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
>>> michael@fullcontact.com
>>>
>>> On Thu, Dec 4, 2014 at 1:34 PM, Eyal Golan <eg...@gmail.com> wrote:
>>>
>>>> Michael,
>>>> I understand your solution, however, if bolt of type A "wins" the
>>>> initialization, will bolts of type B see the same instance?
>>>>
>>>> Sent from my Android phone
>>>> On Dec 4, 2014 10:26 PM, "Eyal Golan" <eg...@gmail.com> wrote:
>>>>
>>>>> Thanks Michael and Ben !
>>>>>
>>>>> Sent from my Android phone
>>>>> On Dec 4, 2014 9:53 PM, "Flint, Ben" <bf...@popcap.com> wrote:
>>>>>
>>>>>> I’ve used Hazelcast for use cases like this with great success. It
>>>>>> makes it relatively straightforward to implement a distributed data
>>>>>> structure.
>>>>>>
>>>>>> From: Eyal Golan <eg...@gmail.com>
>>>>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>> Date: Thursday, December 4, 2014 at 11:40 AM
>>>>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>>>>> Subject: Re: Shared static data between bolts
>>>>>>
>>>>>> Thanks!
>>>>>> All the answers give me good idea.
>>>>>> Here's why I need this list be sharable.
>>>>>> One bolt needs the list for computation.
>>>>>> Another bolt will periodically update the list.
>>>>>> In other words, the list is changeable.
>>>>>>
>>>>>> I tried passing it in conf but there are some 3rd party libs in the
>>>>>> wrapper.
>>>>>>
>>>>>> Anyway, I also need to improve the topology. Separate bolt to two
>>>>>> etc. That way, it would be easier to handle that cache.
>>>>>>
>>>>>> One more question, I thought about giving hazelcast a try.
>>>>>> Create local node and then the bolts will have clients.
>>>>>>
>>>>>> What do you think?
>>>>>>
>>>>>> Sent from my Android phone
>>>>>> On Dec 4, 2014 9:14 PM, "Kushan Maskey" <
>>>>>> kushan.maskey@mmillerassociates.com> wrote:
>>>>>>
>>>>>>> Hi Eyal,
>>>>>>>
>>>>>>> I had the similar issue earlier. Static objects are not a good way
>>>>>>> of passing the variables around in Storm due to clustered nature of the
>>>>>>> Storm. You will see null for all the static variables.
>>>>>>>
>>>>>>> This is how I implemented mine, for instance I wanted to pass around
>>>>>>> the server properties such and host name and username/password. I read the
>>>>>>> properties file and then created a map of these properties then set it into
>>>>>>> the Config object. Like below.
>>>>>>>
>>>>>>> *In topology:*
>>>>>>>
>>>>>>> Map<Stirng, Stirng> map = new HashMap();
>>>>>>> map.put("hostname", "localhost");
>>>>>>> map.put("username", "testuser");
>>>>>>> map.put("password", "testpassord");
>>>>>>> map.put("port", "1234");
>>>>>>>
>>>>>>> Config config = new Config();
>>>>>>>
>>>>>>> config.put("MY_MAP", map)
>>>>>>> *In Bolts:*
>>>>>>>
>>>>>>> public void prepare(Map stormConf, TopologyContext context) {
>>>>>>>
>>>>>>> HashMap<Stirng, Stirng> map = stormConfig.get("MY_MAP");
>>>>>>>
>>>>>>> }
>>>>>>> This how i resolved my problem.Try that if that helps.
>>>>>>> --
>>>>>>> Kushan Maskey
>>>>>>> 817.403.7500
>>>>>>> M. Miller & Associates <http://mmillerassociates.com/>
>>>>>>> kushan.maskey@mmillerassociates.com
>>>>>>>
>>>>>>> On Thu, Dec 4, 2014 at 1:01 PM, Eyal Golan <eg...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>> I am newbie with Storm and having some issues with static objects
>>>>>>>> in the topology.
>>>>>>>> In the main class, where I set the topology, I also set a static
>>>>>>>> object. It's actually a wrapper for a collection of strings.
>>>>>>>> This object needs to he shared between different bolts.
>>>>>>>> It's big so I want it to be only one instance.
>>>>>>>>
>>>>>>>> The problem is that when a bolt tries to get this static instance:
>>>>>>>> example - MyMain.Shared... It's null. Although I instantiated it in the
>>>>>>>> main method.
>>>>>>>>
>>>>>>>> I guess I'm doing something wrong.
>>>>>>>>
>>>>>>>> Thanks for any assistance.
>>>>>>>>
>>>>>>>> Eyal
>>>>>>>>
>>>>>>>> Sent from my Android phone
>>>>>>>>
>>>>>>>
>>>>>>>
>>>