You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Navin Ipe <na...@searchlighthealth.com> on 2016/07/13 05:09:52 UTC
Allocating separate memory and workers to topologies of a single jar?
Hi,
I have a program *MyProg.java* inside which I'm creating 5 topologies and
using *stormSubmitter* to submit it to Storm. The storm UI shows the
assigned memory for each topology as:
*Assigned Mem (MB)*
832
0
832
832
832
One of the topologies was assigned 0MB. Assuming that it was because of
setting a single Config for all of them, I gave them separate configs (as
below), but nothing changed.
*StormConfigFactory stormConfigFactory = new StormConfigFactory();*
*StormSubmitter.submitTopology(upTopologyName,
stormConfigFactory.GetNewConfig(),
upBuilder.createTopology());StormSubmitter.submitTopology(viTopologyName,
stormConfigFactory.GetNewConfig(),
viBuilder.createTopology());StormSubmitter.submitTopology(prTopologyName,
stormConfigFactory.GetNewConfig(),
prBuilder.createTopology());StormSubmitter.submitTopology(opTopologyName,
stormConfigFactory.GetNewConfig(),
opBuilder.createTopology());StormSubmitter.submitTopology(poTopologyName,
stormConfigFactory.GetNewConfig(), poBuilder.createTopology());*
*And the StormConfigFactory class:public class StormConfigFactory {
public Config GetNewConfig() { Config stormConfig = new
Config(); stormConfig.setNumWorkers(1);
stormConfig.setNumAckers(1);
stormConfig.put(Config.TOPOLOGY_DEBUG, false);
stormConfig.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 64);
stormConfig.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE,
65536);
stormConfig.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE,
65536); stormConfig.put(Config.TOPOLOGY_MAX_SPOUT_PENDING,
50); stormConfig.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS,
60); stormConfig.put(Config.STORM_ZOOKEEPER_SERVERS,
Arrays.asList(new String[]{"localhost"}));
return stormConfig; }}*
*How do I allocate separate memory and workers for each topology?*
--
Regards,
Navin
Re: Allocating separate memory and workers to topologies of a single jar?
Posted by Navin Ipe <na...@searchlighthealth.com>.
*Solved:*
All of your solutions worked. Thanks.
Mentioning the solution more elaborately for posterity:
According to NathamMarz
<https://groups.google.com/forum/#!topic/storm-user/rH0TG--1FMQ>:
*The slots on each machine is determined by the storm.yaml that the
supervisor has, not the storm.yaml on your machine or the topology config.
You have to restart the supervisor for it to pick up any changes.*
Another person with the same problem and solution:
https://groups.google.com/forum/#!topic/storm-user/CtErpjCPDWA
*What to do if you want to run more than the default 4 topologies:*
The number of topologies you can run on each supervisor is determined by
the number of slots available for it. By default it is 4. If you want to
increase it, you have to add some code to the storm.yaml file on the
supervisor nodes.
I searched for the storm.yaml file and added this:
*supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703 -
6704 - 6705 *
So I got 6 slots and was able to run 5 topologies. On the storm UI, the
supervisor summary now shows:
Slots Used slots
6 5
On Wed, Jul 13, 2016 at 3:39 PM, Navin Ipe <na...@searchlighthealth.com>
wrote:
> Thanks Satish, but that didn't work either. Although 4 of the topologies
> got 1088MB assigned, the 5th topology got 0MB.
>
> Num workers Num executors Assigned Mem (MB)
> 1 12 1088
> 1 4 1088
> 1 27 1088
> 1 8 1088
> 0 0 0
>
>
>
> On Wed, Jul 13, 2016 at 3:00 PM, Satish Duggana <sa...@gmail.com>
> wrote:
>
>> Config.TOPOLOGY_WORKER_CHILDOPTS: Options which can override
>> WORKER_CHILDOPTS for a topology. You can configure any java options like
>> memory, gc etc
>>
>> In your case it can be
>> config.put(Config.TOPOLOGY_WORKER_CHILDOPTS, "-Xmx1g");
>>
>> Thanks,
>> Satish.
>>
>>
>> On Wed, Jul 13, 2016 at 1:45 PM, Navin Ipe <
>> navin.ipe@searchlighthealth.com> wrote:
>>
>>> Using *stormConfig.put("topology.**worker.childopts",1024)* gave me
>>> this error:
>>> *java.lang.IllegalArgumentException: Field TOPOLOGY_WORKER_CHILDOPTS
>>> must be an Iterable but was a class java.lang.Integer*
>>>
>>> A bit of looking around showed me that this might be the right syntax: *config.put(Config.TOPOLOGY_WORKER_CHILDOPTS,
>>> SOME_OPTS);*
>>>
>>> But couldn't find any example on what Storm expects as SOME_OPTS.
>>>
>>> On Wed, Jul 13, 2016 at 12:38 PM, Spico Florin <sp...@gmail.com>
>>> wrote:
>>>
>>>> Hello!
>>>> For the the topology that you have 0MB allocated, for me it seems
>>>> that you don't have enough slots available. Check out the storm.yaml file
>>>> (on your worker machines) how many slots you have allocated.
>>>> (by default the are 4 slots available supervisor.slots.ports:
>>>> - 6700
>>>> - 6701
>>>> - 6702
>>>> - 6703) You have 5 topologies, therefore one is not ran.
>>>>
>>>> Regarding the memory allocation, you allocate memory per each worker
>>>> (slot available), not per topology. If you set up for your topology a
>>>> number of workers equal to 1, then you topology will run on a single worker
>>>> (available slot) and will receive the configuration that you gave for your
>>>> worker. If you configure to spread your spout and bolts to multiple
>>>> workers,(that are available in as configured slots) then all the workers
>>>> will receive the same amount of memory configured globally via
>>>> worker.childopts property in the storm yaml . So you don't configure the
>>>> meory per topology but per worker.
>>>>
>>>> If you want use different memory allocation for workers depending on
>>>> your topology components memory consumption, then you should use the
>>>> property
>>>>
>>>> stormConfig.put("topology.worker.childopts",1024)
>>>>
>>>> I hope it helps.
>>>>
>>>> Regards,
>>>> Florin
>>>>
>>>> On Wed, Jul 13, 2016 at 9:23 AM, Navin Ipe <
>>>> navin.ipe@searchlighthealth.com> wrote:
>>>>
>>>>> I tried setting stormConfig.put(Config.WORKER_HEAP_MEMORY_MB, 1024);
>>>>> and now *all topologies* Assigned memory is 0MB.
>>>>> Does anyone know why it works this way? How do we assign memory to
>>>>> topologies in this situation?
>>>>>
>>>>> On Wed, Jul 13, 2016 at 10:39 AM, Navin Ipe <
>>>>> navin.ipe@searchlighthealth.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have a program *MyProg.java* inside which I'm creating 5
>>>>>> topologies and using *stormSubmitter* to submit it to Storm. The
>>>>>> storm UI shows the assigned memory for each topology as:
>>>>>> *Assigned Mem (MB)*
>>>>>> 832
>>>>>> 0
>>>>>> 832
>>>>>> 832
>>>>>> 832
>>>>>>
>>>>>> One of the topologies was assigned 0MB. Assuming that it was because
>>>>>> of setting a single Config for all of them, I gave them separate configs
>>>>>> (as below), but nothing changed.
>>>>>>
>>>>>> *StormConfigFactory stormConfigFactory = new StormConfigFactory();*
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *StormSubmitter.submitTopology(upTopologyName,
>>>>>> stormConfigFactory.GetNewConfig(),
>>>>>> upBuilder.createTopology());StormSubmitter.submitTopology(viTopologyName,
>>>>>> stormConfigFactory.GetNewConfig(),
>>>>>> viBuilder.createTopology());StormSubmitter.submitTopology(prTopologyName,
>>>>>> stormConfigFactory.GetNewConfig(),
>>>>>> prBuilder.createTopology());StormSubmitter.submitTopology(opTopologyName,
>>>>>> stormConfigFactory.GetNewConfig(),
>>>>>> opBuilder.createTopology());StormSubmitter.submitTopology(poTopologyName,
>>>>>> stormConfigFactory.GetNewConfig(), poBuilder.createTopology());*
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *And the StormConfigFactory class:public class StormConfigFactory
>>>>>> { public Config GetNewConfig() { Config stormConfig = new
>>>>>> Config(); stormConfig.setNumWorkers(1);
>>>>>> stormConfig.setNumAckers(1);
>>>>>> stormConfig.put(Config.TOPOLOGY_DEBUG, false);
>>>>>> stormConfig.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 64);
>>>>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE,
>>>>>> 65536);
>>>>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE,
>>>>>> 65536); stormConfig.put(Config.TOPOLOGY_MAX_SPOUT_PENDING,
>>>>>> 50); stormConfig.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS,
>>>>>> 60); stormConfig.put(Config.STORM_ZOOKEEPER_SERVERS,
>>>>>> Arrays.asList(new String[]{"localhost"}));
>>>>>> return stormConfig; }}*
>>>>>>
>>>>>>
>>>>>> *How do I allocate separate memory and workers for each topology?*
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Navin
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Navin
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Navin
>>>
>>
>>
>
>
> --
> Regards,
> Navin
>
--
Regards,
Navin
Re: Allocating separate memory and workers to topologies of a single jar?
Posted by Navin Ipe <na...@searchlighthealth.com>.
Thanks Satish, but that didn't work either. Although 4 of the topologies
got 1088MB assigned, the 5th topology got 0MB.
Num workers Num executors Assigned Mem (MB)
1 12 1088
1 4 1088
1 27 1088
1 8 1088
0 0 0
On Wed, Jul 13, 2016 at 3:00 PM, Satish Duggana <sa...@gmail.com>
wrote:
> Config.TOPOLOGY_WORKER_CHILDOPTS: Options which can override
> WORKER_CHILDOPTS for a topology. You can configure any java options like
> memory, gc etc
>
> In your case it can be
> config.put(Config.TOPOLOGY_WORKER_CHILDOPTS, "-Xmx1g");
>
> Thanks,
> Satish.
>
>
> On Wed, Jul 13, 2016 at 1:45 PM, Navin Ipe <
> navin.ipe@searchlighthealth.com> wrote:
>
>> Using *stormConfig.put("topology.**worker.childopts",1024)* gave me this
>> error:
>> *java.lang.IllegalArgumentException: Field TOPOLOGY_WORKER_CHILDOPTS must
>> be an Iterable but was a class java.lang.Integer*
>>
>> A bit of looking around showed me that this might be the right syntax: *config.put(Config.TOPOLOGY_WORKER_CHILDOPTS,
>> SOME_OPTS);*
>>
>> But couldn't find any example on what Storm expects as SOME_OPTS.
>>
>> On Wed, Jul 13, 2016 at 12:38 PM, Spico Florin <sp...@gmail.com>
>> wrote:
>>
>>> Hello!
>>> For the the topology that you have 0MB allocated, for me it seems that
>>> you don't have enough slots available. Check out the storm.yaml file (on
>>> your worker machines) how many slots you have allocated.
>>> (by default the are 4 slots available supervisor.slots.ports:
>>> - 6700
>>> - 6701
>>> - 6702
>>> - 6703) You have 5 topologies, therefore one is not ran.
>>>
>>> Regarding the memory allocation, you allocate memory per each worker
>>> (slot available), not per topology. If you set up for your topology a
>>> number of workers equal to 1, then you topology will run on a single worker
>>> (available slot) and will receive the configuration that you gave for your
>>> worker. If you configure to spread your spout and bolts to multiple
>>> workers,(that are available in as configured slots) then all the workers
>>> will receive the same amount of memory configured globally via
>>> worker.childopts property in the storm yaml . So you don't configure the
>>> meory per topology but per worker.
>>>
>>> If you want use different memory allocation for workers depending on
>>> your topology components memory consumption, then you should use the
>>> property
>>>
>>> stormConfig.put("topology.worker.childopts",1024)
>>>
>>> I hope it helps.
>>>
>>> Regards,
>>> Florin
>>>
>>> On Wed, Jul 13, 2016 at 9:23 AM, Navin Ipe <
>>> navin.ipe@searchlighthealth.com> wrote:
>>>
>>>> I tried setting stormConfig.put(Config.WORKER_HEAP_MEMORY_MB, 1024);
>>>> and now *all topologies* Assigned memory is 0MB.
>>>> Does anyone know why it works this way? How do we assign memory to
>>>> topologies in this situation?
>>>>
>>>> On Wed, Jul 13, 2016 at 10:39 AM, Navin Ipe <
>>>> navin.ipe@searchlighthealth.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have a program *MyProg.java* inside which I'm creating 5 topologies
>>>>> and using *stormSubmitter* to submit it to Storm. The storm UI shows
>>>>> the assigned memory for each topology as:
>>>>> *Assigned Mem (MB)*
>>>>> 832
>>>>> 0
>>>>> 832
>>>>> 832
>>>>> 832
>>>>>
>>>>> One of the topologies was assigned 0MB. Assuming that it was because
>>>>> of setting a single Config for all of them, I gave them separate configs
>>>>> (as below), but nothing changed.
>>>>>
>>>>> *StormConfigFactory stormConfigFactory = new StormConfigFactory();*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *StormSubmitter.submitTopology(upTopologyName,
>>>>> stormConfigFactory.GetNewConfig(),
>>>>> upBuilder.createTopology());StormSubmitter.submitTopology(viTopologyName,
>>>>> stormConfigFactory.GetNewConfig(),
>>>>> viBuilder.createTopology());StormSubmitter.submitTopology(prTopologyName,
>>>>> stormConfigFactory.GetNewConfig(),
>>>>> prBuilder.createTopology());StormSubmitter.submitTopology(opTopologyName,
>>>>> stormConfigFactory.GetNewConfig(),
>>>>> opBuilder.createTopology());StormSubmitter.submitTopology(poTopologyName,
>>>>> stormConfigFactory.GetNewConfig(), poBuilder.createTopology());*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *And the StormConfigFactory class:public class StormConfigFactory {
>>>>> public Config GetNewConfig() { Config stormConfig = new
>>>>> Config(); stormConfig.setNumWorkers(1);
>>>>> stormConfig.setNumAckers(1);
>>>>> stormConfig.put(Config.TOPOLOGY_DEBUG, false);
>>>>> stormConfig.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 64);
>>>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE,
>>>>> 65536);
>>>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE,
>>>>> 65536); stormConfig.put(Config.TOPOLOGY_MAX_SPOUT_PENDING,
>>>>> 50); stormConfig.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS,
>>>>> 60); stormConfig.put(Config.STORM_ZOOKEEPER_SERVERS,
>>>>> Arrays.asList(new String[]{"localhost"}));
>>>>> return stormConfig; }}*
>>>>>
>>>>>
>>>>> *How do I allocate separate memory and workers for each topology?*
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Navin
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Navin
>>>>
>>>
>>>
>>
>>
>> --
>> Regards,
>> Navin
>>
>
>
--
Regards,
Navin
Re: Allocating separate memory and workers to topologies of a single jar?
Posted by Satish Duggana <sa...@gmail.com>.
Config.TOPOLOGY_WORKER_CHILDOPTS: Options which can override
WORKER_CHILDOPTS for a topology. You can configure any java options like
memory, gc etc
In your case it can be
config.put(Config.TOPOLOGY_WORKER_CHILDOPTS, "-Xmx1g");
Thanks,
Satish.
On Wed, Jul 13, 2016 at 1:45 PM, Navin Ipe <na...@searchlighthealth.com>
wrote:
> Using *stormConfig.put("topology.**worker.childopts",1024)* gave me this
> error:
> *java.lang.IllegalArgumentException: Field TOPOLOGY_WORKER_CHILDOPTS must
> be an Iterable but was a class java.lang.Integer*
>
> A bit of looking around showed me that this might be the right syntax: *config.put(Config.TOPOLOGY_WORKER_CHILDOPTS,
> SOME_OPTS);*
>
> But couldn't find any example on what Storm expects as SOME_OPTS.
>
> On Wed, Jul 13, 2016 at 12:38 PM, Spico Florin <sp...@gmail.com>
> wrote:
>
>> Hello!
>> For the the topology that you have 0MB allocated, for me it seems that
>> you don't have enough slots available. Check out the storm.yaml file (on
>> your worker machines) how many slots you have allocated.
>> (by default the are 4 slots available supervisor.slots.ports:
>> - 6700
>> - 6701
>> - 6702
>> - 6703) You have 5 topologies, therefore one is not ran.
>>
>> Regarding the memory allocation, you allocate memory per each worker
>> (slot available), not per topology. If you set up for your topology a
>> number of workers equal to 1, then you topology will run on a single worker
>> (available slot) and will receive the configuration that you gave for your
>> worker. If you configure to spread your spout and bolts to multiple
>> workers,(that are available in as configured slots) then all the workers
>> will receive the same amount of memory configured globally via
>> worker.childopts property in the storm yaml . So you don't configure the
>> meory per topology but per worker.
>>
>> If you want use different memory allocation for workers depending on your
>> topology components memory consumption, then you should use the property
>>
>> stormConfig.put("topology.worker.childopts",1024)
>>
>> I hope it helps.
>>
>> Regards,
>> Florin
>>
>> On Wed, Jul 13, 2016 at 9:23 AM, Navin Ipe <
>> navin.ipe@searchlighthealth.com> wrote:
>>
>>> I tried setting stormConfig.put(Config.WORKER_HEAP_MEMORY_MB, 1024); and
>>> now *all topologies* Assigned memory is 0MB.
>>> Does anyone know why it works this way? How do we assign memory to
>>> topologies in this situation?
>>>
>>> On Wed, Jul 13, 2016 at 10:39 AM, Navin Ipe <
>>> navin.ipe@searchlighthealth.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a program *MyProg.java* inside which I'm creating 5 topologies
>>>> and using *stormSubmitter* to submit it to Storm. The storm UI shows
>>>> the assigned memory for each topology as:
>>>> *Assigned Mem (MB)*
>>>> 832
>>>> 0
>>>> 832
>>>> 832
>>>> 832
>>>>
>>>> One of the topologies was assigned 0MB. Assuming that it was because of
>>>> setting a single Config for all of them, I gave them separate configs (as
>>>> below), but nothing changed.
>>>>
>>>> *StormConfigFactory stormConfigFactory = new StormConfigFactory();*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *StormSubmitter.submitTopology(upTopologyName,
>>>> stormConfigFactory.GetNewConfig(),
>>>> upBuilder.createTopology());StormSubmitter.submitTopology(viTopologyName,
>>>> stormConfigFactory.GetNewConfig(),
>>>> viBuilder.createTopology());StormSubmitter.submitTopology(prTopologyName,
>>>> stormConfigFactory.GetNewConfig(),
>>>> prBuilder.createTopology());StormSubmitter.submitTopology(opTopologyName,
>>>> stormConfigFactory.GetNewConfig(),
>>>> opBuilder.createTopology());StormSubmitter.submitTopology(poTopologyName,
>>>> stormConfigFactory.GetNewConfig(), poBuilder.createTopology());*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *And the StormConfigFactory class:public class StormConfigFactory {
>>>> public Config GetNewConfig() { Config stormConfig = new
>>>> Config(); stormConfig.setNumWorkers(1);
>>>> stormConfig.setNumAckers(1);
>>>> stormConfig.put(Config.TOPOLOGY_DEBUG, false);
>>>> stormConfig.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 64);
>>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE,
>>>> 65536);
>>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE,
>>>> 65536); stormConfig.put(Config.TOPOLOGY_MAX_SPOUT_PENDING,
>>>> 50); stormConfig.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS,
>>>> 60); stormConfig.put(Config.STORM_ZOOKEEPER_SERVERS,
>>>> Arrays.asList(new String[]{"localhost"}));
>>>> return stormConfig; }}*
>>>>
>>>>
>>>> *How do I allocate separate memory and workers for each topology?*
>>>>
>>>> --
>>>> Regards,
>>>> Navin
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Navin
>>>
>>
>>
>
>
> --
> Regards,
> Navin
>
Re: Allocating separate memory and workers to topologies of a single jar?
Posted by Navin Ipe <na...@searchlighthealth.com>.
Using *stormConfig.put("topology.**worker.childopts",1024)* gave me this
error:
*java.lang.IllegalArgumentException: Field TOPOLOGY_WORKER_CHILDOPTS must
be an Iterable but was a class java.lang.Integer*
A bit of looking around showed me that this might be the right syntax:
*config.put(Config.TOPOLOGY_WORKER_CHILDOPTS,
SOME_OPTS);*
But couldn't find any example on what Storm expects as SOME_OPTS.
On Wed, Jul 13, 2016 at 12:38 PM, Spico Florin <sp...@gmail.com>
wrote:
> Hello!
> For the the topology that you have 0MB allocated, for me it seems that
> you don't have enough slots available. Check out the storm.yaml file (on
> your worker machines) how many slots you have allocated.
> (by default the are 4 slots available supervisor.slots.ports:
> - 6700
> - 6701
> - 6702
> - 6703) You have 5 topologies, therefore one is not ran.
>
> Regarding the memory allocation, you allocate memory per each worker (slot
> available), not per topology. If you set up for your topology a number of
> workers equal to 1, then you topology will run on a single worker
> (available slot) and will receive the configuration that you gave for your
> worker. If you configure to spread your spout and bolts to multiple
> workers,(that are available in as configured slots) then all the workers
> will receive the same amount of memory configured globally via
> worker.childopts property in the storm yaml . So you don't configure the
> meory per topology but per worker.
>
> If you want use different memory allocation for workers depending on your
> topology components memory consumption, then you should use the property
>
> stormConfig.put("topology.worker.childopts",1024)
>
> I hope it helps.
>
> Regards,
> Florin
>
> On Wed, Jul 13, 2016 at 9:23 AM, Navin Ipe <
> navin.ipe@searchlighthealth.com> wrote:
>
>> I tried setting stormConfig.put(Config.WORKER_HEAP_MEMORY_MB, 1024); and
>> now *all topologies* Assigned memory is 0MB.
>> Does anyone know why it works this way? How do we assign memory to
>> topologies in this situation?
>>
>> On Wed, Jul 13, 2016 at 10:39 AM, Navin Ipe <
>> navin.ipe@searchlighthealth.com> wrote:
>>
>>> Hi,
>>>
>>> I have a program *MyProg.java* inside which I'm creating 5 topologies
>>> and using *stormSubmitter* to submit it to Storm. The storm UI shows
>>> the assigned memory for each topology as:
>>> *Assigned Mem (MB)*
>>> 832
>>> 0
>>> 832
>>> 832
>>> 832
>>>
>>> One of the topologies was assigned 0MB. Assuming that it was because of
>>> setting a single Config for all of them, I gave them separate configs (as
>>> below), but nothing changed.
>>>
>>> *StormConfigFactory stormConfigFactory = new StormConfigFactory();*
>>>
>>>
>>>
>>>
>>>
>>>
>>> *StormSubmitter.submitTopology(upTopologyName,
>>> stormConfigFactory.GetNewConfig(),
>>> upBuilder.createTopology());StormSubmitter.submitTopology(viTopologyName,
>>> stormConfigFactory.GetNewConfig(),
>>> viBuilder.createTopology());StormSubmitter.submitTopology(prTopologyName,
>>> stormConfigFactory.GetNewConfig(),
>>> prBuilder.createTopology());StormSubmitter.submitTopology(opTopologyName,
>>> stormConfigFactory.GetNewConfig(),
>>> opBuilder.createTopology());StormSubmitter.submitTopology(poTopologyName,
>>> stormConfigFactory.GetNewConfig(), poBuilder.createTopology());*
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *And the StormConfigFactory class:public class StormConfigFactory {
>>> public Config GetNewConfig() { Config stormConfig = new
>>> Config(); stormConfig.setNumWorkers(1);
>>> stormConfig.setNumAckers(1);
>>> stormConfig.put(Config.TOPOLOGY_DEBUG, false);
>>> stormConfig.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 64);
>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE,
>>> 65536);
>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE,
>>> 65536); stormConfig.put(Config.TOPOLOGY_MAX_SPOUT_PENDING,
>>> 50); stormConfig.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS,
>>> 60); stormConfig.put(Config.STORM_ZOOKEEPER_SERVERS,
>>> Arrays.asList(new String[]{"localhost"}));
>>> return stormConfig; }}*
>>>
>>>
>>> *How do I allocate separate memory and workers for each topology?*
>>>
>>> --
>>> Regards,
>>> Navin
>>>
>>
>>
>>
>> --
>> Regards,
>> Navin
>>
>
>
--
Regards,
Navin
Re: Allocating separate memory and workers to topologies of a single jar?
Posted by Spico Florin <sp...@gmail.com>.
Hello!
For the the topology that you have 0MB allocated, for me it seems that
you don't have enough slots available. Check out the storm.yaml file (on
your worker machines) how many slots you have allocated.
(by default the are 4 slots available supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703) You have 5 topologies, therefore one is not ran.
Regarding the memory allocation, you allocate memory per each worker (slot
available), not per topology. If you set up for your topology a number of
workers equal to 1, then you topology will run on a single worker
(available slot) and will receive the configuration that you gave for your
worker. If you configure to spread your spout and bolts to multiple
workers,(that are available in as configured slots) then all the workers
will receive the same amount of memory configured globally via
worker.childopts property in the storm yaml . So you don't configure the
meory per topology but per worker.
If you want use different memory allocation for workers depending on your
topology components memory consumption, then you should use the property
stormConfig.put("topology.worker.childopts",1024)
I hope it helps.
Regards,
Florin
On Wed, Jul 13, 2016 at 9:23 AM, Navin Ipe <na...@searchlighthealth.com>
wrote:
> I tried setting stormConfig.put(Config.WORKER_HEAP_MEMORY_MB, 1024); and
> now *all topologies* Assigned memory is 0MB.
> Does anyone know why it works this way? How do we assign memory to
> topologies in this situation?
>
> On Wed, Jul 13, 2016 at 10:39 AM, Navin Ipe <
> navin.ipe@searchlighthealth.com> wrote:
>
>> Hi,
>>
>> I have a program *MyProg.java* inside which I'm creating 5 topologies
>> and using *stormSubmitter* to submit it to Storm. The storm UI shows the
>> assigned memory for each topology as:
>> *Assigned Mem (MB)*
>> 832
>> 0
>> 832
>> 832
>> 832
>>
>> One of the topologies was assigned 0MB. Assuming that it was because of
>> setting a single Config for all of them, I gave them separate configs (as
>> below), but nothing changed.
>>
>> *StormConfigFactory stormConfigFactory = new StormConfigFactory();*
>>
>>
>>
>>
>>
>>
>> *StormSubmitter.submitTopology(upTopologyName,
>> stormConfigFactory.GetNewConfig(),
>> upBuilder.createTopology());StormSubmitter.submitTopology(viTopologyName,
>> stormConfigFactory.GetNewConfig(),
>> viBuilder.createTopology());StormSubmitter.submitTopology(prTopologyName,
>> stormConfigFactory.GetNewConfig(),
>> prBuilder.createTopology());StormSubmitter.submitTopology(opTopologyName,
>> stormConfigFactory.GetNewConfig(),
>> opBuilder.createTopology());StormSubmitter.submitTopology(poTopologyName,
>> stormConfigFactory.GetNewConfig(), poBuilder.createTopology());*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *And the StormConfigFactory class:public class StormConfigFactory {
>> public Config GetNewConfig() { Config stormConfig = new
>> Config(); stormConfig.setNumWorkers(1);
>> stormConfig.setNumAckers(1);
>> stormConfig.put(Config.TOPOLOGY_DEBUG, false);
>> stormConfig.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 64);
>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE,
>> 65536);
>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE,
>> 65536); stormConfig.put(Config.TOPOLOGY_MAX_SPOUT_PENDING,
>> 50); stormConfig.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS,
>> 60); stormConfig.put(Config.STORM_ZOOKEEPER_SERVERS,
>> Arrays.asList(new String[]{"localhost"}));
>> return stormConfig; }}*
>>
>>
>> *How do I allocate separate memory and workers for each topology?*
>>
>> --
>> Regards,
>> Navin
>>
>
>
>
> --
> Regards,
> Navin
>
Re: Allocating separate memory and workers to topologies of a single jar?
Posted by Navin Ipe <na...@searchlighthealth.com>.
I tried setting stormConfig.put(Config.WORKER_HEAP_MEMORY_MB, 1024); and
now *all topologies* Assigned memory is 0MB.
Does anyone know why it works this way? How do we assign memory to
topologies in this situation?
On Wed, Jul 13, 2016 at 10:39 AM, Navin Ipe <navin.ipe@searchlighthealth.com
> wrote:
> Hi,
>
> I have a program *MyProg.java* inside which I'm creating 5 topologies and
> using *stormSubmitter* to submit it to Storm. The storm UI shows the
> assigned memory for each topology as:
> *Assigned Mem (MB)*
> 832
> 0
> 832
> 832
> 832
>
> One of the topologies was assigned 0MB. Assuming that it was because of
> setting a single Config for all of them, I gave them separate configs (as
> below), but nothing changed.
>
> *StormConfigFactory stormConfigFactory = new StormConfigFactory();*
>
>
>
>
>
>
> *StormSubmitter.submitTopology(upTopologyName,
> stormConfigFactory.GetNewConfig(),
> upBuilder.createTopology());StormSubmitter.submitTopology(viTopologyName,
> stormConfigFactory.GetNewConfig(),
> viBuilder.createTopology());StormSubmitter.submitTopology(prTopologyName,
> stormConfigFactory.GetNewConfig(),
> prBuilder.createTopology());StormSubmitter.submitTopology(opTopologyName,
> stormConfigFactory.GetNewConfig(),
> opBuilder.createTopology());StormSubmitter.submitTopology(poTopologyName,
> stormConfigFactory.GetNewConfig(), poBuilder.createTopology());*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *And the StormConfigFactory class:public class StormConfigFactory {
> public Config GetNewConfig() { Config stormConfig = new
> Config(); stormConfig.setNumWorkers(1);
> stormConfig.setNumAckers(1);
> stormConfig.put(Config.TOPOLOGY_DEBUG, false);
> stormConfig.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 64);
> stormConfig.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE,
> 65536);
> stormConfig.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE,
> 65536); stormConfig.put(Config.TOPOLOGY_MAX_SPOUT_PENDING,
> 50); stormConfig.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS,
> 60); stormConfig.put(Config.STORM_ZOOKEEPER_SERVERS,
> Arrays.asList(new String[]{"localhost"}));
> return stormConfig; }}*
>
>
> *How do I allocate separate memory and workers for each topology?*
>
> --
> Regards,
> Navin
>
--
Regards,
Navin