You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Kushan Maskey <ku...@mmillerassociates.com> on 2014/12/04 17:58:05 UTC
supervisor.slots.port setup from storm config
Is there a way I can set supervisor_slots_port in the config in the
topology class? I tried to set it up like below
*config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
but I get error as below.
*InvalidTopologyException(msg:Field supervisor.slots.ports must be an
Iterable of java.lang.Number*
Thanks.
--
Kushan Maskey
817.403.7500
M. Miller & Associates <http://mmillerassociates.com/>
kushan.maskey@mmillerassociates.com
Re: supervisor.slots.port setup from storm config
Posted by Kushan Maskey <ku...@mmillerassociates.com>.
Hi Harsh,
I need a little help. I made change to the cluster.xml file to make changes
like this. I actually copied and added the following lines of code to
address this issue.
<appender name="MY_APP"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${storm.home}/logs/cexp.rtsale.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy"
>
<fileNamePattern>${storm.home}/logs/myapp.log.%i</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>9</maxIndex>
</rollingPolicy>
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>10MB</maxFileSize>
</triggeringPolicy>
<encoder>
<pattern>%d %-8r %m%n</pattern>
</encoder>
</appender>
<logger name="MY_PACKAGE_STRUCTURE" additivity="false">
<level value="INFO" />
<appender-ref ref="MY_APP" />
</logger>
I restarted storm after making these changes. But I dont see log file being
created and logs been written. Can you help find the issue? Thanks.
--
Kushan Maskey
817.403.7500
M. Miller & Associates <http://mmillerassociates.com/>
kushan.maskey@mmillerassociates.com
On Sat, Dec 6, 2014 at 2:03 AM, John Reilly <jr...@inconspicuous.org> wrote:
>
> I meant that it is not in 0.9.3 release. It looks like it was merged to
> master from the security branch but is not part of a release yet.
>
>
> On Fri Dec 05 2014 at 8:22:48 PM Harsha <st...@harsha.io> wrote:
>
>>
>> Yep my bad its in 0.9.3 and not in 0.9.2. In 0.9.2 workers uses
>> cluster.xml so you can add your log4j changes there.. In 0.9.2 workers
>> uses cluster.xml so you can add your log4j changes there.
>>
>> On Fri, Dec 5, 2014, at 04:39 PM, John Reilly wrote:
>>
>> Hi Harsha,
>> It looks like worker.xml was introduced after the 0.9.3 release.
>> Cheers,
>> John
>>
>> On Fri Dec 05 2014 at 8:38:55 AM Harsha <ma...@harsha.io> wrote:
>>
>>
>>
>> my bad its in 0.9.3 and not in 0.9.2. In 0.9.2 workers uses cluster.xml
>> so you can add your log4j changes there.
>>
>> On Fri, Dec 5, 2014, at 07:55 AM, Kushan Maskey wrote:
>>
>> Harsh,
>>
>> I could not find worker.xml in my storm package. Does it come with storm
>> 0.9.2 package? Thanks.
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>> M. Miller & Associates <http://mmillerassociates.com/>
>> kushan.maskey@mmillerassociates.com
>>
>> On Thu, Dec 4, 2014 at 12:11 PM, Kushan Maskey <kushan.maskey@
>> mmillerassociates.com> wrote:
>>
>> I have not and I think I will try that. I will keep you posted.
>>
>> Thanks.
>>
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>>
>> M. Miller & Associates <http://mmillerassociates.com/>
>> kushan.maskey@mmillerassociates.com
>>
>>
>> On Thu, Dec 4, 2014 at 11:30 AM, Harsha <st...@harsha.io> wrote:
>>
>>
>>
>> Kushan,
>> Did you try modifying logback/worker.xml . Since you have
>> multiple topologies if they have distinct java packages you can add per
>> package log4j config to worker.xml . You probably might see duplicate
>> logging in worker-port.log files additional to the topology log files.
>> Thats one way I would try separating out topology logging .
>>
>> -Harsha
>>
>>
>> On Thu, Dec 4, 2014, at 09:23 AM, Parth Brahmbhatt wrote:
>>
>> Harsha is right, if you are trying to do this as part of topology
>> submission, even though the param will get accepted it will have no effect.
>> On Dec 4, 2014, at 9:01 AM, Parth Brahmbhatt <pb...@hortonworks.com>
>> wrote:
>>
>>
>> As the error suggests it needs to be an utterable, you can add it by
>> using the following line, the second arg must be a collection.
>> *config.put(Config.SUPERVISOR_SLOTS_PORTS, Lists.newArrayList(7000));*
>>
>> On Dec 4, 2014, at 8:58 AM, Kushan Maskey <kushan.maskey@
>> mmillerassociates.com> wrote:
>>
>>
>> Is there a way I can set supervisor_slots_port in the config in the
>> topology class? I tried to set it up like below
>>
>> *config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
>>
>> but I get error as below.
>>
>> *InvalidTopologyException(msg:Field supervisor.slots.ports must be an
>> Iterable of java.lang.Number*
>> Thanks.
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>> M. Miller & Associates <http://mmillerassociates.com/>
>> kushan.maskey@mmillerassociates.com
>>
>>
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
Re: supervisor.slots.port setup from storm config
Posted by John Reilly <jr...@inconspicuous.org>.
I meant that it is not in 0.9.3 release. It looks like it was merged to
master from the security branch but is not part of a release yet.
On Fri Dec 05 2014 at 8:22:48 PM Harsha <st...@harsha.io> wrote:
>
> Yep my bad its in 0.9.3 and not in 0.9.2. In 0.9.2 workers uses
> cluster.xml so you can add your log4j changes there.. In 0.9.2 workers
> uses cluster.xml so you can add your log4j changes there.
>
> On Fri, Dec 5, 2014, at 04:39 PM, John Reilly wrote:
>
> Hi Harsha,
> It looks like worker.xml was introduced after the 0.9.3 release.
> Cheers,
> John
>
> On Fri Dec 05 2014 at 8:38:55 AM Harsha <ma...@harsha.io> wrote:
>
>
>
> my bad its in 0.9.3 and not in 0.9.2. In 0.9.2 workers uses cluster.xml
> so you can add your log4j changes there.
>
> On Fri, Dec 5, 2014, at 07:55 AM, Kushan Maskey wrote:
>
> Harsh,
>
> I could not find worker.xml in my storm package. Does it come with storm
> 0.9.2 package? Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>
> On Thu, Dec 4, 2014 at 12:11 PM, Kushan Maskey <kushan.maskey@
> mmillerassociates.com> wrote:
>
> I have not and I think I will try that. I will keep you posted.
>
> Thanks.
>
>
> --
> Kushan Maskey
> 817.403.7500
>
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>
>
> On Thu, Dec 4, 2014 at 11:30 AM, Harsha <st...@harsha.io> wrote:
>
>
>
> Kushan,
> Did you try modifying logback/worker.xml . Since you have
> multiple topologies if they have distinct java packages you can add per
> package log4j config to worker.xml . You probably might see duplicate
> logging in worker-port.log files additional to the topology log files.
> Thats one way I would try separating out topology logging .
>
> -Harsha
>
>
> On Thu, Dec 4, 2014, at 09:23 AM, Parth Brahmbhatt wrote:
>
> Harsha is right, if you are trying to do this as part of topology
> submission, even though the param will get accepted it will have no effect.
> On Dec 4, 2014, at 9:01 AM, Parth Brahmbhatt <pb...@hortonworks.com>
> wrote:
>
>
> As the error suggests it needs to be an utterable, you can add it by using
> the following line, the second arg must be a collection.
> *config.put(Config.SUPERVISOR_SLOTS_PORTS, Lists.newArrayList(7000));*
>
> On Dec 4, 2014, at 8:58 AM, Kushan Maskey <kushan.maskey@
> mmillerassociates.com> wrote:
>
>
> Is there a way I can set supervisor_slots_port in the config in the
> topology class? I tried to set it up like below
>
> *config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
>
> but I get error as below.
>
> *InvalidTopologyException(msg:Field supervisor.slots.ports must be an
> Iterable of java.lang.Number*
> Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
>
>
>
>
>
>
>
>
>
>
Re: supervisor.slots.port setup from storm config
Posted by Harsha <st...@harsha.io>.
Yep my bad its in 0.9.3 and not in 0.9.2. In 0.9.2 workers uses
cluster.xml so you can add your log4j changes there.. In 0.9.2 workers
uses cluster.xml so you can add your log4j changes there.
On Fri, Dec 5, 2014, at 04:39 PM, John Reilly wrote:
> Hi Harsha, It looks like worker.xml was introduced after the 0.9.3
> release. Cheers, John
>
> On Fri Dec 05 2014 at 8:38:55 AM Harsha <ma...@harsha.io> wrote:
>> __
>>
>> my bad its in 0.9.3 and not in 0.9.2. In 0.9.2 workers uses
>> cluster.xml so you can add your log4j changes there.
>>
>> On Fri, Dec 5, 2014, at 07:55 AM, Kushan Maskey wrote:
>>> Harsh,
>>>
>>> I could not find worker.xml in my storm package. Does it come with
>>> storm 0.9.2 package? Thanks.
>>>
>>> --
>>> Kushan Maskey
>>> 817.403.7500
>>> M. Miller & Associates[1] kushan.maskey@__mmillerassociates.com
>>>
>>> On Thu, Dec 4, 2014 at 12:11 PM, Kushan Maskey
>>> <ku...@__mmillerassociates.com> wrote:
>>>> I have not and I think I will try that. I will keep you posted.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> --
>>>> Kushan Maskey
>>>> 817.403.7500
>>>>
>>>> M. Miller & Associates[2] kushan.maskey@__mmillerassociates.com
>>>>
>>>>
>>>> On Thu, Dec 4, 2014 at 11:30 AM, Harsha <st...@harsha.io> wrote:
>>>>> __
>>>>>
>>>>> Kushan, Did you try modifying logback/worker.xml . Since you have
>>>>> multiple topologies if they have distinct java packages you can
>>>>> add per package log4j config to worker.xml . You probably might
>>>>> see duplicate logging in worker-port.log files additional to the
>>>>> topology log files. Thats one way I would try separating out
>>>>> topology logging .
>>>>>
>>>>> -Harsha
>>>>>
>>>>>
>>>>> On Thu, Dec 4, 2014, at 09:23 AM, Parth Brahmbhatt wrote:
>>>>>> Harsha is right, if you are trying to do this as part of topology
>>>>>> submission, even though the param will get accepted it will have
>>>>>> no effect. On Dec 4, 2014, at 9:01 AM, Parth Brahmbhatt
>>>>>> <pb...@hortonworks.com> wrote:
>>>>>>
>>>>>>> As the error suggests it needs to be an utterable, you can add
>>>>>>> it by using the following line, the second arg must be a
>>>>>>> collection. *config.put(Config.SUPERVISOR___SLOTS_PORTS,
>>>>>>> Lists.newArrayList(7000));*
>>>>>>>
>>>>>>> On Dec 4, 2014, at 8:58 AM, Kushan Maskey
>>>>>>> <ku...@__mmillerassociates.com> wrote:
>>>>>>>
>>>>>>>> Is there a way I can set supervisor_slots_port in the config in
>>>>>>>> the topology class? I tried to set it up like below
>>>>>>>>
>>>>>>>> *config.put(Config.SUPERVISOR___SLOTS_PORTS, 7000);*
>>>>>>>>
>>>>>>>> but I get error as below. *InvalidTopologyException(msg:__Field
>>>>>>>> supervisor.slots.ports must be an Iterable of java.lang.Number*
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Kushan Maskey
>>>>>>>> 817.403.7500
>>>>>>>> M. Miller & Associates[3] kushan.maskey@__mmillerassociates.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>> CONFIDENTIALITY NOTICE NOTICE: This message is intended for the
>>>>>> use of the individual or entity to which it is addressed and may
>>>>>> contain information that is confidential, privileged and exempt
>>>>>> from disclosure under applicable law. If the reader of this
>>>>>> message is not the intended recipient, you are hereby notified
>>>>>> that any printing, copying, dissemination, distribution,
>>>>>> disclosure or forwarding of this communication is strictly
>>>>>> prohibited. If you have received this communication in error,
>>>>>> please contact the sender immediately and delete it from your
>>>>>> system. Thank You.
>>>>>
>>>>
>>>
>>
Links:
1. http://mmillerassociates.com/
2. http://mmillerassociates.com/
3. http://mmillerassociates.com/
Re: supervisor.slots.port setup from storm config
Posted by John Reilly <jr...@inconspicuous.org>.
Hi Harsha,
It looks like worker.xml was introduced after the 0.9.3 release.
Cheers,
John
On Fri Dec 05 2014 at 8:38:55 AM Harsha <ma...@harsha.io> wrote:
>
> my bad its in 0.9.3 and not in 0.9.2. In 0.9.2 workers uses cluster.xml
> so you can add your log4j changes there.
>
> On Fri, Dec 5, 2014, at 07:55 AM, Kushan Maskey wrote:
>
> Harsh,
>
> I could not find worker.xml in my storm package. Does it come with storm
> 0.9.2 package? Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>
> On Thu, Dec 4, 2014 at 12:11 PM, Kushan Maskey <kushan.maskey@
> mmillerassociates.com> wrote:
>
> I have not and I think I will try that. I will keep you posted.
>
> Thanks.
>
>
> --
> Kushan Maskey
> 817.403.7500
>
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>
>
> On Thu, Dec 4, 2014 at 11:30 AM, Harsha <st...@harsha.io> wrote:
>
>
>
> Kushan,
> Did you try modifying logback/worker.xml . Since you have
> multiple topologies if they have distinct java packages you can add per
> package log4j config to worker.xml . You probably might see duplicate
> logging in worker-port.log files additional to the topology log files.
> Thats one way I would try separating out topology logging .
>
> -Harsha
>
>
> On Thu, Dec 4, 2014, at 09:23 AM, Parth Brahmbhatt wrote:
>
> Harsha is right, if you are trying to do this as part of topology
> submission, even though the param will get accepted it will have no effect.
> On Dec 4, 2014, at 9:01 AM, Parth Brahmbhatt <pb...@hortonworks.com>
> wrote:
>
>
> As the error suggests it needs to be an utterable, you can add it by using
> the following line, the second arg must be a collection.
> *config.put(Config.SUPERVISOR_SLOTS_PORTS, Lists.newArrayList(7000));*
>
> On Dec 4, 2014, at 8:58 AM, Kushan Maskey <kushan.maskey@
> mmillerassociates.com> wrote:
>
>
> Is there a way I can set supervisor_slots_port in the config in the
> topology class? I tried to set it up like below
>
> *config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
>
> but I get error as below.
>
> *InvalidTopologyException(msg:Field supervisor.slots.ports must be an
> Iterable of java.lang.Number*
> Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
>
>
>
>
>
>
>
>
Re: supervisor.slots.port setup from storm config
Posted by Harsha <ma...@harsha.io>.
my bad its in 0.9.3 and not in 0.9.2. In 0.9.2 workers uses cluster.xml
so you can add your log4j changes there.
On Fri, Dec 5, 2014, at 07:55 AM, Kushan Maskey wrote:
> Harsh,
>
> I could not find worker.xml in my storm package. Does it come with
> storm 0.9.2 package? Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates[1] kushan.maskey@mmillerassociates.com
>
> On Thu, Dec 4, 2014 at 12:11 PM, Kushan Maskey
> <ku...@mmillerassociates.com> wrote:
>> I have not and I think I will try that. I will keep you posted.
>>
>> Thanks.
>>
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>>
>> M. Miller & Associates[2] kushan.maskey@mmillerassociates.com
>>
>>
>> On Thu, Dec 4, 2014 at 11:30 AM, Harsha <st...@harsha.io> wrote:
>>> __
>>>
>>> Kushan, Did you try modifying logback/worker.xml . Since you have
>>> multiple topologies if they have distinct java packages you can add
>>> per package log4j config to worker.xml . You probably might see
>>> duplicate logging in worker-port.log files additional to the
>>> topology log files. Thats one way I would try separating out
>>> topology logging .
>>>
>>> -Harsha
>>>
>>>
>>> On Thu, Dec 4, 2014, at 09:23 AM, Parth Brahmbhatt wrote:
>>>> Harsha is right, if you are trying to do this as part of topology
>>>> submission, even though the param will get accepted it will have no
>>>> effect. On Dec 4, 2014, at 9:01 AM, Parth Brahmbhatt
>>>> <pb...@hortonworks.com> wrote:
>>>>
>>>>> As the error suggests it needs to be an utterable, you can add it
>>>>> by using the following line, the second arg must be a collection.
>>>>> *config.put(Config.SUPERVISOR_SLOTS_PORTS,
>>>>> Lists.newArrayList(7000));*
>>>>>
>>>>> On Dec 4, 2014, at 8:58 AM, Kushan Maskey
>>>>> <ku...@mmillerassociates.com> wrote:
>>>>>
>>>>>> Is there a way I can set supervisor_slots_port in the config in
>>>>>> the topology class? I tried to set it up like below
>>>>>>
>>>>>> *config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
>>>>>>
>>>>>> but I get error as below. *InvalidTopologyException(msg:Field
>>>>>> supervisor.slots.ports must be an Iterable of java.lang.Number*
>>>>>> Thanks.
>>>>>>
>>>>>> --
>>>>>> Kushan Maskey
>>>>>> 817.403.7500
>>>>>> M. Miller & Associates[3] kushan.maskey@mmillerassociates.com
>>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use
>>>> of the individual or entity to which it is addressed and may
>>>> contain information that is confidential, privileged and exempt
>>>> from disclosure under applicable law. If the reader of this message
>>>> is not the intended recipient, you are hereby notified that any
>>>> printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you
>>>> have received this communication in error, please contact the
>>>> sender immediately and delete it from your system. Thank You.
>>>
>>
>
Links:
1. http://mmillerassociates.com/
2. http://mmillerassociates.com/
3. http://mmillerassociates.com/
Re: supervisor.slots.port setup from storm config
Posted by Kushan Maskey <ku...@mmillerassociates.com>.
Harsh,
I could not find worker.xml in my storm package. Does it come with storm
0.9.2 package? Thanks.
--
Kushan Maskey
817.403.7500
M. Miller & Associates <http://mmillerassociates.com/>
kushan.maskey@mmillerassociates.com
On Thu, Dec 4, 2014 at 12:11 PM, Kushan Maskey <
kushan.maskey@mmillerassociates.com> wrote:
> I have not and I think I will try that. I will keep you posted.
>
> Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>
> On Thu, Dec 4, 2014 at 11:30 AM, Harsha <st...@harsha.io> wrote:
>
>>
>> Kushan,
>> Did you try modifying logback/worker.xml . Since you have
>> multiple topologies if they have distinct java packages you can add per
>> package log4j config to worker.xml . You probably might see duplicate
>> logging in worker-port.log files additional to the topology log files.
>> Thats one way I would try separating out topology logging .
>> -Harsha
>>
>> On Thu, Dec 4, 2014, at 09:23 AM, Parth Brahmbhatt wrote:
>>
>> Harsha is right, if you are trying to do this as part of topology
>> submission, even though the param will get accepted it will have no effect.
>> On Dec 4, 2014, at 9:01 AM, Parth Brahmbhatt <pb...@hortonworks.com>
>> wrote:
>>
>>
>> As the error suggests it needs to be an utterable, you can add it by
>> using the following line, the second arg must be a collection.
>> *config.put(Config.SUPERVISOR_SLOTS_PORTS, Lists.newArrayList(7000));*
>>
>> On Dec 4, 2014, at 8:58 AM, Kushan Maskey <
>> kushan.maskey@mmillerassociates.com> wrote:
>>
>>
>> Is there a way I can set supervisor_slots_port in the config in the
>> topology class? I tried to set it up like below
>>
>> *config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
>>
>> but I get error as below.
>>
>> *InvalidTopologyException(msg:Field supervisor.slots.ports must be an
>> Iterable of java.lang.Number*
>> Thanks.
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>> M. Miller & Associates <http://mmillerassociates.com/>
>> kushan.maskey@mmillerassociates.com
>>
>>
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>>
>>
>
>
Re: supervisor.slots.port setup from storm config
Posted by Susheel Kumar Gadalay <sk...@gmail.com>.
How come a single worker is assigned more than one topology?
I don't think it is possible by storm scheduler.
Yes, there could be multiple executors of same topology running in
different threads of worker and the worker log will have a mix of
these different threads.
On 12/4/14, Kushan Maskey <ku...@mmillerassociates.com> wrote:
> I have not and I think I will try that. I will keep you posted.
>
> Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>
> On Thu, Dec 4, 2014 at 11:30 AM, Harsha <st...@harsha.io> wrote:
>
>>
>> Kushan,
>> Did you try modifying logback/worker.xml . Since you have
>> multiple topologies if they have distinct java packages you can add per
>> package log4j config to worker.xml . You probably might see duplicate
>> logging in worker-port.log files additional to the topology log files.
>> Thats one way I would try separating out topology logging .
>> -Harsha
>>
>> On Thu, Dec 4, 2014, at 09:23 AM, Parth Brahmbhatt wrote:
>>
>> Harsha is right, if you are trying to do this as part of topology
>> submission, even though the param will get accepted it will have no
>> effect.
>> On Dec 4, 2014, at 9:01 AM, Parth Brahmbhatt
>> <pb...@hortonworks.com>
>> wrote:
>>
>>
>> As the error suggests it needs to be an utterable, you can add it by
>> using
>> the following line, the second arg must be a collection.
>> *config.put(Config.SUPERVISOR_SLOTS_PORTS, Lists.newArrayList(7000));*
>>
>> On Dec 4, 2014, at 8:58 AM, Kushan Maskey <
>> kushan.maskey@mmillerassociates.com> wrote:
>>
>>
>> Is there a way I can set supervisor_slots_port in the config in the
>> topology class? I tried to set it up like below
>>
>> *config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
>>
>> but I get error as below.
>>
>> *InvalidTopologyException(msg:Field supervisor.slots.ports must be an
>> Iterable of java.lang.Number*
>> Thanks.
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>> M. Miller & Associates <http://mmillerassociates.com/>
>> kushan.maskey@mmillerassociates.com
>>
>>
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is
>> confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>>
>>
>
Re: supervisor.slots.port setup from storm config
Posted by Kushan Maskey <ku...@mmillerassociates.com>.
I have not and I think I will try that. I will keep you posted.
Thanks.
--
Kushan Maskey
817.403.7500
M. Miller & Associates <http://mmillerassociates.com/>
kushan.maskey@mmillerassociates.com
On Thu, Dec 4, 2014 at 11:30 AM, Harsha <st...@harsha.io> wrote:
>
> Kushan,
> Did you try modifying logback/worker.xml . Since you have
> multiple topologies if they have distinct java packages you can add per
> package log4j config to worker.xml . You probably might see duplicate
> logging in worker-port.log files additional to the topology log files.
> Thats one way I would try separating out topology logging .
> -Harsha
>
> On Thu, Dec 4, 2014, at 09:23 AM, Parth Brahmbhatt wrote:
>
> Harsha is right, if you are trying to do this as part of topology
> submission, even though the param will get accepted it will have no effect.
> On Dec 4, 2014, at 9:01 AM, Parth Brahmbhatt <pb...@hortonworks.com>
> wrote:
>
>
> As the error suggests it needs to be an utterable, you can add it by using
> the following line, the second arg must be a collection.
> *config.put(Config.SUPERVISOR_SLOTS_PORTS, Lists.newArrayList(7000));*
>
> On Dec 4, 2014, at 8:58 AM, Kushan Maskey <
> kushan.maskey@mmillerassociates.com> wrote:
>
>
> Is there a way I can set supervisor_slots_port in the config in the
> topology class? I tried to set it up like below
>
> *config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
>
> but I get error as below.
>
> *InvalidTopologyException(msg:Field supervisor.slots.ports must be an
> Iterable of java.lang.Number*
> Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
>
>
Re: supervisor.slots.port setup from storm config
Posted by Harsha <st...@harsha.io>.
Kushan, Did you try modifying logback/worker.xml . Since you have
multiple topologies if they have distinct java packages you can add per
package log4j config to worker.xml . You probably might see duplicate
logging in worker-port.log files additional to the topology log files.
Thats one way I would try separating out topology logging . -Harsha
On Thu, Dec 4, 2014, at 09:23 AM, Parth Brahmbhatt wrote:
> Harsha is right, if you are trying to do this as part of topology
> submission, even though the param will get accepted it will have no
> effect. On Dec 4, 2014, at 9:01 AM, Parth Brahmbhatt
> <pb...@hortonworks.com> wrote:
>
>> As the error suggests it needs to be an utterable, you can add it by
>> using the following line, the second arg must be a collection.
>> *config.put(Config.SUPERVISOR_SLOTS_PORTS,
>> Lists.newArrayList(7000));*
>>
>> On Dec 4, 2014, at 8:58 AM, Kushan Maskey
>> <ku...@mmillerassociates.com> wrote:
>>
>>> Is there a way I can set supervisor_slots_port in the config in the
>>> topology class? I tried to set it up like below
>>>
>>> *config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
>>>
>>> but I get error as below. *InvalidTopologyException(msg:Field
>>> supervisor.slots.ports must be an Iterable of java.lang.Number*
>>> Thanks.
>>>
>>> --
>>> Kushan Maskey
>>> 817.403.7500
>>> M. Miller & Associates[1] kushan.maskey@mmillerassociates.com
>>
>
>
> CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of
> the individual or entity to which it is addressed and may contain
> information that is confidential, privileged and exempt from
> disclosure under applicable law. If the reader of this message is not
> the intended recipient, you are hereby notified that any printing,
> copying, dissemination, distribution, disclosure or forwarding of this
> communication is strictly prohibited. If you have received this
> communication in error, please contact the sender immediately and
> delete it from your system. Thank You.
Links:
1. http://mmillerassociates.com/
Re: supervisor.slots.port setup from storm config
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
Harsha is right, if you are trying to do this as part of topology submission, even though the param will get accepted it will have no effect.
On Dec 4, 2014, at 9:01 AM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
> As the error suggests it needs to be an utterable, you can add it by using the following line, the second arg must be a collection.
> config.put(Config.SUPERVISOR_SLOTS_PORTS, Lists.newArrayList(7000));
>
> On Dec 4, 2014, at 8:58 AM, Kushan Maskey <ku...@mmillerassociates.com> wrote:
>
>> Is there a way I can set supervisor_slots_port in the config in the topology class? I tried to set it up like below
>>
>> config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);
>>
>> but I get error as below.
>> InvalidTopologyException(msg:Field supervisor.slots.ports must be an Iterable of java.lang.Number
>>
>> Thanks.
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>> M. Miller & Associates
>> kushan.maskey@mmillerassociates.com
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: supervisor.slots.port setup from storm config
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
As the error suggests it needs to be an utterable, you can add it by using the following line, the second arg must be a collection.
config.put(Config.SUPERVISOR_SLOTS_PORTS, Lists.newArrayList(7000));
On Dec 4, 2014, at 8:58 AM, Kushan Maskey <ku...@mmillerassociates.com> wrote:
> Is there a way I can set supervisor_slots_port in the config in the topology class? I tried to set it up like below
>
> config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);
>
> but I get error as below.
> InvalidTopologyException(msg:Field supervisor.slots.ports must be an Iterable of java.lang.Number
>
> Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates
> kushan.maskey@mmillerassociates.com
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: supervisor.slots.port setup from storm config
Posted by Kushan Maskey <ku...@mmillerassociates.com>.
What would be your suggestion to achieve this so i can separate my logs for
each topology. I tried to implement log4j but does not seem to create the
log as expected. I have read about storm 0.9.3 has this fix for log4j but i
am not in to position to migrate to that as of yet. We are in the phase of
production testing on storm 0.9.2. Thanks.
--
Kushan Maskey
817.403.7500
M. Miller & Associates <http://mmillerassociates.com/>
kushan.maskey@mmillerassociates.com
On Thu, Dec 4, 2014 at 11:16 AM, Nathan Marz <na...@nathanmarz.com> wrote:
> No, that's not possible.
>
> On Thu, Dec 4, 2014 at 9:13 AM, Kushan Maskey <
> kushan.maskey@mmillerassociates.com> wrote:
>
>> Thanks Parth for your response. That issue has been resolved.
>>
>> Harsh, I have bunch of topologies running in a server. I set the
>> supervisor.slots.ports in the storm.yaml file as
>>
>> supervisor.slots.ports:
>> 6700,6701,6702,6703,6704,6705,6706,6707,6708,6709,6710,6711,6712,6713,6714,6715,6716,6717,6718,6719,6720,6721,6722,6723,6724,6725,6726,6727,6728
>>
>> All my topologies write logs to the work-6700.log and so on which is
>> making it difficult to debug any exception. So I was thinking I will set it
>> up in the topology to write it on a specific logfile based on the port. for
>> example, i wanna write the logs in the series of 7000, 7001 ... for the
>> topoligy 1 and then 6000, 6001 for topology 2 and so on.
>> Is this not possible?
>>
>> Thanks.
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>> M. Miller & Associates <http://mmillerassociates.com/>
>> kushan.maskey@mmillerassociates.com
>>
>> On Thu, Dec 4, 2014 at 11:03 AM, Harsha <st...@harsha.io> wrote:
>>
>>> Kushan,
>>> You won't be able to modify supervisor_slots_ports from the
>>> topology config. Any reason for doing this in topology as you can define
>>> this in storm.yaml.
>>> -Harsha
>>>
>>>
>>> On Thu, Dec 4, 2014, at 08:58 AM, Kushan Maskey wrote:
>>>
>>> Is there a way I can set supervisor_slots_port in the config in the
>>> topology class? I tried to set it up like below
>>>
>>> *config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
>>>
>>> but I get error as below.
>>>
>>> *InvalidTopologyException(msg:Field supervisor.slots.ports must be an
>>> Iterable of java.lang.Number*
>>> Thanks.
>>>
>>> --
>>> Kushan Maskey
>>> 817.403.7500
>>> M. Miller & Associates <http://mmillerassociates.com/>
>>> kushan.maskey@mmillerassociates.com
>>>
>>>
>>>
>>
>>
>
>
> --
> Twitter: @nathanmarz
> http://nathanmarz.com
>
Re: supervisor.slots.port setup from storm config
Posted by Nathan Marz <na...@nathanmarz.com>.
No, that's not possible.
On Thu, Dec 4, 2014 at 9:13 AM, Kushan Maskey <
kushan.maskey@mmillerassociates.com> wrote:
> Thanks Parth for your response. That issue has been resolved.
>
> Harsh, I have bunch of topologies running in a server. I set the
> supervisor.slots.ports in the storm.yaml file as
>
> supervisor.slots.ports:
> 6700,6701,6702,6703,6704,6705,6706,6707,6708,6709,6710,6711,6712,6713,6714,6715,6716,6717,6718,6719,6720,6721,6722,6723,6724,6725,6726,6727,6728
>
> All my topologies write logs to the work-6700.log and so on which is
> making it difficult to debug any exception. So I was thinking I will set it
> up in the topology to write it on a specific logfile based on the port. for
> example, i wanna write the logs in the series of 7000, 7001 ... for the
> topoligy 1 and then 6000, 6001 for topology 2 and so on.
> Is this not possible?
>
> Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>
> On Thu, Dec 4, 2014 at 11:03 AM, Harsha <st...@harsha.io> wrote:
>
>> Kushan,
>> You won't be able to modify supervisor_slots_ports from the
>> topology config. Any reason for doing this in topology as you can define
>> this in storm.yaml.
>> -Harsha
>>
>>
>> On Thu, Dec 4, 2014, at 08:58 AM, Kushan Maskey wrote:
>>
>> Is there a way I can set supervisor_slots_port in the config in the
>> topology class? I tried to set it up like below
>>
>> *config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
>>
>> but I get error as below.
>>
>> *InvalidTopologyException(msg:Field supervisor.slots.ports must be an
>> Iterable of java.lang.Number*
>> Thanks.
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>> M. Miller & Associates <http://mmillerassociates.com/>
>> kushan.maskey@mmillerassociates.com
>>
>>
>>
>
>
--
Twitter: @nathanmarz
http://nathanmarz.com
Re: supervisor.slots.port setup from storm config
Posted by Hefeng Yuan <hf...@rhapsody.com>.
Related question on this, what’s a more preferable deployment pattern
- use lots of ports like the one mentioned below on a super powerful physical machine
- V.S. maybe, one port per node, but have lots of less powerful VM to run each node
Thanks,
Hefeng
> On Dec 4, 2014, at 9:13 AM, Kushan Maskey <ku...@mmillerassociates.com> wrote:
>
> Thanks Parth for your response. That issue has been resolved.
>
> Harsh, I have bunch of topologies running in a server. I set the supervisor.slots.ports in the storm.yaml file as
>
> supervisor.slots.ports: 6700,6701,6702,6703,6704,6705,6706,6707,6708,6709,6710,6711,6712,6713,6714,6715,6716,6717,6718,6719,6720,6721,6722,6723,6724,6725,6726,6727,6728
>
> All my topologies write logs to the work-6700.log and so on which is making it difficult to debug any exception. So I was thinking I will set it up in the topology to write it on a specific logfile based on the port. for example, i wanna write the logs in the series of 7000, 7001 ... for the topoligy 1 and then 6000, 6001 for topology 2 and so on.
> Is this not possible?
>
> Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com <ma...@mmillerassociates.com>
> On Thu, Dec 4, 2014 at 11:03 AM, Harsha <storm@harsha.io <ma...@harsha.io>> wrote:
> Kushan,
> You won't be able to modify supervisor_slots_ports from the topology config. Any reason for doing this in topology as you can define this in storm.yaml.
> -Harsha
>
>
> On Thu, Dec 4, 2014, at 08:58 AM, Kushan Maskey wrote:
>> Is there a way I can set supervisor_slots_port in the config in the topology class? I tried to set it up like below
>>
>> config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);
>>
>> but I get error as below.
>> InvalidTopologyException(msg:Field supervisor.slots.ports must be an Iterable of java.lang.Number
>>
>> Thanks.
>>
>> --
>> Kushan Maskey
>> 817.403.7500 <tel:817.403.7500>
>> M. Miller & Associates <http://mmillerassociates.com/>
>> kushan.maskey@mmillerassociates.com <ma...@mmillerassociates.com>
>
>
Re: supervisor.slots.port setup from storm config
Posted by Kushan Maskey <ku...@mmillerassociates.com>.
Thanks Parth for your response. That issue has been resolved.
Harsh, I have bunch of topologies running in a server. I set the
supervisor.slots.ports in the storm.yaml file as
supervisor.slots.ports:
6700,6701,6702,6703,6704,6705,6706,6707,6708,6709,6710,6711,6712,6713,6714,6715,6716,6717,6718,6719,6720,6721,6722,6723,6724,6725,6726,6727,6728
All my topologies write logs to the work-6700.log and so on which is making
it difficult to debug any exception. So I was thinking I will set it up in
the topology to write it on a specific logfile based on the port. for
example, i wanna write the logs in the series of 7000, 7001 ... for the
topoligy 1 and then 6000, 6001 for topology 2 and so on.
Is this not possible?
Thanks.
--
Kushan Maskey
817.403.7500
M. Miller & Associates <http://mmillerassociates.com/>
kushan.maskey@mmillerassociates.com
On Thu, Dec 4, 2014 at 11:03 AM, Harsha <st...@harsha.io> wrote:
> Kushan,
> You won't be able to modify supervisor_slots_ports from the
> topology config. Any reason for doing this in topology as you can define
> this in storm.yaml.
> -Harsha
>
>
> On Thu, Dec 4, 2014, at 08:58 AM, Kushan Maskey wrote:
>
> Is there a way I can set supervisor_slots_port in the config in the
> topology class? I tried to set it up like below
>
> *config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
>
> but I get error as below.
>
> *InvalidTopologyException(msg:Field supervisor.slots.ports must be an
> Iterable of java.lang.Number*
> Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>
>
>
Re: supervisor.slots.port setup from storm config
Posted by Kushan Maskey <ku...@mmillerassociates.com>.
Can we define this on storm.yaml for each topology?
--
Kushan Maskey
817.403.7500
M. Miller & Associates <http://mmillerassociates.com/>
kushan.maskey@mmillerassociates.com
On Thu, Dec 4, 2014 at 11:03 AM, Harsha <st...@harsha.io> wrote:
> Kushan,
> You won't be able to modify supervisor_slots_ports from the
> topology config. Any reason for doing this in topology as you can define
> this in storm.yaml.
> -Harsha
>
>
> On Thu, Dec 4, 2014, at 08:58 AM, Kushan Maskey wrote:
>
> Is there a way I can set supervisor_slots_port in the config in the
> topology class? I tried to set it up like below
>
> *config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
>
> but I get error as below.
>
> *InvalidTopologyException(msg:Field supervisor.slots.ports must be an
> Iterable of java.lang.Number*
> Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>
>
>
Re: supervisor.slots.port setup from storm config
Posted by Harsha <st...@harsha.io>.
Kushan, You won't be able to modify supervisor_slots_ports from the
topology config. Any reason for doing this in topology as you can define
this in storm.yaml. -Harsha
On Thu, Dec 4, 2014, at 08:58 AM, Kushan Maskey wrote:
> Is there a way I can set supervisor_slots_port in the config in the
> topology class? I tried to set it up like below
>
> *config.put(Config.SUPERVISOR_SLOTS_PORTS, 7000);*
>
> but I get error as below. *InvalidTopologyException(msg:Field
> supervisor.slots.ports must be an Iterable of java.lang.Number*
> Thanks.
>
> --
> Kushan Maskey
> 817.403.7500
> M. Miller & Associates[1] kushan.maskey@mmillerassociates.com
Links:
1. http://mmillerassociates.com/