how to print topology logs into a seperate log file rather than worker.log

[hong mao <ma...@gmail.com>]

When I run a topology on storm 0.10.0, each conponent's log is written to
***-worker-***.log, mixed together with logs generated by storm framework.

I found it is very inconvenient to debug my topology, I have to search
carefully to find out which logs are my spout/bolt's logs .

Does anyone have a good solution for this case? Or are there any log4j
configuration to support this feature? So that I can print my log to files
in a seperated directory.

Thank you!

Re: how to print topology logs into a seperate log file rather than worker.log

[Justin Hopper <ju...@dev9.com>]

How exactly are you configuring log4j2? I am seeing in the os process list that the storm is started with '-Dlog4j.configurationFile=<install_dir>/log4j2/worker.xml’. If you are configuring it programmatically, I don’t see why Log4J would prevent a “reconfiguration”. In fact, configuration file “watcher” (which does this on change) is a feature of Log4J.

Justin

On Dec 23, 2015, at 23:38, hong mao <ma...@gmail.com>> wrote:

Justin,
I've tryed configuring log4j2 in my own custorm log4j2.xml with in my topology. When running locally, my custom logs could be written to the location I specified. But when running in storm cluster, all logs are written to ***-worker-***.log, the reason may be supervisor has already loaded its system-wide log4j2.xml, my custom configuration has no right to override the default configuration.

2015-12-24 15:12 GMT+08:00 Justin Hopper <ju...@dev9.com>>:
Hong,

Re-reading your message…you will need to change the configuration of the log4j/worker.xml file so that components/loggers under your package name (and those that you are interested go into a separate appender.

So you’ll have to add a <Logger name=“org.maohong.tech”> and a <Logger name=“com.my.thirdparty.package”> etc etc. and then create Appenders for Files that are separate from those already declared (e.g. A1).

As an alternative approach have you tried configuring Log4J2 with your own custom log4j2.xml from within your topology?

Thanks,

Justin

> On Dec 23, 2015, at 22:52, hong mao <ma...@gmail.com>> wrote:
>
> When I run a topology on storm 0.10.0, each conponent's log is written to ***-worker-***.log, mixed together with logs generated by storm framework.
>
> I found it is very inconvenient to debug my topology, I have to search carefully to find out which logs are my spout/bolt's logs .
>
> Does anyone have a good solution for this case? Or are there any log4j configuration to support this feature? So that I can print my log to files in a seperated directory.
>
> Thank you!

Re: how to print topology logs into a seperate log file rather than worker.log

[hong mao <ma...@gmail.com>]

Justin,
I've tryed configuring log4j2 in my own custorm log4j2.xml with in my
topology. When running locally, my custom logs could be written to the
location I specified. But when running in storm cluster, all logs are
written to ***-worker-***.log, the reason may be supervisor has already
loaded its system-wide log4j2.xml, my custom configuration has no right to
override the default configuration.

2015-12-24 15:12 GMT+08:00 Justin Hopper <ju...@dev9.com>:

> Hong,
>
> Re-reading your message…you will need to change the configuration of the
> log4j/worker.xml file so that components/loggers under your package name
> (and those that you are interested go into a separate appender.
>
> So you’ll have to add a <Logger name=“org.maohong.tech”> and a <Logger
> name=“com.my.thirdparty.package”> etc etc. and then create Appenders for
> Files that are separate from those already declared (e.g. A1).
>
> As an alternative approach have you tried configuring Log4J2 with your own
> custom log4j2.xml from within your topology?
>
> Thanks,
>
> Justin
>
> > On Dec 23, 2015, at 22:52, hong mao <ma...@gmail.com> wrote:
> >
> > When I run a topology on storm 0.10.0, each conponent's log is written
> to ***-worker-***.log, mixed together with logs generated by storm
> framework.
> >
> > I found it is very inconvenient to debug my topology, I have to search
> carefully to find out which logs are my spout/bolt's logs .
> >
> > Does anyone have a good solution for this case? Or are there any log4j
> configuration to support this feature? So that I can print my log to files
> in a seperated directory.
> >
> > Thank you!
>
>

Re: how to print topology logs into a seperate log file rather than worker.log

[Justin Hopper <ju...@dev9.com>]

Hong,

Re-reading your message…you will need to change the configuration of the log4j/worker.xml file so that components/loggers under your package name (and those that you are interested go into a separate appender.

So you’ll have to add a <Logger name=“org.maohong.tech”> and a <Logger name=“com.my.thirdparty.package”> etc etc. and then create Appenders for Files that are separate from those already declared (e.g. A1).

As an alternative approach have you tried configuring Log4J2 with your own custom log4j2.xml from within your topology?

Thanks,

Justin

> On Dec 23, 2015, at 22:52, hong mao <ma...@gmail.com> wrote:
> 
> When I run a topology on storm 0.10.0, each conponent's log is written to ***-worker-***.log, mixed together with logs generated by storm framework.
> 
> I found it is very inconvenient to debug my topology, I have to search carefully to find out which logs are my spout/bolt's logs .
> 
> Does anyone have a good solution for this case? Or are there any log4j configuration to support this feature? So that I can print my log to files in a seperated directory.
> 
> Thank you!

Re: how to print topology logs into a seperate log file rather than worker.log

[Justin Hopper <ju...@dev9.com>]

I was just looking through this.

If you look under the storm-install-dir/log4j2 directory there is a worker.xml file in there. You can remove the parameters for the log file name and hard code it. 

Alternatively, in the storm-install-dir/conf directory there is a storm_env.ini file. If you set STORM_JAR_JVM_OPTS: -Dlogfile.name=<some standard name>, then all files create will use this value.

I believe this will need to be configured on each supervisor but I am a bit uncertain about that.

Thanks,

Justin

> On Dec 23, 2015, at 22:52, hong mao <ma...@gmail.com> wrote:
> 
> When I run a topology on storm 0.10.0, each conponent's log is written to ***-worker-***.log, mixed together with logs generated by storm framework.
> 
> I found it is very inconvenient to debug my topology, I have to search carefully to find out which logs are my spout/bolt's logs .
> 
> Does anyone have a good solution for this case? Or are there any log4j configuration to support this feature? So that I can print my log to files in a seperated directory.
> 
> Thank you!