You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Jeffery Maass <ma...@gmail.com> on 2015/05/14 23:41:37 UTC
Re: storm topology logsaggregate and processing them
To separate out the worker logs per topology, you need :
* create separate cluster.xml files per application.
** currently, cluster.xml is hardcoded to the storm worker application
** cluster.xml is found in ${STORM_HOME}/logback
** deploy the custom xml files to all of the storm nodes
** change the A1 file name from ${logfile.name} to ${logfile.name}_${
storm.id}
* alter the python file, storm. pass the custom cluster_{program_name} to
the storm application. search the file for "cluster.xml"
The downside is that logback's log rotation as a means of keeping your disk
from overfilling is no longer practical. This becomes more and more true
whenever you are loading and killing topologies on a regular basis.
Basically, your disk will become full of logs. You will need to implement
some other method of log cleanup.
Thank you for your time!
+++++++++++++++++++++
Jeff Maass <ma...@gmail.com>
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+++++++++++++++++++++
On Wed, May 13, 2015 at 1:06 PM, <Ra...@dellteam.com> wrote:
> *Dell - Internal Use - Confidential *
>
> what is every one using to process the storm logs from all over the
> cluster for troubleshooting and debugging purposes.
>
>
>
> The accessing storm logs via the storm UI or tailing worker logs is
> getting to be pretty painful.
>
>
>
> I guess this could be said of a lot of other Hadoop components like Kafka,
> ZooKeeper, HBase and such.
>
>
>
> P.S: Did anyone get logback working properly to separate out topology
> level logs from worker-xxxx.log if so what are the gotchas you ran into.
>
Re: storm topology logsaggregate and processing them
Posted by Jeffery Maass <ma...@gmail.com>.
I do not at this time have suggestions.
Thank you for your time!
+++++++++++++++++++++
Jeff Maass <ma...@gmail.com>
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+++++++++++++++++++++
On Thu, May 14, 2015 at 9:14 PM, <Ra...@dellteam.com> wrote:
> *Dell - Internal Use - Confidential *
>
> Thanks Jeffery, great to see enthusiastic community around storm.
>
>
>
> Do you have any suggestions on the log retention and indexing mechanisms
> (logstash etc?)
>
>
>
> *From:* Jeffery Maass [mailto:maassql@gmail.com]
> *Sent:* Thursday, May 14, 2015 4:42 PM
> *To:* user@storm.apache.org
> *Subject:* Re: storm topology logsaggregate and processing them
>
>
>
> To separate out the worker logs per topology, you need :
>
> * create separate cluster.xml files per application.
> ** currently, cluster.xml is hardcoded to the storm worker application
>
> ** cluster.xml is found in ${STORM_HOME}/logback
>
> ** deploy the custom xml files to all of the storm nodes
>
> ** change the A1 file name from ${logfile.name} to ${logfile.name}_${
> storm.id}
>
> * alter the python file, storm. pass the custom cluster_{program_name} to
> the storm application. search the file for "cluster.xml"
>
>
>
> The downside is that logback's log rotation as a means of keeping your
> disk from overfilling is no longer practical. This becomes more and more
> true whenever you are loading and killing topologies on a regular basis.
> Basically, your disk will become full of logs. You will need to implement
> some other method of log cleanup.
>
>
>
>
> Thank you for your time!
>
> +++++++++++++++++++++
> Jeff Maass <ma...@gmail.com>
> linkedin.com/in/jeffmaass
> stackoverflow.com/users/373418/maassql
> +++++++++++++++++++++
>
>
>
> On Wed, May 13, 2015 at 1:06 PM, <Ra...@dellteam.com> wrote:
>
> *Dell - Internal Use - Confidential *
>
> what is every one using to process the storm logs from all over the
> cluster for troubleshooting and debugging purposes.
>
>
>
> The accessing storm logs via the storm UI or tailing worker logs is
> getting to be pretty painful.
>
>
>
> I guess this could be said of a lot of other Hadoop components like Kafka,
> ZooKeeper, HBase and such.
>
>
>
> P.S: Did anyone get logback working properly to separate out topology
> level logs from worker-xxxx.log if so what are the gotchas you ran into.
>
>
>
RE: storm topology logsaggregate and processing them
Posted by Ra...@DellTeam.com.
Dell - Internal Use - Confidential
Thanks Jeffery, great to see enthusiastic community around storm.
Do you have any suggestions on the log retention and indexing mechanisms (logstash etc?)
From: Jeffery Maass [mailto:maassql@gmail.com]
Sent: Thursday, May 14, 2015 4:42 PM
To: user@storm.apache.org
Subject: Re: storm topology logsaggregate and processing them
To separate out the worker logs per topology, you need :
* create separate cluster.xml files per application.
** currently, cluster.xml is hardcoded to the storm worker application
** cluster.xml is found in ${STORM_HOME}/logback
** deploy the custom xml files to all of the storm nodes
** change the A1 file name from ${logfile.name<http://logfile.name>} to ${logfile.name<http://logfile.name>}_${storm.id<http://storm.id>}
* alter the python file, storm. pass the custom cluster_{program_name} to the storm application. search the file for "cluster.xml"
The downside is that logback's log rotation as a means of keeping your disk from overfilling is no longer practical. This becomes more and more true whenever you are loading and killing topologies on a regular basis. Basically, your disk will become full of logs. You will need to implement some other method of log cleanup.
Thank you for your time!
+++++++++++++++++++++
Jeff Maass<ma...@gmail.com>
linkedin.com/in/jeffmaass<http://linkedin.com/in/jeffmaass>
stackoverflow.com/users/373418/maassql<http://stackoverflow.com/users/373418/maassql>
+++++++++++++++++++++
On Wed, May 13, 2015 at 1:06 PM, <Ra...@dellteam.com>> wrote:
Dell - Internal Use - Confidential
what is every one using to process the storm logs from all over the cluster for troubleshooting and debugging purposes.
The accessing storm logs via the storm UI or tailing worker logs is getting to be pretty painful.
I guess this could be said of a lot of other Hadoop components like Kafka, ZooKeeper, HBase and such.
P.S: Did anyone get logback working properly to separate out topology level logs from worker-xxxx.log if so what are the gotchas you ran into.