You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Fang, Yiming " <yi...@citi.com> on 2014/08/15 17:32:08 UTC
Storm PROD Server log folder configuration issue
Hi All,
I am currently working on a PROD server setup task on storm 0.9.2 .
Trying to configure storm server log folder inside storm python script:
def exec_storm_class(klass, jvmtype="-server", jvmopts=[], extrajars=[], args=[], fork=False):
global CONFFILE
all_args = [
"java", jvmtype, get_config_opts(),
"-Dstorm.home=" + STORM_DIR,
"-Dstorm.server.log.path=" + STORM_SERVER_LOG_PATH,
"-Djava.library.path=" + confvalue("java.library.path", extrajars),
"-Dstorm.conf.file=" + CONFFILE,
"-cp", get_classpath(extrajars),
] + jvmopts + [klass] + list(args)
print "Running: " + " ".join(all_args)
if fork:
os.spawnvp(os.P_WAIT, "java", all_args)
else:
os.execvp("java", all_args) # replaces the current process and never returns
I pass STORM_SERVER_LOG_PATH as system env parameter and then inside cluster.xml I
Replace the file with new config
<configuration scan="true" scanPeriod="60 seconds">
<appender name="A1" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${storm.server.log.path}/logs/${logfile.name}</file>
Result:
I could have all following logs in new place:
access.log, metrics.log, ui.log, nimbus.log, supervisor.log
but the worker-6702 6703 log just in the original server location at the following place
when I setup my topology:
bash-3.2$ cd /opt/gpf/realtime/storm/0.9.2/bin
bash-3.2$ ls
storm storm.cmd storm-config.cmd storm.server.log.path_IS_UNDEFINED
bash-3.2$ cd storm.server.log.path_IS_UNDEFINED/
bash-3.2$ ls
logs
bash-3.2$ cd logs
bash-3.2$ ls
access.log metrics.log worker-6702.log worker-6703.log
Can anyone help?
Thanks and regards,
Yiming
Re: Storm PROD Server log folder configuration issue
Posted by Harsha <st...@harsha.io>.
Hi Yiming,
Looks like we only have tag for the last release.
you can checkout a tag "git checkout
tags/[1]v0.9.2-incubating".
Thanks,
-Harsha
On Sun, Aug 17, 2014, at 08:15 PM, Fang, Yiming wrote:
Hi Harsha,
Thanks a lot for the help.
As long as we are working on 0.9.2. I will try building one
storm-core-0.9.2.jar to replace existing one on server.
BTW just to confirm, we do not have a 0.9.2 branch that I could
check out from GIT? Seems only thing we get is a 0.9.2 tag.
Regards,
Yiming
From: Harsha [mailto:storm@harsha.io]
Sent: Friday, August 15, 2014 11:44 PM
To: user@storm.incubator.apache.org
Subject: Re: Storm PROD Server log folder configuration issue
Hi Yiming,
This is a known bug in 0.9.2
. [2]https://issues.apache.org/jira/browse/STORM-279.
The bug was that supervisor not forwarding
storm.server.log.path opt to the worker.
[3]https://github.com/apache/incubator-storm/commit/598acf97109
20028ed0c240dc6add02a895f2f48#diff-8a8d97993ededcb27c19504b9e88
9e6f . From 0.9.3 users can define STORM_LOG_DIR and all the
logs will be in that location. By default this would be under
STORM_HOME/logs.
-Harsha
On Fri, Aug 15, 2014, at 08:32 AM, Fang, Yiming wrote:
Hi All,
I am currently working on a PROD server setup task on storm
0.9.2 .
Trying to configure storm server log folder inside storm python
script:
def exec_storm_class(klass, jvmtype="-server", jvmopts=[],
extrajars=[], args=[], fork=False):
global CONFFILE
all_args = [
"java", jvmtype, get_config_opts(),
"-Dstorm.home=" + STORM_DIR,
"-Dstorm.server.log.path=" + STORM_SERVER_LOG_PATH,
"-Djava.library.path=" + confvalue("java.library.path",
extrajars),
"-Dstorm.conf.file=" + CONFFILE,
"-cp", get_classpath(extrajars),
] + jvmopts + [klass] + list(args)
print "Running: " + " ".join(all_args)
if fork:
os.spawnvp(os.P_WAIT, "java", all_args)
else:
os.execvp("java", all_args) # replaces the current
process and never returns
I pass STORM_SERVER_LOG_PATHas system env parameter and then
inside cluster.xml I
Replace the file with new config
<configurationscan="true"scanPeriod="60 seconds">
<appendername="A1"class="ch.qos.logback.core.rolling.RollingFil
eAppender">
<file>${storm.server.log.path}/logs/${logfile.name}</file>
Result:
I could have all following logs in new place:
access.log, metrics.log, ui.log, nimbus.log, supervisor.log
but the worker-6702 6703 log just in the original server
location at the following place
when I setup my topology:
bash-3.2$ cd /opt/gpf/realtime/storm/0.9.2/bin
bash-3.2$ ls
storm storm.cmd storm-config.cmd
storm.server.log.path_IS_UNDEFINED
bash-3.2$ cd storm.server.log.path_IS_UNDEFINED/
bash-3.2$ ls
logs
bash-3.2$ cd logs
bash-3.2$ ls
access.log metrics.log worker-6702.log worker-6703.log
Can anyone help?
Thanks and regards,
Yiming
References
1. https://github.com/apache/incubator-storm/commit/24d4a14de310cbbfebdc4a50d8cc9d86f9943087
2. https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/browse/STORM-279&k=wdHsQuqY0Mqq1fNjZGIYnA%3D%3D%0A&r=CRkaly%2Bvupx2pvTJzswpCvi%2F4%2BxH3geu9hee3ZD15Go%3D%0A&m=4QJR%2Fbp%2B9R22nGkeLzSyLrtQRw7ypah7qAvbo%2F%2F6o0c%3D%0A&s=e36e79c44edc3fd64d244944e74fb86355fbfb0c3db94b53eb09f08a437370e5
3. https://urldefense.proofpoint.com/v1/url?u=https://github.com/apache/incubator-storm/commit/598acf9710920028ed0c240dc6add02a895f2f48%23diff-8a8d97993ededcb27c19504b9e889e6f&k=wdHsQuqY0Mqq1fNjZGIYnA%3D%3D%0A&r=CRkaly%2Bvupx2pvTJzswpCvi%2F4%2BxH3geu9hee3ZD15Go%3D%0A&m=4QJR%2Fbp%2B9R22nGkeLzSyLrtQRw7ypah7qAvbo%2F%2F6o0c%3D%0A&s=1dfdba91e6603d009cbf355850a7787a882e0f351baf79c4795080d964005637
RE: Storm PROD Server log folder configuration issue
Posted by "Fang, Yiming " <yi...@citi.com>.
Hi Harsha,
Thanks a lot for the help.
As long as we are working on 0.9.2. I will try building one storm-core-0.9.2.jar to replace existing one on server.
BTW just to confirm, we do not have a 0.9.2 branch that I could check out from GIT? Seems only thing we get is a 0.9.2 tag.
Regards,
Yiming
From: Harsha [mailto:storm@harsha.io]
Sent: Friday, August 15, 2014 11:44 PM
To: user@storm.incubator.apache.org
Subject: Re: Storm PROD Server log folder configuration issue
Hi Yiming,
This is a known bug in 0.9.2 . https://issues.apache.org/jira/browse/STORM-279<https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/browse/STORM-279&k=wdHsQuqY0Mqq1fNjZGIYnA%3D%3D%0A&r=CRkaly%2Bvupx2pvTJzswpCvi%2F4%2BxH3geu9hee3ZD15Go%3D%0A&m=4QJR%2Fbp%2B9R22nGkeLzSyLrtQRw7ypah7qAvbo%2F%2F6o0c%3D%0A&s=e36e79c44edc3fd64d244944e74fb86355fbfb0c3db94b53eb09f08a437370e5>.
The bug was that supervisor not forwarding storm.server.log.path opt to the worker.
https://github.com/apache/incubator-storm/commit/598acf9710920028ed0c240dc6add02a895f2f48#diff-8a8d97993ededcb27c19504b9e889e6f<https://urldefense.proofpoint.com/v1/url?u=https://github.com/apache/incubator-storm/commit/598acf9710920028ed0c240dc6add02a895f2f48%23diff-8a8d97993ededcb27c19504b9e889e6f&k=wdHsQuqY0Mqq1fNjZGIYnA%3D%3D%0A&r=CRkaly%2Bvupx2pvTJzswpCvi%2F4%2BxH3geu9hee3ZD15Go%3D%0A&m=4QJR%2Fbp%2B9R22nGkeLzSyLrtQRw7ypah7qAvbo%2F%2F6o0c%3D%0A&s=1dfdba91e6603d009cbf355850a7787a882e0f351baf79c4795080d964005637> . From 0.9.3 users can define STORM_LOG_DIR and all the logs will be in that location. By default this would be under STORM_HOME/logs.
-Harsha
On Fri, Aug 15, 2014, at 08:32 AM, Fang, Yiming wrote:
Hi All,
I am currently working on a PROD server setup task on storm 0.9.2 .
Trying to configure storm server log folder inside storm python script:
def exec_storm_class(klass, jvmtype="-server", jvmopts=[], extrajars=[], args=[], fork=False):
global CONFFILE
all_args = [
"java", jvmtype, get_config_opts(),
"-Dstorm.home=" + STORM_DIR,
"-Dstorm.server.log.path=" + STORM_SERVER_LOG_PATH,
"-Djava.library.path=" + confvalue("java.library.path", extrajars),
"-Dstorm.conf.file=" + CONFFILE,
"-cp", get_classpath(extrajars),
] + jvmopts + [klass] + list(args)
print "Running: " + " ".join(all_args)
if fork:
os.spawnvp(os.P_WAIT, "java", all_args)
else:
os.execvp("java", all_args) # replaces the current process and never returns
I pass STORM_SERVER_LOG_PATHas system env parameter and then inside cluster.xml I
Replace the file with new config
<configurationscan="true"scanPeriod="60 seconds">
<appender name="A1" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${storm.server.log.path}/logs/${logfile.name}</file>
Result:
I could have all following logs in new place:
access.log, metrics.log, ui.log, nimbus.log, supervisor.log
but the worker-6702 6703 log just in the original server location at the following place
when I setup my topology:
bash-3.2$ cd /opt/gpf/realtime/storm/0.9.2/bin
bash-3.2$ ls
storm storm.cmd storm-config.cmd storm.server.log.path_IS_UNDEFINED
bash-3.2$ cd storm.server.log.path_IS_UNDEFINED/
bash-3.2$ ls
logs
bash-3.2$ cd logs
bash-3.2$ ls
access.log metrics.log worker-6702.log worker-6703.log
Can anyone help?
Thanks and regards,
Yiming
Re: Storm PROD Server log folder configuration issue
Posted by Harsha <st...@harsha.io>.
Hi Yiming,
This is a known bug in 0.9.2
. [1]https://issues.apache.org/jira/browse/STORM-279.
The bug was that supervisor not forwarding
storm.server.log.path opt to the worker.
[2]https://github.com/apache/incubator-storm/commit/598acf97109
20028ed0c240dc6add02a895f2f48#diff-8a8d97993ededcb27c19504b9e88
9e6f . From 0.9.3 users can define STORM_LOG_DIR and all the
logs will be in that location. By default this would be under
STORM_HOME/logs.
-Harsha
On Fri, Aug 15, 2014, at 08:32 AM, Fang, Yiming wrote:
Hi All,
I am currently working on a PROD server setup task on storm
0.9.2 .
Trying to configure storm server log folder inside storm python
script:
def exec_storm_class(klass, jvmtype="-server", jvmopts=[],
extrajars=[], args=[], fork=False):
global CONFFILE
all_args = [
"java", jvmtype, get_config_opts(),
"-Dstorm.home=" + STORM_DIR,
"-Dstorm.server.log.path=" + STORM_SERVER_LOG_PATH,
"-Djava.library.path=" + confvalue("java.library.path",
extrajars),
"-Dstorm.conf.file=" + CONFFILE,
"-cp", get_classpath(extrajars),
] + jvmopts + [klass] + list(args)
print "Running: " + " ".join(all_args)
if fork:
os.spawnvp(os.P_WAIT, "java", all_args)
else:
os.execvp("java", all_args) # replaces the current
process and never returns
I pass STORM_SERVER_LOG_PATHas system env parameter and then
inside cluster.xml I
Replace the file with new config
<configurationscan="true"scanPeriod="60 seconds">
<appender name="A1"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${storm.server.log.path}/logs/${logfile.name}</file>
Result:
I could have all following logs in new place:
access.log, metrics.log, ui.log, nimbus.log, supervisor.log
but the worker-6702 6703 log just in the original server
location at the following place
when I setup my topology:
bash-3.2$ cd /opt/gpf/realtime/storm/0.9.2/bin
bash-3.2$ ls
storm storm.cmd storm-config.cmd
storm.server.log.path_IS_UNDEFINED
bash-3.2$ cd storm.server.log.path_IS_UNDEFINED/
bash-3.2$ ls
logs
bash-3.2$ cd logs
bash-3.2$ ls
access.log metrics.log worker-6702.log worker-6703.log
Can anyone help?
Thanks and regards,
Yiming
References
1. https://issues.apache.org/jira/browse/STORM-279
2. https://github.com/apache/incubator-storm/commit/598acf9710920028ed0c240dc6add02a895f2f48#diff-8a8d97993ededcb27c19504b9e889e6f