You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Laurens Vets <la...@daemon.be> on 2017/06/09 16:15:32 UTC
[DISCUSSION] How to profile storm component taking massive load.
Hi list,
I've installed metron (pre-)0.4.0 on CentOS 7 (VM) and when I start all
services, the load is really high (as in 20+ on 4 and 8 CPUs). The same
install on a CentOS 6 VM is fine with a load of 0.5.
When I look at htop, the pid 27867 has 300%+ CPU% which is:
storm 27867 328 3.2 6653844 1056312 ? Sl 11:53 8:52
/usr/jdk64/jdk1.8.0_77/bin/java -server -Xmx768m
-javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM
-Djava.library.path=/data1/hadoop/storm/supervisor/stormdist/enrichment-1-1496861469/resources/Linux-amd64:/data1/hadoop/storm/supervisor/stormdist/enrichment-1-1496861469/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib
-Dlogfile.name=worker.log -Dstorm.home=/usr/hdp/2.5.3.0-37/storm
-Dworkers.artifacts=/var/log/storm/workers-artifacts -Dstorm.conf.file=
-Dstorm.options= -Dstorm.log.dir=/var/log/storm
-Djava.io.tmpdir=/data1/hadoop/storm/workers/44e84652-9940-430a-ae10-7ba5b1962a84/tmp
-Dlogging.sensitivity=S3
-Dlog4j.configurationFile=/usr/hdp/2.5.3.0-37/storm/log4j2/worker.xml
-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector
-Dstorm.id=enrichment-1-1496861469
-Dworker.id=44e84652-9940-430a-ae10-7ba5b1962a84 -Dworker.port=6700 -cp
/usr/hdp/2.5.3.0-37/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.3.0-37/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-core-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-rename-hack-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/zookeeper.jar:/usr/hdp/2.5.3.0-37/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/current/storm-supervisor/conf:/data1/hadoop/storm/supervisor/
stormdist/enrichment-1-1496861469/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf
org.apache.storm.daemon.worker enrichment-1-1496861469
9c90251f-54cc-42d7-8c4e-6f3bb2528752 6700
44e84652-9940-430a-ae10-7ba5b1962a84
Any idea what might cause this or how I can further troubleshoot this
component?
Again, same install on CentOS 6, everything's fine...
Re: [DISCUSSION] How to profile storm component taking massive load.
Posted by Otto Fowler <ot...@gmail.com>.
https://storm.apache.org/releases/1.0.0/dynamic-worker-profiling.html ?
On June 9, 2017 at 12:15:51, Laurens Vets (laurens@daemon.be) wrote:
Hi list,
I've installed metron (pre-)0.4.0 on CentOS 7 (VM) and when I start all
services, the load is really high (as in 20+ on 4 and 8 CPUs). The same
install on a CentOS 6 VM is fine with a load of 0.5.
When I look at htop, the pid 27867 has 300%+ CPU% which is:
storm 27867 328 3.2 6653844 1056312 ? Sl 11:53 8:52
/usr/jdk64/jdk1.8.0_77/bin/java -server -Xmx768m
-javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM
-Djava.library.path=/data1/hadoop/storm/supervisor/stormdist/enrichment-1-1496861469/resources/Linux-amd64:/data1/hadoop/storm/supervisor/stormdist/enrichment-1-1496861469/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib
-Dlogfile.name=worker.log -Dstorm.home=/usr/hdp/2.5.3.0-37/storm
-Dworkers.artifacts=/var/log/storm/workers-artifacts -Dstorm.conf.file=
-Dstorm.options= -Dstorm.log.dir=/var/log/storm
-Djava.io.tmpdir=/data1/hadoop/storm/workers/44e84652-9940-430a-ae10-7ba5b1962a84/tmp
-Dlogging.sensitivity=S3
-Dlog4j.configurationFile=/usr/hdp/2.5.3.0-37/storm/log4j2/worker.xml
-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector
-Dstorm.id=enrichment-1-1496861469
-Dworker.id=44e84652-9940-430a-ae10-7ba5b1962a84 -Dworker.port=6700 -cp
/usr/hdp/2.5.3.0-37/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.3.0-37/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-core-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-rename-hack-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/zookeeper.jar:/usr/hdp/2.5.3.0-37/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/current/storm-supervisor/conf:/data1/hadoop/storm/supervisor/
stormdist/enrichment-1-1496861469/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf
org.apache.storm.daemon.worker enrichment-1-1496861469
9c90251f-54cc-42d7-8c4e-6f3bb2528752 6700
44e84652-9940-430a-ae10-7ba5b1962a84
Any idea what might cause this or how I can further troubleshoot this
component?
Again, same install on CentOS 6, everything's fine...