You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Tyler Moore <tm...@goflyball.com> on 2017/01/11 19:16:34 UTC

Offset lag for Kafka - Storm Throwing error

Hi All,

I am still receiving the storm kafka "Offset lag for Kafka" in storm after
building metron 0.3.0 with HDP2.5 profile. It seems odd that It built fine
and shows the profile when running:
/usr/bin/storm version
Yet my storm ui shows version 1.0.1.2.5.0.0-1245 which I believe should be
1.1.0 as per the HDP2.5 profile.

Error from storm: Unable to get offset lags for kafka. Reason:
java.lang.NullPointerException at
org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:269)
at
org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:127)

Output of /usr/bin/storm version :
Running: /usr/jdk64/jdk1.8.0_77/bin/java -client -Ddaemon.name=
-Dstorm.options= -Dstorm.home=/usr/hdp/2.5.0.0-1245/storm
-Dstorm.log.dir=/var/log/storm
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib
-Dstorm.conf.file= -cp
/usr/hdp/2.5.0.0-1245/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.0.0-1245/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.0.0-1245/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1245.jar:/usr/hdp/2.5.0.0-1245/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/zookeeper.jar:/usr/hdp/2.5.0.0-1245/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.0.0-1245/storm/lib/storm-core-1.0.1.2.5.0.0-1245.jar:/usr/hdp/2.5.0.0-1245/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.0.0-1245/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.0.0-1245/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.0.0-1245/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.0.0-1245/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.0.0-1245/storm/extlib-daemon/ranger-storm-plugin-shim-0.6.0.2.5.0.0-1245.jar:/usr/hdp/2.5.0.0-1245/storm/extlib-daemon/ranger-plugin-classloader-0.6.0.2.5.0.0-1245.jar:/usr/hdp/2.5.0.0-1245/storm/extlib-daemon/ojdbc6.jar:/usr/hdp/current/storm-supervisor/conf
org.apache.storm.utils.VersionInfo
Storm 1.0.1.2.5.0.0-1245
URL git@github.com:hortonworks/storm.git -r
bab5eeb67b1912439afa029917db0be345bfc1bc
Branch (HEAD detached at bab5eeb)
Compiled by jenkins on 2016-08-26T02:31Z
From source with checksum 6279a7d1c048ffd8c48af6c05fe37020

Pretty sure i'm just missing something here, do I need to update the
storm-kafka dependency?

Regards,

Tyler Moore
Software Engineer
Phone: 248-909-2769
Email: moore.tyler@goflyball.com

Re: Offset lag for Kafka - Storm Throwing error

Posted by Dima Kovalyov <Di...@sstech.us>.
Hello Tyler,

I have the same version of the Storm, so yours is fine.

This error should not be a problem, I was confused by it as well but it appears it just means that you kafka topic is empty or does not exist (the one from which topology should be getting data), here is the quote from James from here https://github.com/apache/incubator-metron/pull/318:
Once you have data in your kafka queue this should go away.

Please stream some data through your parser and it should work just fine.

p.s. I have updated Troubleshooting section in installation manual: https://cwiki.apache.org/confluence/display/METRON/Metron+with+HDP+2.5+bare-metal+install

- Dima

On 01/11/2017 09:17 PM, Tyler Moore wrote:
Hi All,

I am still receiving the storm kafka "Offset lag for Kafka" in storm after building metron 0.3.0 with HDP2.5 profile. It seems odd that It built fine and shows the profile when running:
/usr/bin/storm version
Yet my storm ui shows version 1.0.1.2.5.0.0-1245 which I believe should be 1.1.0 as per the HDP2.5 profile.

Error from storm: Unable to get offset lags for kafka. Reason: java.lang.NullPointerException at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:269) at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:127)

Output of /usr/bin/storm version :
Running: /usr/jdk64/jdk1.8.0_77/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.5.0.0-1245/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.5.0.0-1245/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.0.0-1245/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.0.0-1245/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1245.jar:/usr/hdp/2.5.0.0-1245/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/zookeeper.jar:/usr/hdp/2.5.0.0-1245/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.0.0-1245/storm/lib/storm-core-1.0.1.2.5.0.0-1245.jar:/usr/hdp/2.5.0.0-1245/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.0.0-1245/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.0.0-1245/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.0.0-1245/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.0.0-1245/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.0.0-1245/storm/extlib-daemon/ranger-storm-plugin-shim-0.6.0.2.5.0.0-1245.jar:/usr/hdp/2.5.0.0-1245/storm/extlib-daemon/ranger-plugin-classloader-0.6.0.2.5.0.0-1245.jar:/usr/hdp/2.5.0.0-1245/storm/extlib-daemon/ojdbc6.jar:/usr/hdp/current/storm-supervisor/conf org.apache.storm.utils.VersionInfo
Storm 1.0.1.2.5.0.0-1245
URL git@github.com:hortonworks/storm.git<mailto:git@github.com:hortonworks/storm.git> -r bab5eeb67b1912439afa029917db0be345bfc1bc
Branch (HEAD detached at bab5eeb)
Compiled by jenkins on 2016-08-26T02:31Z
From source with checksum 6279a7d1c048ffd8c48af6c05fe37020

Pretty sure i'm just missing something here, do I need to update the storm-kafka dependency?

Regards,

Tyler Moore
Software Engineer
Phone: 248-909-2769
Email: moore.tyler@goflyball.com<ma...@goflyball.com>
[https://s3-us-west-2.amazonaws.com/s.cdpn.io/581149/flyball-logo.png]

Re: Offset lag for Kafka - Storm Throwing error

Posted by Matt Foley <mf...@hortonworks.com>.
Hi Tyler,
HDP-2.5 uses Storm 1.0.1, not 1.1.0.  Specifically,

-          HDP-2.5.0.0 has Storm build 1.0.1.2.5.0.0-1245

-          HDP-2.5.3.0 (the December maintenance release) has Storm build 1.0.1.2.5.3.0-37
The “1.0.1” is the Apache version number, the “2.5.x.y” is the HDP version number, and the “-NNNN” is the unique build number.

Which profile are you referring to that states Storm 1.1.0 ?
Thanks,
--Matt

From: Tyler Moore <tm...@goflyball.com>
Reply-To: "user@metron.incubator.apache.org" <us...@metron.incubator.apache.org>
Date: Wednesday, January 11, 2017 at 11:16 AM
To: "user@metron.incubator.apache.org" <us...@metron.incubator.apache.org>
Subject: Offset lag for Kafka - Storm Throwing error

Hi All,

I am still receiving the storm kafka "Offset lag for Kafka" in storm after building metron 0.3.0 with HDP2.5 profile. It seems odd that It built fine and shows the profile when running:
/usr/bin/storm version
Yet my storm ui shows version 1.0.1.2.5.0.0-1245 which I believe should be 1.1.0 as per the HDP2.5 profile.

Error from storm: Unable to get offset lags for kafka. Reason: java.lang.NullPointerException at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:269) at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:127)

Output of /usr/bin/storm version :
Running: /usr/jdk64/jdk1.8.0_77/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.5.0.0-1245/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.5.0.0-1245/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.0.0-1245/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.0.0-1245/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1245.jar:/usr/hdp/2.5.0.0-1245/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/zookeeper.jar:/usr/hdp/2.5.0.0-1245/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.0.0-1245/storm/lib/storm-core-1.0.1.2.5.0.0-1245.jar:/usr/hdp/2.5.0.0-1245/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.0.0-1245/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.0.0-1245/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.0.0-1245/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.0.0-1245/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.0.0-1245/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.0.0-1245/storm/extlib-daemon/ranger-storm-plugin-shim-0.6.0.2.5.0.0-1245.jar:/usr/hdp/2.5.0.0-1245/storm/extlib-daemon/ranger-plugin-classloader-0.6.0.2.5.0.0-1245.jar:/usr/hdp/2.5.0.0-1245/storm/extlib-daemon/ojdbc6.jar:/usr/hdp/current/storm-supervisor/conf org.apache.storm.utils.VersionInfo
Storm 1.0.1.2.5.0.0-1245
URL git@github.com:hortonworks/storm.git -r bab5eeb67b1912439afa029917db0be345bfc1bc
Branch (HEAD detached at bab5eeb)
Compiled by jenkins on 2016-08-26T02:31Z
From source with checksum 6279a7d1c048ffd8c48af6c05fe37020

Pretty sure i'm just missing something here, do I need to update the storm-kafka dependency?

Regards,

Tyler Moore
Software Engineer
Phone: 248-909-2769
Email: moore.tyler@goflyball.com<ma...@goflyball.com>
[https://s3-us-west-2.amazonaws.com/s.cdpn.io/581149/flyball-logo.png]