You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Ayub Khan (JIRA)" <ji...@apache.org> on 2016/08/16 14:42:20 UTC
[jira] [Comment Edited] (ATLAS-1121) NPE while submitting topology
in StormHook
[ https://issues.apache.org/jira/browse/ATLAS-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422817#comment-15422817 ]
Ayub Khan edited comment on ATLAS-1121 at 8/16/16 2:41 PM:
-----------------------------------------------------------
Submitting the patch to handle NPE with respect to jackson-databind. [~madhan.neethiraj], [~shwethags], [~suma.shivaprasad] please review the patch
I have tested the patch and the topology submission is successful and the entities are created over Atlas. You will still see the error but the topology metadata submission to Atlas succeeds..
{noformat}
[root@vimal-erie-6-1 storm-topology]# storm jar hcube-storm-topology-0.0.1.jar org.hw.hcube.storm.LogStreamingToplogyV2 phoenix.properties
Running: /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name= -Dstorm.options= -Dstorm.home=/grid/0/hdp/2.5.0.0-1181/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-over-slf4j-1.6.6.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/zookeeper.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/servlet-api-2.5.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/reflectasm-1.10.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/ambari-metrics-storm-sink.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/objenesis-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/clojure-1.7.0.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-slf4j-impl-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/disruptor-3.3.2.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-core-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/minlog-1.3.0.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/slf4j-api-1.7.7.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/kryo-3.0.3.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/asm-5.0.3.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/ring-cors-0.1.5.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/storm-core-1.0.1.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-api-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/extlib/storm-bridge-shim-0.7.0.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/extlib/atlas-plugin-classloader-0.7.0.2.5.0.0-1181.jar org.apache.storm.daemon.ClientJarTransformerRunner org.apache.storm.hack.StormShadeTransformer hcube-storm-topology-0.0.1.jar /tmp/6a601f5663ba11e698cffa163ef6032d.jar
Running: /usr/jdk64/jdk1.8.0_60/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/grid/0/hdp/2.5.0.0-1181/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-over-slf4j-1.6.6.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/zookeeper.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/servlet-api-2.5.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/reflectasm-1.10.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/ambari-metrics-storm-sink.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/objenesis-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/clojure-1.7.0.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-slf4j-impl-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/disruptor-3.3.2.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-core-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/minlog-1.3.0.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/slf4j-api-1.7.7.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/kryo-3.0.3.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/asm-5.0.3.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/ring-cors-0.1.5.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/storm-core-1.0.1.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-api-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/extlib/storm-bridge-shim-0.7.0.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/extlib/atlas-plugin-classloader-0.7.0.2.5.0.0-1181.jar:/tmp/6a601f5663ba11e698cffa163ef6032d.jar:/usr/hdp/current/storm-supervisor/conf:/grid/0/hdp/2.5.0.0-1181/storm/bin -Dstorm.jar=/tmp/6a601f5663ba11e698cffa163ef6032d.jar org.hw.hcube.storm.LogStreamingToplogyV2 phoenix.properties
2339 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -7910590019420719599:-5664763422779979058
2456 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
2543 [main] INFO o.a.s.StormSubmitter - Uploading topology jar /tmp/6a601f5663ba11e698cffa163ef6032d.jar to assigned location: /grid/0/hadoop/storm/nimbus/inbox/stormjar-36b9f10a-bd32-4368-bed8-75ae19104fd9.jar
Start uploading file '/tmp/6a601f5663ba11e698cffa163ef6032d.jar' to '/grid/0/hadoop/storm/nimbus/inbox/stormjar-36b9f10a-bd32-4368-bed8-75ae19104fd9.jar' (161473843 bytes)
[==================================================] 161473843 / 161473843
File '/tmp/6a601f5663ba11e698cffa163ef6032d.jar' uploaded to '/grid/0/hadoop/storm/nimbus/inbox/stormjar-36b9f10a-bd32-4368-bed8-75ae19104fd9.jar' (161473843 bytes)
4418 [main] INFO o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /grid/0/hadoop/storm/nimbus/inbox/stormjar-36b9f10a-bd32-4368-bed8-75ae19104fd9.jar
4419 [main] INFO o.a.s.StormSubmitter - Submitting topology hcube-log-streaming-phoenix in distributed mode with conf {"topology.workers":10,"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-7910590019420719599:-5664763422779979058"}
8342 [main] INFO o.a.s.StormSubmitter - Finished submitting topology: hcube-log-streaming-phoenix
8349 [main] INFO o.a.s.StormSubmitter - Initializing the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
8924 [main] INFO o.a.a.ApplicationProperties - Looking for atlas-application.properties in classpath
8924 [main] INFO o.a.a.ApplicationProperties - Loading atlas-application.properties from file:/etc/storm/2.5.0.0-1181/0/atlas-application.properties
log4j:WARN No appenders could be found for logger (org.apache.atlas.ApplicationProperties).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
10171 [main] INFO o.a.a.h.AtlasHook - Created Atlas Hook
10522 [main] INFO o.a.s.StormSubmitter - Invoking the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
10523 [main] INFO o.a.a.s.h.StormAtlasHook - Collecting metadata for a new storm topology: hcube-log-streaming-phoenix
10634 [main] ERROR o.a.a.s.h.StormAtlasHook - storm topology exception
java.lang.NullPointerException
at java.text.DateFormat.hashCode(DateFormat.java:739) ~[?:1.8.0_60]
at java.util.HashMap.hash(HashMap.java:338) ~[?:1.8.0_60]
at java.util.HashMap.put(HashMap.java:611) ~[?:1.8.0_60]
at java.util.HashSet.add(HashSet.java:219) ~[?:1.8.0_60]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:141) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.createDataSet(StormAtlasHook.java:187) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyInputs(StormAtlasHook.java:149) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyDataSets(StormAtlasHook.java:132) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:89) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:57) [storm-bridge-shim-0.7.0.2.5.0.0-1181.jar:0.7.0.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:285) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:258) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:311) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:347) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:328) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.hw.hcube.storm.LogStreamingToplogyV2.main(LogStreamingToplogyV2.java:164) [6a601f5663ba11e698cffa163ef6032d.jar:?]
10709 [main] ERROR o.a.a.s.h.StormAtlasHook - storm topology exception
java.lang.NullPointerException
at java.text.DateFormat.hashCode(DateFormat.java:739) ~[?:1.8.0_60]
at java.util.HashMap.hash(HashMap.java:338) ~[?:1.8.0_60]
at java.util.HashMap.put(HashMap.java:611) ~[?:1.8.0_60]
at java.util.HashSet.add(HashSet.java:219) ~[?:1.8.0_60]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:141) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.createSpoutInstance(StormAtlasHook.java:310) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.addSpouts(StormAtlasHook.java:294) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.createTopologyGraph(StormAtlasHook.java:280) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:95) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:57) [storm-bridge-shim-0.7.0.2.5.0.0-1181.jar:0.7.0.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:285) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:258) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:311) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:347) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:328) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.hw.hcube.storm.LogStreamingToplogyV2.main(LogStreamingToplogyV2.java:164) [6a601f5663ba11e698cffa163ef6032d.jar:?]
10718 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: org.apache.storm.jdbc.bolt.JdbcInsertBolt@7a5b769b
10718 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: org.apache.storm.jdbc.mapper.SimpleJdbcMapper@f4c0e4e
10718 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='host', val=null, sqlType=12}
10718 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='type', val=null, sqlType=12}
10718 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='seq_num', val=null, sqlType=-5}
10718 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='id', val=null, sqlType=12}
10719 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='cluster', val=null, sqlType=12}
10719 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='event_count', val=null, sqlType=-5}
10719 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='message_md5', val=null, sqlType=12}
10719 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='event_md5', val=null, sqlType=12}
10723 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='ip', val=null, sqlType=12}
10725 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='path', val=null, sqlType=12}
10725 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='file', val=null, sqlType=12}
10725 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='method', val=null, sqlType=12}
10725 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='logger_name', val=null, sqlType=12}
10725 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='logtime', val=null, sqlType=93}
10725 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='line_number', val=null, sqlType=-5}
10732 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='logfile_line_number', val=null, sqlType=-5}
10732 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='level', val=null, sqlType=12}
10732 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='log_message', val=null, sqlType=12}
10732 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: org.apache.storm.jdbc.common.HikariCPConnectionProvider@24361cfc
10737 [main] INFO o.a.a.h.AtlasHook - Adding entity for type: storm_topology
23946 [main] INFO o.a.k.c.p.ProducerConfig - ProducerConfig values:
metric.reporters = []
metadata.max.age.ms = 300000
reconnect.backoff.ms = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
bootstrap.servers = [vimal-erie-6-1.openstacklocal:6667]
ssl.keystore.type = JKS
sasl.mechanism = GSSAPI
max.block.ms = 60000
interceptor.classes = null
ssl.truststore.password = null
client.id =
ssl.endpoint.identification.algorithm = null
request.timeout.ms = 30000
acks = 1
receive.buffer.bytes = 32768
ssl.truststore.type = JKS
retries = 0
ssl.truststore.location = null
ssl.keystore.password = null
send.buffer.bytes = 131072
compression.type = none
metadata.fetch.timeout.ms = 60000
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
buffer.memory = 33554432
timeout.ms = 30000
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.trustmanager.algorithm = PKIX
block.on.buffer.full = false
ssl.key.password = null
sasl.kerberos.min.time.before.relogin = 60000
connections.max.idle.ms = 540000
max.in.flight.requests.per.connection = 5
metrics.num.samples = 2
ssl.protocol = TLS
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
batch.size = 16384
ssl.keystore.location = null
ssl.cipher.suites = null
security.protocol = PLAINTEXT
max.request.size = 1048576
value.serializer = class org.apache.kafka.common.serialization.StringSerializer
ssl.keymanager.algorithm = SunX509
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
linger.ms = 0
24084 [main] INFO o.a.k.c.p.ProducerConfig - ProducerConfig values:
metric.reporters = []
metadata.max.age.ms = 300000
reconnect.backoff.ms = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
bootstrap.servers = [vimal-erie-6-1.openstacklocal:6667]
ssl.keystore.type = JKS
sasl.mechanism = GSSAPI
max.block.ms = 60000
interceptor.classes = null
ssl.truststore.password = null
client.id = producer-1
ssl.endpoint.identification.algorithm = null
request.timeout.ms = 30000
acks = 1
receive.buffer.bytes = 32768
ssl.truststore.type = JKS
retries = 0
ssl.truststore.location = null
ssl.keystore.password = null
send.buffer.bytes = 131072
compression.type = none
metadata.fetch.timeout.ms = 60000
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
buffer.memory = 33554432
timeout.ms = 30000
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.trustmanager.algorithm = PKIX
block.on.buffer.full = false
ssl.key.password = null
sasl.kerberos.min.time.before.relogin = 60000
connections.max.idle.ms = 540000
max.in.flight.requests.per.connection = 5
metrics.num.samples = 2
ssl.protocol = TLS
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
batch.size = 16384
ssl.keystore.location = null
ssl.cipher.suites = null
security.protocol = PLAINTEXT
max.request.size = 1048576
value.serializer = class org.apache.kafka.common.serialization.StringSerializer
ssl.keymanager.algorithm = SunX509
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
linger.ms = 0
24089 [main] WARN o.a.k.c.p.ProducerConfig - The configuration key.deserializer = org.apache.kafka.common.serialization.StringDeserializer was supplied but isn't a known config.
24090 [main] WARN o.a.k.c.p.ProducerConfig - The configuration value.deserializer = org.apache.kafka.common.serialization.StringDeserializer was supplied but isn't a known config.
24090 [main] WARN o.a.k.c.p.ProducerConfig - The configuration hook.group.id = atlas was supplied but isn't a known config.
24090 [main] WARN o.a.k.c.p.ProducerConfig - The configuration partition.assignment.strategy = roundrobin was supplied but isn't a known config.
24090 [main] WARN o.a.k.c.p.ProducerConfig - The configuration zookeeper.connection.timeout.ms = 200 was supplied but isn't a known config.
24091 [main] WARN o.a.k.c.p.ProducerConfig - The configuration zookeeper.session.timeout.ms = 400 was supplied but isn't a known config.
24091 [main] WARN o.a.k.c.p.ProducerConfig - The configuration zookeeper.connect = vimal-erie-6-1.openstacklocal:2181 was supplied but isn't a known config.
24091 [main] WARN o.a.k.c.p.ProducerConfig - The configuration zookeeper.sync.time.ms = 20 was supplied but isn't a known config.
24091 [main] WARN o.a.k.c.p.ProducerConfig - The configuration auto.offset.reset = smallest was supplied but isn't a known config.
24093 [main] INFO o.a.k.c.u.AppInfoParser - Kafka version : 0.10.0.2.5.0.0-1181
24093 [main] INFO o.a.k.c.u.AppInfoParser - Kafka commitId : 022ed507ec080025
{noformat}
Attaching Atlas UI snapshot showing the entities created as part of storm topology submission.
!Atlas 2016-08-16 19-48-59.png|thumbnail!
was (Author: ayubkhan):
Submitting the patch to handle NPE with respect to jackson-databind. [~madhan.neethiraj][~shwethags] [~suma.shivaprasad] please review the patch
I have tested the patch and the topology submission is successful and the entities are created over Atlas. You will still see the error but the topology metadata submission to Atlas succeeds..
{noformat}
[root@vimal-erie-6-1 storm-topology]# storm jar hcube-storm-topology-0.0.1.jar org.hw.hcube.storm.LogStreamingToplogyV2 phoenix.properties
Running: /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name= -Dstorm.options= -Dstorm.home=/grid/0/hdp/2.5.0.0-1181/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-over-slf4j-1.6.6.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/zookeeper.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/servlet-api-2.5.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/reflectasm-1.10.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/ambari-metrics-storm-sink.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/objenesis-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/clojure-1.7.0.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-slf4j-impl-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/disruptor-3.3.2.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-core-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/minlog-1.3.0.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/slf4j-api-1.7.7.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/kryo-3.0.3.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/asm-5.0.3.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/ring-cors-0.1.5.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/storm-core-1.0.1.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-api-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/extlib/storm-bridge-shim-0.7.0.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/extlib/atlas-plugin-classloader-0.7.0.2.5.0.0-1181.jar org.apache.storm.daemon.ClientJarTransformerRunner org.apache.storm.hack.StormShadeTransformer hcube-storm-topology-0.0.1.jar /tmp/6a601f5663ba11e698cffa163ef6032d.jar
Running: /usr/jdk64/jdk1.8.0_60/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/grid/0/hdp/2.5.0.0-1181/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-over-slf4j-1.6.6.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/zookeeper.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/servlet-api-2.5.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/reflectasm-1.10.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/ambari-metrics-storm-sink.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/objenesis-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/clojure-1.7.0.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-slf4j-impl-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/disruptor-3.3.2.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-core-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/minlog-1.3.0.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/slf4j-api-1.7.7.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/kryo-3.0.3.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/asm-5.0.3.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/ring-cors-0.1.5.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/storm-core-1.0.1.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/lib/log4j-api-2.1.jar:/grid/0/hdp/2.5.0.0-1181/storm/extlib/storm-bridge-shim-0.7.0.2.5.0.0-1181.jar:/grid/0/hdp/2.5.0.0-1181/storm/extlib/atlas-plugin-classloader-0.7.0.2.5.0.0-1181.jar:/tmp/6a601f5663ba11e698cffa163ef6032d.jar:/usr/hdp/current/storm-supervisor/conf:/grid/0/hdp/2.5.0.0-1181/storm/bin -Dstorm.jar=/tmp/6a601f5663ba11e698cffa163ef6032d.jar org.hw.hcube.storm.LogStreamingToplogyV2 phoenix.properties
2339 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -7910590019420719599:-5664763422779979058
2456 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
2543 [main] INFO o.a.s.StormSubmitter - Uploading topology jar /tmp/6a601f5663ba11e698cffa163ef6032d.jar to assigned location: /grid/0/hadoop/storm/nimbus/inbox/stormjar-36b9f10a-bd32-4368-bed8-75ae19104fd9.jar
Start uploading file '/tmp/6a601f5663ba11e698cffa163ef6032d.jar' to '/grid/0/hadoop/storm/nimbus/inbox/stormjar-36b9f10a-bd32-4368-bed8-75ae19104fd9.jar' (161473843 bytes)
[==================================================] 161473843 / 161473843
File '/tmp/6a601f5663ba11e698cffa163ef6032d.jar' uploaded to '/grid/0/hadoop/storm/nimbus/inbox/stormjar-36b9f10a-bd32-4368-bed8-75ae19104fd9.jar' (161473843 bytes)
4418 [main] INFO o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /grid/0/hadoop/storm/nimbus/inbox/stormjar-36b9f10a-bd32-4368-bed8-75ae19104fd9.jar
4419 [main] INFO o.a.s.StormSubmitter - Submitting topology hcube-log-streaming-phoenix in distributed mode with conf {"topology.workers":10,"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-7910590019420719599:-5664763422779979058"}
8342 [main] INFO o.a.s.StormSubmitter - Finished submitting topology: hcube-log-streaming-phoenix
8349 [main] INFO o.a.s.StormSubmitter - Initializing the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
8924 [main] INFO o.a.a.ApplicationProperties - Looking for atlas-application.properties in classpath
8924 [main] INFO o.a.a.ApplicationProperties - Loading atlas-application.properties from file:/etc/storm/2.5.0.0-1181/0/atlas-application.properties
log4j:WARN No appenders could be found for logger (org.apache.atlas.ApplicationProperties).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
10171 [main] INFO o.a.a.h.AtlasHook - Created Atlas Hook
10522 [main] INFO o.a.s.StormSubmitter - Invoking the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
10523 [main] INFO o.a.a.s.h.StormAtlasHook - Collecting metadata for a new storm topology: hcube-log-streaming-phoenix
10634 [main] ERROR o.a.a.s.h.StormAtlasHook - storm topology exception
java.lang.NullPointerException
at java.text.DateFormat.hashCode(DateFormat.java:739) ~[?:1.8.0_60]
at java.util.HashMap.hash(HashMap.java:338) ~[?:1.8.0_60]
at java.util.HashMap.put(HashMap.java:611) ~[?:1.8.0_60]
at java.util.HashSet.add(HashSet.java:219) ~[?:1.8.0_60]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:141) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.createDataSet(StormAtlasHook.java:187) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyInputs(StormAtlasHook.java:149) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyDataSets(StormAtlasHook.java:132) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:89) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:57) [storm-bridge-shim-0.7.0.2.5.0.0-1181.jar:0.7.0.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:285) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:258) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:311) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:347) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:328) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.hw.hcube.storm.LogStreamingToplogyV2.main(LogStreamingToplogyV2.java:164) [6a601f5663ba11e698cffa163ef6032d.jar:?]
10709 [main] ERROR o.a.a.s.h.StormAtlasHook - storm topology exception
java.lang.NullPointerException
at java.text.DateFormat.hashCode(DateFormat.java:739) ~[?:1.8.0_60]
at java.util.HashMap.hash(HashMap.java:338) ~[?:1.8.0_60]
at java.util.HashMap.put(HashMap.java:611) ~[?:1.8.0_60]
at java.util.HashSet.add(HashSet.java:219) ~[?:1.8.0_60]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:141) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:198) ~[storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.createSpoutInstance(StormAtlasHook.java:310) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.addSpouts(StormAtlasHook.java:294) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.createTopologyGraph(StormAtlasHook.java:280) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:95) [storm-bridge-0.7.0.2.5.0.0-1181.jar:0.8-incubating-SNAPSHOT]
at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:57) [storm-bridge-shim-0.7.0.2.5.0.0-1181.jar:0.7.0.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:285) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:258) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:311) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:347) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:328) [storm-core-1.0.1.2.5.0.0-1181.jar:1.0.1.2.5.0.0-1181]
at org.hw.hcube.storm.LogStreamingToplogyV2.main(LogStreamingToplogyV2.java:164) [6a601f5663ba11e698cffa163ef6032d.jar:?]
10718 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: org.apache.storm.jdbc.bolt.JdbcInsertBolt@7a5b769b
10718 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: org.apache.storm.jdbc.mapper.SimpleJdbcMapper@f4c0e4e
10718 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='host', val=null, sqlType=12}
10718 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='type', val=null, sqlType=12}
10718 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='seq_num', val=null, sqlType=-5}
10718 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='id', val=null, sqlType=12}
10719 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='cluster', val=null, sqlType=12}
10719 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='event_count', val=null, sqlType=-5}
10719 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='message_md5', val=null, sqlType=12}
10719 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='event_md5', val=null, sqlType=12}
10723 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='ip', val=null, sqlType=12}
10725 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='path', val=null, sqlType=12}
10725 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='file', val=null, sqlType=12}
10725 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='method', val=null, sqlType=12}
10725 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='logger_name', val=null, sqlType=12}
10725 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='logtime', val=null, sqlType=93}
10725 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='line_number', val=null, sqlType=-5}
10732 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='logfile_line_number', val=null, sqlType=-5}
10732 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='level', val=null, sqlType=12}
10732 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: Column{columnName='log_message', val=null, sqlType=12}
10732 [main] INFO o.a.a.s.h.StormTopologyUtil - Processing instance: org.apache.storm.jdbc.common.HikariCPConnectionProvider@24361cfc
10737 [main] INFO o.a.a.h.AtlasHook - Adding entity for type: storm_topology
23946 [main] INFO o.a.k.c.p.ProducerConfig - ProducerConfig values:
metric.reporters = []
metadata.max.age.ms = 300000
reconnect.backoff.ms = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
bootstrap.servers = [vimal-erie-6-1.openstacklocal:6667]
ssl.keystore.type = JKS
sasl.mechanism = GSSAPI
max.block.ms = 60000
interceptor.classes = null
ssl.truststore.password = null
client.id =
ssl.endpoint.identification.algorithm = null
request.timeout.ms = 30000
acks = 1
receive.buffer.bytes = 32768
ssl.truststore.type = JKS
retries = 0
ssl.truststore.location = null
ssl.keystore.password = null
send.buffer.bytes = 131072
compression.type = none
metadata.fetch.timeout.ms = 60000
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
buffer.memory = 33554432
timeout.ms = 30000
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.trustmanager.algorithm = PKIX
block.on.buffer.full = false
ssl.key.password = null
sasl.kerberos.min.time.before.relogin = 60000
connections.max.idle.ms = 540000
max.in.flight.requests.per.connection = 5
metrics.num.samples = 2
ssl.protocol = TLS
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
batch.size = 16384
ssl.keystore.location = null
ssl.cipher.suites = null
security.protocol = PLAINTEXT
max.request.size = 1048576
value.serializer = class org.apache.kafka.common.serialization.StringSerializer
ssl.keymanager.algorithm = SunX509
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
linger.ms = 0
24084 [main] INFO o.a.k.c.p.ProducerConfig - ProducerConfig values:
metric.reporters = []
metadata.max.age.ms = 300000
reconnect.backoff.ms = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
bootstrap.servers = [vimal-erie-6-1.openstacklocal:6667]
ssl.keystore.type = JKS
sasl.mechanism = GSSAPI
max.block.ms = 60000
interceptor.classes = null
ssl.truststore.password = null
client.id = producer-1
ssl.endpoint.identification.algorithm = null
request.timeout.ms = 30000
acks = 1
receive.buffer.bytes = 32768
ssl.truststore.type = JKS
retries = 0
ssl.truststore.location = null
ssl.keystore.password = null
send.buffer.bytes = 131072
compression.type = none
metadata.fetch.timeout.ms = 60000
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
buffer.memory = 33554432
timeout.ms = 30000
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.trustmanager.algorithm = PKIX
block.on.buffer.full = false
ssl.key.password = null
sasl.kerberos.min.time.before.relogin = 60000
connections.max.idle.ms = 540000
max.in.flight.requests.per.connection = 5
metrics.num.samples = 2
ssl.protocol = TLS
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
batch.size = 16384
ssl.keystore.location = null
ssl.cipher.suites = null
security.protocol = PLAINTEXT
max.request.size = 1048576
value.serializer = class org.apache.kafka.common.serialization.StringSerializer
ssl.keymanager.algorithm = SunX509
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
linger.ms = 0
24089 [main] WARN o.a.k.c.p.ProducerConfig - The configuration key.deserializer = org.apache.kafka.common.serialization.StringDeserializer was supplied but isn't a known config.
24090 [main] WARN o.a.k.c.p.ProducerConfig - The configuration value.deserializer = org.apache.kafka.common.serialization.StringDeserializer was supplied but isn't a known config.
24090 [main] WARN o.a.k.c.p.ProducerConfig - The configuration hook.group.id = atlas was supplied but isn't a known config.
24090 [main] WARN o.a.k.c.p.ProducerConfig - The configuration partition.assignment.strategy = roundrobin was supplied but isn't a known config.
24090 [main] WARN o.a.k.c.p.ProducerConfig - The configuration zookeeper.connection.timeout.ms = 200 was supplied but isn't a known config.
24091 [main] WARN o.a.k.c.p.ProducerConfig - The configuration zookeeper.session.timeout.ms = 400 was supplied but isn't a known config.
24091 [main] WARN o.a.k.c.p.ProducerConfig - The configuration zookeeper.connect = vimal-erie-6-1.openstacklocal:2181 was supplied but isn't a known config.
24091 [main] WARN o.a.k.c.p.ProducerConfig - The configuration zookeeper.sync.time.ms = 20 was supplied but isn't a known config.
24091 [main] WARN o.a.k.c.p.ProducerConfig - The configuration auto.offset.reset = smallest was supplied but isn't a known config.
24093 [main] INFO o.a.k.c.u.AppInfoParser - Kafka version : 0.10.0.2.5.0.0-1181
24093 [main] INFO o.a.k.c.u.AppInfoParser - Kafka commitId : 022ed507ec080025
{noformat}
Attaching Atlas UI snapshot showing the entities created as part of storm topology submission.
!Atlas 2016-08-16 19-48-59.png|thumbnail!
> NPE while submitting topology in StormHook
> ------------------------------------------
>
> Key: ATLAS-1121
> URL: https://issues.apache.org/jira/browse/ATLAS-1121
> Project: Atlas
> Issue Type: Bug
> Affects Versions: trunk
> Reporter: Ayub Khan
> Assignee: Ayub Khan
>
> NPE while submitting topology in StormHook
> {code}
> [storm@ctr-e25-1471039652053-0001-01-000009 erie]$ storm jar hcube-storm-topology-0.0.1.jar org.hw.hcube.storm.LogStreamingToplogyV2 phoenix.properties
> Running: /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.5.0.0-1201/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/hdp/2.5.0.0-1201/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.0.0-1201/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.0.0-1201/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.0.0-1201/storm/lib/storm-core-1.0.1.2.5.0.0-1201.jar:/usr/hdp/2.5.0.0-1201/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.0.0-1201/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.0.0-1201/storm/lib/zookeeper.jar:/usr/hdp/2.5.0.0-1201/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.0.0-1201/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.0.0-1201/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.0.0-1201/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.0.0-1201/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1201.jar:/usr/hdp/2.5.0.0-1201/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.5.0.0-1201/storm/extlib/storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:/usr/hdp/2.5.0.0-1201/storm/extlib/atlas-plugin-classloader-0.7.0.2.5.0.0-1201.jar org.apache.storm.daemon.ClientJarTransformerRunner org.apache.storm.hack.StormShadeTransformer hcube-storm-topology-0.0.1.jar /tmp/9e206b70636a11e685530242ac1b1bc0.jar
> Running: /usr/jdk64/jdk1.8.0_60/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.5.0.0-1201/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.5.0.0-1201/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.0.0-1201/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.0.0-1201/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.0.0-1201/storm/lib/storm-core-1.0.1.2.5.0.0-1201.jar:/usr/hdp/2.5.0.0-1201/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.0.0-1201/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.0.0-1201/storm/lib/zookeeper.jar:/usr/hdp/2.5.0.0-1201/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.0.0-1201/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.0.0-1201/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.0.0-1201/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.0.0-1201/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.0.0-1201/storm/lib/storm-rename-hack-1.0.1.2.5.0.0-1201.jar:/usr/hdp/2.5.0.0-1201/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.5.0.0-1201/storm/extlib/storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:/usr/hdp/2.5.0.0-1201/storm/extlib/atlas-plugin-classloader-0.7.0.2.5.0.0-1201.jar:/tmp/9e206b70636a11e685530242ac1b1bc0.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.5.0.0-1201/storm/bin -Dstorm.jar=/tmp/9e206b70636a11e685530242ac1b1bc0.jar org.hw.hcube.storm.LogStreamingToplogyV2 phoenix.properties
> 1269 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -5580857394466738431:-5449170806113196196
> 1320 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
> 1533 [main] INFO o.a.s.m.n.Login - successfully logged in.
> 1682 [main] INFO o.a.s.m.n.Login - successfully logged in.
> 1710 [main] INFO o.a.s.m.n.Login - successfully logged in.
> 1738 [main] INFO o.a.s.m.n.Login - successfully logged in.
> 1765 [main] INFO o.a.s.StormSubmitter - Uploading topology jar /tmp/9e206b70636a11e685530242ac1b1bc0.jar to assigned location: /hadoop/storm/nimbus/inbox/stormjar-2d8edcfa-000b-41d3-a170-9982b45867c0.jar
> Start uploading file '/tmp/9e206b70636a11e685530242ac1b1bc0.jar' to '/hadoop/storm/nimbus/inbox/stormjar-2d8edcfa-000b-41d3-a170-9982b45867c0.jar' (161437864 bytes)
> [==================================================] 161437864 / 161437864
> File '/tmp/9e206b70636a11e685530242ac1b1bc0.jar' uploaded to '/hadoop/storm/nimbus/inbox/stormjar-2d8edcfa-000b-41d3-a170-9982b45867c0.jar' (161437864 bytes)
> 5154 [main] INFO o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /hadoop/storm/nimbus/inbox/stormjar-2d8edcfa-000b-41d3-a170-9982b45867c0.jar
> 5155 [main] INFO o.a.s.StormSubmitter - Submitting topology hcube-log-streaming-phoenix in distributed mode with conf {"topology.workers":10,"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-5580857394466738431:-5449170806113196196"}
> 5164 [main] INFO o.a.s.m.n.Login - successfully logged in.
> 5194 [main] INFO o.a.s.m.n.Login - successfully logged in.
> 6269 [main] INFO o.a.s.StormSubmitter - Finished submitting topology: hcube-log-streaming-phoenix
> 6274 [main] INFO o.a.s.StormSubmitter - Initializing the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
> 6366 [main] INFO o.a.a.ApplicationProperties - Looking for atlas-application.properties in classpath
> 6366 [main] INFO o.a.a.ApplicationProperties - Loading atlas-application.properties from file:/etc/storm/2.5.0.0-1201/0/atlas-application.properties
> log4j:WARN No appenders could be found for logger (org.apache.atlas.ApplicationProperties).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> 6736 [main] INFO o.a.a.h.AtlasHook - Created Atlas Hook
> 6741 [main] INFO o.a.s.m.n.Login - successfully logged in.
> 6771 [main] INFO o.a.s.m.n.Login - successfully logged in.
> 6828 [main] INFO o.a.s.StormSubmitter - Invoking the registered ISubmitterHook [org.apache.atlas.storm.hook.StormAtlasHook]
> 6829 [main] INFO o.a.a.s.h.StormAtlasHook - Collecting metadata for a new storm topology: hcube-log-streaming-phoenix
> 6859 [main] WARN o.a.s.StormSubmitter - Error occurred in invoking submitter hook:[org.apache.atlas.storm.hook.StormAtlasHook]
> java.lang.RuntimeException: Atlas hook is unable to process the topology.
> at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:105) ~[storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:0.7.0.2.5.0.0-1201]
> at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:57) ~[storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:0.7.0.2.5.0.0-1201]
> at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:285) [storm-core-1.0.1.2.5.0.0-1201.jar:1.0.1.2.5.0.0-1201]
> at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:258) [storm-core-1.0.1.2.5.0.0-1201.jar:1.0.1.2.5.0.0-1201]
> at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:311) [storm-core-1.0.1.2.5.0.0-1201.jar:1.0.1.2.5.0.0-1201]
> at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:347) [storm-core-1.0.1.2.5.0.0-1201.jar:1.0.1.2.5.0.0-1201]
> at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:328) [storm-core-1.0.1.2.5.0.0-1201.jar:1.0.1.2.5.0.0-1201]
> at org.hw.hcube.storm.LogStreamingToplogyV2.main(LogStreamingToplogyV2.java:164) [9e206b70636a11e685530242ac1b1bc0.jar:?]
> Caused by: java.lang.NullPointerException
> at java.text.DateFormat.hashCode(DateFormat.java:739) ~[?:1.8.0_60]
> at java.util.HashMap.hash(HashMap.java:338) ~[?:1.8.0_60]
> at java.util.HashMap.put(HashMap.java:611) ~[?:1.8.0_60]
> at java.util.HashSet.add(HashSet.java:219) ~[?:1.8.0_60]
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:139) ~[?:?]
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197) ~[?:?]
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197) ~[?:?]
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197) ~[?:?]
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197) ~[?:?]
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197) ~[?:?]
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197) ~[?:?]
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197) ~[?:?]
> at org.apache.atlas.storm.hook.StormAtlasHook.createDataSet(StormAtlasHook.java:182) ~[storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:0.7.0.2.5.0.0-1201]
> at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyInputs(StormAtlasHook.java:149) ~[storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:0.7.0.2.5.0.0-1201]
> at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyDataSets(StormAtlasHook.java:132) ~[storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:0.7.0.2.5.0.0-1201]
> at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:89) ~[storm-bridge-shim-0.7.0.2.5.0.0-1201.jar:0.7.0.2.5.0.0-1201]
> ... 7 more
> Exception in thread "main" org.apache.storm.hooks.SubmitterHookException: java.lang.RuntimeException: Atlas hook is unable to process the topology.
> at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:289)
> at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:258)
> at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:311)
> at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:347)
> at org.apache.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:328)
> at org.hw.hcube.storm.LogStreamingToplogyV2.main(LogStreamingToplogyV2.java:164)
> Caused by: java.lang.RuntimeException: Atlas hook is unable to process the topology.
> at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:105)
> at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:57)
> at org.apache.storm.StormSubmitter.invokeSubmitterHook(StormSubmitter.java:285)
> ... 5 more
> Caused by: java.lang.NullPointerException
> at java.text.DateFormat.hashCode(DateFormat.java:739)
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:139)
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> at org.apache.atlas.storm.hook.StormTopologyUtil.getFieldValues(StormTopologyUtil.java:197)
> at org.apache.atlas.storm.hook.StormAtlasHook.createDataSet(StormAtlasHook.java:182)
> at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyInputs(StormAtlasHook.java:149)
> at org.apache.atlas.storm.hook.StormAtlasHook.addTopologyDataSets(StormAtlasHook.java:132)
> at org.apache.atlas.storm.hook.StormAtlasHook.notify(StormAtlasHook.java:89)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)