You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Jungtaek Lim (JIRA)" <ji...@apache.org> on 2017/02/03 00:22:51 UTC
[jira] [Resolved] (STORM-2321) Nimbus did not come up after restart
[ https://issues.apache.org/jira/browse/STORM-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim resolved STORM-2321.
---------------------------------
Resolution: Fixed
Fix Version/s: 1.1.0
2.0.0
Merged into master and 1.x branch via [~sriharsha].
I'll handle porting back to 1.0.x after 1.0.3 RC1 vote.
> Nimbus did not come up after restart
> ------------------------------------
>
> Key: STORM-2321
> URL: https://issues.apache.org/jira/browse/STORM-2321
> Project: Apache Storm
> Issue Type: Bug
> Components: blobstore
> Affects Versions: 1.0.0, 2.0.0, 1.0.1, 1.0.2, 1.1.0
> Reporter: Raghav Kumar Gautam
> Assignee: Jungtaek Lim
> Priority: Critical
> Labels: ha
> Fix For: 2.0.0, 1.1.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The nimbus was restarted during HA testing. After the restart the nimbus failed to come up.
> {code}
> 2017-01-18 04:57:58.231 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
> 2017-01-18 04:57:58.247 o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with keyKillLeaderThenSubmitNewTopology1-1-1484715309-stormjar.jar
> 2017-01-18 04:57:58.273 o.a.s.b.KeySequenceNumber [ERROR] Exception {}
> org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /blobstore/KillLeaderThenSubmitNewTopology1-1-1484715309-stormjar.jar
> at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590)
> at org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214)
> at org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:203)
> at org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:108)
> at org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:200)
> at org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191)
> at org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:38)
> at org.apache.storm.blobstore.KeySequenceNumber.getKeySequenceNumber(KeySequenceNumber.java:149)
> at org.apache.storm.daemon.nimbus$get_version_for_key.invoke(nimbus.clj:456)
> at org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__9548.createStateInZookeeper(nimbus.clj:2056)
> at org.apache.storm.generated.Nimbus$Processor$createStateInZookeeper.getResult(Nimbus.java:3755)
> at org.apache.storm.generated.Nimbus$Processor$createStateInZookeeper.getResult(Nimbus.java:3740)
> at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.storm.security.auth.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:144)
> at org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2017-01-18 04:57:58.274 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting
> 2017-01-18 04:57:58.296 o.a.s.m.n.Login [INFO] successfully logged in.
> 2017-01-18 04:57:58.309 o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x359afc1eaa2009b closed
> 2017-01-18 04:57:58.309 o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
> 2017-01-18 04:57:58.310 o.a.s.t.s.TThreadPoolServer [ERROR] Error occurred during processing of message.
> java.util.NoSuchElementException
> at java.util.TreeMap.key(TreeMap.java:1327)
> at java.util.TreeMap.lastKey(TreeMap.java:297)
> at java.util.TreeSet.last(TreeSet.java:401)
> at org.apache.storm.blobstore.KeySequenceNumber.getKeySequenceNumber(KeySequenceNumber.java:206)
> at org.apache.storm.daemon.nimbus$get_version_for_key.invoke(nimbus.clj:456)
> at org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__9548.createStateInZookeeper(nimbus.clj:2056)
> at org.apache.storm.generated.Nimbus$Processor$createStateInZookeeper.getResult(Nimbus.java:3755)
> at org.apache.storm.generated.Nimbus$Processor$createStateInZookeeper.getResult(Nimbus.java:3740)
> at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.storm.security.auth.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:144)
> at org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2017-01-18 04:57:58.311 o.a.s.d.nimbus [ERROR] Error when processing event
> java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException
> at org.apache.storm.blobstore.BlobSynchronizer.syncBlobs(BlobSynchronizer.java:92)
> at org.apache.storm.daemon.nimbus$fn__9373.invoke(nimbus.clj:1452)
> at clojure.lang.MultiFn.invoke(MultiFn.java:233)
> at org.apache.storm.daemon.nimbus$fn__9770$exec_fn__3656__auto____9771$fn__9786.invoke(nimbus.clj:2452)
> at org.apache.storm.timer$schedule_recurring$this__2188.invoke(timer.clj:105)
> at org.apache.storm.timer$mk_timer$fn__2171$fn__2172.invoke(timer.clj:50)
> at org.apache.storm.timer$mk_timer$fn__2171.invoke(timer.clj:42)
> at clojure.lang.AFn.run(AFn.java:22)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException
> at org.apache.storm.blobstore.BlobSynchronizer.updateKeySetForBlobStore(BlobSynchronizer.java:114)
> at org.apache.storm.blobstore.BlobSynchronizer.syncBlobs(BlobSynchronizer.java:76)
> ... 8 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException
> at org.apache.storm.blobstore.BlobStoreUtils.updateKeyForBlobStore(BlobStoreUtils.java:252)
> at org.apache.storm.blobstore.BlobSynchronizer.updateKeySetForBlobStore(BlobSynchronizer.java:111)
> ... 9 more
> Caused by: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException
> at org.apache.storm.blobstore.NimbusBlobStore.createStateInZookeeper(NimbusBlobStore.java:349)
> at org.apache.storm.blobstore.BlobStoreUtils.createStateInZookeeper(BlobStoreUtils.java:217)
> at org.apache.storm.blobstore.BlobStoreUtils.updateKeyForBlobStore(BlobStoreUtils.java:249)
> ... 10 more
> Caused by: org.apache.storm.thrift.transport.TTransportException
> at org.apache.storm.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86)
> at org.apache.storm.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
> at org.apache.storm.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
> at org.apache.storm.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
> at org.apache.storm.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86)
> at org.apache.storm.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at org.apache.storm.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at org.apache.storm.generated.Nimbus$Client.recv_createStateInZookeeper(Nimbus.java:1000)
> at org.apache.storm.generated.Nimbus$Client.createStateInZookeeper(Nimbus.java:987)
> at org.apache.storm.blobstore.NimbusBlobStore.createStateInZookeeper(NimbusBlobStore.java:346)
> ... 12 more
> 2017-01-18 04:57:58.314 o.a.s.util [ERROR] Halting process: ("Error when processing an event")
> java.lang.RuntimeException: ("Error when processing an event")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> at clojure.lang.RestFn.invoke(RestFn.java:423)
> at org.apache.storm.daemon.nimbus$nimbus_data$fn__8579.invoke(nimbus.clj:212)
> at org.apache.storm.timer$mk_timer$fn__2171$fn__2172.invoke(timer.clj:71)
> at org.apache.storm.timer$mk_timer$fn__2171.invoke(timer.clj:42)
> at clojure.lang.AFn.run(AFn.java:22)
> at java.lang.Thread.run(Thread.java:745)
> 2017-01-18 04:57:58,317 FATAL Ignoring log event after log4j was shut down
> 2017-01-18 04:57:58,317 FATAL Ignoring log event after log4j was shut down
> 2017-01-18 04:57:58,317 FATAL Ignoring log event after log4j was shut down
> 2017-01-18 04:57:58,318 FATAL Ignoring log event after log4j was shut down
> 2017-01-18 04:57:58,318 FATAL Ignoring log event after log4j was shut down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)