Storm 0.9.0.1 - supervisor failing to start repeatedly (EOFException)

["Evered, Randolf" <ra...@lmco.com>]

We are using Storm 0.9.0.1 with Netty and Trident topologies on a single machine (nimbus, supervisor, and drpc running on the same machine).  Supervisor keeps dying and gets restarted after 7-8 seconds by Supervisord (the service that restarts storm and zookeeper processes).  Here is the error in supervisor.log we see over and over:

2014-04-15 21:13:13 b.s.event [ERROR] Error when processing event
java.lang.RuntimeException: java.io.EOFException
        at backtype.storm.utils.Utils.deserialize(Utils.java:69) ~[storm-core-0.9.0.1.jar:na]
        at backtype.storm.utils.LocalState.snapshot(LocalState.java:28) ~[storm-core-0.9.0.1.jar:na]
        at backtype.storm.utils.LocalState.get(LocalState.java:39) ~[storm-core-0.9.0.1.jar:na]
        at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:187) ~[storm-core-0.9.0.1.jar:na]
        at clojure.lang.AFn.applyToHelper(AFn.java:161) [clojure-1.4.0.jar:na]
        at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
        at clojure.core$apply.invoke(core.clj:603) ~[clojure-1.4.0.jar:na]
        at clojure.core$partial$fn__4070.doInvoke(core.clj:2343) ~[clojure-1.4.0.jar:na]
        at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.4.0.jar:na]
        at backtype.storm.event$event_manager$fn__3072.invoke(event.clj:24) ~[storm-core-0.9.0.1.jar:na]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
        at java.lang.Thread.run(Unknown Source) [na:1.7.0_45]
Caused by: java.io.EOFException: null
        at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source) ~[na:1.7.0_45]
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source) ~[na:1.7.0_45]
        at java.io.ObjectInputStream.readStreamHeader(Unknown Source) ~[na:1.7.0_45]
        at java.io.ObjectInputStream.<init>(Unknown Source) ~[na:1.7.0_45]
        at backtype.storm.utils.Utils.deserialize(Utils.java:64) ~[storm-core-0.9.0.1.jar:na]
        ... 11 common frames omitted
2014-04-15 21:13:13 b.s.util [INFO] Halting process: ("Error when processing an event")

Any ideas why supervisor might be dying?

Per recommendation from the post "Supervisor throwing error on start up" from https://groups.google.com/forum/#!topic/storm-user/2gapTYTRrX8, we stopped storm processes, cleared the storm and zookeeper data directories, and it was fine (after we loaded the topologies again).  However, we would like to know how to prevent this bug from happening in a production system environment.

We are also getting a ton of Connection refused errors in the Nimbus and Worker logs.  I expect this would be the case if Supervisor can't start up.

Thank you,
Randy