Topology hanging (activation, ipv6?)

[Stig Døssing <sd...@it-minds.dk>]

Hi,

I'm running a few Kafka spout based topologies that seem to sometimes stop
processing tuples. When I look at the thread dump for the workers, it looks
like the executing threads are sleeping in the pre-activation part of
spout/bolt setup.

https://github.com/apache/storm/blob/v0.10.0/storm-core/src/clj/backtype/storm/daemon/executor.clj#L512

I've also seen some spouts stuck in the following line, which I take to
mean the topology has been deactivated.

https://github.com/apache/storm/blob/v0.10.0/storm-core/src/clj/backtype/storm/daemon/executor.clj#L617

Activating and deactivating the topology doesn't restart the hanging
executors, but killing and redeploying does. I'm not really sure where to
start debugging this, but the documentation mentions that Storm doesn't
support ipv6, and mentions that it is enough to add
"-Djava.net.preferIPv4Stack=true"
to supervisor.childopts and restart. I did that, but when I look at lsof -i
:6703 on a machine, it still appears to use ipv6. Should I add the option
to nimbus/worker/topology.worker childopts as well?

Any other ideas for debugging are also welcome :)

Med venlig hilsen | Best regards

*Stig Rohde Døssing*
Konsulent


   Aarhus   I     København

     *+45 61 35 93 66*  I    www.it-minds.dk   I     *+45 42 59 00 84*