You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@storm.apache.org by Keith Bourgoin <ke...@gmail.com> on 2015/01/14 17:41:16 UTC

0.8.2 ZMQ Question

Hey Everyone,

This will be a blast from the past, but some problems have prevented us
from upgrading to 0.9.x at the moment.

The problem we're seeing is ZMQ trying to use an address already in use:

2015-01-14 04:39:38 util [ERROR] Async loop died!
org.zeromq.ZMQException: Address already in use(0x62)
    at org.zeromq.ZMQ$Socket.bind(Native Method)
    at zilch.mq$bind.invoke(mq.clj:69)
    at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)
    at backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26)
    at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
    at clojure.lang.AFn.run(AFn.java:24)
    at java.lang.Thread.run(Thread.java:745)


I googled around and couldn't find anything. It looks like this tends to
affect some workers on a machine, but not all. Those workers are
functionally dead, and eventually Storm will switch to using a different
workers. However, as our machines get more full, these get hit more often,
and the time to rebalance usually means failed tuples.

I know it's an old version, but any help would be greatly appreciated.

Thanks,

Keith.

Re: 0.8.2 ZMQ Question

Posted by Keith Bourgoin <ke...@gmail.com>.

Thanks for the response, Tim. This had been troubling us for a while, but I
think we finally figured it out. It was indeed a port conflict, but not
with another ZMQ process. It turns out our ephemeral port range was messed
up on the machines:

$ cat /proc/sys/net/ipv4/ip_local_port_range
1024    65000


We fixed that, and now everything's great.  Hopefully this will help
someone else down the road, because other threads on this error were dead
when I looked.

Thanks again!

Keith.

On Wed Jan 14 2015 at 12:21:40 PM Tim Molter <ti...@gmail.com> wrote:

> Hi Keith,
>
> Some ZeroMQ process is trying to start up, but some other ZeroMQ process
> is already using the port, and it could have entered into an unrecoverable
> state. To fix it, I always restarted everything.
>
> ~Tim
>
> On 2015_01_14 5:41 PM, Keith Bourgoin wrote:
> > Hey Everyone,
> >
> > This will be a blast from the past, but some problems have prevented us
> > from upgrading to 0.9.x at the moment.
> >
> > The problem we're seeing is ZMQ trying to use an address already in use:
> >
> > |2015-01-14 04:39:38 util [ERROR] Async loop died!
> > org.zeromq.ZMQException: Address already in use(0x62)
> >     at org.zeromq.ZMQ$Socket.bind(Native Method)
> >     at zilch.mq <http://zilch.mq/>$bind.invoke(mq.clj:69)
> >     at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)
> >     at backtype.storm.messaging.loader$launch_receive_thread_
> BANG_$fn__1629.invoke(loader.clj:26)
> >     at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> >     at clojure.lang.AFn.run(AFn.java:24)
> >     at java.lang.Thread.run(Thread.java:745)
> > |
> >
> >
> > I googled around and couldn't find anything. It looks like this tends to
> > affect some workers on a machine, but not all. Those workers are
> > functionally dead, and eventually Storm will switch to using a different
> > workers. However, as our machines get more full, these get hit more
> > often, and the time to rebalance usually means failed tuples.
> >
> > I know it's an old version, but any help would be greatly appreciated.
> >
> > Thanks,
> >
> > Keith.
>
>

Re: 0.8.2 ZMQ Question

Posted by Tim Molter <ti...@gmail.com>.

Hi Keith,

Some ZeroMQ process is trying to start up, but some other ZeroMQ process is already using the port, and it could have entered into an unrecoverable state. To fix it, I always restarted everything.

~Tim

On 2015_01_14 5:41 PM, Keith Bourgoin wrote:
> Hey Everyone,
>
> This will be a blast from the past, but some problems have prevented us
> from upgrading to 0.9.x at the moment.
>
> The problem we're seeing is ZMQ trying to use an address already in use:
>
> |2015-01-14 04:39:38 util [ERROR] Async loop died!
> org.zeromq.ZMQException: Address already in use(0x62)
>     at org.zeromq.ZMQ$Socket.bind(Native Method)
>     at zilch.mq <http://zilch.mq/>$bind.invoke(mq.clj:69)
>     at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)
>     at backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26)
>     at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
>     at clojure.lang.AFn.run(AFn.java:24)
>     at java.lang.Thread.run(Thread.java:745)
> |
>
>
> I googled around and couldn't find anything. It looks like this tends to
> affect some workers on a machine, but not all. Those workers are
> functionally dead, and eventually Storm will switch to using a different
> workers. However, as our machines get more full, these get hit more
> often, and the time to rebalance usually means failed tuples.
>
> I know it's an old version, but any help would be greatly appreciated.
>
> Thanks,
>
> Keith.