[jira] [Created] (STORM-537) A worker reconnects infinitely to another dead worker

["Sergey Tryuber (JIRA)" <ji...@apache.org>]

Sergey Tryuber created STORM-537:
------------------------------------

             Summary: A worker reconnects infinitely to another dead worker
                 Key: STORM-537
                 URL: https://issues.apache.org/jira/browse/STORM-537
             Project: Apache Storm
          Issue Type: Bug
    Affects Versions: 0.9.3
            Reporter: Sergey Tryuber


We're using 0.9.3-rc1. Most probably this wrong behavior was introduced as a side efffect for STORM-409. When I kill a worker, another worker starts to print messages like:
{noformat}
2014-10-20 11:45:03 b.s.m.n.Client [INFO] Reconnect started for Netty-Client-<HOST>:4706... [0]
2014-10-20 11:45:03 b.s.m.n.Client [INFO] Reconnect started for Netty-Client-<HOST>:4706... [1]
2014-10-20 11:45:03 b.s.m.n.Client [INFO] Reconnect started for Netty-Client-<HOST>:4706... [2]
..... so on
{noformat}
Then it reaches default 300 max_retries and starts the cycle again:
{noformat}
2014-10-20 11:54:38 b.s.m.n.Client [INFO] connection established to a remote host Netty-Client-<HOST>:4706, [id: 
0xec088412, /<HOST>:39795 :> <HOST>:4706]
2014-10-20 11:54:38 b.s.m.n.Client [INFO] Reconnect started for Netty-Client-<HOST>:4706... [0]
2014-10-20 11:54:38 b.s.m.n.Client [INFO] Reconnect started for Netty-Client-<HOST>:4706... [1]
2014-10-20 11:54:38 b.s.m.n.Client [INFO] Reconnect started for Netty-Client-<HOST>:4706... [2]
{noformat}
And so on infinitely... 

An issue most probably is in backtype.storm.messaging.netty.Client#connect method in following place which determines that we give up on reconnection:
{code}
if (null != channel) {
    LOG.info("connection established to a remote host " + name() + ", " + channel.toString());
    channelRef.set(channel);
} else {
    close();
    throw new RuntimeException("Remote address is not reachable. We will close this client " + name());
}
{code}
I guess (not tried yet), that _channel_ object is not _null_ if this is a real reconnection. So the method return a _channel_ object and then reconnection starts again and again.

This might be fixed by adding explicity *current = null;* into following code block of the same method:
{code}
if (!future.isSuccess()) {
    if (null != current) {
        current.close();
    }
}
{code}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)