A possible fix for Netty connect issue

[Bikrant Neupane <bi...@gmail.com>]

Apologies if this has already been discussed.

I was getting 'Netty connect'[1] error every time I started a topology
with more than 1 worker. I'm using Storm 0.9.5. Because of this issue
I was forced to run 1 worker with multiple spout/bolt threads.

I had default kernel settings in my servers, which I think were too
low for the amount of connection/data thrown out to them. I was able
to see high number of Tcp receive queue full and tcp failed accepts
with netstat -s command. Now after I tweaked few linux kernel network
parameters [2], my topologies with 4 works each have been running
without a single error since last few days. This seems to have solved
the reconnect issues I was having.

Let me know if you have questions.

[1]
2015-09-17T21:59:58.790+0000 b.s.m.n.Client [ERROR] connection attempt
103 to Netty-Client-storm-host-name/1.2.3.4:6710 failed:
java.lang.RuntimeException: Returned channel was actually not
established

[2]
net.ipv4.tcp_window_scaling = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_slow_start_after_idle = 0