You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Dimitris Sarlis <sa...@gmail.com> on 2015/07/27 18:22:11 UTC
Storm Netty connection problem
Hi all,
I'm trying to run a Storm topology and I'm getting the following errors
in my bolts logs:
2015-07-27T15:55:29.239+0000 b.s.m.n.Client [ERROR] connection attempt 7
to Netty-Client-slave7/192.168.5.163:6703 failed:
java.lang.RuntimeException: Returned channel was actually not established
2015-07-27T15:55:29.240+0000 b.s.m.n.Client [INFO] connection attempt 8
to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 239 ms
2015-07-27T15:55:29.243+0000 b.s.m.n.Client [INFO] connection
established to Netty-Client-slave4/192.168.5.160:6703
2015-07-27T15:55:29.483+0000 b.s.m.n.Client [ERROR] connection attempt 8
to Netty-Client-slave7/192.168.5.163:6703 failed:
java.lang.RuntimeException: Returned channel was actually not established
2015-07-27T15:55:29.483+0000 b.s.m.n.Client [INFO] connection attempt 9
to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 356 ms
2015-07-27T15:55:29.842+0000 b.s.m.n.Client [ERROR] connection attempt 9
to Netty-Client-slave7/192.168.5.163:6703 failed:
java.lang.RuntimeException: Returned channel was actually not established
2015-07-27T15:55:29.842+0000 b.s.m.n.Client [INFO] connection attempt 10
to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 359 ms
2015-07-27T15:55:30.203+0000 b.s.m.n.Client [ERROR] connection attempt
10 to Netty-Client-slave7/192.168.5.163:6703 failed:
java.lang.RuntimeException: Returned channel was actually not established
2015-07-27T15:55:30.204+0000 b.s.m.n.Client [INFO] connection attempt 11
to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 361 ms
2015-07-27T15:55:30.568+0000 b.s.m.n.Client [ERROR] connection attempt
11 to Netty-Client-slave7/192.168.5.163:6703 failed:
java.lang.RuntimeException: Returned channel was actually not established
2015-07-27T15:55:30.568+0000 b.s.m.n.Client [INFO] connection attempt 12
to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 362 ms
2015-07-27T15:55:30.932+0000 b.s.m.n.Client [ERROR] connection attempt
12 to Netty-Client-slave7/192.168.5.163:6703 failed:
java.lang.RuntimeException: Returned channel was actually not established
2015-07-27T15:55:30.933+0000 b.s.m.n.Client [INFO] connection attempt 13
to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 364 ms
2015-07-27T15:55:31.301+0000 b.s.m.n.Client [ERROR] connection attempt
13 to Netty-Client-slave7/192.168.5.163:6703 failed:
java.lang.RuntimeException: Returned channel was actually not established
2015-07-27T15:55:31.301+0000 b.s.m.n.Client [INFO] connection attempt 14
to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 366 ms
2015-07-27T15:55:31.670+0000 b.s.m.n.Client [ERROR] connection attempt
14 to Netty-Client-slave7/192.168.5.163:6703 failed:
java.lang.RuntimeException: Returned channel was actually not established
2015-07-27T15:55:31.671+0000 b.s.m.n.Client [INFO] connection attempt 15
to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 368 ms
2015-07-27T15:55:32.041+0000 b.s.m.n.Client [INFO] connection
established to Netty-Client-slave7/192.168.5.163:6703
I understand that at the end the connection has been re-established.
However, is it normal to get these errors? Does this mean that there is
a network error which is not related to Storm?
Thanks in advance,
Dimitris Sarlis
Re: Storm Netty connection problem
Posted by John Reilly <jr...@inconspicuous.org>.
As far as I know, if a worker comes up before another downstream worker
that it is trying to send tuples to, you will see messages like
that....also if a downstream worker dies and has to restart you may see
this.
On Mon, Jul 27, 2015 at 10:19 AM Enno Shioji <es...@gmail.com> wrote:
> Same here, I'm guessing it's some benign race condition.
> There was some code change around this area for reducing the amount of
> ERROR logs, so in future releases you should at least see less ERROR logs.
>
> On Mon, Jul 27, 2015 at 6:11 PM, Nick R. Katsipoulakis <
> nick.katsip@gmail.com> wrote:
>
>> Hello Dimitri,
>>
>> I also get similar errors on my AWS cluster and I have not figured out
>> how to resolve them. However, I do not think that they necessarily affect
>> your topologies.
>>
>> Cheers,
>> Nikos
>>
>> 2015-07-27 12:22 GMT-04:00 Dimitris Sarlis <sa...@gmail.com>:
>>
>>> Hi all,
>>>
>>> I'm trying to run a Storm topology and I'm getting the following errors
>>> in my bolts logs:
>>>
>>> 2015-07-27T15:55:29.239+0000 b.s.m.n.Client [ERROR] connection attempt 7
>>> to Netty-Client-slave7/192.168.5.163:6703 failed:
>>> java.lang.RuntimeException: Returned channel was actually not established
>>> 2015-07-27T15:55:29.240+0000 b.s.m.n.Client [INFO] connection attempt 8
>>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 239 ms
>>> 2015-07-27T15:55:29.243+0000 b.s.m.n.Client [INFO] connection
>>> established to Netty-Client-slave4/192.168.5.160:6703
>>> 2015-07-27T15:55:29.483+0000 b.s.m.n.Client [ERROR] connection attempt 8
>>> to Netty-Client-slave7/192.168.5.163:6703 failed:
>>> java.lang.RuntimeException: Returned channel was actually not established
>>> 2015-07-27T15:55:29.483+0000 b.s.m.n.Client [INFO] connection attempt 9
>>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 356 ms
>>> 2015-07-27T15:55:29.842+0000 b.s.m.n.Client [ERROR] connection attempt 9
>>> to Netty-Client-slave7/192.168.5.163:6703 failed:
>>> java.lang.RuntimeException: Returned channel was actually not established
>>> 2015-07-27T15:55:29.842+0000 b.s.m.n.Client [INFO] connection attempt 10
>>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 359 ms
>>> 2015-07-27T15:55:30.203+0000 b.s.m.n.Client [ERROR] connection attempt
>>> 10 to Netty-Client-slave7/192.168.5.163:6703 failed:
>>> java.lang.RuntimeException: Returned channel was actually not established
>>> 2015-07-27T15:55:30.204+0000 b.s.m.n.Client [INFO] connection attempt 11
>>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 361 ms
>>> 2015-07-27T15:55:30.568+0000 b.s.m.n.Client [ERROR] connection attempt
>>> 11 to Netty-Client-slave7/192.168.5.163:6703 failed:
>>> java.lang.RuntimeException: Returned channel was actually not established
>>> 2015-07-27T15:55:30.568+0000 b.s.m.n.Client [INFO] connection attempt 12
>>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 362 ms
>>> 2015-07-27T15:55:30.932+0000 b.s.m.n.Client [ERROR] connection attempt
>>> 12 to Netty-Client-slave7/192.168.5.163:6703 failed:
>>> java.lang.RuntimeException: Returned channel was actually not established
>>> 2015-07-27T15:55:30.933+0000 b.s.m.n.Client [INFO] connection attempt 13
>>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 364 ms
>>> 2015-07-27T15:55:31.301+0000 b.s.m.n.Client [ERROR] connection attempt
>>> 13 to Netty-Client-slave7/192.168.5.163:6703 failed:
>>> java.lang.RuntimeException: Returned channel was actually not established
>>> 2015-07-27T15:55:31.301+0000 b.s.m.n.Client [INFO] connection attempt 14
>>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 366 ms
>>> 2015-07-27T15:55:31.670+0000 b.s.m.n.Client [ERROR] connection attempt
>>> 14 to Netty-Client-slave7/192.168.5.163:6703 failed:
>>> java.lang.RuntimeException: Returned channel was actually not established
>>> 2015-07-27T15:55:31.671+0000 b.s.m.n.Client [INFO] connection attempt 15
>>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 368 ms
>>> 2015-07-27T15:55:32.041+0000 b.s.m.n.Client [INFO] connection
>>> established to Netty-Client-slave7/192.168.5.163:6703
>>>
>>> I understand that at the end the connection has been re-established.
>>> However, is it normal to get these errors? Does this mean that there is a
>>> network error which is not related to Storm?
>>>
>>> Thanks in advance,
>>> Dimitris Sarlis
>>>
>>
>>
>>
>> --
>> Nikolaos Romanos Katsipoulakis,
>> University of Pittsburgh, PhD candidate
>>
>
>
Re: Storm Netty connection problem
Posted by Enno Shioji <es...@gmail.com>.
Same here, I'm guessing it's some benign race condition.
There was some code change around this area for reducing the amount of
ERROR logs, so in future releases you should at least see less ERROR logs.
On Mon, Jul 27, 2015 at 6:11 PM, Nick R. Katsipoulakis <
nick.katsip@gmail.com> wrote:
> Hello Dimitri,
>
> I also get similar errors on my AWS cluster and I have not figured out how
> to resolve them. However, I do not think that they necessarily affect your
> topologies.
>
> Cheers,
> Nikos
>
> 2015-07-27 12:22 GMT-04:00 Dimitris Sarlis <sa...@gmail.com>:
>
>> Hi all,
>>
>> I'm trying to run a Storm topology and I'm getting the following errors
>> in my bolts logs:
>>
>> 2015-07-27T15:55:29.239+0000 b.s.m.n.Client [ERROR] connection attempt 7
>> to Netty-Client-slave7/192.168.5.163:6703 failed:
>> java.lang.RuntimeException: Returned channel was actually not established
>> 2015-07-27T15:55:29.240+0000 b.s.m.n.Client [INFO] connection attempt 8
>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 239 ms
>> 2015-07-27T15:55:29.243+0000 b.s.m.n.Client [INFO] connection established
>> to Netty-Client-slave4/192.168.5.160:6703
>> 2015-07-27T15:55:29.483+0000 b.s.m.n.Client [ERROR] connection attempt 8
>> to Netty-Client-slave7/192.168.5.163:6703 failed:
>> java.lang.RuntimeException: Returned channel was actually not established
>> 2015-07-27T15:55:29.483+0000 b.s.m.n.Client [INFO] connection attempt 9
>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 356 ms
>> 2015-07-27T15:55:29.842+0000 b.s.m.n.Client [ERROR] connection attempt 9
>> to Netty-Client-slave7/192.168.5.163:6703 failed:
>> java.lang.RuntimeException: Returned channel was actually not established
>> 2015-07-27T15:55:29.842+0000 b.s.m.n.Client [INFO] connection attempt 10
>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 359 ms
>> 2015-07-27T15:55:30.203+0000 b.s.m.n.Client [ERROR] connection attempt 10
>> to Netty-Client-slave7/192.168.5.163:6703 failed:
>> java.lang.RuntimeException: Returned channel was actually not established
>> 2015-07-27T15:55:30.204+0000 b.s.m.n.Client [INFO] connection attempt 11
>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 361 ms
>> 2015-07-27T15:55:30.568+0000 b.s.m.n.Client [ERROR] connection attempt 11
>> to Netty-Client-slave7/192.168.5.163:6703 failed:
>> java.lang.RuntimeException: Returned channel was actually not established
>> 2015-07-27T15:55:30.568+0000 b.s.m.n.Client [INFO] connection attempt 12
>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 362 ms
>> 2015-07-27T15:55:30.932+0000 b.s.m.n.Client [ERROR] connection attempt 12
>> to Netty-Client-slave7/192.168.5.163:6703 failed:
>> java.lang.RuntimeException: Returned channel was actually not established
>> 2015-07-27T15:55:30.933+0000 b.s.m.n.Client [INFO] connection attempt 13
>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 364 ms
>> 2015-07-27T15:55:31.301+0000 b.s.m.n.Client [ERROR] connection attempt 13
>> to Netty-Client-slave7/192.168.5.163:6703 failed:
>> java.lang.RuntimeException: Returned channel was actually not established
>> 2015-07-27T15:55:31.301+0000 b.s.m.n.Client [INFO] connection attempt 14
>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 366 ms
>> 2015-07-27T15:55:31.670+0000 b.s.m.n.Client [ERROR] connection attempt 14
>> to Netty-Client-slave7/192.168.5.163:6703 failed:
>> java.lang.RuntimeException: Returned channel was actually not established
>> 2015-07-27T15:55:31.671+0000 b.s.m.n.Client [INFO] connection attempt 15
>> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 368 ms
>> 2015-07-27T15:55:32.041+0000 b.s.m.n.Client [INFO] connection established
>> to Netty-Client-slave7/192.168.5.163:6703
>>
>> I understand that at the end the connection has been re-established.
>> However, is it normal to get these errors? Does this mean that there is a
>> network error which is not related to Storm?
>>
>> Thanks in advance,
>> Dimitris Sarlis
>>
>
>
>
> --
> Nikolaos Romanos Katsipoulakis,
> University of Pittsburgh, PhD candidate
>
Re: Storm Netty connection problem
Posted by "Nick R. Katsipoulakis" <ni...@gmail.com>.
Hello Dimitri,
I also get similar errors on my AWS cluster and I have not figured out how
to resolve them. However, I do not think that they necessarily affect your
topologies.
Cheers,
Nikos
2015-07-27 12:22 GMT-04:00 Dimitris Sarlis <sa...@gmail.com>:
> Hi all,
>
> I'm trying to run a Storm topology and I'm getting the following errors in
> my bolts logs:
>
> 2015-07-27T15:55:29.239+0000 b.s.m.n.Client [ERROR] connection attempt 7
> to Netty-Client-slave7/192.168.5.163:6703 failed:
> java.lang.RuntimeException: Returned channel was actually not established
> 2015-07-27T15:55:29.240+0000 b.s.m.n.Client [INFO] connection attempt 8 to
> Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 239 ms
> 2015-07-27T15:55:29.243+0000 b.s.m.n.Client [INFO] connection established
> to Netty-Client-slave4/192.168.5.160:6703
> 2015-07-27T15:55:29.483+0000 b.s.m.n.Client [ERROR] connection attempt 8
> to Netty-Client-slave7/192.168.5.163:6703 failed:
> java.lang.RuntimeException: Returned channel was actually not established
> 2015-07-27T15:55:29.483+0000 b.s.m.n.Client [INFO] connection attempt 9 to
> Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 356 ms
> 2015-07-27T15:55:29.842+0000 b.s.m.n.Client [ERROR] connection attempt 9
> to Netty-Client-slave7/192.168.5.163:6703 failed:
> java.lang.RuntimeException: Returned channel was actually not established
> 2015-07-27T15:55:29.842+0000 b.s.m.n.Client [INFO] connection attempt 10
> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 359 ms
> 2015-07-27T15:55:30.203+0000 b.s.m.n.Client [ERROR] connection attempt 10
> to Netty-Client-slave7/192.168.5.163:6703 failed:
> java.lang.RuntimeException: Returned channel was actually not established
> 2015-07-27T15:55:30.204+0000 b.s.m.n.Client [INFO] connection attempt 11
> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 361 ms
> 2015-07-27T15:55:30.568+0000 b.s.m.n.Client [ERROR] connection attempt 11
> to Netty-Client-slave7/192.168.5.163:6703 failed:
> java.lang.RuntimeException: Returned channel was actually not established
> 2015-07-27T15:55:30.568+0000 b.s.m.n.Client [INFO] connection attempt 12
> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 362 ms
> 2015-07-27T15:55:30.932+0000 b.s.m.n.Client [ERROR] connection attempt 12
> to Netty-Client-slave7/192.168.5.163:6703 failed:
> java.lang.RuntimeException: Returned channel was actually not established
> 2015-07-27T15:55:30.933+0000 b.s.m.n.Client [INFO] connection attempt 13
> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 364 ms
> 2015-07-27T15:55:31.301+0000 b.s.m.n.Client [ERROR] connection attempt 13
> to Netty-Client-slave7/192.168.5.163:6703 failed:
> java.lang.RuntimeException: Returned channel was actually not established
> 2015-07-27T15:55:31.301+0000 b.s.m.n.Client [INFO] connection attempt 14
> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 366 ms
> 2015-07-27T15:55:31.670+0000 b.s.m.n.Client [ERROR] connection attempt 14
> to Netty-Client-slave7/192.168.5.163:6703 failed:
> java.lang.RuntimeException: Returned channel was actually not established
> 2015-07-27T15:55:31.671+0000 b.s.m.n.Client [INFO] connection attempt 15
> to Netty-Client-slave7/192.168.5.163:6703 scheduled to run in 368 ms
> 2015-07-27T15:55:32.041+0000 b.s.m.n.Client [INFO] connection established
> to Netty-Client-slave7/192.168.5.163:6703
>
> I understand that at the end the connection has been re-established.
> However, is it normal to get these errors? Does this mean that there is a
> network error which is not related to Storm?
>
> Thanks in advance,
> Dimitris Sarlis
>
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate