You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "B. Todd Burruss" <bb...@real.com> on 2010/04/23 22:34:58 UTC
MESSAGE-STREAMING-POOL exception
i see these exceptions on 4 out of the 7 nodes in my cluster. in
addition those same four nodes all show AE-SERVICE-STAGE with pending
work, and been showing this for several hours now. each node in the
cluster has less than 2gb, so it should be finished by now.
when i do nodetool streams on these nodes i see streams with byte counts
that are never increasing.
2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
[DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
java.lang.RuntimeException: java.net.ConnectException: Connection timed out
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection timed out
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
at
org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
[CassandraDaemon.java:78] Fatal exception in thread
Thread[MESSAGE-STREAMING-POOL:1,5,main]
java.lang.RuntimeException: java.net.ConnectException: Connection timed out
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection timed out
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
at
org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
Re: MESSAGE-STREAMING-POOL exception
Posted by "B. Todd Burruss" <bb...@real.com>.
https://issues.apache.org/jira/browse/CASSANDRA-1019
Jonathan Ellis wrote:
> Can you create a ticket?
>
> On Fri, Apr 23, 2010 at 3:50 PM, B. Todd Burruss <bb...@real.com> wrote:
>
>> i agree, but it seems to have implications on the streaming service.
>>
>> Jonathan Ellis wrote:
>>
>>> java.net.ConnectException: Connection timed out at
>>> sun.nio.ch.Net.connect is an os-level connection problem.
>>>
>>> On Fri, Apr 23, 2010 at 3:34 PM, B. Todd Burruss <bb...@real.com>
>>> wrote:
>>>
>>>
>>>> i see these exceptions on 4 out of the 7 nodes in my cluster. in
>>>> addition
>>>> those same four nodes all show AE-SERVICE-STAGE with pending work, and
>>>> been
>>>> showing this for several hours now. each node in the cluster has less
>>>> than
>>>> 2gb, so it should be finished by now.
>>>>
>>>> when i do nodetool streams on these nodes i see streams with byte counts
>>>> that are never increasing.
>>>>
>>>>
>>>> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
>>>> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
>>>> java.lang.RuntimeException: java.net.ConnectException: Connection timed
>>>> out
>>>> at
>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: java.net.ConnectException: Connection timed out
>>>> at sun.nio.ch.Net.connect(Native Method)
>>>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
>>>> at
>>>>
>>>> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
>>>> at
>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>> ... 3 more
>>>> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
>>>> [CassandraDaemon.java:78] Fatal exception in thread
>>>> Thread[MESSAGE-STREAMING-POOL:1,5,main]
>>>> java.lang.RuntimeException: java.net.ConnectException: Connection timed
>>>> out
>>>> at
>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: java.net.ConnectException: Connection timed out
>>>> at sun.nio.ch.Net.connect(Native Method)
>>>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
>>>> at
>>>>
>>>> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
>>>> at
>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>> ... 3 more
>>>>
>>>>
>>>>
>>>>
Re: MESSAGE-STREAMING-POOL exception
Posted by Jonathan Ellis <jb...@gmail.com>.
Can you create a ticket?
On Fri, Apr 23, 2010 at 3:50 PM, B. Todd Burruss <bb...@real.com> wrote:
> i agree, but it seems to have implications on the streaming service.
>
> Jonathan Ellis wrote:
>>
>> java.net.ConnectException: Connection timed out at
>> sun.nio.ch.Net.connect is an os-level connection problem.
>>
>> On Fri, Apr 23, 2010 at 3:34 PM, B. Todd Burruss <bb...@real.com>
>> wrote:
>>
>>>
>>> i see these exceptions on 4 out of the 7 nodes in my cluster. in
>>> addition
>>> those same four nodes all show AE-SERVICE-STAGE with pending work, and
>>> been
>>> showing this for several hours now. each node in the cluster has less
>>> than
>>> 2gb, so it should be finished by now.
>>>
>>> when i do nodetool streams on these nodes i see streams with byte counts
>>> that are never increasing.
>>>
>>>
>>> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
>>> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
>>> java.lang.RuntimeException: java.net.ConnectException: Connection timed
>>> out
>>> at
>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: java.net.ConnectException: Connection timed out
>>> at sun.nio.ch.Net.connect(Native Method)
>>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
>>> at
>>>
>>> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
>>> at
>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>> ... 3 more
>>> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
>>> [CassandraDaemon.java:78] Fatal exception in thread
>>> Thread[MESSAGE-STREAMING-POOL:1,5,main]
>>> java.lang.RuntimeException: java.net.ConnectException: Connection timed
>>> out
>>> at
>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: java.net.ConnectException: Connection timed out
>>> at sun.nio.ch.Net.connect(Native Method)
>>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
>>> at
>>>
>>> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
>>> at
>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>> ... 3 more
>>>
>>>
>>>
>
Re: MESSAGE-STREAMING-POOL exception
Posted by "B. Todd Burruss" <bb...@real.com>.
i agree, but it seems to have implications on the streaming service.
Jonathan Ellis wrote:
> java.net.ConnectException: Connection timed out at
> sun.nio.ch.Net.connect is an os-level connection problem.
>
> On Fri, Apr 23, 2010 at 3:34 PM, B. Todd Burruss <bb...@real.com> wrote:
>
>> i see these exceptions on 4 out of the 7 nodes in my cluster. in addition
>> those same four nodes all show AE-SERVICE-STAGE with pending work, and been
>> showing this for several hours now. each node in the cluster has less than
>> 2gb, so it should be finished by now.
>>
>> when i do nodetool streams on these nodes i see streams with byte counts
>> that are never increasing.
>>
>>
>> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
>> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
>> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.net.ConnectException: Connection timed out
>> at sun.nio.ch.Net.connect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
>> at
>> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> ... 3 more
>> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
>> [CassandraDaemon.java:78] Fatal exception in thread
>> Thread[MESSAGE-STREAMING-POOL:1,5,main]
>> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.net.ConnectException: Connection timed out
>> at sun.nio.ch.Net.connect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
>> at
>> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> ... 3 more
>>
>>
>>
Re: MESSAGE-STREAMING-POOL exception
Posted by Jonathan Ellis <jb...@gmail.com>.
java.net.ConnectException: Connection timed out at
sun.nio.ch.Net.connect is an os-level connection problem.
On Fri, Apr 23, 2010 at 3:34 PM, B. Todd Burruss <bb...@real.com> wrote:
> i see these exceptions on 4 out of the 7 nodes in my cluster. in addition
> those same four nodes all show AE-SERVICE-STAGE with pending work, and been
> showing this for several hours now. each node in the cluster has less than
> 2gb, so it should be finished by now.
>
> when i do nodetool streams on these nodes i see streams with byte counts
> that are never increasing.
>
>
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at
> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at
> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
>
>