You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "B. Todd Burruss (JIRA)" <ji...@apache.org> on 2010/04/23 23:13:49 UTC
[jira] Created: (CASSANDRA-1019) "java.net.ConnectException:
Connection timed out" in MESSAGE-STREAMING-POOL:1
"java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
-----------------------------------------------------------------------------
Key: CASSANDRA-1019
URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.6.1
Reporter: B. Todd Burruss
after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
Mode: Normal
Streaming to: /192.168.132.117
/data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
Not receiving any streams.
in addition those same four nodes all show AE-SERVICE-STAGE with pending
work, and been showing this for several hours now. each node in the
cluster has less than 2gb, so it should be finished by now.
here is the exception:
2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
[DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
java.lang.RuntimeException: java.net.ConnectException: Connection timed out
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection timed out
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
[CassandraDaemon.java:78] Fatal exception in thread
Thread[MESSAGE-STREAMING-POOL:1,5,main]
java.lang.RuntimeException: java.net.ConnectException: Connection timed out
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection timed out
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1019) "java.net.ConnectException:
Connection timed out" in MESSAGE-STREAMING-POOL:1
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871777#action_12871777 ]
Gary Dusbabek commented on CASSANDRA-1019:
------------------------------------------
+6 on the 0.6 change. Re trunk: would it make sense to rename FileStatus.STREAM -> FileStatus.RESTREAM?
> "java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-1019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: B. Todd Burruss
> Assignee: Stu Hood
> Fix For: 0.6.3, 0.7
>
> Attachments: 1019-for-0.6-0001-Add-exponentially-backed-off-retry-to-FileStreamTask.patch, 1019-for-trunk-0001-Rename-CompletedFileStatus-to-FileStatus-to-indicate.patch, 1019-for-trunk-0002-Rename-StreamCompletionHandler-to-FileStatusHandler-.patch, 1019-for-trunk-0003-Rename-StreamCompletionAction-to-Action-and-change-d.patch, 1019-for-trunk-0004-Add-exponentially-backed-off-retry-to-FileStreamTask.patch
>
>
> after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
> Mode: Normal
> Streaming to: /192.168.132.117
> /data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
> Not receiving any streams.
> in addition those same four nodes all show AE-SERVICE-STAGE with pending
> work, and been showing this for several hours now. each node in the
> cluster has less than 2gb, so it should be finished by now.
> here is the exception:
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-1019) "java.net.ConnectException:
Connection timed out" in MESSAGE-STREAMING-POOL:1
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-1019:
-----------------------------------------
Assignee: (was: Stu Hood)
> "java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-1019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6
> Reporter: B. Todd Burruss
> Fix For: 0.6.2
>
>
> after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
> Mode: Normal
> Streaming to: /192.168.132.117
> /data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
> Not receiving any streams.
> in addition those same four nodes all show AE-SERVICE-STAGE with pending
> work, and been showing this for several hours now. each node in the
> cluster has less than 2gb, so it should be finished by now.
> here is the exception:
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-1019) "java.net.ConnectException:
Connection timed out" in MESSAGE-STREAMING-POOL:1
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-1019:
-----------------------------------------
Assignee: Stu Hood
Stu, could you add some "retry N (3? 10?) times before aborting the stream" logic?
> "java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-1019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6
> Reporter: B. Todd Burruss
> Assignee: Stu Hood
> Fix For: 0.6.2
>
>
> after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
> Mode: Normal
> Streaming to: /192.168.132.117
> /data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
> Not receiving any streams.
> in addition those same four nodes all show AE-SERVICE-STAGE with pending
> work, and been showing this for several hours now. each node in the
> cluster has less than 2gb, so it should be finished by now.
> here is the exception:
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1019) "java.net.ConnectException:
Connection timed out" in MESSAGE-STREAMING-POOL:1
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated CASSANDRA-1019:
--------------------------------
Attachment: 1019-for-trunk-0004-Add-exponentially-backed-off-retry-to-FileStreamTask.patch
...and a set that does the same thing for trunk. 1019-for-0.6-0001 should be the same as 1019-for-trunk-0004.
The rest of the set performs some cleanups in the Streaming package to give things more appropriate names and document them a little better.
> "java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-1019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: B. Todd Burruss
> Assignee: Stu Hood
> Fix For: 0.6.3, 0.7
>
> Attachments: 1019-for-0.6-0001-Add-exponentially-backed-off-retry-to-FileStreamTask.patch, 1019-for-trunk-0001-Rename-CompletedFileStatus-to-FileStatus-to-indicate.patch, 1019-for-trunk-0002-Rename-StreamCompletionHandler-to-FileStatusHandler-.patch, 1019-for-trunk-0003-Rename-StreamCompletionAction-to-Action-and-change-d.patch, 1019-for-trunk-0004-Add-exponentially-backed-off-retry-to-FileStreamTask.patch
>
>
> after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
> Mode: Normal
> Streaming to: /192.168.132.117
> /data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
> Not receiving any streams.
> in addition those same four nodes all show AE-SERVICE-STAGE with pending
> work, and been showing this for several hours now. each node in the
> cluster has less than 2gb, so it should be finished by now.
> here is the exception:
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1019) "java.net.ConnectException:
Connection timed out" in MESSAGE-STREAMING-POOL:1
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872200#action_12872200 ]
Hudson commented on CASSANDRA-1019:
-----------------------------------
Integrated in Cassandra #447 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/447/])
rename StreamCompletionAction to Action. patch by stuhood, reviwed by gdusbabek. CASSANDRA-1019
rename StreamCompletionHandler to FileStatusHandler. patch by stuhood, reviwed by gdusbabek. CASSANDRA-1019
rename CompletedFileStatus to FileStatus. patch by stuhood, reviwed by gdusbabek. CASSANDRA-1019
> "java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-1019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: B. Todd Burruss
> Assignee: Stu Hood
> Fix For: 0.6.3, 0.7
>
> Attachments: 1019-for-0.6-0001-Add-exponentially-backed-off-retry-to-FileStreamTask.patch, 1019-for-trunk-0001-Rename-CompletedFileStatus-to-FileStatus-to-indicate.patch, 1019-for-trunk-0002-Rename-StreamCompletionHandler-to-FileStatusHandler-.patch, 1019-for-trunk-0003-Rename-StreamCompletionAction-to-Action-and-change-d.patch, 1019-for-trunk-0004-Add-exponentially-backed-off-retry-to-FileStreamTask.patch
>
>
> after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
> Mode: Normal
> Streaming to: /192.168.132.117
> /data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
> Not receiving any streams.
> in addition those same four nodes all show AE-SERVICE-STAGE with pending
> work, and been showing this for several hours now. each node in the
> cluster has less than 2gb, so it should be finished by now.
> here is the exception:
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1019) "java.net.ConnectException:
Connection timed out" in MESSAGE-STREAMING-POOL:1
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1019:
--------------------------------------
Fix Version/s: 0.6.2
Affects Version/s: 0.6
(was: 0.6.1)
> "java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-1019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6
> Reporter: B. Todd Burruss
> Fix For: 0.6.2
>
>
> after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
> Mode: Normal
> Streaming to: /192.168.132.117
> /data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
> Not receiving any streams.
> in addition those same four nodes all show AE-SERVICE-STAGE with pending
> work, and been showing this for several hours now. each node in the
> cluster has less than 2gb, so it should be finished by now.
> here is the exception:
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1019) "java.net.ConnectException:
Connection timed out" in MESSAGE-STREAMING-POOL:1
Posted by "B. Todd Burruss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860975#action_12860975 ]
B. Todd Burruss commented on CASSANDRA-1019:
--------------------------------------------
some more info that may assist. we have just purchased new machines for our test cluster and we are having lots of trouble with the NICs going down. this causes an extremely long timeout situation and could have been the catalyst for this problem.
this situation does cause the cluster to behave very poorly because the connection takes several minutes to timeout. this type of situation makes me want the ability to manually take a node out of the cluster and prevent nodes from gossiping to it. is this something that has been talked about?
> "java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-1019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6
> Reporter: B. Todd Burruss
> Assignee: Stu Hood
> Fix For: 0.6.2
>
>
> after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
> Mode: Normal
> Streaming to: /192.168.132.117
> /data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
> Not receiving any streams.
> in addition those same four nodes all show AE-SERVICE-STAGE with pending
> work, and been showing this for several hours now. each node in the
> cluster has less than 2gb, so it should be finished by now.
> here is the exception:
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1019) "java.net.ConnectException:
Connection timed out" in MESSAGE-STREAMING-POOL:1
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated CASSANDRA-1019:
--------------------------------
Attachment: 1019-for-trunk-0001-Rename-CompletedFileStatus-to-FileStatus-to-indicate.patch
1019-for-trunk-0002-Rename-StreamCompletionHandler-to-FileStatusHandler-.patch
1019-for-trunk-0003-Rename-StreamCompletionAction-to-Action-and-change-d.patch
> "java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-1019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6
> Reporter: B. Todd Burruss
> Assignee: Stu Hood
> Fix For: 0.6.3
>
> Attachments: 1019-for-0.6-0001-Add-exponentially-backed-off-retry-to-FileStreamTask.patch, 1019-for-trunk-0001-Rename-CompletedFileStatus-to-FileStatus-to-indicate.patch, 1019-for-trunk-0002-Rename-StreamCompletionHandler-to-FileStatusHandler-.patch, 1019-for-trunk-0003-Rename-StreamCompletionAction-to-Action-and-change-d.patch
>
>
> after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
> Mode: Normal
> Streaming to: /192.168.132.117
> /data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
> Not receiving any streams.
> in addition those same four nodes all show AE-SERVICE-STAGE with pending
> work, and been showing this for several hours now. each node in the
> cluster has less than 2gb, so it should be finished by now.
> here is the exception:
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-1019) "java.net.ConnectException:
Connection timed out" in MESSAGE-STREAMING-POOL:1
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood reassigned CASSANDRA-1019:
-----------------------------------
Assignee: Stu Hood
> "java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-1019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6
> Reporter: B. Todd Burruss
> Assignee: Stu Hood
> Fix For: 0.6.3
>
>
> after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
> Mode: Normal
> Streaming to: /192.168.132.117
> /data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
> Not receiving any streams.
> in addition those same four nodes all show AE-SERVICE-STAGE with pending
> work, and been showing this for several hours now. each node in the
> cluster has less than 2gb, so it should be finished by now.
> here is the exception:
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1019)
"java.net.ConnectException: Connection timed out" in
MESSAGE-STREAMING-POOL:1
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871777#action_12871777 ]
Gary Dusbabek edited comment on CASSANDRA-1019 at 5/26/10 11:41 AM:
--------------------------------------------------------------------
+1 on the 0.6 change. Re trunk: would it make sense to rename FileStatus.STREAM -> FileStatus.RESTREAM?
was (Author: gdusbabek):
+6 on the 0.6 change. Re trunk: would it make sense to rename FileStatus.STREAM -> FileStatus.RESTREAM?
> "java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-1019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: B. Todd Burruss
> Assignee: Stu Hood
> Fix For: 0.6.3, 0.7
>
> Attachments: 1019-for-0.6-0001-Add-exponentially-backed-off-retry-to-FileStreamTask.patch, 1019-for-trunk-0001-Rename-CompletedFileStatus-to-FileStatus-to-indicate.patch, 1019-for-trunk-0002-Rename-StreamCompletionHandler-to-FileStatusHandler-.patch, 1019-for-trunk-0003-Rename-StreamCompletionAction-to-Action-and-change-d.patch, 1019-for-trunk-0004-Add-exponentially-backed-off-retry-to-FileStreamTask.patch
>
>
> after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
> Mode: Normal
> Streaming to: /192.168.132.117
> /data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
> Not receiving any streams.
> in addition those same four nodes all show AE-SERVICE-STAGE with pending
> work, and been showing this for several hours now. each node in the
> cluster has less than 2gb, so it should be finished by now.
> here is the exception:
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1019) "java.net.ConnectException:
Connection timed out" in MESSAGE-STREAMING-POOL:1
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871785#action_12871785 ]
Stu Hood commented on CASSANDRA-1019:
-------------------------------------
> Re trunk: would it make sense to rename FileStatus.STREAM -> FileStatus.RESTREAM?
I debated doing that, but since the FileStatus object exists for the lifetime of the transfer it needs to have an initial Action/status that indicates it is streaming.
> "java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-1019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: B. Todd Burruss
> Assignee: Stu Hood
> Fix For: 0.6.3, 0.7
>
> Attachments: 1019-for-0.6-0001-Add-exponentially-backed-off-retry-to-FileStreamTask.patch, 1019-for-trunk-0001-Rename-CompletedFileStatus-to-FileStatus-to-indicate.patch, 1019-for-trunk-0002-Rename-StreamCompletionHandler-to-FileStatusHandler-.patch, 1019-for-trunk-0003-Rename-StreamCompletionAction-to-Action-and-change-d.patch, 1019-for-trunk-0004-Add-exponentially-backed-off-retry-to-FileStreamTask.patch
>
>
> after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
> Mode: Normal
> Streaming to: /192.168.132.117
> /data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
> Not receiving any streams.
> in addition those same four nodes all show AE-SERVICE-STAGE with pending
> work, and been showing this for several hours now. each node in the
> cluster has less than 2gb, so it should be finished by now.
> here is the exception:
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1019) "java.net.ConnectException:
Connection timed out" in MESSAGE-STREAMING-POOL:1
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated CASSANDRA-1019:
--------------------------------
Attachment: 1019-for-0.6-0001-Add-exponentially-backed-off-retry-to-FileStreamTask.patch
Adds retries with exponential backoff to the connect step in FileStreamTask.
> "java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-1019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6
> Reporter: B. Todd Burruss
> Assignee: Stu Hood
> Fix For: 0.6.3
>
> Attachments: 1019-for-0.6-0001-Add-exponentially-backed-off-retry-to-FileStreamTask.patch, 1019-for-trunk-0001-Rename-CompletedFileStatus-to-FileStatus-to-indicate.patch, 1019-for-trunk-0002-Rename-StreamCompletionHandler-to-FileStatusHandler-.patch, 1019-for-trunk-0003-Rename-StreamCompletionAction-to-Action-and-change-d.patch
>
>
> after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
> Mode: Normal
> Streaming to: /192.168.132.117
> /data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
> Not receiving any streams.
> in addition those same four nodes all show AE-SERVICE-STAGE with pending
> work, and been showing this for several hours now. each node in the
> cluster has less than 2gb, so it should be finished by now.
> here is the exception:
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1019) "java.net.ConnectException:
Connection timed out" in MESSAGE-STREAMING-POOL:1
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860768#action_12860768 ]
Stu Hood commented on CASSANDRA-1019:
-------------------------------------
I should be able to tackle this Tuesday or Wednesday.
> "java.net.ConnectException: Connection timed out" in MESSAGE-STREAMING-POOL:1
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-1019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6
> Reporter: B. Todd Burruss
> Assignee: Stu Hood
> Fix For: 0.6.2
>
>
> after doing a nodetool repair on a node in my cluster, i see the following exception on 4 out of the 7 nodes. replication factor is 3. no compactions happening. no client traffic to the cluster. nodetool streams (on one of the nodes not repaired) shows the following which is not ever increasing:
> Mode: Normal
> Streaming to: /192.168.132.117
> /data/cassandra-data/data/UdsProfiles/stream/UdsProfiles-43-Data.db 0/523088443
> Not receiving any streams.
> in addition those same four nodes all show AE-SERVICE-STAGE with pending
> work, and been showing this for several hours now. each node in the
> cluster has less than 2gb, so it should be finished by now.
> here is the exception:
> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
> [CassandraDaemon.java:78] Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:1,5,main]
> java.lang.RuntimeException: java.net.ConnectException: Connection timed out
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.