You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stephen Walsh (JIRA)" <ji...@apache.org> on 2015/09/29 17:28:04 UTC
[jira] [Comment Edited] (CASSANDRA-10303) streaming for 'nodetool
rebuild' fails after adding a datacenter
[ https://issues.apache.org/jira/browse/CASSANDRA-10303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935324#comment-14935324 ]
Stephen Walsh edited comment on CASSANDRA-10303 at 9/29/15 3:27 PM:
--------------------------------------------------------------------
I've seen similar problems in 2.1.6 on a single DC
For example we have 100 keyspaces with 5 CF in each over 4 Cassandra nodes. - No data in any. (all data had a TTL of 10 seconds)
I tried to add another node to the system and after 2 hours 13 min in i get...
ERROR 12:25:19 [Stream #c7b822a0-6692-11e5-94b5-e758b0554964] Streaming error occurred
java.io.IOException: Connection timed out
I'm now 5 hours into trying to add in a 5th node in a Cassandra cluster with no data.
We are running in AWS so its unlikely to be network issues
was (Author: darkslice):
I've seen similar problems in 2.1.6.
For example we have 100 keyspaces with 5 CF in each over 4 Cassandra nodes. - No data in any. (all data had a TTL of 10 seconds)
I tried to add another node to the system and after 2 hours 13 min in i get...
ERROR 12:25:19 [Stream #c7b822a0-6692-11e5-94b5-e758b0554964] Streaming error occurred
java.io.IOException: Connection timed out
I'm now 5 hours into trying to add in a 5th node in a Cassandra cluster with no data.
We are running in AWS so its unlikely to be network issues
> streaming for 'nodetool rebuild' fails after adding a datacenter
> -----------------------------------------------------------------
>
> Key: CASSANDRA-10303
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10303
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: jdk1.7
> cassandra 2.1.8
> Reporter: zhaoyan
>
> we add another datacenter.
> use nodetool rebuild DC1
> stream from some node of old datacenter always hang up with these exception:
> {code}
> ERROR [Thread-1472] 2015-09-10 19:24:53,091 CassandraDaemon.java:223 - Exception in thread Thread[Thread-1472,5,RMI Runtime]
> java.lang.RuntimeException: java.io.IOException: Connection timed out
> at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.1.8.jar:2.1.8]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
> Caused by: java.io.IOException: Connection timed out
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_60]
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.7.0_60]
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.7.0_60]
> at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.7.0_60]
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[na:1.7.0_60]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) ~[na:1.7.0_60]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.7.0_60]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.7.0_60]
> at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172) ~[apache-cassandra-2.1.8.jar:2.1.8]
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.8.jar:2.1.8]
> ... 1 common frames omitted
> {code}
> i must restart node to stop current rebuild, and rebuild agagin and again to success....
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)