You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by cfavero <ch...@tricast.com> on 2013/09/20 19:03:02 UTC
sstableloader hangs at progress. never finishes
Hello,
I have tried unsuccessfully to stream snapshots from a 4 node cluster to a 2
node cluster. I have set up a different machine to run sstableloader on and
I can see in the logs that it starts the stream and that the tmp files are
created in the correct columnfamily folders.
we are on 1.2.5 of datastax community
destination server is 10.10.104.80 and 81
sstableloader server is 10.10.105.190
*the issue is that the command:*
sstableloader -v -t 1 --debug -d 10.10.104.80,10.10.104.81
/root/sstables/keyspace/ColFamily
*Outputs the following and never completes.*
[user@sstableloader user]# sstableloader -v -t 1 --debug -d
10.10.104.80,10.10.104.81 /user/sstables/keyspace/columfamily
Streaming revelant part of
/user/sstables/keyspace/columfamily/keysp-columnfam-ic-69-Data.db
/user/sstables/keyspace/columfamily/keysp-columnfam-ic-70-Data.db to
[/10.10.104.81, /10.10.104.80]
progress: [/10.10.104.81 1/2 (35)] [/10.10.104.80 1/2 (35)] [total: 35 -
0MB/s (avg: 0MB/s)]
*the destination server logs show the following:*
*when streaming starts:*
DEBUG [Thread-50] 2013-09-20 11:55:49,661 IncomingTcpConnection.java (line
75) Connection version 6 from /10.10.105.190
DEBUG [Thread-50] 2013-09-20 11:55:49,662 StreamInSession.java (line 104)
Adding file
/user/sstables/keyspace/columnfam/keyspace-columnfam-ic-70-Data.db to Stream
Request queue
DEBUG [Thread-50] 2013-09-20 11:55:49,662 StreamInSession.java (line 104)
Adding file
/user/sstables/keyspace/columnfam/keyspace-columnfam-ic-69-Data.db to Stream
Request queue
DEBUG [Thread-50] 2013-09-20 11:55:49,662 IncomingStreamReader.java (line
112) Receiving stream
DEBUG [Thread-50] 2013-09-20 11:55:49,662 IncomingStreamReader.java (line
113) Creating file for
/var/lib/cassandra/data/keyspace/columnfam/keyspace-columnfam-tmp-ic-7-Data.db
with 128 estimated keys
DEBUG [Thread-50] 2013-09-20 11:55:49,663 ColumnFamilyStore.java (line 863)
Checking for sstables overlapping []
*When I finally ctrl-c the hung process:*
DEBUG [Thread-50] 2013-09-20 11:55:54,195 FileUtils.java (line 110) Deleting
keyspace-columnfam-tmp-ic-7-Data.db
DEBUG [Thread-50] 2013-09-20 11:55:54,196 FileUtils.java (line 110) Deleting
keyspace-columnfam-tmp-ic-7-Index.db
DEBUG [Thread-50] 2013-09-20 11:55:54,196 FileUtils.java (line 110) Deleting
keyspace-columnfam-tmp-ic-7-Filter.db
DEBUG [Thread-50] 2013-09-20 11:55:54,196 FileUtils.java (line 110) Deleting
keyspace-columnfam-tmp-ic-7-TOC.txt
DEBUG [Thread-50] 2013-09-20 11:55:54,196 FileUtils.java (line 110) Deleting
keyspace-columnfam-tmp-ic-7-CompressionInfo.db
DEBUG [Thread-50] 2013-09-20 11:55:54,196 SSTable.java (line 154) Deleted
/var/lib/cassandra/data/keyspace/columnfam/keyspace-columnfam-tmp-ic-7
INFO [Thread-50] 2013-09-20 11:55:54,197 StreamInSession.java (line 136)
Streaming of file
/user/sstables/keyspace/columnfam/keyspace-columnfam-ic-70-Data.db
sections=1 progress=0/1617 - 0% for
org.apache.cassandra.streaming.StreamInSession@672baef4 failed: requesting a
retry.
ERROR [Thread-50] 2013-09-20 11:55:54,197 StreamInSession.java (line 143)
Sending retry message failed, closing session.
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:94)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
at java.nio.channels.Channels.writeFully(Channels.java:98)
at java.nio.channels.Channels.access$000(Channels.java:61)
at java.nio.channels.Channels$1.write(Channels.java:174)
at java.io.OutputStream.write(OutputStream.java:75)
at java.nio.channels.Channels$1.write(Channels.java:155)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:198)
at
org.apache.cassandra.streaming.StreamReply$FileStatusSerializer.serialize(StreamReply.java:75)
at
org.apache.cassandra.streaming.StreamReply$FileStatusSerializer.serialize(StreamReply.java:69)
at
org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:131)
at
org.apache.cassandra.net.OutboundTcpConnection.write(OutboundTcpConnection.java:240)
at
org.apache.cassandra.streaming.StreamInSession.sendMessage(StreamInSession.java:151)
at
org.apache.cassandra.streaming.StreamInSession.retry(StreamInSession.java:139)
at
org.apache.cassandra.streaming.IncomingStreamReader.retry(IncomingStreamReader.java:219)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:127)
at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
at
org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
ERROR [Thread-50] 2013-09-20 11:55:54,198 FailureDetector.java (line 154)
unknown endpoint /10.10.105.190
DEBUG [Thread-50] 2013-09-20 11:55:54,198 AbstractStreamSession.java (line
74) closing with status false
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/sstableloader-hangs-at-progress-never-finishes-tp7590464.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.