You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Joe Obernberger <jo...@gmail.com> on 2024/08/14 20:50:02 UTC
Bootstrap error - Cassandra 4.1.5
Hi all - when adding a node to our existing 15 node cluster, I get:
DEBUG [NonPeriodicTasks:1] 2024-08-14 20:34:10,383
StreamCoordinator.java:152 - Finished connecting all sessions
WARN [NonPeriodicTasks:1] 2024-08-14 20:34:10,385
StreamResultFuture.java:242 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Stream failed
DEBUG [NonPeriodicTasks:1] 2024-08-14 20:34:10,386
StreamSession.java:529 - [Stream #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f]
Will close attached inbound
{3d52d53f=org.apache.cassandra.streaming.async.NettyStreamingChannel@2e6b7635,
67655215=org.apache.cassandra.streaming.async.NettyStreamingChannel@11e1a7e5,
d4d397f3=org.apache.cassandra.streaming.async.NettyStreamingChannel@11959fa4,
50e7cefb=org.apache.cassandra.streaming.async.NettyStreamingChannel@7a69edc0,
dfbbe8cd=org.apache.cassandra.streaming.async.NettyStreamingChannel@f46e666,
55d116ad=org.apache.cassandra.streaming.async.NettyStreamingChannel@1304a6e4,
5cf05913=org.apache.cassandra.streaming.async.NettyStreamingChannel@2ed1739f,
833323b4=org.apache.cassandra.streaming.async.NettyStreamingChannel@1be38c68,
1146e1c5=org.apache.cassandra.streaming.async.NettyStreamingChannel@232e3ca4,
f983d95c=org.apache.cassandra.streaming.async.NettyStreamingChannel@152b8da3,
3f061317=org.apache.cassandra.streaming.async.NettyStreamingChannel@30434912,
0c0b1395=org.apache.cassandra.streaming.async.NettyStreamingChannel@374403be,
30dafd8c=org.apache.cassandra.streaming.async.NettyStreamingChannel@65ddc2ee,
28c0fe3c=org.apache.cassandra.streaming.async.NettyStreamingChannel@2cd20d63,
55d1221c=org.apache.cassandra.streaming.async.NettyStreamingChannel@6f3ca32,
4fca9827=org.apache.cassandra.streaming.async.NettyStreamingChannel@113d70a1,
bea79e0c=org.apache.cassandra.streaming.async.NettyStreamingChannel@1afc0ada}
and outbound
{0c0b1395=org.apache.cassandra.streaming.async.NettyStreamingChannel@374403be}
channels
DEBUG [Stream-Deserializer-/192.168.189.127:7000-833323b4] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-50e7cefb] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-dfbbe8cd] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-67655215] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-55d116ad] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-1146e1c5] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-f983d95c] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-5cf05913] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-3f061317] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-0c0b1395] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-d4d397f3] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-28c0fe3c] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-3d52d53f] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-55d1221c] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-30dafd8c] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-4fca9827] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
DEBUG [Stream-Deserializer-/192.168.189.127:7000-bea79e0c] 2024-08-14
20:34:10,387 StreamSession.java:677 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
completed with state COMPLETE
ERROR [main] 2024-08-14 20:34:10,387 StorageService.java:2041 - Error
while waiting on bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
at
org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2034)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1185)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1145)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:936)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:854)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:243)
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
at
org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:517)
at
org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
at
org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at
org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
DEBUG [Messaging-EventLoop-3-15] 2024-08-14 20:34:10,388
StreamingMultiplexedChannel.java:513 - [Stream
#d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Closing stream connection
channels on /192.168.189.127:7000
WARN [main] 2024-08-14 20:34:10,456 StorageService.java:1221 - Some
data streaming failed. Use nodetool to check bootstrap state and resume.
For more, see `nodetool help bootstrap`. IN_PROGRESS
INFO [main] 2024-08-14 20:34:10,458 Gossiper.java:2293 - Waiting for
gossip to settle...
DEBUG [main] 2024-08-14 20:34:16,458 Gossiper.java:2305 - Gossip looks
settled.
root@cassandra-15:/# nodetool status -r
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN cassandra-6.cassandra.cassandra-jos.svc.cluster.local 2.07 TiB
256 20.0% 34a419fb-4db1-4fb4-8ea7-c0988b2a3d5a rack1
UN cassandra-3.cassandra.cassandra-jos.svc.cluster.local 2.08 TiB
256 20.0% 354a9e1f-71b4-4aba-9f03-26778aaa17e2 rack1
UN cassandra-14.cassandra.cassandra-jos.svc.cluster.local 1.95 TiB
256 20.0% 6e748875-1101-4bc1-a132-c6fe643c6148 rack1
UN cassandra-5.cassandra.cassandra-jos.svc.cluster.local 2.07 TiB
256 20.0% ead07b83-a927-4904-bc6c-e8e4249f7560 rack1
UN cassandra-7.cassandra.cassandra-jos.svc.cluster.local 2.08 TiB
256 20.0% b5fbc09e-565e-402d-a6d0-3feac5922a13 rack1
UN cassandra-4.cassandra.cassandra-jos.svc.cluster.local 2.27 TiB
256 20.0% 14fa3f70-a8e2-4f60-8bc9-42c7b3294718 rack1
UN cassandra-1.cassandra.cassandra-jos.svc.cluster.local 2.07 TiB
256 20.0% ea5979b8-71fe-4af5-b856-db06e008ced3 rack1
UN cassandra-11.cassandra.cassandra-jos.svc.cluster.local 2.08 TiB
256 20.0% f2bb43f7-8381-4b28-8ced-194060267c29 rack1
UN cassandra-9.cassandra.cassandra-jos.svc.cluster.local 2.07 TiB
256 20.0% ed1ce133-4473-4bc1-a0cc-a242d6f67acc rack1
UN cassandra-2.cassandra.cassandra-jos.svc.cluster.local 2.08 TiB
256 20.0% 2b5db930-2bfe-4829-b353-3b1db0ca368f rack1
UN cassandra-12.cassandra.cassandra-jos.svc.cluster.local 2.07 TiB
256 20.0% 1d3ed71f-18b4-4cc0-9c6e-3a5479328496 rack1
UJ cassandra-15.cassandra.cassandra-jos.svc.cluster.local 874.82 GiB
256 ? 9d68cd5a-6e48-4156-92c1-5c51f02bce65 rack1
UN cassandra-0.cassandra.cassandra-jos.svc.cluster.local 2.14 TiB
256 20.0% 5c3d3b46-300d-49f2-b01d-cbdb44d98022 rack1
UN cassandra-8.cassandra.cassandra-jos.svc.cluster.local 2.07 TiB
256 20.0% fb2e8220-549e-4316-bb78-53f6cd07318a rack1
UN cassandra-10.cassandra.cassandra-jos.svc.cluster.local 2.07 TiB
256 20.0% a2ab4d22-b564-4d6b-b743-2d78598fe53c rack1
UN cassandra-13.cassandra.cassandra-jos.svc.cluster.local 2.07 TiB
256 20.0% ed865baf-5333-40e5-8c73-9de2df2bd330 rack1
If I restart the bootstrap, it usually completes, but I'd like to double
the size of the cluster, and that's a very long operation. Is there
anyway to add multiple nodes at once?
Thanks!
-Joe
--
This email has been checked for viruses by AVG antivirus software.
www.avg.com
Re: Bootstrap error - Cassandra 4.1.5
Posted by Bowen Song via user <us...@cassandra.apache.org>.
ClosedChannelException is often the result of a network connectivity
issue. It's worth check the OS logs and the configuration of network
devices between the nodes. For example, has the conntrack table filled
up? Does any router between the two nodes have short TCP session timeout?
A crashed process or server can also cause the ClosedChannelException on
the other end, but since you have logs from both ends showing the same
error, this is definitely not the case.
On 15/08/2024 22:14, Joe Obernberger wrote:
> Thank you Bowen. Adding more nodes, but the latest one keeps failing.
>
> On the node being added:ERROR
> [Stream-Deserializer-/192.168.115.187:7000-83045bad] 2024-08-15
> 21:08:08,315 CassandraEntireSSTableStreamReader.java:146 - [Stream
> 59fa6610-5b48-11ef-8ffc-a13bcc763c65] Error while reading sstable from
> stream for table = doc.embeddings
> java.nio.channels.ClosedChannelException: null
> at
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
> at
> org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:139)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:228)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:217)
> at
> org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:124)
> at
> org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:50)
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36)
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
> at
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
> ERROR [Stream-Deserializer-/192.168.115.187:7000-83045bad] 2024-08-15
> 21:08:08,318 StreamSession.java:725 - [Stream
> #59fa6610-5b48-11ef-8ffc-a13bcc763c65] Streaming error occurred on
> session with peer 192.168.115.187:7000
> org.apache.cassandra.streaming.StreamReceiveException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:59)
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36)
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
> at
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: java.nio.channels.ClosedChannelException: null
> at
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
> at
> org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:139)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:228)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:217)
> at
> org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:124)
> at
> org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:50)
> ... 5 common frames omitted
>
> On the192.168.115.187:
>
> DEBUG [Stream-Deserializer-/192.168.100.69:7000-7fd4b735] 2024-08-15
> 21:07:35,683 StreamDeserializingTask.java:74 - [Stream
> #59fa6610-5b48-11ef-8ffc-a13bcc763c65 channel: 7fd4b735] Received
> Session Failed
> ERROR [Stream-Deserializer-/192.168.100.69:7000-d0937d5a] 2024-08-15
> 21:07:35,686 CassandraEntireSSTableStreamReader.java:146 - [Stream
> 59fa6610-5b48-11ef-8ffc-a13bcc763c65] Error while reading sstable from
> stream for table = doc.embeddings
> java.nio.channels.ClosedChannelException: null
> at
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
> at
> org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:139)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:228)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:217)
> at
> org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:124)
> at
> org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:50)
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36)
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
> at
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
> ERROR [Stream-Deserializer-/192.168.100.69:7000-0e3bae15] 2024-08-15
> 21:07:35,686 CassandraEntireSSTableStreamReader.java:146 - [Stream
> 59fa6610-5b48-11ef-8ffc-a13bcc763c65] Error while reading sstable from
> stream for table = doc.embeddings
> java.nio.channels.ClosedChannelException: null
> at
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
> at
> org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:139)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:228)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:217)
> at
> org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:124)
> at
> org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:50)
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36)
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
> at
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
> ERROR [Stream-Deserializer-/192.168.100.69:7000-d0937d5a] 2024-08-15
> 21:07:35,690 StreamSession.java:725 - [Stream
> #59fa6610-5b48-11ef-8ffc-a13bcc763c65] Streaming error occurred on
> session with peer 192.168.100.69:7000
> org.apache.cassandra.streaming.StreamReceiveException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:59)
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36)
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
> at
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: java.nio.channels.ClosedChannelException: null
> at
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
> at
> org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:139)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:228)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:217)
> at
> org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:124)
> at
> org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:50)
> ... 5 common frames omitted
> ERROR [Stream-Deserializer-/192.168.100.69:7000-0e3bae15] 2024-08-15
> 21:07:35,690 StreamSession.java:725 - [Stream
> #59fa6610-5b48-11ef-8ffc-a13bcc763c65] Streaming error occurred on
> session with peer 192.168.100.69:7000
> org.apache.cassandra.streaming.StreamReceiveException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:59)
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36)
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
> at
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: java.nio.channels.ClosedChannelException: null
> at
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
> at
> org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:139)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:228)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:217)
> at
> org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:124)
> at
> org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
> at
> org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:50)
> ... 5 common frames omitted
> DEBUG [Stream-Deserializer-/192.168.21.246:7000-bbc1f78c] 2024-08-15
> 21:07:36,012 CassandraEntireSSTableStreamReader.java:128 - [Stream
> #59fa6610-5b48-11ef-8ffc-a13bcc763c65] Finished receiving Index.db
> component from /192.168.21.246:7000, componentSize = 1.773MiB,
> readBytes = 161.935MiB, totalSize = 162.050MiB
> DEBUG [Stream-Deserializer-/192.168.21.246:7000-bbc1f78c] 2024-08-15
> 21:07:36,012 CassandraEntireSSTableStreamReader.java:116 - [Stream
> #59fa6610-5b48-11ef-8ffc-a13bcc763c65] Started receiving Statistics.db
> component from /192.168.21.246:7000, componentSize = 10.388KiB,
> readBytes = 161.935MiB, totalSize = 162.050MiB
> 53721,1 99%
>
> Any ideas?
>
> Thank you!
>
> -Joe
>
> On 8/15/2024 9:03 AM, Bowen Song via user wrote:
>> You may need to look at the zipped log files if the streaming had
>> been running for a while before failing. The error could have
>> happened hours or days before the final failure.
>>
>> If your cluster is already experiencing performance issues (e.g. due
>> to CPU bottleneck or GC pauses), it's highly likely that these are
>> related to the streaming failure.
>>
>> On 15/08/2024 13:41, Joe Obernberger wrote:
>>> Thank you Bowen - yeah the only ERROR I see in
>>> /var/log/cassandra/debug.log is:
>>>
>>> ERROR [main] 2024-08-15 04:48:23,374 StorageService.java:2041 -
>>> Error while waiting on bootstrap to complete. Bootstrap will have to
>>> be restarted.
>>> java.util.concurrent.ExecutionException:
>>> org.apache.cassandra.streaming.StreamException: Stream failed
>>> at
>>> org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
>>> at
>>> org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246)
>>> at
>>> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2034)
>>> at
>>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1185)
>>> at
>>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1145)
>>> at
>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:936)
>>> at
>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:854)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
>>> Caused by: org.apache.cassandra.streaming.StreamException: Stream
>>> failed
>>> at
>>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:243)
>>> at
>>> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
>>> at
>>> org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:517)
>>> at
>>> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
>>> at
>>> org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
>>> at
>>> org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
>>> at
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>> Source)
>>> at java.base/java.util.concurrent.FutureTask.run(Unknown
>>> Source)
>>> at
>>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>> Source)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>>> Source)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>> Source)
>>> at
>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>> at java.base/java.lang.Thread.run(Unknown Source)
>>>
>>> The cluster is running inside of kubernetes on bare metal with a
>>> netapp for storage. I'd love a way to double the number of nodes,
>>> but sounds like I shouldn't have let it get this far. We're having
>>> some odd performance issues on reads, that I'm diagnosing.
>>>
>>> -Joe
>>>
>>> On 8/14/2024 5:07 PM, Bowen Song via user wrote:
>>>> It looks like all your nodes are in the same DC and the same rack
>>>> with 256 vnodes each. It's very hard (if not impossible) to add
>>>> multiple nodes to the same DC concurrently and safely in this
>>>> setup. You are better off adding one node at a time to this cluster.
>>>>
>>>> Try search for "ERROR" in the logs, it should tell you why did the
>>>> streaming session fail. If you can find the cause of the failure,
>>>> you may be able to prevent or reduce the chance of it happening
>>>> again in the future.
>>>>
>>>> On 14/08/2024 21:50, Joe Obernberger wrote:
>>>>> Hi all - when adding a node to our existing 15 node cluster, I get:
>>>>>
>>>>>
>>>>> DEBUG [NonPeriodicTasks:1] 2024-08-14 20:34:10,383
>>>>> StreamCoordinator.java:152 - Finished connecting all sessions
>>>>> WARN [NonPeriodicTasks:1] 2024-08-14 20:34:10,385
>>>>> StreamResultFuture.java:242 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Stream failed
>>>>> DEBUG [NonPeriodicTasks:1] 2024-08-14 20:34:10,386
>>>>> StreamSession.java:529 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Will close attached inbound
>>>>> {3d52d53f=org.apache.cassandra.streaming.async.NettyStreamingChannel@2e6b7635,
>>>>> 67655215=org.apache.cassandra.streaming.async.NettyStreamingChannel@11e1a7e5,
>>>>> d4d397f3=org.apache.cassandra.streaming.async.NettyStreamingChannel@11959fa4,
>>>>> 50e7cefb=org.apache.cassandra.streaming.async.NettyStreamingChannel@7a69edc0,
>>>>> dfbbe8cd=org.apache.cassandra.streaming.async.NettyStreamingChannel@f46e666,
>>>>> 55d116ad=org.apache.cassandra.streaming.async.NettyStreamingChannel@1304a6e4,
>>>>> 5cf05913=org.apache.cassandra.streaming.async.NettyStreamingChannel@2ed1739f,
>>>>> 833323b4=org.apache.cassandra.streaming.async.NettyStreamingChannel@1be38c68,
>>>>> 1146e1c5=org.apache.cassandra.streaming.async.NettyStreamingChannel@232e3ca4,
>>>>> f983d95c=org.apache.cassandra.streaming.async.NettyStreamingChannel@152b8da3,
>>>>> 3f061317=org.apache.cassandra.streaming.async.NettyStreamingChannel@30434912,
>>>>> 0c0b1395=org.apache.cassandra.streaming.async.NettyStreamingChannel@374403be,
>>>>> 30dafd8c=org.apache.cassandra.streaming.async.NettyStreamingChannel@65ddc2ee,
>>>>> 28c0fe3c=org.apache.cassandra.streaming.async.NettyStreamingChannel@2cd20d63,
>>>>> 55d1221c=org.apache.cassandra.streaming.async.NettyStreamingChannel@6f3ca32,
>>>>> 4fca9827=org.apache.cassandra.streaming.async.NettyStreamingChannel@113d70a1,
>>>>> bea79e0c=org.apache.cassandra.streaming.async.NettyStreamingChannel@1afc0ada}
>>>>> and outbound
>>>>> {0c0b1395=org.apache.cassandra.streaming.async.NettyStreamingChannel@374403be}
>>>>> channels
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-833323b4]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-50e7cefb]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-dfbbe8cd]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-67655215]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-55d116ad]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-1146e1c5]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-f983d95c]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-5cf05913]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-3f061317]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-0c0b1395]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-d4d397f3]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-28c0fe3c]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-3d52d53f]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-55d1221c]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-30dafd8c]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-4fca9827]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-bea79e0c]
>>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>>> completed with state COMPLETE
>>>>> ERROR [main] 2024-08-14 20:34:10,387 StorageService.java:2041 -
>>>>> Error while waiting on bootstrap to complete. Bootstrap will have
>>>>> to be restarted.
>>>>> java.util.concurrent.ExecutionException:
>>>>> org.apache.cassandra.streaming.StreamException: Stream failed
>>>>> at
>>>>> org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
>>>>> at
>>>>> org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246)
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2034)
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1185)
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1145)
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:936)
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:854)
>>>>> at
>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
>>>>> at
>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
>>>>> at
>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
>>>>> Caused by: org.apache.cassandra.streaming.StreamException: Stream
>>>>> failed
>>>>> at
>>>>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:243)
>>>>> at
>>>>> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
>>>>> at
>>>>> org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:517)
>>>>> at
>>>>> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
>>>>> at
>>>>> org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
>>>>> at
>>>>> org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
>>>>> at
>>>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>>>> Source)
>>>>> at java.base/java.util.concurrent.FutureTask.run(Unknown
>>>>> Source)
>>>>> at
>>>>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>>>> Source)
>>>>> at
>>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>>>>> Source)
>>>>> at
>>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>>>> Source)
>>>>> at
>>>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>>>> at java.base/java.lang.Thread.run(Unknown Source)
>>>>> DEBUG [Messaging-EventLoop-3-15] 2024-08-14 20:34:10,388
>>>>> StreamingMultiplexedChannel.java:513 - [Stream
>>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Closing stream connection
>>>>> channels on /192.168.189.127:7000
>>>>> WARN [main] 2024-08-14 20:34:10,456 StorageService.java:1221 -
>>>>> Some data streaming failed. Use nodetool to check bootstrap state
>>>>> and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
>>>>> INFO [main] 2024-08-14 20:34:10,458 Gossiper.java:2293 - Waiting
>>>>> for gossip to settle...
>>>>> DEBUG [main] 2024-08-14 20:34:16,458 Gossiper.java:2305 - Gossip
>>>>> looks settled.
>>>>>
>>>>>
>>>>> root@cassandra-15:/# nodetool status -r
>>>>> Datacenter: dc1
>>>>> ===============
>>>>> Status=Up/Down
>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>> -- Address Load Tokens Owns (effective) Host
>>>>> ID Rack
>>>>> UN cassandra-6.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>>> TiB 256 20.0% 34a419fb-4db1-4fb4-8ea7-c0988b2a3d5a rack1
>>>>> UN cassandra-3.cassandra.cassandra-jos.svc.cluster.local 2.08
>>>>> TiB 256 20.0% 354a9e1f-71b4-4aba-9f03-26778aaa17e2 rack1
>>>>> UN cassandra-14.cassandra.cassandra-jos.svc.cluster.local 1.95
>>>>> TiB 256 20.0% 6e748875-1101-4bc1-a132-c6fe643c6148 rack1
>>>>> UN cassandra-5.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>>> TiB 256 20.0% ead07b83-a927-4904-bc6c-e8e4249f7560 rack1
>>>>> UN cassandra-7.cassandra.cassandra-jos.svc.cluster.local 2.08
>>>>> TiB 256 20.0% b5fbc09e-565e-402d-a6d0-3feac5922a13 rack1
>>>>> UN cassandra-4.cassandra.cassandra-jos.svc.cluster.local 2.27
>>>>> TiB 256 20.0% 14fa3f70-a8e2-4f60-8bc9-42c7b3294718 rack1
>>>>> UN cassandra-1.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>>> TiB 256 20.0% ea5979b8-71fe-4af5-b856-db06e008ced3 rack1
>>>>> UN cassandra-11.cassandra.cassandra-jos.svc.cluster.local 2.08
>>>>> TiB 256 20.0% f2bb43f7-8381-4b28-8ced-194060267c29 rack1
>>>>> UN cassandra-9.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>>> TiB 256 20.0% ed1ce133-4473-4bc1-a0cc-a242d6f67acc rack1
>>>>> UN cassandra-2.cassandra.cassandra-jos.svc.cluster.local 2.08
>>>>> TiB 256 20.0% 2b5db930-2bfe-4829-b353-3b1db0ca368f rack1
>>>>> UN cassandra-12.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>>> TiB 256 20.0% 1d3ed71f-18b4-4cc0-9c6e-3a5479328496 rack1
>>>>> UJ cassandra-15.cassandra.cassandra-jos.svc.cluster.local 874.82
>>>>> GiB 256 ? 9d68cd5a-6e48-4156-92c1-5c51f02bce65 rack1
>>>>> UN cassandra-0.cassandra.cassandra-jos.svc.cluster.local 2.14
>>>>> TiB 256 20.0% 5c3d3b46-300d-49f2-b01d-cbdb44d98022 rack1
>>>>> UN cassandra-8.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>>> TiB 256 20.0% fb2e8220-549e-4316-bb78-53f6cd07318a rack1
>>>>> UN cassandra-10.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>>> TiB 256 20.0% a2ab4d22-b564-4d6b-b743-2d78598fe53c rack1
>>>>> UN cassandra-13.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>>> TiB 256 20.0% ed865baf-5333-40e5-8c73-9de2df2bd330 rack1
>>>>>
>>>>>
>>>>> If I restart the bootstrap, it usually completes, but I'd like to
>>>>> double the size of the cluster, and that's a very long operation.
>>>>> Is there anyway to add multiple nodes at once?
>>>>> Thanks!
>>>>>
>>>>> -Joe
>>>>>
>>>>>
>>>
>
Re: Bootstrap error - Cassandra 4.1.5
Posted by Joe Obernberger <jo...@gmail.com>.
Thank you Bowen. Adding more nodes, but the latest one keeps failing.
On the node being added:ERROR
[Stream-Deserializer-/192.168.115.187:7000-83045bad] 2024-08-15
21:08:08,315 CassandraEntireSSTableStreamReader.java:146 - [Stream
59fa6610-5b48-11ef-8ffc-a13bcc763c65] Error while reading sstable from
stream for table = doc.embeddings
java.nio.channels.ClosedChannelException: null
at
org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
at
org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:139)
at
org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:228)
at
org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:217)
at
org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:124)
at
org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:50)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36)
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
at
org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
ERROR [Stream-Deserializer-/192.168.115.187:7000-83045bad] 2024-08-15
21:08:08,318 StreamSession.java:725 - [Stream
#59fa6610-5b48-11ef-8ffc-a13bcc763c65] Streaming error occurred on
session with peer 192.168.115.187:7000
org.apache.cassandra.streaming.StreamReceiveException:
java.nio.channels.ClosedChannelException
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:59)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36)
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
at
org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.nio.channels.ClosedChannelException: null
at
org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
at
org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:139)
at
org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:228)
at
org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:217)
at
org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:124)
at
org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:50)
... 5 common frames omitted
On the192.168.115.187:
DEBUG [Stream-Deserializer-/192.168.100.69:7000-7fd4b735] 2024-08-15
21:07:35,683 StreamDeserializingTask.java:74 - [Stream
#59fa6610-5b48-11ef-8ffc-a13bcc763c65 channel: 7fd4b735] Received
Session Failed
ERROR [Stream-Deserializer-/192.168.100.69:7000-d0937d5a] 2024-08-15
21:07:35,686 CassandraEntireSSTableStreamReader.java:146 - [Stream
59fa6610-5b48-11ef-8ffc-a13bcc763c65] Error while reading sstable from
stream for table = doc.embeddings
java.nio.channels.ClosedChannelException: null
at
org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
at
org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:139)
at
org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:228)
at
org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:217)
at
org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:124)
at
org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:50)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36)
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
at
org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
ERROR [Stream-Deserializer-/192.168.100.69:7000-0e3bae15] 2024-08-15
21:07:35,686 CassandraEntireSSTableStreamReader.java:146 - [Stream
59fa6610-5b48-11ef-8ffc-a13bcc763c65] Error while reading sstable from
stream for table = doc.embeddings
java.nio.channels.ClosedChannelException: null
at
org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
at
org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:139)
at
org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:228)
at
org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:217)
at
org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:124)
at
org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:50)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36)
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
at
org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
ERROR [Stream-Deserializer-/192.168.100.69:7000-d0937d5a] 2024-08-15
21:07:35,690 StreamSession.java:725 - [Stream
#59fa6610-5b48-11ef-8ffc-a13bcc763c65] Streaming error occurred on
session with peer 192.168.100.69:7000
org.apache.cassandra.streaming.StreamReceiveException:
java.nio.channels.ClosedChannelException
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:59)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36)
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
at
org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.nio.channels.ClosedChannelException: null
at
org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
at
org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:139)
at
org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:228)
at
org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:217)
at
org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:124)
at
org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:50)
... 5 common frames omitted
ERROR [Stream-Deserializer-/192.168.100.69:7000-0e3bae15] 2024-08-15
21:07:35,690 StreamSession.java:725 - [Stream
#59fa6610-5b48-11ef-8ffc-a13bcc763c65] Streaming error occurred on
session with peer 192.168.100.69:7000
org.apache.cassandra.streaming.StreamReceiveException:
java.nio.channels.ClosedChannelException
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:59)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36)
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
at
org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.nio.channels.ClosedChannelException: null
at
org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
at
org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:139)
at
org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:228)
at
org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:217)
at
org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:124)
at
org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:50)
... 5 common frames omitted
DEBUG [Stream-Deserializer-/192.168.21.246:7000-bbc1f78c] 2024-08-15
21:07:36,012 CassandraEntireSSTableStreamReader.java:128 - [Stream
#59fa6610-5b48-11ef-8ffc-a13bcc763c65] Finished receiving Index.db
component from /192.168.21.246:7000, componentSize = 1.773MiB, readBytes
= 161.935MiB, totalSize = 162.050MiB
DEBUG [Stream-Deserializer-/192.168.21.246:7000-bbc1f78c] 2024-08-15
21:07:36,012 CassandraEntireSSTableStreamReader.java:116 - [Stream
#59fa6610-5b48-11ef-8ffc-a13bcc763c65] Started receiving Statistics.db
component from /192.168.21.246:7000, componentSize = 10.388KiB,
readBytes = 161.935MiB, totalSize = 162.050MiB
53721,1 99%
Any ideas?
Thank you!
-Joe
On 8/15/2024 9:03 AM, Bowen Song via user wrote:
> You may need to look at the zipped log files if the streaming had been
> running for a while before failing. The error could have happened
> hours or days before the final failure.
>
> If your cluster is already experiencing performance issues (e.g. due
> to CPU bottleneck or GC pauses), it's highly likely that these are
> related to the streaming failure.
>
> On 15/08/2024 13:41, Joe Obernberger wrote:
>> Thank you Bowen - yeah the only ERROR I see in
>> /var/log/cassandra/debug.log is:
>>
>> ERROR [main] 2024-08-15 04:48:23,374 StorageService.java:2041 - Error
>> while waiting on bootstrap to complete. Bootstrap will have to be
>> restarted.
>> java.util.concurrent.ExecutionException:
>> org.apache.cassandra.streaming.StreamException: Stream failed
>> at
>> org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
>> at
>> org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246)
>> at
>> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2034)
>> at
>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1185)
>> at
>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1145)
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:936)
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:854)
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
>> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>> at
>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:243)
>> at
>> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
>> at
>> org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:517)
>> at
>> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
>> at
>> org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
>> at
>> org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
>> at
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>> Source)
>> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
>> at
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>> Source)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>> Source)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>> Source)
>> at
>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>> at java.base/java.lang.Thread.run(Unknown Source)
>>
>> The cluster is running inside of kubernetes on bare metal with a
>> netapp for storage. I'd love a way to double the number of nodes,
>> but sounds like I shouldn't have let it get this far. We're having
>> some odd performance issues on reads, that I'm diagnosing.
>>
>> -Joe
>>
>> On 8/14/2024 5:07 PM, Bowen Song via user wrote:
>>> It looks like all your nodes are in the same DC and the same rack
>>> with 256 vnodes each. It's very hard (if not impossible) to add
>>> multiple nodes to the same DC concurrently and safely in this setup.
>>> You are better off adding one node at a time to this cluster.
>>>
>>> Try search for "ERROR" in the logs, it should tell you why did the
>>> streaming session fail. If you can find the cause of the failure,
>>> you may be able to prevent or reduce the chance of it happening
>>> again in the future.
>>>
>>> On 14/08/2024 21:50, Joe Obernberger wrote:
>>>> Hi all - when adding a node to our existing 15 node cluster, I get:
>>>>
>>>>
>>>> DEBUG [NonPeriodicTasks:1] 2024-08-14 20:34:10,383
>>>> StreamCoordinator.java:152 - Finished connecting all sessions
>>>> WARN [NonPeriodicTasks:1] 2024-08-14 20:34:10,385
>>>> StreamResultFuture.java:242 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Stream failed
>>>> DEBUG [NonPeriodicTasks:1] 2024-08-14 20:34:10,386
>>>> StreamSession.java:529 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Will close attached inbound
>>>> {3d52d53f=org.apache.cassandra.streaming.async.NettyStreamingChannel@2e6b7635,
>>>> 67655215=org.apache.cassandra.streaming.async.NettyStreamingChannel@11e1a7e5,
>>>> d4d397f3=org.apache.cassandra.streaming.async.NettyStreamingChannel@11959fa4,
>>>> 50e7cefb=org.apache.cassandra.streaming.async.NettyStreamingChannel@7a69edc0,
>>>> dfbbe8cd=org.apache.cassandra.streaming.async.NettyStreamingChannel@f46e666,
>>>> 55d116ad=org.apache.cassandra.streaming.async.NettyStreamingChannel@1304a6e4,
>>>> 5cf05913=org.apache.cassandra.streaming.async.NettyStreamingChannel@2ed1739f,
>>>> 833323b4=org.apache.cassandra.streaming.async.NettyStreamingChannel@1be38c68,
>>>> 1146e1c5=org.apache.cassandra.streaming.async.NettyStreamingChannel@232e3ca4,
>>>> f983d95c=org.apache.cassandra.streaming.async.NettyStreamingChannel@152b8da3,
>>>> 3f061317=org.apache.cassandra.streaming.async.NettyStreamingChannel@30434912,
>>>> 0c0b1395=org.apache.cassandra.streaming.async.NettyStreamingChannel@374403be,
>>>> 30dafd8c=org.apache.cassandra.streaming.async.NettyStreamingChannel@65ddc2ee,
>>>> 28c0fe3c=org.apache.cassandra.streaming.async.NettyStreamingChannel@2cd20d63,
>>>> 55d1221c=org.apache.cassandra.streaming.async.NettyStreamingChannel@6f3ca32,
>>>> 4fca9827=org.apache.cassandra.streaming.async.NettyStreamingChannel@113d70a1,
>>>> bea79e0c=org.apache.cassandra.streaming.async.NettyStreamingChannel@1afc0ada}
>>>> and outbound
>>>> {0c0b1395=org.apache.cassandra.streaming.async.NettyStreamingChannel@374403be}
>>>> channels
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-833323b4]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-50e7cefb]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-dfbbe8cd]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-67655215]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-55d116ad]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-1146e1c5]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-f983d95c]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-5cf05913]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-3f061317]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-0c0b1395]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-d4d397f3]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-28c0fe3c]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-3d52d53f]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-55d1221c]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-30dafd8c]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-4fca9827]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-bea79e0c]
>>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>>> completed with state COMPLETE
>>>> ERROR [main] 2024-08-14 20:34:10,387 StorageService.java:2041 -
>>>> Error while waiting on bootstrap to complete. Bootstrap will have
>>>> to be restarted.
>>>> java.util.concurrent.ExecutionException:
>>>> org.apache.cassandra.streaming.StreamException: Stream failed
>>>> at
>>>> org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
>>>> at
>>>> org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246)
>>>> at
>>>> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2034)
>>>> at
>>>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1185)
>>>> at
>>>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1145)
>>>> at
>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:936)
>>>> at
>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:854)
>>>> at
>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
>>>> at
>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
>>>> at
>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
>>>> Caused by: org.apache.cassandra.streaming.StreamException: Stream
>>>> failed
>>>> at
>>>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:243)
>>>> at
>>>> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
>>>> at
>>>> org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:517)
>>>> at
>>>> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
>>>> at
>>>> org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
>>>> at
>>>> org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
>>>> at
>>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>>> Source)
>>>> at java.base/java.util.concurrent.FutureTask.run(Unknown
>>>> Source)
>>>> at
>>>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>>> Source)
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>>>> Source)
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>>> Source)
>>>> at
>>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>>> at java.base/java.lang.Thread.run(Unknown Source)
>>>> DEBUG [Messaging-EventLoop-3-15] 2024-08-14 20:34:10,388
>>>> StreamingMultiplexedChannel.java:513 - [Stream
>>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Closing stream connection
>>>> channels on /192.168.189.127:7000
>>>> WARN [main] 2024-08-14 20:34:10,456 StorageService.java:1221 -
>>>> Some data streaming failed. Use nodetool to check bootstrap state
>>>> and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
>>>> INFO [main] 2024-08-14 20:34:10,458 Gossiper.java:2293 - Waiting
>>>> for gossip to settle...
>>>> DEBUG [main] 2024-08-14 20:34:16,458 Gossiper.java:2305 - Gossip
>>>> looks settled.
>>>>
>>>>
>>>> root@cassandra-15:/# nodetool status -r
>>>> Datacenter: dc1
>>>> ===============
>>>> Status=Up/Down
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>> -- Address Load Tokens Owns (effective) Host
>>>> ID Rack
>>>> UN cassandra-6.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>> TiB 256 20.0% 34a419fb-4db1-4fb4-8ea7-c0988b2a3d5a rack1
>>>> UN cassandra-3.cassandra.cassandra-jos.svc.cluster.local 2.08
>>>> TiB 256 20.0% 354a9e1f-71b4-4aba-9f03-26778aaa17e2 rack1
>>>> UN cassandra-14.cassandra.cassandra-jos.svc.cluster.local 1.95
>>>> TiB 256 20.0% 6e748875-1101-4bc1-a132-c6fe643c6148 rack1
>>>> UN cassandra-5.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>> TiB 256 20.0% ead07b83-a927-4904-bc6c-e8e4249f7560 rack1
>>>> UN cassandra-7.cassandra.cassandra-jos.svc.cluster.local 2.08
>>>> TiB 256 20.0% b5fbc09e-565e-402d-a6d0-3feac5922a13 rack1
>>>> UN cassandra-4.cassandra.cassandra-jos.svc.cluster.local 2.27
>>>> TiB 256 20.0% 14fa3f70-a8e2-4f60-8bc9-42c7b3294718 rack1
>>>> UN cassandra-1.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>> TiB 256 20.0% ea5979b8-71fe-4af5-b856-db06e008ced3 rack1
>>>> UN cassandra-11.cassandra.cassandra-jos.svc.cluster.local 2.08
>>>> TiB 256 20.0% f2bb43f7-8381-4b28-8ced-194060267c29 rack1
>>>> UN cassandra-9.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>> TiB 256 20.0% ed1ce133-4473-4bc1-a0cc-a242d6f67acc rack1
>>>> UN cassandra-2.cassandra.cassandra-jos.svc.cluster.local 2.08
>>>> TiB 256 20.0% 2b5db930-2bfe-4829-b353-3b1db0ca368f rack1
>>>> UN cassandra-12.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>> TiB 256 20.0% 1d3ed71f-18b4-4cc0-9c6e-3a5479328496 rack1
>>>> UJ cassandra-15.cassandra.cassandra-jos.svc.cluster.local 874.82
>>>> GiB 256 ? 9d68cd5a-6e48-4156-92c1-5c51f02bce65 rack1
>>>> UN cassandra-0.cassandra.cassandra-jos.svc.cluster.local 2.14
>>>> TiB 256 20.0% 5c3d3b46-300d-49f2-b01d-cbdb44d98022 rack1
>>>> UN cassandra-8.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>> TiB 256 20.0% fb2e8220-549e-4316-bb78-53f6cd07318a rack1
>>>> UN cassandra-10.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>> TiB 256 20.0% a2ab4d22-b564-4d6b-b743-2d78598fe53c rack1
>>>> UN cassandra-13.cassandra.cassandra-jos.svc.cluster.local 2.07
>>>> TiB 256 20.0% ed865baf-5333-40e5-8c73-9de2df2bd330 rack1
>>>>
>>>>
>>>> If I restart the bootstrap, it usually completes, but I'd like to
>>>> double the size of the cluster, and that's a very long operation.
>>>> Is there anyway to add multiple nodes at once?
>>>> Thanks!
>>>>
>>>> -Joe
>>>>
>>>>
>>
--
This email has been checked for viruses by AVG antivirus software.
www.avg.com
Re: Bootstrap error - Cassandra 4.1.5
Posted by Bowen Song via user <us...@cassandra.apache.org>.
You may need to look at the zipped log files if the streaming had been
running for a while before failing. The error could have happened hours
or days before the final failure.
If your cluster is already experiencing performance issues (e.g. due to
CPU bottleneck or GC pauses), it's highly likely that these are related
to the streaming failure.
On 15/08/2024 13:41, Joe Obernberger wrote:
> Thank you Bowen - yeah the only ERROR I see in
> /var/log/cassandra/debug.log is:
>
> ERROR [main] 2024-08-15 04:48:23,374 StorageService.java:2041 - Error
> while waiting on bootstrap to complete. Bootstrap will have to be
> restarted.
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
> at
> org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246)
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2034)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1185)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1145)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:936)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:854)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:243)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
> at
> org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:517)
> at
> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at
> org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at
> org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
>
> The cluster is running inside of kubernetes on bare metal with a
> netapp for storage. I'd love a way to double the number of nodes, but
> sounds like I shouldn't have let it get this far. We're having some
> odd performance issues on reads, that I'm diagnosing.
>
> -Joe
>
> On 8/14/2024 5:07 PM, Bowen Song via user wrote:
>> It looks like all your nodes are in the same DC and the same rack
>> with 256 vnodes each. It's very hard (if not impossible) to add
>> multiple nodes to the same DC concurrently and safely in this setup.
>> You are better off adding one node at a time to this cluster.
>>
>> Try search for "ERROR" in the logs, it should tell you why did the
>> streaming session fail. If you can find the cause of the failure, you
>> may be able to prevent or reduce the chance of it happening again in
>> the future.
>>
>> On 14/08/2024 21:50, Joe Obernberger wrote:
>>> Hi all - when adding a node to our existing 15 node cluster, I get:
>>>
>>>
>>> DEBUG [NonPeriodicTasks:1] 2024-08-14 20:34:10,383
>>> StreamCoordinator.java:152 - Finished connecting all sessions
>>> WARN [NonPeriodicTasks:1] 2024-08-14 20:34:10,385
>>> StreamResultFuture.java:242 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Stream failed
>>> DEBUG [NonPeriodicTasks:1] 2024-08-14 20:34:10,386
>>> StreamSession.java:529 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Will close attached inbound
>>> {3d52d53f=org.apache.cassandra.streaming.async.NettyStreamingChannel@2e6b7635,
>>> 67655215=org.apache.cassandra.streaming.async.NettyStreamingChannel@11e1a7e5,
>>> d4d397f3=org.apache.cassandra.streaming.async.NettyStreamingChannel@11959fa4,
>>> 50e7cefb=org.apache.cassandra.streaming.async.NettyStreamingChannel@7a69edc0,
>>> dfbbe8cd=org.apache.cassandra.streaming.async.NettyStreamingChannel@f46e666,
>>> 55d116ad=org.apache.cassandra.streaming.async.NettyStreamingChannel@1304a6e4,
>>> 5cf05913=org.apache.cassandra.streaming.async.NettyStreamingChannel@2ed1739f,
>>> 833323b4=org.apache.cassandra.streaming.async.NettyStreamingChannel@1be38c68,
>>> 1146e1c5=org.apache.cassandra.streaming.async.NettyStreamingChannel@232e3ca4,
>>> f983d95c=org.apache.cassandra.streaming.async.NettyStreamingChannel@152b8da3,
>>> 3f061317=org.apache.cassandra.streaming.async.NettyStreamingChannel@30434912,
>>> 0c0b1395=org.apache.cassandra.streaming.async.NettyStreamingChannel@374403be,
>>> 30dafd8c=org.apache.cassandra.streaming.async.NettyStreamingChannel@65ddc2ee,
>>> 28c0fe3c=org.apache.cassandra.streaming.async.NettyStreamingChannel@2cd20d63,
>>> 55d1221c=org.apache.cassandra.streaming.async.NettyStreamingChannel@6f3ca32,
>>> 4fca9827=org.apache.cassandra.streaming.async.NettyStreamingChannel@113d70a1,
>>> bea79e0c=org.apache.cassandra.streaming.async.NettyStreamingChannel@1afc0ada}
>>> and outbound
>>> {0c0b1395=org.apache.cassandra.streaming.async.NettyStreamingChannel@374403be}
>>> channels
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-833323b4]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-50e7cefb]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-dfbbe8cd]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-67655215]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-55d116ad]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-1146e1c5]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-f983d95c]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-5cf05913]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-3f061317]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-0c0b1395]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-d4d397f3]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-28c0fe3c]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-3d52d53f]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-55d1221c]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-30dafd8c]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-4fca9827]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-bea79e0c]
>>> 2024-08-14 20:34:10,387 StreamSession.java:677 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>>> completed with state COMPLETE
>>> ERROR [main] 2024-08-14 20:34:10,387 StorageService.java:2041 -
>>> Error while waiting on bootstrap to complete. Bootstrap will have to
>>> be restarted.
>>> java.util.concurrent.ExecutionException:
>>> org.apache.cassandra.streaming.StreamException: Stream failed
>>> at
>>> org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
>>> at
>>> org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246)
>>> at
>>> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2034)
>>> at
>>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1185)
>>> at
>>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1145)
>>> at
>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:936)
>>> at
>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:854)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
>>> Caused by: org.apache.cassandra.streaming.StreamException: Stream
>>> failed
>>> at
>>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:243)
>>> at
>>> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
>>> at
>>> org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:517)
>>> at
>>> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
>>> at
>>> org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
>>> at
>>> org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
>>> at
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>> Source)
>>> at java.base/java.util.concurrent.FutureTask.run(Unknown
>>> Source)
>>> at
>>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>> Source)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>>> Source)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>> Source)
>>> at
>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>> at java.base/java.lang.Thread.run(Unknown Source)
>>> DEBUG [Messaging-EventLoop-3-15] 2024-08-14 20:34:10,388
>>> StreamingMultiplexedChannel.java:513 - [Stream
>>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Closing stream connection
>>> channels on /192.168.189.127:7000
>>> WARN [main] 2024-08-14 20:34:10,456 StorageService.java:1221 - Some
>>> data streaming failed. Use nodetool to check bootstrap state and
>>> resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
>>> INFO [main] 2024-08-14 20:34:10,458 Gossiper.java:2293 - Waiting
>>> for gossip to settle...
>>> DEBUG [main] 2024-08-14 20:34:16,458 Gossiper.java:2305 - Gossip
>>> looks settled.
>>>
>>>
>>> root@cassandra-15:/# nodetool status -r
>>> Datacenter: dc1
>>> ===============
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> -- Address Load Tokens Owns (effective) Host
>>> ID Rack
>>> UN cassandra-6.cassandra.cassandra-jos.svc.cluster.local 2.07
>>> TiB 256 20.0% 34a419fb-4db1-4fb4-8ea7-c0988b2a3d5a rack1
>>> UN cassandra-3.cassandra.cassandra-jos.svc.cluster.local 2.08
>>> TiB 256 20.0% 354a9e1f-71b4-4aba-9f03-26778aaa17e2 rack1
>>> UN cassandra-14.cassandra.cassandra-jos.svc.cluster.local 1.95
>>> TiB 256 20.0% 6e748875-1101-4bc1-a132-c6fe643c6148 rack1
>>> UN cassandra-5.cassandra.cassandra-jos.svc.cluster.local 2.07
>>> TiB 256 20.0% ead07b83-a927-4904-bc6c-e8e4249f7560 rack1
>>> UN cassandra-7.cassandra.cassandra-jos.svc.cluster.local 2.08
>>> TiB 256 20.0% b5fbc09e-565e-402d-a6d0-3feac5922a13 rack1
>>> UN cassandra-4.cassandra.cassandra-jos.svc.cluster.local 2.27
>>> TiB 256 20.0% 14fa3f70-a8e2-4f60-8bc9-42c7b3294718 rack1
>>> UN cassandra-1.cassandra.cassandra-jos.svc.cluster.local 2.07
>>> TiB 256 20.0% ea5979b8-71fe-4af5-b856-db06e008ced3 rack1
>>> UN cassandra-11.cassandra.cassandra-jos.svc.cluster.local 2.08
>>> TiB 256 20.0% f2bb43f7-8381-4b28-8ced-194060267c29 rack1
>>> UN cassandra-9.cassandra.cassandra-jos.svc.cluster.local 2.07
>>> TiB 256 20.0% ed1ce133-4473-4bc1-a0cc-a242d6f67acc rack1
>>> UN cassandra-2.cassandra.cassandra-jos.svc.cluster.local 2.08
>>> TiB 256 20.0% 2b5db930-2bfe-4829-b353-3b1db0ca368f rack1
>>> UN cassandra-12.cassandra.cassandra-jos.svc.cluster.local 2.07
>>> TiB 256 20.0% 1d3ed71f-18b4-4cc0-9c6e-3a5479328496 rack1
>>> UJ cassandra-15.cassandra.cassandra-jos.svc.cluster.local 874.82
>>> GiB 256 ? 9d68cd5a-6e48-4156-92c1-5c51f02bce65 rack1
>>> UN cassandra-0.cassandra.cassandra-jos.svc.cluster.local 2.14
>>> TiB 256 20.0% 5c3d3b46-300d-49f2-b01d-cbdb44d98022 rack1
>>> UN cassandra-8.cassandra.cassandra-jos.svc.cluster.local 2.07
>>> TiB 256 20.0% fb2e8220-549e-4316-bb78-53f6cd07318a rack1
>>> UN cassandra-10.cassandra.cassandra-jos.svc.cluster.local 2.07
>>> TiB 256 20.0% a2ab4d22-b564-4d6b-b743-2d78598fe53c rack1
>>> UN cassandra-13.cassandra.cassandra-jos.svc.cluster.local 2.07
>>> TiB 256 20.0% ed865baf-5333-40e5-8c73-9de2df2bd330 rack1
>>>
>>>
>>> If I restart the bootstrap, it usually completes, but I'd like to
>>> double the size of the cluster, and that's a very long operation. Is
>>> there anyway to add multiple nodes at once?
>>> Thanks!
>>>
>>> -Joe
>>>
>>>
>
Re: Bootstrap error - Cassandra 4.1.5
Posted by Joe Obernberger <jo...@gmail.com>.
Thank you Bowen - yeah the only ERROR I see in
/var/log/cassandra/debug.log is:
ERROR [main] 2024-08-15 04:48:23,374 StorageService.java:2041 - Error
while waiting on bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
at
org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2034)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1185)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1145)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:936)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:854)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:243)
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
at
org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:517)
at
org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
at
org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at
org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
The cluster is running inside of kubernetes on bare metal with a netapp
for storage. I'd love a way to double the number of nodes, but sounds
like I shouldn't have let it get this far. We're having some odd
performance issues on reads, that I'm diagnosing.
-Joe
On 8/14/2024 5:07 PM, Bowen Song via user wrote:
> It looks like all your nodes are in the same DC and the same rack with
> 256 vnodes each. It's very hard (if not impossible) to add multiple
> nodes to the same DC concurrently and safely in this setup. You are
> better off adding one node at a time to this cluster.
>
> Try search for "ERROR" in the logs, it should tell you why did the
> streaming session fail. If you can find the cause of the failure, you
> may be able to prevent or reduce the chance of it happening again in
> the future.
>
> On 14/08/2024 21:50, Joe Obernberger wrote:
>> Hi all - when adding a node to our existing 15 node cluster, I get:
>>
>>
>> DEBUG [NonPeriodicTasks:1] 2024-08-14 20:34:10,383
>> StreamCoordinator.java:152 - Finished connecting all sessions
>> WARN [NonPeriodicTasks:1] 2024-08-14 20:34:10,385
>> StreamResultFuture.java:242 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Stream failed
>> DEBUG [NonPeriodicTasks:1] 2024-08-14 20:34:10,386
>> StreamSession.java:529 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Will close attached inbound
>> {3d52d53f=org.apache.cassandra.streaming.async.NettyStreamingChannel@2e6b7635,
>> 67655215=org.apache.cassandra.streaming.async.NettyStreamingChannel@11e1a7e5,
>> d4d397f3=org.apache.cassandra.streaming.async.NettyStreamingChannel@11959fa4,
>> 50e7cefb=org.apache.cassandra.streaming.async.NettyStreamingChannel@7a69edc0,
>> dfbbe8cd=org.apache.cassandra.streaming.async.NettyStreamingChannel@f46e666,
>> 55d116ad=org.apache.cassandra.streaming.async.NettyStreamingChannel@1304a6e4,
>> 5cf05913=org.apache.cassandra.streaming.async.NettyStreamingChannel@2ed1739f,
>> 833323b4=org.apache.cassandra.streaming.async.NettyStreamingChannel@1be38c68,
>> 1146e1c5=org.apache.cassandra.streaming.async.NettyStreamingChannel@232e3ca4,
>> f983d95c=org.apache.cassandra.streaming.async.NettyStreamingChannel@152b8da3,
>> 3f061317=org.apache.cassandra.streaming.async.NettyStreamingChannel@30434912,
>> 0c0b1395=org.apache.cassandra.streaming.async.NettyStreamingChannel@374403be,
>> 30dafd8c=org.apache.cassandra.streaming.async.NettyStreamingChannel@65ddc2ee,
>> 28c0fe3c=org.apache.cassandra.streaming.async.NettyStreamingChannel@2cd20d63,
>> 55d1221c=org.apache.cassandra.streaming.async.NettyStreamingChannel@6f3ca32,
>> 4fca9827=org.apache.cassandra.streaming.async.NettyStreamingChannel@113d70a1,
>> bea79e0c=org.apache.cassandra.streaming.async.NettyStreamingChannel@1afc0ada}
>> and outbound
>> {0c0b1395=org.apache.cassandra.streaming.async.NettyStreamingChannel@374403be}
>> channels
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-833323b4] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-50e7cefb] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-dfbbe8cd] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-67655215] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-55d116ad] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-1146e1c5] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-f983d95c] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-5cf05913] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-3f061317] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-0c0b1395] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-d4d397f3] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-28c0fe3c] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-3d52d53f] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-55d1221c] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-30dafd8c] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-4fca9827] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> DEBUG [Stream-Deserializer-/192.168.189.127:7000-bea79e0c] 2024-08-14
>> 20:34:10,387 StreamSession.java:677 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
>> completed with state COMPLETE
>> ERROR [main] 2024-08-14 20:34:10,387 StorageService.java:2041 - Error
>> while waiting on bootstrap to complete. Bootstrap will have to be
>> restarted.
>> java.util.concurrent.ExecutionException:
>> org.apache.cassandra.streaming.StreamException: Stream failed
>> at
>> org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
>> at
>> org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246)
>> at
>> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2034)
>> at
>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1185)
>> at
>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1145)
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:936)
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:854)
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
>> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>> at
>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:243)
>> at
>> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
>> at
>> org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:517)
>> at
>> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
>> at
>> org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
>> at
>> org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
>> at
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>> Source)
>> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
>> at
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>> Source)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>> Source)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>> Source)
>> at
>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>> at java.base/java.lang.Thread.run(Unknown Source)
>> DEBUG [Messaging-EventLoop-3-15] 2024-08-14 20:34:10,388
>> StreamingMultiplexedChannel.java:513 - [Stream
>> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Closing stream connection
>> channels on /192.168.189.127:7000
>> WARN [main] 2024-08-14 20:34:10,456 StorageService.java:1221 - Some
>> data streaming failed. Use nodetool to check bootstrap state and
>> resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
>> INFO [main] 2024-08-14 20:34:10,458 Gossiper.java:2293 - Waiting for
>> gossip to settle...
>> DEBUG [main] 2024-08-14 20:34:16,458 Gossiper.java:2305 - Gossip
>> looks settled.
>>
>>
>> root@cassandra-15:/# nodetool status -r
>> Datacenter: dc1
>> ===============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host
>> ID Rack
>> UN cassandra-6.cassandra.cassandra-jos.svc.cluster.local 2.07
>> TiB 256 20.0% 34a419fb-4db1-4fb4-8ea7-c0988b2a3d5a rack1
>> UN cassandra-3.cassandra.cassandra-jos.svc.cluster.local 2.08
>> TiB 256 20.0% 354a9e1f-71b4-4aba-9f03-26778aaa17e2 rack1
>> UN cassandra-14.cassandra.cassandra-jos.svc.cluster.local 1.95
>> TiB 256 20.0% 6e748875-1101-4bc1-a132-c6fe643c6148 rack1
>> UN cassandra-5.cassandra.cassandra-jos.svc.cluster.local 2.07
>> TiB 256 20.0% ead07b83-a927-4904-bc6c-e8e4249f7560 rack1
>> UN cassandra-7.cassandra.cassandra-jos.svc.cluster.local 2.08
>> TiB 256 20.0% b5fbc09e-565e-402d-a6d0-3feac5922a13 rack1
>> UN cassandra-4.cassandra.cassandra-jos.svc.cluster.local 2.27
>> TiB 256 20.0% 14fa3f70-a8e2-4f60-8bc9-42c7b3294718 rack1
>> UN cassandra-1.cassandra.cassandra-jos.svc.cluster.local 2.07
>> TiB 256 20.0% ea5979b8-71fe-4af5-b856-db06e008ced3 rack1
>> UN cassandra-11.cassandra.cassandra-jos.svc.cluster.local 2.08
>> TiB 256 20.0% f2bb43f7-8381-4b28-8ced-194060267c29 rack1
>> UN cassandra-9.cassandra.cassandra-jos.svc.cluster.local 2.07
>> TiB 256 20.0% ed1ce133-4473-4bc1-a0cc-a242d6f67acc rack1
>> UN cassandra-2.cassandra.cassandra-jos.svc.cluster.local 2.08
>> TiB 256 20.0% 2b5db930-2bfe-4829-b353-3b1db0ca368f rack1
>> UN cassandra-12.cassandra.cassandra-jos.svc.cluster.local 2.07
>> TiB 256 20.0% 1d3ed71f-18b4-4cc0-9c6e-3a5479328496 rack1
>> UJ cassandra-15.cassandra.cassandra-jos.svc.cluster.local 874.82
>> GiB 256 ? 9d68cd5a-6e48-4156-92c1-5c51f02bce65 rack1
>> UN cassandra-0.cassandra.cassandra-jos.svc.cluster.local 2.14
>> TiB 256 20.0% 5c3d3b46-300d-49f2-b01d-cbdb44d98022 rack1
>> UN cassandra-8.cassandra.cassandra-jos.svc.cluster.local 2.07
>> TiB 256 20.0% fb2e8220-549e-4316-bb78-53f6cd07318a rack1
>> UN cassandra-10.cassandra.cassandra-jos.svc.cluster.local 2.07
>> TiB 256 20.0% a2ab4d22-b564-4d6b-b743-2d78598fe53c rack1
>> UN cassandra-13.cassandra.cassandra-jos.svc.cluster.local 2.07
>> TiB 256 20.0% ed865baf-5333-40e5-8c73-9de2df2bd330 rack1
>>
>>
>> If I restart the bootstrap, it usually completes, but I'd like to
>> double the size of the cluster, and that's a very long operation. Is
>> there anyway to add multiple nodes at once?
>> Thanks!
>>
>> -Joe
>>
>>
--
This email has been checked for viruses by AVG antivirus software.
www.avg.com
Re: Bootstrap error - Cassandra 4.1.5
Posted by Bowen Song via user <us...@cassandra.apache.org>.
It looks like all your nodes are in the same DC and the same rack with
256 vnodes each. It's very hard (if not impossible) to add multiple
nodes to the same DC concurrently and safely in this setup. You are
better off adding one node at a time to this cluster.
Try search for "ERROR" in the logs, it should tell you why did the
streaming session fail. If you can find the cause of the failure, you
may be able to prevent or reduce the chance of it happening again in the
future.
On 14/08/2024 21:50, Joe Obernberger wrote:
> Hi all - when adding a node to our existing 15 node cluster, I get:
>
>
> DEBUG [NonPeriodicTasks:1] 2024-08-14 20:34:10,383
> StreamCoordinator.java:152 - Finished connecting all sessions
> WARN [NonPeriodicTasks:1] 2024-08-14 20:34:10,385
> StreamResultFuture.java:242 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Stream failed
> DEBUG [NonPeriodicTasks:1] 2024-08-14 20:34:10,386
> StreamSession.java:529 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Will close attached inbound
> {3d52d53f=org.apache.cassandra.streaming.async.NettyStreamingChannel@2e6b7635,
> 67655215=org.apache.cassandra.streaming.async.NettyStreamingChannel@11e1a7e5,
> d4d397f3=org.apache.cassandra.streaming.async.NettyStreamingChannel@11959fa4,
> 50e7cefb=org.apache.cassandra.streaming.async.NettyStreamingChannel@7a69edc0,
> dfbbe8cd=org.apache.cassandra.streaming.async.NettyStreamingChannel@f46e666,
> 55d116ad=org.apache.cassandra.streaming.async.NettyStreamingChannel@1304a6e4,
> 5cf05913=org.apache.cassandra.streaming.async.NettyStreamingChannel@2ed1739f,
> 833323b4=org.apache.cassandra.streaming.async.NettyStreamingChannel@1be38c68,
> 1146e1c5=org.apache.cassandra.streaming.async.NettyStreamingChannel@232e3ca4,
> f983d95c=org.apache.cassandra.streaming.async.NettyStreamingChannel@152b8da3,
> 3f061317=org.apache.cassandra.streaming.async.NettyStreamingChannel@30434912,
> 0c0b1395=org.apache.cassandra.streaming.async.NettyStreamingChannel@374403be,
> 30dafd8c=org.apache.cassandra.streaming.async.NettyStreamingChannel@65ddc2ee,
> 28c0fe3c=org.apache.cassandra.streaming.async.NettyStreamingChannel@2cd20d63,
> 55d1221c=org.apache.cassandra.streaming.async.NettyStreamingChannel@6f3ca32,
> 4fca9827=org.apache.cassandra.streaming.async.NettyStreamingChannel@113d70a1,
> bea79e0c=org.apache.cassandra.streaming.async.NettyStreamingChannel@1afc0ada}
> and outbound
> {0c0b1395=org.apache.cassandra.streaming.async.NettyStreamingChannel@374403be}
> channels
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-833323b4] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-50e7cefb] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-dfbbe8cd] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-67655215] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-55d116ad] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-1146e1c5] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-f983d95c] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-5cf05913] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-3f061317] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-0c0b1395] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-d4d397f3] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-28c0fe3c] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-3d52d53f] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-55d1221c] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-30dafd8c] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-4fca9827] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> DEBUG [Stream-Deserializer-/192.168.189.127:7000-bea79e0c] 2024-08-14
> 20:34:10,387 StreamSession.java:677 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Socket closed after session
> completed with state COMPLETE
> ERROR [main] 2024-08-14 20:34:10,387 StorageService.java:2041 - Error
> while waiting on bootstrap to complete. Bootstrap will have to be
> restarted.
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
> at
> org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246)
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2034)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1185)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1145)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:936)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:854)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:243)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
> at
> org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:517)
> at
> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at
> org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at
> org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
> DEBUG [Messaging-EventLoop-3-15] 2024-08-14 20:34:10,388
> StreamingMultiplexedChannel.java:513 - [Stream
> #d7bf9f60-5a5e-11ef-aa71-51cb94e3c01f] Closing stream connection
> channels on /192.168.189.127:7000
> WARN [main] 2024-08-14 20:34:10,456 StorageService.java:1221 - Some
> data streaming failed. Use nodetool to check bootstrap state and
> resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
> INFO [main] 2024-08-14 20:34:10,458 Gossiper.java:2293 - Waiting for
> gossip to settle...
> DEBUG [main] 2024-08-14 20:34:16,458 Gossiper.java:2305 - Gossip looks
> settled.
>
>
> root@cassandra-15:/# nodetool status -r
> Datacenter: dc1
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host
> ID Rack
> UN cassandra-6.cassandra.cassandra-jos.svc.cluster.local 2.07
> TiB 256 20.0% 34a419fb-4db1-4fb4-8ea7-c0988b2a3d5a rack1
> UN cassandra-3.cassandra.cassandra-jos.svc.cluster.local 2.08
> TiB 256 20.0% 354a9e1f-71b4-4aba-9f03-26778aaa17e2 rack1
> UN cassandra-14.cassandra.cassandra-jos.svc.cluster.local 1.95
> TiB 256 20.0% 6e748875-1101-4bc1-a132-c6fe643c6148 rack1
> UN cassandra-5.cassandra.cassandra-jos.svc.cluster.local 2.07
> TiB 256 20.0% ead07b83-a927-4904-bc6c-e8e4249f7560 rack1
> UN cassandra-7.cassandra.cassandra-jos.svc.cluster.local 2.08
> TiB 256 20.0% b5fbc09e-565e-402d-a6d0-3feac5922a13 rack1
> UN cassandra-4.cassandra.cassandra-jos.svc.cluster.local 2.27
> TiB 256 20.0% 14fa3f70-a8e2-4f60-8bc9-42c7b3294718 rack1
> UN cassandra-1.cassandra.cassandra-jos.svc.cluster.local 2.07
> TiB 256 20.0% ea5979b8-71fe-4af5-b856-db06e008ced3 rack1
> UN cassandra-11.cassandra.cassandra-jos.svc.cluster.local 2.08
> TiB 256 20.0% f2bb43f7-8381-4b28-8ced-194060267c29 rack1
> UN cassandra-9.cassandra.cassandra-jos.svc.cluster.local 2.07
> TiB 256 20.0% ed1ce133-4473-4bc1-a0cc-a242d6f67acc rack1
> UN cassandra-2.cassandra.cassandra-jos.svc.cluster.local 2.08
> TiB 256 20.0% 2b5db930-2bfe-4829-b353-3b1db0ca368f rack1
> UN cassandra-12.cassandra.cassandra-jos.svc.cluster.local 2.07
> TiB 256 20.0% 1d3ed71f-18b4-4cc0-9c6e-3a5479328496 rack1
> UJ cassandra-15.cassandra.cassandra-jos.svc.cluster.local 874.82
> GiB 256 ? 9d68cd5a-6e48-4156-92c1-5c51f02bce65 rack1
> UN cassandra-0.cassandra.cassandra-jos.svc.cluster.local 2.14
> TiB 256 20.0% 5c3d3b46-300d-49f2-b01d-cbdb44d98022 rack1
> UN cassandra-8.cassandra.cassandra-jos.svc.cluster.local 2.07
> TiB 256 20.0% fb2e8220-549e-4316-bb78-53f6cd07318a rack1
> UN cassandra-10.cassandra.cassandra-jos.svc.cluster.local 2.07
> TiB 256 20.0% a2ab4d22-b564-4d6b-b743-2d78598fe53c rack1
> UN cassandra-13.cassandra.cassandra-jos.svc.cluster.local 2.07
> TiB 256 20.0% ed865baf-5333-40e5-8c73-9de2df2bd330 rack1
>
>
> If I restart the bootstrap, it usually completes, but I'd like to
> double the size of the cluster, and that's a very long operation. Is
> there anyway to add multiple nodes at once?
> Thanks!
>
> -Joe
>
>