Error in bootstrapping node

[Jean Carlo <je...@gmail.com>]

Hello guys,

I got a node in joining status for almost more than 12 hours even if the
streaming seems to be over.

Checking the logs I found this error

ERROR [main] 2021-12-15 19:39:43,886  InsightsModule.java:166 - Unable to
initialize insights client due to error
org.apache.cassandra.exceptions.IsBootstrappingException: Cannot read from
a bootstrapping node
        at
org.apache.cassandra.service.StorageProxy.checkNotBootstrappingOrSystemQuery(StorageProxy.java:1295)
        at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1306)
        at
org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1239)
        at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:371)
        at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:579)
        at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:312)
        at
com.datastax.bdp.util.QueryProcessorUtil.processPreparedSelect(QueryProcessorUtil.java:96)
        at
com.datastax.bdp.util.QueryProcessorUtil.processPreparedSelect(QueryProcessorUtil.java:84)
        at
com.datastax.bdp.insights.storage.config.DseLocalInsightsRuntimeConfigAccess.selectConfigValue(DseLocalInsightsRuntimeConfigAccess.java:239)
        at
com.datastax.bdp.insights.storage.config.InsightsRuntimeConfigManager.populateCachedConfigWithLocalConfig(InsightsRuntimeConfigManager.java:97)
        at
com.datastax.bdp.insights.storage.config.InsightsRuntimeConfigManager.getRuntimeConfig(InsightsRuntimeConfigManager.java:328)
        at
com.datastax.bdp.insights.InsightsModule.insightsClient(InsightsModule.java:154)

Before I got some errors in the streaming

ERROR [STREAM-IN-/10.17.74.94:7000] 2021-12-15 10:10:58,181
 StreamSession.java:650 - [Stream #840346f0-5daf-11ec-a92c-2728dd47bba2]
Streaming error occurred on session with peer x.x.x.x
java.lang.NullPointerException: null
        at
org.apache.cassandra.utils.ByteSource.optionalSignedFixedLengthNumber(ByteSource.java:111)
        at
org.apache.cassandra.db.marshal.LongType.asByteComparableSource(LongType.java:53)
        at
org.apache.cassandra.db.ClusteringComparator$ClusteringSource.next(ClusteringComparator.java:284)
        at
org.apache.cassandra.utils.ByteSource$Separator.next(ByteSource.java:572)
        at
org.apache.cassandra.io.tries.IncrementalTrieWriterBase.add(IncrementalTrieWriterBase.java:72)
        at
org.apache.cassandra.io.sstable.format.trieindex.RowIndexWriter.add(RowIndexWriter.java:73)
        at
org.apache.cassandra.io.sstable.format.trieindex.PartitionWriter.addIndexBlock(PartitionWriter.java:139)
        at
org.apache.cassandra.io.sstable.format.trieindex.PartitionWriter.finish(PartitionWriter.java:190)
        at
org.apache.cassandra.io.sstable.format.trieindex.PartitionWriter.writePartition(PartitionWriter.java:107)
        at
org.apache.cassandra.io.sstable.format.trieindex.TrieIndexSSTableWriter.append(TrieIndexSSTableWriter.java:218)
        at
org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:47)
        at
org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:105)
        at
org.apache.cassandra.streaming.StreamReader.writePartition(StreamReader.java:168)
        at
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:102)
        at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:52)
        at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:41)
        at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:97)
        at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318)
        at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

WARN  [Thread-1400] 2021-12-15 10:10:58,220  CompressedInputStream.java:232
- Error while reading compressed input stream.
java.nio.channels.ClosedChannelException: null
        at
sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:254)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:297)
        at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:219)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
        at
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
        at
org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:221)
        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

ERROR [main] 2021-12-15 19:39:40,756  StorageService.java:1889 - Error
while waiting on bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed
        at
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
        at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
        at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1882)
        at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1115)
        at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:689)
        at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:613)
        at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:402)
        at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:527)
        at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:702)

After using nodetool bootstrap resume, I got the same error with

INFO  [OptionalTasks:1] 2021-12-16 04:41:34,078
 CassandraRoleManager.java:432 - Setup task failed with error, rescheduling
org.apache.cassandra.exceptions.IsBootstrappingException: Cannot read from
a bootstrapping node

Thanks


Jean Carlo

"The best way to predict the future is to invent it" Alan Kay

Re: Error in bootstrapping node

[Erick Ramirez <er...@datastax.com>]

The error you're seeing is specific to DSE so your best course of action is
to log a ticket with DataStax Support (https://support.datastax.com).
Cheers!