You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yongle Zhang (Jira)" <ji...@apache.org> on 2020/11/10 01:12:00 UTC
[jira] [Created] (CASSANDRA-16258) Running read on a mixed cluster
with C* version 3.11.6 and version 4.0-alpha1 failed with multiple
exceptions
Yongle Zhang created CASSANDRA-16258:
----------------------------------------
Summary: Running read on a mixed cluster with C* version 3.11.6 and version 4.0-alpha1 failed with multiple exceptions
Key: CASSANDRA-16258
URL: https://issues.apache.org/jira/browse/CASSANDRA-16258
Project: Cassandra
Issue Type: Bug
Reporter: Yongle Zhang
Steps to reproduce:
# Set up a cluster with 2 nodes (1 seed using 3.11.6, the other node using 4.0-alpha1).
# Run stress testing on the seed node running C* 3.11.6.
Error messages on each node:
Seed (3.11.6):
{code:java}
ERROR [RequestResponseStage-1] 2020-06-25 01:58:56,745 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[RequestResponseStage-1,5,main]
java.lang.IllegalArgumentException: Unknown request failure reason error code: 3
at org.apache.cassandra.exceptions.RequestFailureReason.fromCode(RequestFailureReason.java:49) ~[main/:na]
at org.apache.cassandra.net.MessageIn.getFailureReason(MessageIn.java:189) ~[main/:na]
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:47) ~[main/:na]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) ~[main/:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_222]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165) ~[main/:na]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137) [main/:na]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:113) [main/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222]{code}
Looks like this error message might be because the enum class RequestFailureReason is incompatible between 3.11.6 and 4.0-alpha1: In 3.11.6 it contains only 2 possible values ([https://github.com/apache/cassandra/blob/cassandra-3.11.6/src/java/org/apache/cassandra/exceptions/RequestFailureReason.java#L21]), while in 4.0-alpha1 it contains 4 possible values ([https://github.com/apache/cassandra/blob/cassandra-4.0-alpha1/src/java/org/apache/cassandra/exceptions/RequestFailureReason.java#L33]).
Non-seed node (4.0-alpha1):
{code:java}
INFO [Messaging-EventLoop-3-5] 2020-06-25 01:58:56,743 NoSpamLogger.java:91 - 250.16.238.1:7000->250.16.238.2:7000-LEGACY_MESSAGES-0e4d7ae2 incompatible schema encountered while deserializing a message
org.apache.cassandra.exceptions.UnknownTableException: Couldn't find table with id 5bc52802-de25-35ed-aeab-188eecebb090. If a table was just created, this is likely due to the schemanot being fully propagated. Please wait for schema agreement on table creation.
at org.apache.cassandra.schema.Schema.getExistingTableMetadata(Schema.java:456)
at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:614)
at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:344)
at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:362)
at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:326)
at org.apache.cassandra.net.Message$Serializer.deserializePayloadPre40(Message.java:949)
at org.apache.cassandra.net.Message$Serializer.deserializePre40(Message.java:930)
at org.apache.cassandra.net.Message$Serializer.deserializePre40(Message.java:918)
at org.apache.cassandra.net.Message$Serializer.deserialize(Message.java:618)
at org.apache.cassandra.net.InboundMessageHandler.processSmallMessage(InboundMessageHandler.java:321)
at org.apache.cassandra.net.InboundMessageHandler.processOneContainedMessage(InboundMessageHandler.java:304)
at org.apache.cassandra.net.InboundMessageHandler.processFrameOfContainedMessages(InboundMessageHandler.java:271)
at org.apache.cassandra.net.InboundMessageHandler.processIntactFrame(InboundMessageHandler.java:256)
at org.apache.cassandra.net.InboundMessageHandler.process(InboundMessageHandler.java:247)
at org.apache.cassandra.net.FrameDecoder.deliver(FrameDecoder.java:320)
at org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:284)
at org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:268)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748){code}
Not exactly sure why how is this UnknownTableException triggered.
Client:
{code:java}
WARN 01:59:41,160 No schema agreement from live replicas after 10 s. The schema may not be up to date on some nodes.
com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace 'stresscql' does not exist
at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at org.apache.cassandra.stress.util.JavaDriverClient.execute(JavaDriverClient.java:189)
at org.apache.cassandra.stress.StressProfile.maybeCreateSchema(StressProfile.java:238)
at org.apache.cassandra.stress.StressProfile.newGenerator(StressProfile.java:681)
at org.apache.cassandra.stress.StressProfile.printSettings(StressProfile.java:129)
at org.apache.cassandra.stress.settings.StressSettings.printSettings(StressSettings.java:383)
at org.apache.cassandra.stress.Stress.run(Stress.java:95)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace 'stresscql' does not exist
at com.datastax.driver.core.Responses$Error.asException(Responses.java:136)
at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179)
at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:177)
at com.datastax.driver.core.RequestHandler.access$2500(RequestHandler.java:43)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:792)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:611)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1013)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:936)
at com.datastax.shaded.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
at com.datastax.shaded.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
at com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
at com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)
at com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)
at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
at com.datastax.shaded.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
at com.datastax.shaded.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org