You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yongle Zhang (Jira)" <ji...@apache.org> on 2020/11/10 01:13:00 UTC
[jira] [Updated] (CASSANDRA-16258) Running read on a mixed cluster
with C* version 3.11.6 and version 4.0-alpha1 failed with multiple
exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-16258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yongle Zhang updated CASSANDRA-16258:
-------------------------------------
Severity: Normal
> Running read on a mixed cluster with C* version 3.11.6 and version 4.0-alpha1 failed with multiple exceptions
> -------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16258
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16258
> Project: Cassandra
> Issue Type: Bug
> Reporter: Yongle Zhang
> Priority: Normal
>
> Steps to reproduce:
> # Set up a cluster with 2 nodes (1 seed using 3.11.6, the other node using 4.0-alpha1).
> # Run stress testing on the seed node running C* 3.11.6.
> Error messages on each node:
> Seed (3.11.6):
>
> {code:java}
> ERROR [RequestResponseStage-1] 2020-06-25 01:58:56,745 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[RequestResponseStage-1,5,main]
> java.lang.IllegalArgumentException: Unknown request failure reason error code: 3
> at org.apache.cassandra.exceptions.RequestFailureReason.fromCode(RequestFailureReason.java:49) ~[main/:na]
> at org.apache.cassandra.net.MessageIn.getFailureReason(MessageIn.java:189) ~[main/:na]
> at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:47) ~[main/:na]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) ~[main/:na]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_222]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165) ~[main/:na]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137) [main/:na]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:113) [main/:na]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222]{code}
>
>
> Looks like this error message might be because the enum class RequestFailureReason is incompatible between 3.11.6 and 4.0-alpha1: In 3.11.6 it contains only 2 possible values ([https://github.com/apache/cassandra/blob/cassandra-3.11.6/src/java/org/apache/cassandra/exceptions/RequestFailureReason.java#L21]), while in 4.0-alpha1 it contains 4 possible values ([https://github.com/apache/cassandra/blob/cassandra-4.0-alpha1/src/java/org/apache/cassandra/exceptions/RequestFailureReason.java#L33]).
>
> Non-seed node (4.0-alpha1):
>
> {code:java}
> INFO [Messaging-EventLoop-3-5] 2020-06-25 01:58:56,743 NoSpamLogger.java:91 - 250.16.238.1:7000->250.16.238.2:7000-LEGACY_MESSAGES-0e4d7ae2 incompatible schema encountered while deserializing a message
> org.apache.cassandra.exceptions.UnknownTableException: Couldn't find table with id 5bc52802-de25-35ed-aeab-188eecebb090. If a table was just created, this is likely due to the schemanot being fully propagated. Please wait for schema agreement on table creation.
> at org.apache.cassandra.schema.Schema.getExistingTableMetadata(Schema.java:456)
> at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:614)
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:344)
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:362)
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:326)
> at org.apache.cassandra.net.Message$Serializer.deserializePayloadPre40(Message.java:949)
> at org.apache.cassandra.net.Message$Serializer.deserializePre40(Message.java:930)
> at org.apache.cassandra.net.Message$Serializer.deserializePre40(Message.java:918)
> at org.apache.cassandra.net.Message$Serializer.deserialize(Message.java:618)
> at org.apache.cassandra.net.InboundMessageHandler.processSmallMessage(InboundMessageHandler.java:321)
> at org.apache.cassandra.net.InboundMessageHandler.processOneContainedMessage(InboundMessageHandler.java:304)
> at org.apache.cassandra.net.InboundMessageHandler.processFrameOfContainedMessages(InboundMessageHandler.java:271)
> at org.apache.cassandra.net.InboundMessageHandler.processIntactFrame(InboundMessageHandler.java:256)
> at org.apache.cassandra.net.InboundMessageHandler.process(InboundMessageHandler.java:247)
> at org.apache.cassandra.net.FrameDecoder.deliver(FrameDecoder.java:320)
> at org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:284)
> at org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:268)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
> at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
> at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
> at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748){code}
>
> Not exactly sure why how is this UnknownTableException triggered.
> Client:
>
> {code:java}
> WARN 01:59:41,160 No schema agreement from live replicas after 10 s. The schema may not be up to date on some nodes.
> com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace 'stresscql' does not exist
> at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
> at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
> at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
> at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
> at org.apache.cassandra.stress.util.JavaDriverClient.execute(JavaDriverClient.java:189)
> at org.apache.cassandra.stress.StressProfile.maybeCreateSchema(StressProfile.java:238)
> at org.apache.cassandra.stress.StressProfile.newGenerator(StressProfile.java:681)
> at org.apache.cassandra.stress.StressProfile.printSettings(StressProfile.java:129)
> at org.apache.cassandra.stress.settings.StressSettings.printSettings(StressSettings.java:383)
> at org.apache.cassandra.stress.Stress.run(Stress.java:95)
> at org.apache.cassandra.stress.Stress.main(Stress.java:62)
> Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace 'stresscql' does not exist
> at com.datastax.driver.core.Responses$Error.asException(Responses.java:136)
> at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179)
> at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:177)
> at com.datastax.driver.core.RequestHandler.access$2500(RequestHandler.java:43)
> at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:792)
> at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:611)
> at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1013)
> at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:936)
> at com.datastax.shaded.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
> at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
> at com.datastax.shaded.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
> at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
> at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
> at com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
> at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
> at com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)
> at com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)
> at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
> at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
> at com.datastax.shaded.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
> at com.datastax.shaded.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org