You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Ke Han (Jira)" <ji...@apache.org> on 2022/12/10 17:05:00 UTC
[jira] [Comment Edited] (CASSANDRA-18109) Unknown column cdc during deserialization ERROR when upgrading from 3.0.28 to 3.11.14
[ https://issues.apache.org/jira/browse/CASSANDRA-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645649#comment-17645649 ]
Ke Han edited comment on CASSANDRA-18109 at 12/10/22 5:04 PM:
--------------------------------------------------------------
Thanks for the reply!
[~jmckenzie] The [code|https://github.com/apache/cassandra/blob/cassandra-3.11.14/src/java/org/apache/cassandra/schema/SchemaKeyspace.java#L437-L450] indeed tries to skip cdc column during the Schema serialization at [line 448|https://github.com/apache/cassandra/blob/cassandra-3.11.14/src/java/org/apache/cassandra/schema/SchemaKeyspace.java#L448]. But this issue is different. It happens during rolling upgrade when N0 and N1 are in 3.11.14 with cdc enabled, while N2 is in 3.0.28 with default settings (cdc disabled).
Therefore, the function actually returns at [line 441|https://github.com/apache/cassandra/blob/cassandra-3.11.14/src/java/org/apache/cassandra/schema/SchemaKeyspace.java#L441] for the 3.11.14 nodes. cdc column is still serialized and sent to 3.0.28 nodes. That causes the error.
[~brandon.williams] Thanks for the suggestions! I agree we shouldn't enable new features until the cluster upgrade is completed. It might cause incompatibilities between versions.
was (Author: JIRAUSER289562):
Thanks for the reply!
[~jmckenzie] The [code|https://github.com/apache/cassandra/blob/cassandra-3.11.14/src/java/org/apache/cassandra/schema/SchemaKeyspace.java#L437-L450] indeed tries to skip cdc column during the Schema serialization at [line 448|https://github.com/apache/cassandra/blob/cassandra-3.11.14/src/java/org/apache/cassandra/schema/SchemaKeyspace.java#L448]. But this issue is different. It happens during rolling upgrade when N0 and N1 are in 3.11.14 with cdc enabled, while N2 is in 3.0.28 with default settings (cdc disabled).
Therefore, the function actually returns at [line 441|https://github.com/apache/cassandra/blob/cassandra-3.11.14/src/java/org/apache/cassandra/schema/SchemaKeyspace.java#L441] for the 3.11.14 nodes. cdc column is still serialized and sent to 3.0.28 nodes. That causes the error.
[~brandon.williams] Thanks for the suggestions! I agree we shouldn't enable new features until the cluster upgrade is completed. It might cause incompatibility between versions.
> Unknown column cdc during deserialization ERROR when upgrading from 3.0.28 to 3.11.14
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-18109
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18109
> Project: Cassandra
> Issue Type: Bug
> Reporter: Ke Han
> Priority: Normal
> Attachments: N0-system.log, N1-system.log, N2-system.log
>
>
> When we performed a rolling upgrade from 3.0.28 to 3.11.14, we encountered the following error message during the upgrade process. It doesn't prevent the upgrade process, but it's in ERROR log level and causes concerns for the upgrade process.
> {code:java}
> ERROR [MessagingService-Incoming-/192.168.107.2] 2022-12-10 04:01:12,449 CassandraDaemon.java:265 - Exception in thread Thread[MessagingService-Incoming-/192.168.107.2,5,main]
> java.lang.RuntimeException: Unknown column cdc during deserialization
> at org.apache.cassandra.db.Columns$Serializer.deserialize(Columns.java:445)
> at org.apache.cassandra.db.SerializationHeader$Serializer.deserializeForMessaging(SerializationHeader.java:446)
> at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.deserializeHeader(UnfilteredRowIteratorSerializer.java:190)
> at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:734)
> at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:722)
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:337)
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:346)
> at org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:574)
> at org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:557)
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98)
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:216)
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:193)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:96)
> INFO [MigrationStage:1] 2022-12-10 04:01:13,394 MigrationCoordinator.java:523 - Sending schema pull request to /192.168.107.2 at 1670644873394 with timeout 10000
> ERROR [MessagingService-Incoming-/192.168.107.2] 2022-12-10 04:01:13,427 CassandraDaemon.java:265 - Exception in thread Thread[MessagingService-Incoming-/192.168.107.2,5,main]
> java.lang.RuntimeException: Unknown column cdc during deserialization
> at org.apache.cassandra.db.Columns$Serializer.deserialize(Columns.java:445)
> at org.apache.cassandra.db.SerializationHeader$Serializer.deserializeForMessaging(SerializationHeader.java:446)
> at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.deserializeHeader(UnfilteredRowIteratorSerializer.java:190)
> at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:734)
> at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:722)
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:337)
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:346)
> at org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:574)
> at org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:557)
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98)
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:216)
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:193)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:96){code}
> h1. Steps to reproduce
> Configure a 3-node Cassandra cluster (N0, N1 and N2) in 3.0.28 using the default configuration setting. Configure N0 as the seed node.
> For the new version configuration (3.11.14), set *cdc_enabled: true* in cassandra.yaml. All the other configurations use default values.
> Perform a rolling upgrade.
> # Start up the 3-node cluster in 3.0.28.
> # Drain, shutdown and upgrade N0 to 3.11.14.
> # Drain, shutdown and upgrade N1 to 3.11.14.
> # Drain, shutdown and upgrade N2 to 3.11.14.
> The +Unknown column cdc ERROR message+ occurs in *N2* once both N0 and N1 are upgraded.
> I attached the system.log for our three-node cluster.
> * N0 (Seed): 192.168.86.2
> * N1: 192.168.86.3
> * N2: 192.168.86.4
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org