You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Ke Han (Jira)" <ji...@apache.org> on 2022/12/09 21:51:00 UTC
[jira] [Updated] (CASSANDRA-18108) Data loss after a system restart/upgrade (3.11.14)
[ https://issues.apache.org/jira/browse/CASSANDRA-18108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ke Han updated CASSANDRA-18108:
-------------------------------
Description:
When we upgrade Cassandra from 3.11.14 to 4.0.7, we found a data loss during the upgrade process. This bug can also be triggered if simply performing a system restart.
h1. Steps to reproduce
Start a 3.11.14 or 4.0.7 Cassandra node using default configurations. Execute the following cqlsh commands.
{code:java}
CREATE KEYSPACE ks WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
CREATE TABLE IF NOT EXISTS ks.tb (c1 INT,c3 INT,c2 TEXT, PRIMARY KEY (c1 )) WITH speculative_retry = 'ALWAYS';
INSERT INTO ks.tb (c1, c2) VALUES (2,'val');
ALTER TABLE ks.tb DROP c2 ;
ALTER TABLE ks.tb RENAME c1 TO c2; {code}
Then execute a SELECT command.
{code:java}
cqlsh> SELECT * FROM ks.tb;
c2 | c3
----+------
2 | null
(1 rows){code}
Flush and stop the Cassandra daemon.
{code:java}
bin/nodetool flush
bin/nodetool stopdaemon{code}
Then restart the node.
{code:java}
bin/cassandra{code}
Start cqlsh, and execute the same SELECT command. The data in ks.tb is lost.
{code:java}
cqlsh> SELECT * FROM ks.tb;
c2 | c3
----+----
(0 rows){code}
During the node restart, we also found an error log about initializing the table, but it didn't prevent the system from starting up.
{code:java}
INFO [main] 2022-12-09 21:37:54,234 ColumnFamilyStore.java:432 - Initializing ks.tb
ERROR [SSTableBatchOpen:1] 2022-12-09 21:37:54,237 CassandraDaemon.java:244 - Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.AssertionError: null
at org.apache.cassandra.db.PartitionColumns$Builder.add(PartitionColumns.java:161)
at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:340)
at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:522)
at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:385)
at org.apache.cassandra.io.sstable.format.SSTableReader$3.run(SSTableReader.java:570)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
at java.lang.Thread.run(Thread.java:750) {code}
This bug can also be triggered if we perform an upgrade to 4.0.7 and execute the SELECT command in the new version.
was:
When we upgrade Cassandra from 3.11.14 to 4.0.7, we found a data loss during the upgrade process. This bug can also be triggered if simply performing a system restart.
h1. Steps to reproduce
Start a 3.11.14 or 4.0.7 Cassandra node using default configurations. Execute the following cqlsh commands.
{code:java}
CREATE KEYSPACE ks WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
CREATE TABLE IF NOT EXISTS ks.tb (c1 INT,c3 INT,c2 TEXT, PRIMARY KEY (c1 )) WITH speculative_retry = 'ALWAYS';
INSERT INTO ks.tb (c1, c2) VALUES (2,'val');
ALTER TABLE ks.tb DROP c2 ;
ALTER TABLE ks.tb RENAME c1 TO c2; {code}
Then execute a SELECT read
{code:java}
cqlsh> SELECT * FROM ks.tb;
c2 | c3
----+------
2 | null
(1 rows){code}
Flush and stop the daemon
{code:java}
bin/nodetool flush
bin/nodetool stopdaemon{code}
Then restart the node
{code:java}
bin/cassandra{code}
Start cqlsh, and execute the same SELECT command. The data in ks.tb is lost.
{code:java}
cqlsh> SELECT * FROM ks.tb;
c2 | c3
----+----
(0 rows){code}
During the node restart, we also found an error log about initializing the table, but it didn't prevent the system from starting up.
{code:java}
INFO [main] 2022-12-09 21:37:54,234 ColumnFamilyStore.java:432 - Initializing ks.tb
ERROR [SSTableBatchOpen:1] 2022-12-09 21:37:54,237 CassandraDaemon.java:244 - Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.AssertionError: null
at org.apache.cassandra.db.PartitionColumns$Builder.add(PartitionColumns.java:161)
at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:340)
at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:522)
at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:385)
at org.apache.cassandra.io.sstable.format.SSTableReader$3.run(SSTableReader.java:570)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
at java.lang.Thread.run(Thread.java:750) {code}
This bug can also be triggered if we perform an upgrade to 4.0.7 and execute the read in the new version.
> Data loss after a system restart/upgrade (3.11.14)
> --------------------------------------------------
>
> Key: CASSANDRA-18108
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18108
> Project: Cassandra
> Issue Type: Bug
> Reporter: Ke Han
> Priority: Normal
>
> When we upgrade Cassandra from 3.11.14 to 4.0.7, we found a data loss during the upgrade process. This bug can also be triggered if simply performing a system restart.
> h1. Steps to reproduce
> Start a 3.11.14 or 4.0.7 Cassandra node using default configurations. Execute the following cqlsh commands.
> {code:java}
> CREATE KEYSPACE ks WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
> CREATE TABLE IF NOT EXISTS ks.tb (c1 INT,c3 INT,c2 TEXT, PRIMARY KEY (c1 )) WITH speculative_retry = 'ALWAYS';
> INSERT INTO ks.tb (c1, c2) VALUES (2,'val');
> ALTER TABLE ks.tb DROP c2 ;
> ALTER TABLE ks.tb RENAME c1 TO c2; {code}
> Then execute a SELECT command.
> {code:java}
> cqlsh> SELECT * FROM ks.tb;
> c2 | c3
> ----+------
> 2 | null
> (1 rows){code}
> Flush and stop the Cassandra daemon.
> {code:java}
> bin/nodetool flush
> bin/nodetool stopdaemon{code}
> Then restart the node.
> {code:java}
> bin/cassandra{code}
> Start cqlsh, and execute the same SELECT command. The data in ks.tb is lost.
> {code:java}
> cqlsh> SELECT * FROM ks.tb;
> c2 | c3
> ----+----
> (0 rows){code}
>
> During the node restart, we also found an error log about initializing the table, but it didn't prevent the system from starting up.
> {code:java}
> INFO [main] 2022-12-09 21:37:54,234 ColumnFamilyStore.java:432 - Initializing ks.tb
> ERROR [SSTableBatchOpen:1] 2022-12-09 21:37:54,237 CassandraDaemon.java:244 - Exception in thread Thread[SSTableBatchOpen:1,5,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.db.PartitionColumns$Builder.add(PartitionColumns.java:161)
> at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:340)
> at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:522)
> at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:385)
> at org.apache.cassandra.io.sstable.format.SSTableReader$3.run(SSTableReader.java:570)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
> at java.lang.Thread.run(Thread.java:750) {code}
>
>
> This bug can also be triggered if we perform an upgrade to 4.0.7 and execute the SELECT command in the new version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org