You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Alex Petrov (JIRA)" <ji...@apache.org> on 2016/09/30 16:45:21 UTC
[jira] [Comment Edited] (CASSANDRA-12694) PAXOS Update Corrupted
empty row exception
[ https://issues.apache.org/jira/browse/CASSANDRA-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15536170#comment-15536170 ]
Alex Petrov edited comment on CASSANDRA-12694 at 9/30/16 4:44 PM:
------------------------------------------------------------------
Thanks for the pointer to the commit. Although the problem was not really caused by this addition, it just got surfaced. What we expect there is correct and protects users from reading data from the corrupt SSTable. In the case of an empty row we would have liveness (which is used to distinguish empty from dead row [more details here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/rows/Row.java#L71-L86](.
What happens is that we're following a read path that reads out only the columns that are affected by the LWT (and this column was never set), which results into this check failing, since there's no liveness that would confirm that this row exists but is empty. In fact, we'll run into the same situation if we use "regular" (not LWT) update in the first place, like that:
{code}
update test.lwt_corruption_test set last_updated = 555 where test_id = 'test3';
update test.lwt_corruption_test set last_updated = 555 where test_id = 'test3' if message_id = null;
;; would result into the same error
{code}
However, running the LWT with a column that actually exists would result into successful LWT:
{code}
update test.lwt_corruption_test set last_updated = 555 where test_id = 'test3' if message_id = null and last_updated = 555;
{code}
Now, {{last_updated}} will be returned and non-empty.
In order to avoid this problem and make sure we never have a situation where row is empty, we have to fetch enough columns to satisfy the column condition and distinguish between "all conditions are null" and "row does not exist".
If we update static row, we won't have any conditions (or updates) on regular rows, so we fetch _all_ static rows.
If we update regular rows, we fetch only the static rows that take part in column condition and _all_ regular rows.
|[trunk|https://github.com/ifesdjeen/cassandra/tree/12694-trunk] |[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12694-trunk-testall/] |[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12694-trunk-dtest/] |
was (Author: ifesdjeen):
Thanks for the pointer to the commit. Although the problem was not really caused by this addition, it just got surfaced. What we expect there is correct and protects users from reading data from the corrupt SSTable. In the case of an empty row we would have liveness (which is used to distinguish empty from dead row [more details here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/rows/Row.java#L71-L86](.
What happens is that we're following a read path that reads out only the columns that are affected by the LWT (and this column was never set), which results into this check failing, since there's no liveness that would confirm that this row exists but is empty. In fact, we'll run into the same situation if we use "regular" (not LWT) update in the first place, like that:
{code}
update test.lwt_corruption_test set last_updated = 555 where test_id = 'test3';
update test.lwt_corruption_test set last_updated = 555 where test_id = 'test3' if message_id = null;
;; would result into the same error
{code}
However, running the LWT with a column that actually exists would result into successful LWT:
{code}
update test.lwt_corruption_test set last_updated = 555 where test_id = 'test3' if message_id = null and last_updated = 555;
{code}
Now, {{last_updated}} will be returned and non-empty.
> PAXOS Update Corrupted empty row exception
> ------------------------------------------
>
> Key: CASSANDRA-12694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12694
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: 3 node cluster using RF=3 running on cassandra 3.7
> Reporter: Cameron Zemek
> Assignee: Alex Petrov
>
> {noformat}
> cqlsh> create table test.test (test_id TEXT, last_updated TIMESTAMP, message_id TEXT, PRIMARY KEY(test_id));
> update test.test set last_updated = 1474494363669 where test_id = 'test1' if message_id = null;
> {noformat}
> Then nodetool flush on the all 3 nodes.
> {noformat}
> cqlsh> update test.test set last_updated = 1474494363669 where test_id = 'test1' if message_id = null;
> ServerError: <ErrorMessage code=0000 [Server error] message="java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered partition">
> {noformat}
> From cassandra log
> {noformat}
> ERROR [SharedPool-Worker-1] 2016-09-23 12:09:13,179 Message.java:611 - Unexpected exception during request; channel = [id: 0x7a22599e, L:/127.0.0.1:9042 - R:/127.0.0.1:58297]
> java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered partition
> at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224) ~[main/:na]
> at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212) ~[main/:na]
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[main/:na]
> at org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:125) ~[main/:na]
> at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:249) ~[main/:na]
> at org.apache.cassandra.db.ReadResponse.makeDigest(ReadResponse.java:87) ~[main/:na]
> at org.apache.cassandra.db.ReadResponse$DataResponse.digest(ReadResponse.java:192) ~[main/:na]
> at org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:80) ~[main/:na]
> at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:139) ~[main/:na]
> at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:145) ~[main/:na]
> at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.awaitResultsAndRetryOnDigestMismatch(StorageProxy.java:1714) ~[main/:na]
> at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1663) ~[main/:na]
> at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1604) ~[main/:na]
> at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1523) ~[main/:na]
> at org.apache.cassandra.service.StorageProxy.readOne(StorageProxy.java:1497) ~[main/:na]
> at org.apache.cassandra.service.StorageProxy.readOne(StorageProxy.java:1491) ~[main/:na]
> at org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:249) ~[main/:na]
> at org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:441) ~[main/:na]
> at org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:416) ~[main/:na]
> at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:208) ~[main/:na]
> at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:239) ~[main/:na]
> at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:224) ~[main/:na]
> at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115) ~[main/:na]
> at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [main/:na]
> at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [main/:na]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)