You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Russ Hatch (JIRA)" <ji...@apache.org> on 2015/12/08 23:39:11 UTC
[jira] [Comment Edited] (CASSANDRA-10822) SSTable data loss when
upgrading with row tombstone present
[ https://issues.apache.org/jira/browse/CASSANDRA-10822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15047563#comment-15047563 ]
Russ Hatch edited comment on CASSANDRA-10822 at 12/8/15 10:38 PM:
------------------------------------------------------------------
(caveat: possibly unrelated) After upgrade I run sstableverify on the node and get this exception:
{noformat}
Verifying BigTableReader(path='/home/rhatch/.ccm/financial/node1/data/financial/symbol_history-801411b09df711e581e7ff4e6f1f6740/financial-symbol_history-ka-1-Data.db') (1403 bytes)
Checking computed hash of BigTableReader(path='/home/rhatch/.ccm/financial/node1/data/financial/symbol_history-801411b09df711e581e7ff4e6f1f6740/financial-symbol_history-ka-1-Data.db')
Error verifying BigTableReader(path='/home/rhatch/.ccm/financial/node1/data/financial/symbol_history-801411b09df711e581e7ff4e6f1f6740/financial-symbol_history-ka-1-Data.db'): java.nio.file.NoSuchFileException: /home/rhatch/.ccm/financial/node1/data/financial/symbol_history-801411b09df711e581e7ff4e6f1f6740/financial-symbol_history-ka-1-Digest.adler32
java.lang.RuntimeException: java.nio.file.NoSuchFileException: /home/rhatch/.ccm/financial/node1/data/financial/symbol_history-801411b09df711e581e7ff4e6f1f6740/financial-symbol_history-ka-1-Digest.adler32
at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
at org.apache.cassandra.io.util.ChannelProxy.<init>(ChannelProxy.java:66)
at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:504)
at org.apache.cassandra.io.util.DataIntegrityMetadata$FileDigestValidator.<init>(DataIntegrityMetadata.java:113)
at org.apache.cassandra.io.util.DataIntegrityMetadata.fileDigestValidator(DataIntegrityMetadata.java:98)
at org.apache.cassandra.db.compaction.Verifier.verify(Verifier.java:102)
at org.apache.cassandra.tools.StandaloneVerifier.main(StandaloneVerifier.java:103)
Caused by: java.nio.file.NoSuchFileException: /home/rhatch/.ccm/financial/node1/data/financial/symbol_history-801411b09df711e581e7ff4e6f1f6740/financial-symbol_history-ka-1-Digest.adler32
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51)
... 6 more
{noformat}
After running upgradesstables verify works fine (though the data is still not present).
was (Author: rhatch):
(caveat: possibly unrelated) After upgrade I run sstableverify on the node and get this exception:
{noformat}
Verifying BigTableReader(path='/home/rhatch/.ccm/financial/node1/data/financial/symbol_history-801411b09df711e581e7ff4e6f1f6740/financial-symbol_history-ka-1-Data.db') (1403 bytes)
Checking computed hash of BigTableReader(path='/home/rhatch/.ccm/financial/node1/data/financial/symbol_history-801411b09df711e581e7ff4e6f1f6740/financial-symbol_history-ka-1-Data.db')
Error verifying BigTableReader(path='/home/rhatch/.ccm/financial/node1/data/financial/symbol_history-801411b09df711e581e7ff4e6f1f6740/financial-symbol_history-ka-1-Data.db'): java.nio.file.NoSuchFileException: /home/rhatch/.ccm/financial/node1/data/financial/symbol_history-801411b09df711e581e7ff4e6f1f6740/financial-symbol_history-ka-1-Digest.adler32
java.lang.RuntimeException: java.nio.file.NoSuchFileException: /home/rhatch/.ccm/financial/node1/data/financial/symbol_history-801411b09df711e581e7ff4e6f1f6740/financial-symbol_history-ka-1-Digest.adler32
at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
at org.apache.cassandra.io.util.ChannelProxy.<init>(ChannelProxy.java:66)
at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:504)
at org.apache.cassandra.io.util.DataIntegrityMetadata$FileDigestValidator.<init>(DataIntegrityMetadata.java:113)
at org.apache.cassandra.io.util.DataIntegrityMetadata.fileDigestValidator(DataIntegrityMetadata.java:98)
at org.apache.cassandra.db.compaction.Verifier.verify(Verifier.java:102)
at org.apache.cassandra.tools.StandaloneVerifier.main(StandaloneVerifier.java:103)
Caused by: java.nio.file.NoSuchFileException: /home/rhatch/.ccm/financial/node1/data/financial/symbol_history-801411b09df711e581e7ff4e6f1f6740/financial-symbol_history-ka-1-Digest.adler32
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51)
... 6 more
{noformat}
After running upgradesstables this goes away.
> SSTable data loss when upgrading with row tombstone present
> -----------------------------------------------------------
>
> Key: CASSANDRA-10822
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10822
> Project: Cassandra
> Issue Type: Bug
> Reporter: Andy Tolbert
> Priority: Critical
> Fix For: 3.0.x, 3.x
>
>
> I ran into an issue when upgrading between 2.1.11 to 3.0.0 (and also cassandra-3.0 branch) where subsequent rows were lost within a partition where there is a row tombstone present.
> Here's a scenario that reproduces the issue.
> Using ccm create a single node cluster at 2.1.11:
> {{ccm create -n 1 -v 2.1.11 -s financial}}
> Run the following queries to create schema, populate some data and then delete some data for november:
> {noformat}
> drop keyspace if exists financial;
> create keyspace if not exists financial with replication = {'class': 'SimpleStrategy', 'replication_factor' : 1 };
> create table if not exists financial.symbol_history (
> symbol text,
> name text static,
> year int,
> month int,
> day int,
> volume bigint,
> close double,
> open double,
> low double,
> high double,
> primary key((symbol, year), month, day)
> ) with CLUSTERING ORDER BY (month desc, day desc);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) values ('CORP', 'MegaCorp', 2004, 1, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) values ('CORP', 'MegaCorp', 2004, 2, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) values ('CORP', 'MegaCorp', 2004, 3, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) values ('CORP', 'MegaCorp', 2004, 4, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) values ('CORP', 'MegaCorp', 2004, 5, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) values ('CORP', 'MegaCorp', 2004, 6, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) values ('CORP', 'MegaCorp', 2004, 7, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) values ('CORP', 'MegaCorp', 2004, 8, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) values ('CORP', 'MegaCorp', 2004, 9, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) values ('CORP', 'MegaCorp', 2004, 10, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) values ('CORP', 'MegaCorp', 2004, 11, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) values ('CORP', 'MegaCorp', 2004, 12, 1, 100);
> delete from financial.symbol_history where symbol='CORP' and year = 2004 and month=11;
> {noformat}
> Flush and run sstable2json on the sole Data.db file:
> {noformat}
> ccm node1 flush
> sstable2json /path/to/file.db
> {noformat}
> The output should look like the following:
> {code}
> [
> {"key": "CORP:2004",
> "cells": [["::name","MegaCorp",1449457517033030],
> ["12:1:","",1449457517033030],
> ["12:1:volume","100",1449457517033030],
> ["11:_","11:!",1449457564983269,"t",1449457564],
> ["10:1:","",1449457516313738],
> ["10:1:volume","100",1449457516313738],
> ["9:1:","",1449457516310205],
> ["9:1:volume","100",1449457516310205],
> ["8:1:","",1449457516235664],
> ["8:1:volume","100",1449457516235664],
> ["7:1:","",1449457516233535],
> ["7:1:volume","100",1449457516233535],
> ["6:1:","",1449457516231458],
> ["6:1:volume","100",1449457516231458],
> ["5:1:","",1449457516228307],
> ["5:1:volume","100",1449457516228307],
> ["4:1:","",1449457516225415],
> ["4:1:volume","100",1449457516225415],
> ["3:1:","",1449457516222811],
> ["3:1:volume","100",1449457516222811],
> ["2:1:","",1449457516220301],
> ["2:1:volume","100",1449457516220301],
> ["1:1:","",1449457516210758],
> ["1:1:volume","100",1449457516210758]]}
> ]
> {code}
> Prepare for upgrade
> {noformat}
> ccm node1 nodetool snapshot financial
> ccm node1 nodetool drain
> ccm node1 stop
> {noformat}
> Upgrade to cassandra-3.0 and start the node
> {noformat}
> ccm node1 setdir -v git:cassandra-3.0
> ccm node1 start
> {noformat}
> Run command in cqlsh and observe only 1 row is returned! It appears that all data following november is gone.
> {noformat}
> cqlsh> select * from financial.symbol_history;
> symbol | year | month | day | name | close | high | low | open | volume
> --------+------+-------+-----+----------+-------+------+------+------+--------
> CORP | 2004 | 12 | 1 | MegaCorp | null | null | null | null | 100
> {noformat}
> Upgrade sstables and query again and you'll observe the same problem.
> {noformat}
> ccm node1 nodetool upgradesstables financial
> {noformat}
> I modified the 2.2 version of sstable2json so that it works with 3.0 (couldn't help myself :)), and observed 2 RangeTombstoneBoundMarker occurrences for 1 delete and the rest of the data missing.
> {code}
> [
> {
> "key": "CORP:2004",
> "static": {
> "cells": {
> ["name","MegaCorp",1449457517033030]
> }
> },
> "rows": [
> {
> "clustering": {"month": "12", "day": "1"},
> "cells": {
> ["volume","100",1449457517033030]
> }
> },
> {
> "tombstone": ["11:*",1449457564983269,"t",1449457564]
> },
> {
> "tombstone": ["11:*",1449457564983269,"t",1449457564]
> }
> ]
> }
> ]
> {code}
> I'm not sure why this is happening, but I should point out that I'm using static columns here and that I'm using reverse order for my clustering, so maybe that makes a difference. I'll try without static columns / regular ordering to see if that makes a difference and update the ticket.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)