You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jeffrey Wang <jw...@palantir.com> on 2011/07/12 05:28:54 UTC
commitlog replay missing data
Hey all,
Recently upgraded to 0.8.1 and noticed what seems to be missing data after a
commitlog replay on a single-node cluster. I start the node, insert a bunch
of stuff (~600MB), stop it, and restart it. There are log messages
pertaining to the commitlog replay and no errors, but some of the data is
missing. If I flush before stopping the node, everything is fine, and
running cfstats in the two cases shows different amounts of data in the
SSTables. Moreover, the amount of data that is missing is nondeterministic.
Has anyone run into this? Thanks.
Here is the output of a side-by-side diff between cfstats outputs for a
single CF before restarting (left) and after (right). Somehow a 37MB
memtable became a 2.9MB SSTable (note the difference in write count as
well)?
Column Family: Blocks Column
Family: Blocks
SSTable count: 0 | SSTable
count: 1
Space used (live): 0 | Space used
(live): 2907637
Space used (total): 0 | Space used
(total): 2907637
Memtable Columns Count: 8198 | Memtable
Columns Count: 0
Memtable Data Size: 37550510 | Memtable
Data Size: 0
Memtable Switch Count: 0 | Memtable
Switch Count: 1
Read Count: 0 Read Count:
0
Read Latency: NaN ms. Read
Latency: NaN ms.
Write Count: 8198 | Write Count:
1526
Write Latency: 0.018 ms. | Write
Latency: 0.011 ms.
Pending Tasks: 0 Pending
Tasks: 0
Key cache capacity: 200000 Key cache
capacity: 200000
Key cache size: 0 Key cache
size: 0
Key cache hit rate: NaN Key cache
hit rate: NaN
Row cache: disabled Row cache:
disabled
Compacted row minimum size: 0 | Compacted
row minimum size: 1110
Compacted row maximum size: 0 | Compacted
row maximum size: 2299
Compacted row mean size: 0 | Compacted
row mean size: 1960
Note that I patched https://issues.apache.org/jira/browse/CASSANDRA-2317 in
my version, but there are no deletions involved so I don't think it's
relevant unless I messed something up while patching.
-Jeffrey
Re: commitlog replay missing data
Posted by Aaron Morton <aa...@thelastpickle.com>.
Have you verified that data you expect to see is not in the server after shutdown?
WRT the differed in the difference between the Memtable data size and SSTable live size, don't believe everything you read :)
Memtable live size is increased by the serialised byte size of every column inserted, and is never decremented. Deletes and overwrites will inflate this value. What was your workload like?
As of 0.8 we now have global memory management for cf's that tracks actual JVM bytes used by a CF.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 12/07/2011, at 3:28 PM, Jeffrey Wang <jw...@palantir.com> wrote:
> Hey all,
>
>
>
> Recently upgraded to 0.8.1 and noticed what seems to be missing data after a commitlog replay on a single-node cluster. I start the node, insert a bunch of stuff (~600MB), stop it, and restart it. There are log messages pertaining to the commitlog replay and no errors, but some of the data is missing. If I flush before stopping the node, everything is fine, and running cfstats in the two cases shows different amounts of data in the SSTables. Moreover, the amount of data that is missing is nondeterministic. Has anyone run into this? Thanks.
>
>
>
> Here is the output of a side-by-side diff between cfstats outputs for a single CF before restarting (left) and after (right). Somehow a 37MB memtable became a 2.9MB SSTable (note the difference in write count as well)?
>
>
>
> Column Family: Blocks Column Family: Blocks
>
> SSTable count: 0 | SSTable count: 1
>
> Space used (live): 0 | Space used (live): 2907637
>
> Space used (total): 0 | Space used (total): 2907637
>
> Memtable Columns Count: 8198 | Memtable Columns Count: 0
>
> Memtable Data Size: 37550510 | Memtable Data Size: 0
>
> Memtable Switch Count: 0 | Memtable Switch Count: 1
>
> Read Count: 0 Read Count: 0
>
> Read Latency: NaN ms. Read Latency: NaN ms.
>
> Write Count: 8198 | Write Count: 1526
>
> Write Latency: 0.018 ms. | Write Latency: 0.011 ms.
>
> Pending Tasks: 0 Pending Tasks: 0
>
> Key cache capacity: 200000 Key cache capacity: 200000
>
> Key cache size: 0 Key cache size: 0
>
> Key cache hit rate: NaN Key cache hit rate: NaN
>
> Row cache: disabled Row cache: disabled
>
> Compacted row minimum size: 0 | Compacted row minimum size: 1110
>
> Compacted row maximum size: 0 | Compacted row maximum size: 2299
>
> Compacted row mean size: 0 | Compacted row mean size: 1960
>
>
>
> Note that I patched https://issues.apache.org/jira/browse/CASSANDRA-2317 in my version, but there are no deletions involved so I don’t think it’s relevant unless I messed something up while patching.
>
>
>
> -Jeffrey
>
Re: commitlog replay missing data
Posted by Peter Schuller <pe...@infidyne.com>.
> # wait for a bit until no one is sending it writes anymore
More accurately, until all other nodes have realized it's down
(nodetool ring on each respective host).
--
/ Peter Schuller (@scode on twitter)
Re: commitlog replay missing data
Posted by Peter Schuller <pe...@infidyne.com>.
> What are the other ways to stop Cassandra?
nodetool disablegossip
nodetool disablethrift
# wait for a bit until no one is sending it writes anymore
nodetool flush # only relevant if in periodic mode
# then kill it
> What's the difference between batch vs periodic?
Search for "batch" on http://wiki.apache.org/cassandra/StorageConfiguration
--
/ Peter Schuller (@scode on twitter)
Re: commitlog replay missing data
Posted by mcasandra <mo...@gmail.com>.
Peter Schuller wrote:
>
>> Recently upgraded to 0.8.1 and noticed what seems to be missing data
>> after a
>> commitlog replay on a single-node cluster. I start the node, insert a
>> bunch
>> of stuff (~600MB), stop it, and restart it. There are log messages
>
> If you stop by a kill, make sure you use batched commitlog synch mode
> instead of periodic if you want guarantees on individual writes.
>
What are the other ways to stop Cassandra?
What's the difference between batch vs periodic?
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/commitlog-replay-missing-data-tp6573659p6580886.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: commitlog replay missing data
Posted by Peter Schuller <pe...@infidyne.com>.
> Recently upgraded to 0.8.1 and noticed what seems to be missing data after a
> commitlog replay on a single-node cluster. I start the node, insert a bunch
> of stuff (~600MB), stop it, and restart it. There are log messages
If you stop by a kill, make sure you use batched commitlog synch mode
instead of periodic if you want guarantees on individual writes.
(I don't believe you'd expect a significant disk space discrepancy
though since in practice the delay until write() should be small. But
don't quote me on this because I'd have to check the code to make sure
that commit log reply isn't dependent on some marker that isn't
written until commit log synch.)
--
/ Peter Schuller (@scode on twitter)