You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "T Jake Luciani (JIRA)" <ji...@apache.org> on 2014/06/01 03:21:02 UTC
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress +
CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014880#comment-14014880 ]
T Jake Luciani commented on CASSANDRA-7245:
-------------------------------------------
After much experimentation I've found one of the issues here:
Our version of netty throws away buffers once they have been read. This issue was fixed in the latest netty so buffer with refcounts aren't cannibalized https://github.com/netty/netty/issues/2327
I've tested a with 4.0.19 release and the token corruption is now gone.
I am however seeing some kind of race condition which I think may be netty related as well. It seems like the same Message is being dispatched to multiple threads sometimes and causing refcount exceptions. I've added logging to capture the history when this happens and I'm trying to track this down.
> Out-of-Order keys with stress + CQL3
> ------------------------------------
>
> Key: CASSANDRA-7245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7245
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Pavel Yaskevich
> Assignee: T Jake Luciani
> Fix For: 2.1.0
>
> Attachments: 7245-v2.txt, 7245.txt
>
>
> We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 and found following problem almost in every SSTable generated (~200 GB of data and 821 SSTables).
> We set up they keys to be 10 bytes in size (default) and population between 1 and 600000000.
> Once I ran 'sstablekeys' on the generated SSTable files I got following exceptions:
> _There is a problem with sorting of normal looking keys:_
> 30303039443538353645
> 30303039443745364242
> java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, *30303039443745364242*) > DecoratedKey(-1767746583617597213, *30303039443437454333*)
> 00000a30303033343933
> 37344400001388343933
> java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, *37344400001388343933*) > DecoratedKey(-7565486415339257200, *30303033344639443137*)
> 30303033354244363031
> 30303033354133423742
> java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, *30303033354133423742*) > DecoratedKey(-7838239767410066684, *30303033354145344534*)
> 30303034313442354137
> 30343136353633340000
> java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, *30343136353633340000*) > DecoratedKey(-9106177395653818217, *30303034313333444238*)
> 30303035373044373435
> 30303035373044334631
> java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, *30303035373044334631*) > DecoratedKey(-4296696226469000945, *30303035373132364138*)
> _And completely different ones:_
> 30303041333745373543
> 7cd045c59a90d7587d8d
> java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, *30303041333943303232*)
> 30303033323144444144
> 30303033323346343932
> java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, *30303033323346343932*) > DecoratedKey(5233296131921119414, *53d83e00000012287e03*)
> 30303034314531374431
> 3806734b256c27e41ec2
> java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, *30303034314136413343*)
> _And sometimes there is no problem at all:_
> 30303033353144463637
> 0000002a31b3b31a1c2f
> 5d616dd38211ebb5d6ec
> 44423645000013880000
> 00001388138844463744
> 30303033353143394343
> It's worth to mention that we have got 22 timeout exceptions but number of out-of-order keys is much larger than that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)