You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Soumava Ghosh <so...@cs.utexas.edu> on 2013/07/21 13:22:04 UTC
Cassandra 2.0 - AssertionError in ArrayBackedSortedColumns
Hi,
I'm taking a look at the Check and Set functionalities provided by the
cas() API provided by cassandra 2.0 (the code available on git). I'm
running a few tests on a small sized cluster (replication factor 3,
consistency level quorum) with a few clients. I've observed a lot of cases
seem to hit the TimedOutException while paxos is in progress. On inspection
of the server side logs the following stack was seen:
ERROR [Thread-582] 2013-07-21 01:17:46,169 CassandraDaemon.java (line 196)
Exception in thread Thread[Thread-582,5,main]
java.lang.AssertionError: Added column does not sort as the last column
at
org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
at
org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:119)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:91)
at
org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:139)
at
org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:128)
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:175)
at
org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:135)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
I was updating multiple columns using the same cas() call, and looking at
the assert (code below) I added code to sort them before sending.
However, now I'm seeing the same issue *even with single column* cas()
calls.
int c = internalComparator().compare(columns.get(getColumnCount() -
1).name(), column.name());
// note that we want an assertion here (see addColumn javadoc), but
we also want that if
// assertion are disabled, addColumn works correctly with unsorted
input
assert c <= 0 : "Added column does not sort as the " + (reversed ?
"first" : "last") + " column";
I have also seen this error occur in the getRow() path, as below:
java.lang.AssertionError: Added column does not sort as the last column
at
org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
at
org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
at
org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:151)
at
org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:95)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:57)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1452)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1281)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1195)
at org.apache.cassandra.db.Table.getRow(Table.java:331)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1288)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1813)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Could someone provide me a little more clarity about what the context of
the error is and what the client needs to do to fix it? I understand that
running with assertions disabled would do away with this error, but i'd
like to know if there is a proper way to fix this. Moreover, what could
possibly be going wrong in the get path that is hitting this issue?
Some guidance, on the fact that whether it is expected that the client sort
the columns before calling the API with multiple columns, would be really
appreciated.
Thanks,
Soumava
Re: Cassandra 2.0 - AssertionError in ArrayBackedSortedColumns
Posted by Soumava Ghosh <so...@cs.utexas.edu>.
Thanks for the confirmation!
It looks like the code is capable of handling the data in unsorted order
with assertions disabled; so for the time being I will disable the
assertion for my tests, until the fixes are available.
Thanks,
Soumava
On Mon, Jul 22, 2013 at 1:33 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
> Actually, your second stack trace is due to
> https://issues.apache.org/jira/browse/CASSANDRA-5788.
>
>
> On Mon, Jul 22, 2013 at 9:37 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>>
>> unless data was stored in incorrect order and I don't know if that is
>>> possible through the API..
>>>
>>
>> That's exactly what happens and that's why I say it's a bug.
>>
>>
>>>
>>> ii. I am also seeing some sporadic cases where a single column cas()
>>> update is hitting this same issue (the bug you mentioned) as multiple
>>> columns, do you see any reason for that happening?
>>>
>>
>> That would be slightly weirder, but frankly that's almost surely due to
>> https://issues.apache.org/jira/browse/CASSANDRA-5786 too.
>>
>> --
>> Sylvain
>>
>>
>>> Thank you,
>>> Soumava
>>>
>>>
>>> On Mon, Jul 22, 2013 at 12:11 AM, Sylvain Lebresne <sylvain@datastax.com
>>> > wrote:
>>>
>>>> This is a bug really:
>>>> https://issues.apache.org/jira/browse/CASSANDRA-5786.
>>>>
>>>> This should get fixed in the next beta of 2.0, but if you really want
>>>> to test CAS updates in the meantime, you'll have to provide the columns in
>>>> (column family comparator) sorted order to the thrift cas() method.
>>>>
>>>> --
>>>> Sylvain
>>>>
>>>>
>>>> On Sun, Jul 21, 2013 at 1:22 PM, Soumava Ghosh <so...@cs.utexas.edu>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm taking a look at the Check and Set functionalities provided by the
>>>>> cas() API provided by cassandra 2.0 (the code available on git). I'm
>>>>> running a few tests on a small sized cluster (replication factor 3,
>>>>> consistency level quorum) with a few clients. I've observed a lot of cases
>>>>> seem to hit the TimedOutException while paxos is in progress. On inspection
>>>>> of the server side logs the following stack was seen:
>>>>>
>>>>> ERROR [Thread-582] 2013-07-21 01:17:46,169 CassandraDaemon.java (line
>>>>> 196) Exception in thread Thread[Thread-582,5,main]
>>>>> java.lang.AssertionError: Added column does not sort as the last column
>>>>> at
>>>>> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
>>>>> at
>>>>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
>>>>> at
>>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:119)
>>>>> at
>>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96)
>>>>> at
>>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:91)
>>>>> at
>>>>> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:139)
>>>>> at
>>>>> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:128)
>>>>> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
>>>>> at
>>>>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:175)
>>>>> at
>>>>> org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:135)
>>>>> at
>>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
>>>>>
>>>>> I was updating multiple columns using the same cas() call, and looking
>>>>> at the assert (code below) I added code to sort them before sending.
>>>>> However, now I'm seeing the same issue *even with single column*cas() calls.
>>>>>
>>>>> int c =
>>>>> internalComparator().compare(columns.get(getColumnCount() - 1).name(),
>>>>> column.name());
>>>>> // note that we want an assertion here (see addColumn
>>>>> javadoc), but we also want that if
>>>>> // assertion are disabled, addColumn works correctly with
>>>>> unsorted input
>>>>> assert c <= 0 : "Added column does not sort as the " +
>>>>> (reversed ? "first" : "last") + " column";
>>>>>
>>>>> I have also seen this error occur in the getRow() path, as below:
>>>>>
>>>>> java.lang.AssertionError: Added column does not sort as the last column
>>>>> at
>>>>> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
>>>>> at
>>>>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
>>>>> at
>>>>> org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:151)
>>>>> at
>>>>> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:95)
>>>>> at
>>>>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:57)
>>>>> at
>>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1452)
>>>>> at
>>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1281)
>>>>> at
>>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1195)
>>>>> at org.apache.cassandra.db.Table.getRow(Table.java:331)
>>>>> at
>>>>> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
>>>>> at
>>>>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1288)
>>>>> at
>>>>> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1813)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:722)
>>>>>
>>>>> Could someone provide me a little more clarity about what the context
>>>>> of the error is and what the client needs to do to fix it? I understand
>>>>> that running with assertions disabled would do away with this error, but
>>>>> i'd like to know if there is a proper way to fix this. Moreover, what could
>>>>> possibly be going wrong in the get path that is hitting this issue?
>>>>>
>>>>> Some guidance, on the fact that whether it is expected that the client
>>>>> sort the columns before calling the API with multiple columns, would be
>>>>> really appreciated.
>>>>>
>>>>> Thanks,
>>>>> Soumava
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Cassandra 2.0 - AssertionError in ArrayBackedSortedColumns
Posted by Sylvain Lebresne <sy...@datastax.com>.
Actually, your second stack trace is due to
https://issues.apache.org/jira/browse/CASSANDRA-5788.
On Mon, Jul 22, 2013 at 9:37 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
> unless data was stored in incorrect order and I don't know if that is
>> possible through the API..
>>
>
> That's exactly what happens and that's why I say it's a bug.
>
>
>>
>> ii. I am also seeing some sporadic cases where a single column cas()
>> update is hitting this same issue (the bug you mentioned) as multiple
>> columns, do you see any reason for that happening?
>>
>
> That would be slightly weirder, but frankly that's almost surely due to
> https://issues.apache.org/jira/browse/CASSANDRA-5786 too.
>
> --
> Sylvain
>
>
>> Thank you,
>> Soumava
>>
>>
>> On Mon, Jul 22, 2013 at 12:11 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>
>>> This is a bug really:
>>> https://issues.apache.org/jira/browse/CASSANDRA-5786.
>>>
>>> This should get fixed in the next beta of 2.0, but if you really want to
>>> test CAS updates in the meantime, you'll have to provide the columns in
>>> (column family comparator) sorted order to the thrift cas() method.
>>>
>>> --
>>> Sylvain
>>>
>>>
>>> On Sun, Jul 21, 2013 at 1:22 PM, Soumava Ghosh <so...@cs.utexas.edu>wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm taking a look at the Check and Set functionalities provided by the
>>>> cas() API provided by cassandra 2.0 (the code available on git). I'm
>>>> running a few tests on a small sized cluster (replication factor 3,
>>>> consistency level quorum) with a few clients. I've observed a lot of cases
>>>> seem to hit the TimedOutException while paxos is in progress. On inspection
>>>> of the server side logs the following stack was seen:
>>>>
>>>> ERROR [Thread-582] 2013-07-21 01:17:46,169 CassandraDaemon.java (line
>>>> 196) Exception in thread Thread[Thread-582,5,main]
>>>> java.lang.AssertionError: Added column does not sort as the last column
>>>> at
>>>> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
>>>> at
>>>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
>>>> at
>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:119)
>>>> at
>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96)
>>>> at
>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:91)
>>>> at
>>>> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:139)
>>>> at
>>>> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:128)
>>>> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
>>>> at
>>>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:175)
>>>> at
>>>> org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:135)
>>>> at
>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
>>>>
>>>> I was updating multiple columns using the same cas() call, and looking
>>>> at the assert (code below) I added code to sort them before sending.
>>>> However, now I'm seeing the same issue *even with single column* cas()
>>>> calls.
>>>>
>>>> int c =
>>>> internalComparator().compare(columns.get(getColumnCount() - 1).name(),
>>>> column.name());
>>>> // note that we want an assertion here (see addColumn javadoc),
>>>> but we also want that if
>>>> // assertion are disabled, addColumn works correctly with
>>>> unsorted input
>>>> assert c <= 0 : "Added column does not sort as the " +
>>>> (reversed ? "first" : "last") + " column";
>>>>
>>>> I have also seen this error occur in the getRow() path, as below:
>>>>
>>>> java.lang.AssertionError: Added column does not sort as the last column
>>>> at
>>>> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
>>>> at
>>>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
>>>> at
>>>> org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:151)
>>>> at
>>>> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:95)
>>>> at
>>>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:57)
>>>> at
>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1452)
>>>> at
>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1281)
>>>> at
>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1195)
>>>> at org.apache.cassandra.db.Table.getRow(Table.java:331)
>>>> at
>>>> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
>>>> at
>>>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1288)
>>>> at
>>>> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1813)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:722)
>>>>
>>>> Could someone provide me a little more clarity about what the context
>>>> of the error is and what the client needs to do to fix it? I understand
>>>> that running with assertions disabled would do away with this error, but
>>>> i'd like to know if there is a proper way to fix this. Moreover, what could
>>>> possibly be going wrong in the get path that is hitting this issue?
>>>>
>>>> Some guidance, on the fact that whether it is expected that the client
>>>> sort the columns before calling the API with multiple columns, would be
>>>> really appreciated.
>>>>
>>>> Thanks,
>>>> Soumava
>>>>
>>>
>>>
>>
>
Re: Cassandra 2.0 - AssertionError in ArrayBackedSortedColumns
Posted by Sylvain Lebresne <sy...@datastax.com>.
> unless data was stored in incorrect order and I don't know if that is
> possible through the API..
>
That's exactly what happens and that's why I say it's a bug.
>
> ii. I am also seeing some sporadic cases where a single column cas()
> update is hitting this same issue (the bug you mentioned) as multiple
> columns, do you see any reason for that happening?
>
That would be slightly weirder, but frankly that's almost surely due to
https://issues.apache.org/jira/browse/CASSANDRA-5786 too.
--
Sylvain
> Thank you,
> Soumava
>
>
> On Mon, Jul 22, 2013 at 12:11 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>> This is a bug really:
>> https://issues.apache.org/jira/browse/CASSANDRA-5786.
>>
>> This should get fixed in the next beta of 2.0, but if you really want to
>> test CAS updates in the meantime, you'll have to provide the columns in
>> (column family comparator) sorted order to the thrift cas() method.
>>
>> --
>> Sylvain
>>
>>
>> On Sun, Jul 21, 2013 at 1:22 PM, Soumava Ghosh <so...@cs.utexas.edu>wrote:
>>
>>> Hi,
>>>
>>> I'm taking a look at the Check and Set functionalities provided by the
>>> cas() API provided by cassandra 2.0 (the code available on git). I'm
>>> running a few tests on a small sized cluster (replication factor 3,
>>> consistency level quorum) with a few clients. I've observed a lot of cases
>>> seem to hit the TimedOutException while paxos is in progress. On inspection
>>> of the server side logs the following stack was seen:
>>>
>>> ERROR [Thread-582] 2013-07-21 01:17:46,169 CassandraDaemon.java (line
>>> 196) Exception in thread Thread[Thread-582,5,main]
>>> java.lang.AssertionError: Added column does not sort as the last column
>>> at
>>> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
>>> at
>>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
>>> at
>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:119)
>>> at
>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96)
>>> at
>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:91)
>>> at
>>> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:139)
>>> at
>>> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:128)
>>> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
>>> at
>>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:175)
>>> at
>>> org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:135)
>>> at
>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
>>>
>>> I was updating multiple columns using the same cas() call, and looking
>>> at the assert (code below) I added code to sort them before sending.
>>> However, now I'm seeing the same issue *even with single column* cas()
>>> calls.
>>>
>>> int c =
>>> internalComparator().compare(columns.get(getColumnCount() - 1).name(),
>>> column.name());
>>> // note that we want an assertion here (see addColumn javadoc),
>>> but we also want that if
>>> // assertion are disabled, addColumn works correctly with
>>> unsorted input
>>> assert c <= 0 : "Added column does not sort as the " + (reversed
>>> ? "first" : "last") + " column";
>>>
>>> I have also seen this error occur in the getRow() path, as below:
>>>
>>> java.lang.AssertionError: Added column does not sort as the last column
>>> at
>>> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
>>> at
>>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
>>> at
>>> org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:151)
>>> at
>>> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:95)
>>> at
>>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:57)
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1452)
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1281)
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1195)
>>> at org.apache.cassandra.db.Table.getRow(Table.java:331)
>>> at
>>> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
>>> at
>>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1288)
>>> at
>>> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1813)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:722)
>>>
>>> Could someone provide me a little more clarity about what the context of
>>> the error is and what the client needs to do to fix it? I understand that
>>> running with assertions disabled would do away with this error, but i'd
>>> like to know if there is a proper way to fix this. Moreover, what could
>>> possibly be going wrong in the get path that is hitting this issue?
>>>
>>> Some guidance, on the fact that whether it is expected that the client
>>> sort the columns before calling the API with multiple columns, would be
>>> really appreciated.
>>>
>>> Thanks,
>>> Soumava
>>>
>>
>>
>
Re: Cassandra 2.0 - AssertionError in ArrayBackedSortedColumns
Posted by Soumava Ghosh <so...@cs.utexas.edu>.
Thanks for the reply Sylvain! A couple of follow-up questions:
i. The second stack in my mail originated at a getRow() call. What could be
the cause of that? I am assuming data retrieval should not cause any
issues, unless data was stored in incorrect order and I don't know if that
is possible through the API..
ii. I am also seeing some sporadic cases where a single column cas() update
is hitting this same issue (the bug you mentioned) as multiple columns, do
you see any reason for that happening? I am yet to investigate this though.
Thank you,
Soumava
On Mon, Jul 22, 2013 at 12:11 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
> This is a bug really: https://issues.apache.org/jira/browse/CASSANDRA-5786
> .
>
> This should get fixed in the next beta of 2.0, but if you really want to
> test CAS updates in the meantime, you'll have to provide the columns in
> (column family comparator) sorted order to the thrift cas() method.
>
> --
> Sylvain
>
>
> On Sun, Jul 21, 2013 at 1:22 PM, Soumava Ghosh <so...@cs.utexas.edu>wrote:
>
>> Hi,
>>
>> I'm taking a look at the Check and Set functionalities provided by the
>> cas() API provided by cassandra 2.0 (the code available on git). I'm
>> running a few tests on a small sized cluster (replication factor 3,
>> consistency level quorum) with a few clients. I've observed a lot of cases
>> seem to hit the TimedOutException while paxos is in progress. On inspection
>> of the server side logs the following stack was seen:
>>
>> ERROR [Thread-582] 2013-07-21 01:17:46,169 CassandraDaemon.java (line
>> 196) Exception in thread Thread[Thread-582,5,main]
>> java.lang.AssertionError: Added column does not sort as the last column
>> at
>> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
>> at
>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
>> at
>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:119)
>> at
>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96)
>> at
>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:91)
>> at
>> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:139)
>> at
>> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:128)
>> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:175)
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:135)
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
>>
>> I was updating multiple columns using the same cas() call, and looking at
>> the assert (code below) I added code to sort them before sending.
>> However, now I'm seeing the same issue *even with single column* cas()
>> calls.
>>
>> int c = internalComparator().compare(columns.get(getColumnCount()
>> - 1).name(), column.name());
>> // note that we want an assertion here (see addColumn javadoc),
>> but we also want that if
>> // assertion are disabled, addColumn works correctly with
>> unsorted input
>> assert c <= 0 : "Added column does not sort as the " + (reversed
>> ? "first" : "last") + " column";
>>
>> I have also seen this error occur in the getRow() path, as below:
>>
>> java.lang.AssertionError: Added column does not sort as the last column
>> at
>> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
>> at
>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
>> at
>> org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:151)
>> at
>> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:95)
>> at
>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:57)
>> at
>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1452)
>> at
>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1281)
>> at
>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1195)
>> at org.apache.cassandra.db.Table.getRow(Table.java:331)
>> at
>> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
>> at
>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1288)
>> at
>> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1813)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> Could someone provide me a little more clarity about what the context of
>> the error is and what the client needs to do to fix it? I understand that
>> running with assertions disabled would do away with this error, but i'd
>> like to know if there is a proper way to fix this. Moreover, what could
>> possibly be going wrong in the get path that is hitting this issue?
>>
>> Some guidance, on the fact that whether it is expected that the client
>> sort the columns before calling the API with multiple columns, would be
>> really appreciated.
>>
>> Thanks,
>> Soumava
>>
>
>
Re: Cassandra 2.0 - AssertionError in ArrayBackedSortedColumns
Posted by Sylvain Lebresne <sy...@datastax.com>.
This is a bug really: https://issues.apache.org/jira/browse/CASSANDRA-5786.
This should get fixed in the next beta of 2.0, but if you really want to
test CAS updates in the meantime, you'll have to provide the columns in
(column family comparator) sorted order to the thrift cas() method.
--
Sylvain
On Sun, Jul 21, 2013 at 1:22 PM, Soumava Ghosh <so...@cs.utexas.edu>wrote:
> Hi,
>
> I'm taking a look at the Check and Set functionalities provided by the
> cas() API provided by cassandra 2.0 (the code available on git). I'm
> running a few tests on a small sized cluster (replication factor 3,
> consistency level quorum) with a few clients. I've observed a lot of cases
> seem to hit the TimedOutException while paxos is in progress. On inspection
> of the server side logs the following stack was seen:
>
> ERROR [Thread-582] 2013-07-21 01:17:46,169 CassandraDaemon.java (line 196)
> Exception in thread Thread[Thread-582,5,main]
> java.lang.AssertionError: Added column does not sort as the last column
> at
> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
> at
> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:119)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:91)
> at
> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:139)
> at
> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:128)
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:175)
> at
> org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:135)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
>
> I was updating multiple columns using the same cas() call, and looking at
> the assert (code below) I added code to sort them before sending.
> However, now I'm seeing the same issue *even with single column* cas()
> calls.
>
> int c = internalComparator().compare(columns.get(getColumnCount()
> - 1).name(), column.name());
> // note that we want an assertion here (see addColumn javadoc),
> but we also want that if
> // assertion are disabled, addColumn works correctly with unsorted
> input
> assert c <= 0 : "Added column does not sort as the " + (reversed ?
> "first" : "last") + " column";
>
> I have also seen this error occur in the getRow() path, as below:
>
> java.lang.AssertionError: Added column does not sort as the last column
> at
> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115)
> at
> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
> at
> org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:151)
> at
> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:95)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:57)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1452)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1281)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1195)
> at org.apache.cassandra.db.Table.getRow(Table.java:331)
> at
> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1288)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1813)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
>
> Could someone provide me a little more clarity about what the context of
> the error is and what the client needs to do to fix it? I understand that
> running with assertions disabled would do away with this error, but i'd
> like to know if there is a proper way to fix this. Moreover, what could
> possibly be going wrong in the get path that is hitting this issue?
>
> Some guidance, on the fact that whether it is expected that the client
> sort the columns before calling the API with multiple columns, would be
> really appreciated.
>
> Thanks,
> Soumava
>