[jira] [Comment Edited] (CASSANDRA-7088) Zero length BigInteger exception causing processing to freeze

["Jacek Furmankiewicz (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981248#comment-13981248 ] 

Jacek Furmankiewicz edited comment on CASSANDRA-7088 at 4/25/14 5:05 PM:
-------------------------------------------------------------------------

Hi Brandon, we have good news. I am not sure if you still want to treat it as an issue or not.

The core underlying problem is that we had a mismatch between a composite column type in our Astyanax code (long,long,String) 
vs how the the actual CF was created (CompositeType(LongType,LongType,IntegerType)).

Now one ever noticed because it worked just fine in 1.2 (how I am not sure....it should have bombed on the first query....I suspect Astyanax did some raw binary serialization that somehow hid this underlying problem).

However, once we moved to 2.0, Cassandra must have detected that something does not make sense in the query vs the underlying data representation and hence the zero length BigInteger error.

So it is definitely an issue on our side.

The only potential issue on the Cassandra side is that this error caused the entire query to freeze  until we got an operation timeout exception, which sort of hid the underlying issue for a while.

I will leave  it up to you if you want to leave that particular issue open for further investigation on your side or close it.

Thank you for your help along the way.


was (Author: jfurmankiewicz):
Hi Brandon, we have good news. I am not sure if you still want to treat it as an issue or not.

The core underlying problem is that we had a mismatch between a composite column type in our Astyanax code (long,long,String) 
vs how the the actual CF was created (CompositeType(LongType,LongType,IntegerType)).

Now one ever noticed because it worked just fine in 1.2 (how I am not sure....it should have bombed on the first query....I suspect Astyanax did some raw binary serialization that somehow hid this underlying problem).

However, once we moved to 2.0, Cassandra must have detected that something does not make sense in the query vs the underlying data representation and hence the zero length BigInteger error.

So it is definitely an issue on our side.

The only potential issue on the Cassandra side is that this error caused the entire query to freeze  until we got an operation timeout exception, which sort of hid the underlying issue for a while.

I will leave  it up to you if you want to leave the issue open for further investigation on your side or close it.

Thank you for your help along the way.

>  Zero length BigInteger exception causing processing to freeze
> --------------------------------------------------------------
>
>                 Key: CASSANDRA-7088
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7088
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Linux Mint 16
>            Reporter: Jacek Furmankiewicz
>         Attachments: cassandra.yaml
>
>
> We attempted to migrate our developers to Cassandra 2.0.7 from 1.2.
> Everything worked perfectly, but we have experienced a massive drop in developer velocity.
> We run integration tests with Cucumber BDD and 1000 BDDs went from 7 minutes (Cassandra 1.2) to 15 minutes (2.0.7),
> This is when we run Cassandra of the ramdisk (/dev/shm) to make it run faster on dev boxes.
> When we tried pointed to actual drives  the difference was dramatic: the entire suite took over 70 minutes (!) vs 15 in Cassandra 1.2.
> After investigation, we found that most of the time is spent in the truncation logic between every scenario, where we truncate all the column families and start with a clean DB for the next test case.
> This used to be super fast in 1.2, is now very slow in 2.0.
> It may not seem important, but upgrading to 2.0 has basically cut down developer velocity by 100%, just by more than doubling the time it takes to run our BDD suite.
> We truncate the CFs using the Ruby driver:
>   $cassandra.column_families.each do |column_family|
>         name = column_family[0].to_s
>         $cassandra.truncate! name
>   end
> I am attaching our cassandra.yaml. Please note we already switched off auto_compaction before truncate, just as we did in 1.2 for dev boxes, Made no difference.



--
This message was sent by Atlassian JIRA
(v6.2#6252)