You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Roland Gude <ro...@yoochoose.com> on 2011/08/09 17:33:51 UTC

IndexSliceQuery issue - ReadStage piling up (looks like deadlock/infinite loop or similar)

Hi,

I experience issues when doing a indexslicequery with multiple expressions if one of the expressions is about a non index column

I did the equivalent of this example (but with my data) from
http://www.datastax.com/dev/blog/whats-new-cassandra-07-secondary-indexes


Secondary indexes automate this. Let's add some state data:

[default@demo] set users[bsanderson][state] = 'UT';
[default@demo] set users[prothfuss][state] = 'WI';
[default@demo] set users[htayler][state] = 'UT';

Note that even though state is not indexed yet, we can include the new state data in a query as long as another column in the query is indexed:

[default@demo] get users where state = 'UT';
No indexed columns present in index clause with operator EQ
[default@demo] get users where state = 'UT' and birth_date > 1970;
No indexed columns present in index clause with operator EQ
[default@demo]get users where birth_date = 1968 and state = 'UT';
-------------------
RowKey: htayler
=> (column=birth_date, value=1968, timestamp=1291334765649000)
=> (column=full_name, value=Howard Tayler, timestamp=1291334749160000)
=> (column=state, value=5554, timestamp=1291334890708000)

On On 0.7.8 (with CASSANDRA-2964 applied)
This example will not return any data, but return "0 rows". I repeated the query multiple times with different variations for the values which should all have returned data, but eventually I ended up with the cluster having 80000 reads pending on some of the nodes

On 0.7.5 the query will result in a NullPointerException being thrown and "null" returned in the cli

ERROR [ReadStage:258] 2011-08-09 16:03:27,153 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[ReadStage:258,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:51)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException
        at org.apache.cassandra.db.ColumnFamily.addAll(ColumnFamily.java:131)
        at org.apache.cassandra.db.ColumnFamilyStore.scan(ColumnFamilyStore.java:1615)
        at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:42)
        ... 4 more
ERROR [ReadStage:258] 2011-08-09 16:03:27,153 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[ReadStage:258,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:51)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException
        at org.apache.cassandra.db.ColumnFamily.addAll(ColumnFamily.java:131)
        at org.apache.cassandra.db.ColumnFamilyStore.scan(ColumnFamilyStore.java:1615)
        at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:42)
        ... 4 more



Can anybody reproduce this?

Greetings,
Roland

--
YOOCHOOSE GmbH

Roland Gude
Software Engineer

Im Mediapark 8, 50670 Köln

+49 221 4544151 (Tel)
+49 221 4544159 (Fax)
+49 171 7894057 (Mobil)


Email: roland.gude@yoochoose.com
WWW: www.yoochoose.com<http://www.yoochoose.com/>

YOOCHOOSE GmbH
Geschäftsführer: Dr. Uwe Alkemper, Michael Friedmann
Handelsregister: Amtsgericht Köln HRB 65275
Ust-Ident-Nr: DE 264 773 520
Sitz der Gesellschaft: Köln

AW: IndexSliceQuery issue - ReadStage piling up (looks like deadlock/infinite loop or similar)

Posted by Roland Gude <ro...@yoochoose.com>.
Yes, i can reproduce this behavior

If issue a query like this (on 0.7.8 with patch for CASSANDRA-2964 applied)
[default@demo]get users where birth_date = 1968 and state = 'UT';
with an index on birth_date but no index on state
I do not get results (actually I get '0 rows') even though there are rows which statisfy all clauses.
However, if I repeat this several times several of the nodes start piling up pending reads. (tpstats shows some 80000 reads pending)
And even though the nodes are not able to fulfill (read) requests anymore, they are not marked as down by the gossiper. Overall this results in an unusable cluster.

If I do the same thing on a 0.7.5 cluster
Cassandra logs a nullpointerexception and the cli returns with null, but the cluster stays functional.


Von: aaron morton [mailto:aaron@thelastpickle.com]
Gesendet: Mittwoch, 10. August 2011 23:48
An: user@cassandra.apache.org
Betreff: Re: IndexSliceQuery issue - ReadStage piling up (looks like deadlock/infinite loop or similar)

Are you still having a problem ? I'm a bit confused about what you saying.

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 10 Aug 2011, at 03:33, Roland Gude wrote:


Hi,

I experience issues when doing a indexslicequery with multiple expressions if one of the expressions is about a non index column

I did the equivalent of this example (but with my data) from
http://www.datastax.com/dev/blog/whats-new-cassandra-07-secondary-indexes


Secondary indexes automate this. Let's add some state data:

[default@demo] set users[bsanderson][state] = 'UT';
[default@demo] set users[prothfuss][state] = 'WI';
[default@demo] set users[htayler][state] = 'UT';

Note that even though state is not indexed yet, we can include the new state data in a query as long as another column in the query is indexed:

[default@demo] get users where state = 'UT';
No indexed columns present in index clause with operator EQ
[default@demo] get users where state = 'UT' and birth_date > 1970;
No indexed columns present in index clause with operator EQ
[default@demo]get users where birth_date = 1968 and state = 'UT';
-------------------
RowKey: htayler
=> (column=birth_date, value=1968, timestamp=1291334765649000)
=> (column=full_name, value=Howard Tayler, timestamp=1291334749160000)
=> (column=state, value=5554, timestamp=1291334890708000)

On On 0.7.8 (with CASSANDRA-2964 applied)
This example will not return any data, but return "0 rows". I repeated the query multiple times with different variations for the values which should all have returned data, but eventually I ended up with the cluster having 80000 reads pending on some of the nodes

On 0.7.5 the query will result in a NullPointerException being thrown and "null" returned in the cli

ERROR [ReadStage:258] 2011-08-09 16:03:27,153 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[ReadStage:258,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:51)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException
        at org.apache.cassandra.db.ColumnFamily.addAll(ColumnFamily.java:131)
        at org.apache.cassandra.db.ColumnFamilyStore.scan(ColumnFamilyStore.java:1615)
        at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:42)
        ... 4 more
ERROR [ReadStage:258] 2011-08-09 16:03:27,153 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[ReadStage:258,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:51)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException
        at org.apache.cassandra.db.ColumnFamily.addAll(ColumnFamily.java:131)
        at org.apache.cassandra.db.ColumnFamilyStore.scan(ColumnFamilyStore.java:1615)
        at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:42)
        ... 4 more



Can anybody reproduce this?

Greetings,
Roland

--
YOOCHOOSE GmbH

Roland Gude
Software Engineer

Im Mediapark 8, 50670 Köln

+49 221 4544151 (Tel)
+49 221 4544159 (Fax)
+49 171 7894057 (Mobil)


Email: roland.gude@yoochoose.com<ma...@yoochoose.com>
WWW: www.yoochoose.com<http://www.yoochoose.com/>

YOOCHOOSE GmbH
Geschäftsführer: Dr. Uwe Alkemper, Michael Friedmann
Handelsregister: Amtsgericht Köln HRB 65275
Ust-Ident-Nr: DE 264 773 520
Sitz der Gesellschaft: Köln

Re: IndexSliceQuery issue - ReadStage piling up (looks like deadlock/infinite loop or similar)

Posted by aaron morton <aa...@thelastpickle.com>.
Are you still having a problem ? I'm a bit confused about what you saying. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 10 Aug 2011, at 03:33, Roland Gude wrote:

> Hi,
>  
> I experience issues when doing a indexslicequery with multiple expressions if one of the expressions is about a non index column
>  
> I did the equivalent of this example (but with my data) from
> http://www.datastax.com/dev/blog/whats-new-cassandra-07-secondary-indexes
>  
> Secondary indexes automate this. Let’s add some state data:
> 
> [default@demo] set users[bsanderson][state] = 'UT';
> [default@demo] set users[prothfuss][state] = 'WI';
> [default@demo] set users[htayler][state] = 'UT';
> 
> Note that even though state is not indexed yet, we can include the new state data in a query as long as another column in the query is indexed:
> 
> [default@demo] get users where state = 'UT';
> No indexed columns present in index clause with operator EQ
> [default@demo] get users where state = 'UT' and birth_date > 1970;
> No indexed columns present in index clause with operator EQ
> [default@demo]get users where birth_date = 1968 and state = 'UT';
> -------------------
> RowKey: htayler
> => (column=birth_date, value=1968, timestamp=1291334765649000)
> => (column=full_name, value=Howard Tayler, timestamp=1291334749160000)
> => (column=state, value=5554, timestamp=1291334890708000)
> 
>  
> On On 0.7.8 (with CASSANDRA-2964 applied)
> This example will not return any data, but return “0 rows”. I repeated the query multiple times with different variations for the values which should all have returned data, but eventually I ended up with the cluster having 80000 reads pending on some of the nodes
>  
> On 0.7.5 the query will result in a NullPointerException being thrown and “null” returned in the cli
>  
> ERROR [ReadStage:258] 2011-08-09 16:03:27,153 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[ReadStage:258,5,main]
> java.lang.RuntimeException: java.lang.NullPointerException
>         at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:51)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.NullPointerException
>         at org.apache.cassandra.db.ColumnFamily.addAll(ColumnFamily.java:131)
>         at org.apache.cassandra.db.ColumnFamilyStore.scan(ColumnFamilyStore.java:1615)
>         at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:42)
>         ... 4 more
> ERROR [ReadStage:258] 2011-08-09 16:03:27,153 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[ReadStage:258,5,main]
> java.lang.RuntimeException: java.lang.NullPointerException
>         at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:51)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.NullPointerException
>         at org.apache.cassandra.db.ColumnFamily.addAll(ColumnFamily.java:131)
>         at org.apache.cassandra.db.ColumnFamilyStore.scan(ColumnFamilyStore.java:1615)
>         at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:42)
>         ... 4 more
>  
>  
>  
> Can anybody reproduce this?
>  
> Greetings,
> Roland
>  
> --
> YOOCHOOSE GmbH
>  
> Roland Gude
> Software Engineer
>  
> Im Mediapark 8, 50670 Köln
>  
> +49 221 4544151 (Tel)
> +49 221 4544159 (Fax)
> +49 171 7894057 (Mobil)
>  
>  
> Email: roland.gude@yoochoose.com
> WWW: www.yoochoose.com
>  
> YOOCHOOSE GmbH
> Geschäftsführer: Dr. Uwe Alkemper, Michael Friedmann
> Handelsregister: Amtsgericht Köln HRB 65275
> Ust-Ident-Nr: DE 264 773 520
> Sitz der Gesellschaft: Köln
>