You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Denis Mikhaylov <no...@gmail.com> on 2016/10/28 10:30:54 UTC
Cassandra failure during read query at consistency QUORUM (2 responses were required but only 0 replica responded, 2 failed)
Hi!
We’re running Cassandra 3.9
On the application side I see failed reads with this exception com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency QUORUM (2 responses were required but only 0 replica responded, 2 failed)
On the server side we see:
WARN [SharedPool-Worker-3] 2016-10-28 13:28:22,965 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-3,5,
main]: {}
java.lang.AssertionError: null
at org.apache.cassandra.db.rows.BTreeRow.getCell(BTreeRow.java:212) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.SinglePartitionReadCommand.canRemoveRow(SinglePartitionReadCommand.java:899) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.SinglePartitionReadCommand.reduceFilter(SinglePartitionReadCommand.java:863) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:748) ~[apache-cassan
dra-3.7.jar:3.7]
at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:519) ~[apache-cassandra-3.7.jar:
3.7]
at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:496) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:358) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.7.jar:3.7]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-
3.7.jar:3.7]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache
-cassandra-3.7.jar:3.7]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.7.jar:3.7]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
It’s only affect single table. Sadly both on test (3.9) and production (3.7) deployments of cassandra.
What could be the problem? Please help.
Re: Cassandra failure during read query at consistency QUORUM (2 responses were required but only 0 replica responded, 2 failed)
Posted by Denis Mikhaylov <no...@gmail.com>.
Why does it prevent quorum? How to fix it?
> On 28 Oct 2016, at 16:02, Edward Capriolo <ed...@gmail.com> wrote:
>
> This looks like another case of an assert bubbling through try catch that don't catch assert
>
> On Fri, Oct 28, 2016 at 6:30 AM, Denis Mikhaylov <no...@gmail.com> wrote:
> Hi!
>
> We’re running Cassandra 3.9
>
> On the application side I see failed reads with this exception com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency QUORUM (2 responses were required but only 0 replica responded, 2 failed)
>
> On the server side we see:
>
> WARN [SharedPool-Worker-3] 2016-10-28 13:28:22,965 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-3,5,
> main]: {}
> java.lang.AssertionError: null
> at org.apache.cassandra.db.rows.BTreeRow.getCell(BTreeRow.java:212) ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.db.SinglePartitionReadCommand.canRemoveRow(SinglePartitionReadCommand.java:899) ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.db.SinglePartitionReadCommand.reduceFilter(SinglePartitionReadCommand.java:863) ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:748) ~[apache-cassan
> dra-3.7.jar:3.7]
> at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:519) ~[apache-cassandra-3.7.jar:
> 3.7]
> at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:496) ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:358) ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366) ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48) ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.7.jar:3.7]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-
> 3.7.jar:3.7]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache
> -cassandra-3.7.jar:3.7]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.7.jar:3.7]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>
> It’s only affect single table. Sadly both on test (3.9) and production (3.7) deployments of cassandra.
>
> What could be the problem? Please help.
>
Re: Cassandra failure during read query at consistency QUORUM (2
responses were required but only 0 replica responded, 2 failed)
Posted by Edward Capriolo <ed...@gmail.com>.
This looks like another case of an assert bubbling through try catch that
don't catch assert
On Fri, Oct 28, 2016 at 6:30 AM, Denis Mikhaylov <no...@gmail.com> wrote:
> Hi!
>
> We’re running Cassandra 3.9
>
> On the application side I see failed reads with this exception
> com.datastax.driver.core.exceptions.ReadFailureException: Cassandra
> failure during read query at consistency QUORUM (2 responses were required
> but only 0 replica responded, 2 failed)
>
> On the server side we see:
>
> WARN [SharedPool-Worker-3] 2016-10-28 13:28:22,965
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-3,5,
> main]: {}
> java.lang.AssertionError: null
> at org.apache.cassandra.db.rows.BTreeRow.getCell(BTreeRow.java:212)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.db.SinglePartitionReadCommand.
> canRemoveRow(SinglePartitionReadCommand.java:899)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.db.SinglePartitionReadCommand.
> reduceFilter(SinglePartitionReadCommand.java:863)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.db.SinglePartitionReadCommand.
> queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:748)
> ~[apache-cassan
> dra-3.7.jar:3.7]
> at org.apache.cassandra.db.SinglePartitionReadCommand.
> queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:519)
> ~[apache-cassandra-3.7.jar:
> 3.7]
> at org.apache.cassandra.db.SinglePartitionReadCommand.
> queryMemtableAndDisk(SinglePartitionReadCommand.java:496)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.db.SinglePartitionReadCommand.
> queryStorage(SinglePartitionReadCommand.java:358)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366)
> ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(
> ReadCommandVerbHandler.java:48) ~[apache-cassandra-3.7.jar:3.7]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.7.jar:3.7]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_102]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ
> ice$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-
> 3.7.jar:3.7]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ
> ice$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache
> -cassandra-3.7.jar:3.7]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.7.jar:3.7]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>
> It’s only affect single table. Sadly both on test (3.9) and production
> (3.7) deployments of cassandra.
>
> What could be the problem? Please help.