You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jeff Liu (JIRA)" <ji...@apache.org> on 2015/03/03 19:34:07 UTC
[jira] [Commented] (CASSANDRA-8870) Tombstone overwhelming issue
aborts client queries
[ https://issues.apache.org/jira/browse/CASSANDRA-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14345461#comment-14345461 ]
Jeff Liu commented on CASSANDRA-8870:
-------------------------------------
Another question I have been curious about is that why we would see those tombstone errors. in our application, we are doing insert and update only. Will update ops generate tombstones?
> Tombstone overwhelming issue aborts client queries
> --------------------------------------------------
>
> Key: CASSANDRA-8870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8870
> Project: Cassandra
> Issue Type: Bug
> Environment: cassandra 2.1.2 ubunbtu 12.04
> Reporter: Jeff Liu
>
> We are getting client queries timeout issues on the clients who are trying to query data from cassandra cluster.
> Nodetool status shows that all nodes are still up regardless.
> Logs from client side:
> {noformat}
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cass-chisel01.abc01.abc02.abc.abc.com/10.66.182.113:9042 (com.datastax.driver.core.TransportException: [cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042] Connection has been closed))
> at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108) ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
> at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179) ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
> {noformat}
> Logs from cassandra/system.log
> {noformat}
> ERROR [HintedHandoff:2] 2015-02-23 23:46:28,410 SliceQueryFilter.java:212 - Scanned over 100000 tombstones in system.hints; query aborted (see tombstone_failure_threshold)
> ERROR [HintedHandoff:2] 2015-02-23 23:46:28,417 CassandraDaemon.java:153 - Exception in thread Thread[HintedHandoff:2,1,main]
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException: null
> at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:310) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1858) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1666) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:385) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:344) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:94) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:555) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)