[jira] [Comment Edited] (CASSANDRA-6998) HintedHandoff - expired hints may block future hints deliveries

["Muhammad Adel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979480#comment-13979480 ] 

Muhammad Adel edited comment on CASSANDRA-6998 at 4/24/14 9:03 AM:
-------------------------------------------------------------------

 Instead of ignoring tombstones when counting returned page size, and instead of ignoring irrelevant data, deliver both relevant and Irrelevant columns. This will increase data consistency when the recovering node was down for a time longer than the gc grace period.

To avoid performance problems, when choosing to return both relevant and irrelevant data, the slice query filter will count live, tombstones and irrelevant columns, compared to counting live columns only in the ordinary case. The check for tombstone threshold is ignored in this specific case.

Patch submitted.


was (Author: muhammadadel):
 Instead of ignoring tombstones when counting returned page size, and instead of ignoring irrelevant data, deliver both relevant and Irrelevant columns. This will increase data consistency when the recovering node was down for a time longer than the gc grace period.

To avoid performance problems, when choosing to return both relevant and irrelevant data, the slice query filter will count the tombstones and irrelevant columns, compared to counting live columns only in the ordinary case. The check for tombstone threshold is ignored in this specific case.

Patch submitted.

> HintedHandoff - expired hints may block future hints deliveries
> ---------------------------------------------------------------
>
>                 Key: CASSANDRA-6998
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6998
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: - cluster of two DCs: DC1, DC2
> - keyspace using NetworkTopologyStrategy (replication factors for both DCs)
> - heavy load (write:read, 100:1) with LOCAL_QUORUM using Java driver setup with DC awareness, writing to DC1
>            Reporter: Scooletz
>              Labels: HintedHandoff, TTL
>             Fix For: 2.0.3
>
>         Attachments: 6998
>
>
> For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled with millions of rows. Now, when _HintedHandOffManager_ tries to _doDeliverHintsToEndpoint_  it queries the store with QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws org.apache.cassandra.db.filter.TombstoneOverwhelmingException. 
> Throwing this exception stops the manager from running compaction as it is run only after successful handoff. This leaves the HH practically disabled till administrator runs truncateAllHints. 
> Wouldn't it be nicer if on org.apache.cassandra.db.filter.TombstoneOverwhelmingException run compaction? That would remove TTLed hints leaving whole HH mechanism in a healthy state.
> The stacktrace is:
> {quote}
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException
> 	at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
> 	at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
> 	at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
> 	at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
> 	at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
> 	at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
> 	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487)
> 	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306)
> 	at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
> 	at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
> 	at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
> 	at org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 	at java.lang.Thread.run(Thread.java:722)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.2#6252)