You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Aravindan T <ar...@tcs.com> on 2014/08/22 23:47:11 UTC
.TombstoneOverwhelmingException
Dear All,
COuld you please help on how to resolve the below issue?
1. Only insertion of records are done since the cassandra started. But, when looked into the system.log messages, i see this following error.
ERROR [HintedHandoff:15] 2014-08-22 22:35:17,629 SliceQueryFilter.java (line 200) Scanned over 200000 tombstones in system.hints; query aborted (see tombstone_fail_threshold)
ERROR [HintedHandoff:15] 2014-08-22 22:35:17,630 CassandraDaemon.java (line 196) Exception in thread Thread[HintedHandoff:15,1,main]
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:202)
at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1551)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1380)
at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:374)
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:331)
at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
at org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:558)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Now, could you please tell how the tombstone got created when there are no deletes executed?
2. How and where to monitor the current tombstone counts ?
3. done compaction, nodetool repair, increased the threshold count but still this tombstone is not getting resolved. How to resolve this issue?
Thanks
Aravindan Thangavelu
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
Re: .TombstoneOverwhelmingException
Posted by Aravindan T <ar...@tcs.com>.
Thanks Robert.
I guess the issue is resolved now after doing the mentioned step. Can you tell me how to verify the current tombstone counts in order to do a regular check?
Thanks
Aravindan Thangavelu
-----Robert Coli <rc...@eventbrite.com> wrote: -----
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
From: Robert Coli <rc...@eventbrite.com>
Date: 08/23/2014 03:30AM
Subject: Re: .TombstoneOverwhelmingException
On Fri, Aug 22, 2014 at 2:47 PM, Aravindan T <ar...@tcs.com> wrote:
1. Only insertion of records are done since the cassandra started. But, when looked into the system.log messages, i see this following error.
...
Now, could you please tell how the tombstone got created when there are no deletes executed
You stored a bunch of hints via hinted handoff. Then they got tombstoned.
2. How and where to monitor the current tombstone counts ?
3. done compaction, nodetool repair, increased the threshold count but still this tombstone is not getting resolved. How to resolve this issue?
Either major compact the hints columnfamily (might not be done by nodetool compact because it's system keyspace) or :
1) nodetool drain
2) stop node
3) delete all sstables in system/hintscolumnfamily
4) start node
5) repair
You should also figure out why you're storing so many hints, and stop doing so.
=Rob
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
Re: .TombstoneOverwhelmingException
Posted by Robert Coli <rc...@eventbrite.com>.
On Fri, Aug 22, 2014 at 2:47 PM, Aravindan T <ar...@tcs.com> wrote:
> 1. Only insertion of records are done since the cassandra started. But,
> when looked into the system.log messages, i see this following error.
>
...
> Now, could you please tell how the tombstone got created when there are no
> deletes executed
>
You stored a bunch of hints via hinted handoff. Then they got tombstoned.
> 2. How and where to monitor the current tombstone counts ?
>
> 3. done compaction, nodetool repair, increased the threshold count but
> still this tombstone is not getting resolved. How to resolve this issue?
>
Either major compact the hints columnfamily (might not be done by nodetool
compact because it's system keyspace) or :
1) nodetool drain
2) stop node
3) delete all sstables in system/hintscolumnfamily
4) start node
5) repair
You should also figure out why you're storing so many hints, and stop doing
so.
=Rob