You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jeffrey Wang (JIRA)" <ji...@apache.org> on 2011/03/10 05:39:59 UTC
[jira] Created: (CASSANDRA-2305) Tombstoned CFs not reset after
compaction
Tombstoned CFs not reset after compaction
-----------------------------------------
Key: CASSANDRA-2305
URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.0
Reporter: Jeffrey Wang
>From email to list:
I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
insert row X with timestamp T
delete row X with timestamp T+1
force flush + compaction
insert row X with timestamp T
My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2305) Tombstoned rows not purged from
cache after gcgraceseconds
Posted by "Jeffrey Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005834#comment-13005834 ]
Jeffrey Wang commented on CASSANDRA-2305:
-----------------------------------------
I actually don't have row cache enabled (I just checked cfstats to make sure), so I don't think that's the cause of my problem in particular. Here's some more info that may or may not be correct:
- When I run the compaction, in ColumnFamilyStore.removeDeletedStandard() I see that columns are being removed because of the c.timestamp() <= cf.getMarkedForDeleteAt() condition, which makes sense since I issued a delete on the entire row.
- However, after the compaction, I do the insert, and if I flush/compact again, I still see the columns being removed because of that condition. It seems like the markedForDeleteAt field on the ColumnFamily is persisting across the major compaction which I believe is hiding the newly inserted column.
Also, my initial steps to repro were not correct, which made it hard to figure out the root cause. Here is a proper repro:
- Create a CF with gc_grace_seconds = 0 and no row cache.
- Insert row X, col A with timestamp 0.
- Insert row X, col B with timestamp 2.
- Remove row X with timestamp 1 (expect col A to disappear, col B to stay).
- Wait 1 second.
- Force flush and compaction.
- Insert row X, col A with timestamp 0.
- Read row X, col A (see nothing).
Inserting row X, col B is necessary for this to repro because if all the columns in a row disappear, the ColumnFamily object goes away and the markedForDeleteAt field is reset. Only when a column still exists does the field persist across the compaction. Hope this helps!
> Tombstoned rows not purged from cache after gcgraceseconds
> ----------------------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
> Attachments: 0001-Compaction-test.patch, 0002-Invalidate-row-cache-on-compaction-purge.patch
>
> Original Estimate: 2h
> Time Spent: 2h
> Remaining Estimate: 0h
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Reopened: (CASSANDRA-2305) Tombstoned rows not purged from
cache after gcgraceseconds
Posted by "Jeffrey Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Wang reopened CASSANDRA-2305:
-------------------------------------
I believe the cause to be something else (see latest comment).
> Tombstoned rows not purged from cache after gcgraceseconds
> ----------------------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
> Attachments: 0001-Compaction-test.patch, 0002-Invalidate-row-cache-on-compaction-purge.patch
>
> Original Estimate: 2h
> Time Spent: 2h
> Remaining Estimate: 0h
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2305) Tombstoned rows not purged from
cache after gcgraceseconds
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018136#comment-13018136 ]
Hudson commented on CASSANDRA-2305:
-----------------------------------
Integrated in Cassandra-0.7 #429 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/429/])
Purge tombstone from row cache (0.7 version)
patch by slebresne; reviewed by jbellis for CASSANDRA-2305
> Tombstoned rows not purged from cache after gcgraceseconds
> ----------------------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.5
>
> Attachments: 0001-Compaction-test.patch, 0002-Invalidate-row-cache-on-compaction-purge.patch, 2305_2nd_patch.patch
>
> Original Estimate: 2h
> Time Spent: 2h
> Remaining Estimate: 0h
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2305) Tombstoned CFs not reset after
compaction
Posted by "Jeffrey Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004960#comment-13004960 ]
Jeffrey Wang commented on CASSANDRA-2305:
-----------------------------------------
For a little more info, I think this only happens when you remove an entire row. If you delete specific columns, the tombstones are handled appropriately.
> Tombstoned CFs not reset after compaction
> -----------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2305) Tombstoned rows not purged after
gcgraceseconds
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2305:
--------------------------------------
Priority: Minor (was: Major)
Fix Version/s: 0.7.4
Assignee: Sylvain Lebresne
Summary: Tombstoned rows not purged after gcgraceseconds (was: Tombstoned CFs not reset after compaction)
> Tombstoned rows not purged after gcgraceseconds
> -----------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2305) Tombstoned rows not purged after
gcgraceseconds
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2305:
----------------------------------------
Remaining Estimate: 2h
Original Estimate: 2h
> Tombstoned rows not purged after gcgraceseconds
> -----------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
> Attachments: 0001-Compaction-test.patch, 0002-Invalidate-row-cache-on-compaction-purge.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2305) Tombstoned rows not purged from
cache after gcgraceseconds
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2305:
----------------------------------------
Attachment: 2305_2nd_patch.patch
The first patch was purging the cache when the full row is expired. But we don't remove expired tombstone from the cache.
Attaching a second patch to purge the cache. This has two purposes:
# avoid surprise for client getting tombstones back from query (either sstable2json or rangeSlice) even well after gc_grace and compaction has occured
# reclaim some memory for row sitting in the cache for a very long time
Note that this patch introduces concurrent deletes, and as such SHOULDN'T be applied to a branch that do not have CASSANDRA-1559 (0.7 and 0.8 have it).
Patch is against 0.7
> Tombstoned rows not purged from cache after gcgraceseconds
> ----------------------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
> Attachments: 0001-Compaction-test.patch, 0002-Invalidate-row-cache-on-compaction-purge.patch, 2305_2nd_patch.patch
>
> Original Estimate: 2h
> Time Spent: 2h
> Remaining Estimate: 0h
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2305) Tombstoned rows not purged from
cache after gcgraceseconds
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017673#comment-13017673 ]
Jonathan Ellis commented on CASSANDRA-2305:
-------------------------------------------
+1
> Tombstoned rows not purged from cache after gcgraceseconds
> ----------------------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.5
>
> Attachments: 0001-Compaction-test.patch, 0002-Invalidate-row-cache-on-compaction-purge.patch, 2305_2nd_patch.patch
>
> Original Estimate: 2h
> Time Spent: 2h
> Remaining Estimate: 0h
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2305) Tombstoned rows not purged after
gcgraceseconds
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005228#comment-13005228 ]
Jonathan Ellis commented on CASSANDRA-2305:
-------------------------------------------
bq. The problem is that when you flush + compact and gcGrace has elapsed, if the row is empty (i.e. all tombstone have been collected), the row itself is collected. This means that when you issue the second wave of inserts, there is no trace whatsoever of the row.
That's what's supposed to happen, but Jeffrey is saying that is NOT what he observes.
> Tombstoned rows not purged after gcgraceseconds
> -----------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2305) Tombstoned rows not purged from
cache after gcgraceseconds
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018171#comment-13018171 ]
Hudson commented on CASSANDRA-2305:
-----------------------------------
Integrated in Cassandra #846 (See [https://hudson.apache.org/hudson/job/Cassandra/846/])
> Tombstoned rows not purged from cache after gcgraceseconds
> ----------------------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.5
>
> Attachments: 0001-Compaction-test.patch, 0002-Invalidate-row-cache-on-compaction-purge.patch, 2305_2nd_patch.patch
>
> Original Estimate: 2h
> Time Spent: 2h
> Remaining Estimate: 0h
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2305) Tombstoned rows not purged from
cache after gcgraceseconds
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006030#comment-13006030 ]
Sylvain Lebresne commented on CASSANDRA-2305:
---------------------------------------------
Ok, I understand what you meant. I've created CASSANDRA-2317 with the fix since we have already committed a patch here. Thanks a lot for the report.
> Tombstoned rows not purged from cache after gcgraceseconds
> ----------------------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
> Attachments: 0001-Compaction-test.patch, 0002-Invalidate-row-cache-on-compaction-purge.patch
>
> Original Estimate: 2h
> Time Spent: 2h
> Remaining Estimate: 0h
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Reopened: (CASSANDRA-2305) Tombstoned rows not purged from
cache after gcgraceseconds
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne reopened CASSANDRA-2305:
-----------------------------------------
I'm reopening because the committed patch, while ok, is only a partial fix.
> Tombstoned rows not purged from cache after gcgraceseconds
> ----------------------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
> Attachments: 0001-Compaction-test.patch, 0002-Invalidate-row-cache-on-compaction-purge.patch
>
> Original Estimate: 2h
> Time Spent: 2h
> Remaining Estimate: 0h
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2305) Tombstoned rows not purged after
gcgraceseconds
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2305:
----------------------------------------
Attachment: 0002-Invalidate-row-cache-on-compaction-purge.patch
0001-Compaction-test.patch
I think this is due to row cache. We do not invalidate the row cache when a row is fully collected by compaction.
Jeffrey, can you confirm that you had some row cache enabled when doing your experiments ?
Attaching 2 patch against 0.7. The first one is a unit test showing the failure, the second one is the fix.
> Tombstoned rows not purged after gcgraceseconds
> -----------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
> Attachments: 0001-Compaction-test.patch, 0002-Invalidate-row-cache-on-compaction-purge.patch
>
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2305) Tombstoned rows not purged after
gcgraceseconds
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005236#comment-13005236 ]
Sylvain Lebresne commented on CASSANDRA-2305:
---------------------------------------------
Oups, my mistake. I somehow confused myself. I was not able to reproduce though, but I'll try harder tomorrow.
> Tombstoned rows not purged after gcgraceseconds
> -----------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2305) Tombstoned rows not purged from
cache after gcgraceseconds
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005884#comment-13005884 ]
Hudson commented on CASSANDRA-2305:
-----------------------------------
Integrated in Cassandra-0.7 #375 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/375/])
> Tombstoned rows not purged from cache after gcgraceseconds
> ----------------------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
> Attachments: 0001-Compaction-test.patch, 0002-Invalidate-row-cache-on-compaction-purge.patch
>
> Original Estimate: 2h
> Time Spent: 2h
> Remaining Estimate: 0h
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (CASSANDRA-2305) Tombstoned rows not purged from
cache after gcgraceseconds
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne resolved CASSANDRA-2305.
-----------------------------------------
Resolution: Fixed
Remarking this resolved, the follow up is in CASSANDRA-2317 instead.
> Tombstoned rows not purged from cache after gcgraceseconds
> ----------------------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
> Attachments: 0001-Compaction-test.patch, 0002-Invalidate-row-cache-on-compaction-purge.patch
>
> Original Estimate: 2h
> Time Spent: 2h
> Remaining Estimate: 0h
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2305) Tombstoned rows not purged after
gcgraceseconds
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005211#comment-13005211 ]
Sylvain Lebresne commented on CASSANDRA-2305:
---------------------------------------------
This will also happen if you remove all the columns inside the row (even though you didn't issued a row deletion command).
The problem is that when you flush + compact and gcGrace has elapsed, if the row is empty (i.e. all tombstone have been collected), the row itself is collected. This means that when you issue the second wave of inserts, there is no trace whatsoever of the row.
That's why you are not supposed to have a gcGrace too low and why it is highly advised to use the current time as a timestamp. If so, the scenario above will never happen.
Best thing we can do is probably to edit http://wiki.apache.org/cassandra/DistributedDeletes to add that gcGrace should be such that no insert with a timestamp lower that a delete could reach any given node after gcGrace has elapsed.
> Tombstoned rows not purged after gcgraceseconds
> -----------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2305) Tombstoned rows not purged from
cache after gcgraceseconds
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2305:
----------------------------------------
Summary: Tombstoned rows not purged from cache after gcgraceseconds (was: Tombstoned rows not purged after gcgraceseconds)
> Tombstoned rows not purged from cache after gcgraceseconds
> ----------------------------------------------------------
>
> Key: CASSANDRA-2305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2305
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Jeffrey Wang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.4
>
> Attachments: 0001-Compaction-test.patch, 0002-Invalidate-row-cache-on-compaction-purge.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> From email to list:
> I was wondering if this is the expected behavior of deletes (0.7.0). Let's say I have a 1-node cluster with a single CF which has gc_grace_seconds = 0. The following sequence of operations happens (in the given order):
> insert row X with timestamp T
> delete row X with timestamp T+1
> force flush + compaction
> insert row X with timestamp T
> My understanding is that the tombstone created by the delete (and row X) will disappear with the flush + compaction which means the last insertion should show up. My experimentation, however, suggests otherwise (the last insertion does not show up).
> I believe I have traced this to the fact that the markedForDeleteAt field on the ColumnFamily does not get reset after a compaction (after gc_grace_seconds has passed); is this desirable? I think it introduces an inconsistency in how tombstoned columns work versus tombstoned CFs. Thanks.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira