You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (Created) (JIRA)" <ji...@apache.org> on 2012/02/16 00:57:00 UTC
[jira] [Created] (CASSANDRA-3921) Compaction doesn't clear out
expired tombstones from SerializingCache
Compaction doesn't clear out expired tombstones from SerializingCache
---------------------------------------------------------------------
Key: CASSANDRA-3921
URL: https://issues.apache.org/jira/browse/CASSANDRA-3921
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Jonathan Ellis
Priority: Minor
Fix For: 1.1.0
Compaction calls removeDeletedInCache, which looks like this:
{code}
. public void removeDeletedInCache(DecoratedKey key)
{
ColumnFamily cachedRow = cfs.getRawCachedRow(key);
if (cachedRow != null)
ColumnFamilyStore.removeDeleted(cachedRow, gcBefore);
}
{code}
For the SerializingCache, this means it calls removeDeleted on a temporary, deserialized copy, which leaves the cache contents unaffected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3921) Compaction doesn't clear out
expired tombstones from SerializingCache
Posted by "Peter Schuller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209048#comment-13209048 ]
Peter Schuller commented on CASSANDRA-3921:
-------------------------------------------
With a typical workload, not evicting it would just fail to "optimize" - you have to wait for the data to get dropped out of cache.
But if you do have a use-case where you have a lot of requests to non-existent data, this could "silently" render the row caches almost useless in a way that is very hard to detect. Suppose you have rows with a hot set where you want caching, but the data is made up of "churning" columns (insertions/deletions; the set of columns constantly But, the "damaged" is limited to the ratio of requests for empty rows vs. other requests, and is less prominent the bigger the average requested row in relation to the size of a tombstone.
The above should be true if the rows in cache only contain the tombstones.
But consider a use-case where you have rows (hot) that gets constantly updated with insertion/deletion (let's say it has 1-5 columns at any point in time, with columns constantly "churned"). How you have something which is hot in cache due to reads (for the ones in hot set), data being over-written, and the per-row average size increasing over time. Cache locality becomes worse and worse and no one can see why, and a restart magically fixes it. This should hold true as long as row cache entries aren't entirely re-read on writes, which would in part defeat the purpose of caching (but I realize now I'm not sure what we do here).
> Compaction doesn't clear out expired tombstones from SerializingCache
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-3921
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3921
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Jonathan Ellis
> Priority: Minor
> Fix For: 1.1.0
>
>
> Compaction calls removeDeletedInCache, which looks like this:
> {code}
> . public void removeDeletedInCache(DecoratedKey key)
> {
> ColumnFamily cachedRow = cfs.getRawCachedRow(key);
> if (cachedRow != null)
> ColumnFamilyStore.removeDeleted(cachedRow, gcBefore);
> }
> {code}
> For the SerializingCache, this means it calls removeDeleted on a temporary, deserialized copy, which leaves the cache contents unaffected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3921) Compaction doesn't clear out
expired tombstones from SerializingCache
Posted by "Jonathan Ellis (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-3921.
---------------------------------------
Resolution: Fixed
Reviewer: slebresne
Assignee: Jonathan Ellis
bq. we may want to make removeDeletedInCache be a noop for copying caches just to avoid the useless deserialization
done in 94860c6c3713cda4f17dabb2ac2ce30cfe92f6e2
> Compaction doesn't clear out expired tombstones from SerializingCache
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-3921
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3921
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 1.1.0
>
>
> Compaction calls removeDeletedInCache, which looks like this:
> {code}
> . public void removeDeletedInCache(DecoratedKey key)
> {
> ColumnFamily cachedRow = cfs.getRawCachedRow(key);
> if (cachedRow != null)
> ColumnFamilyStore.removeDeleted(cachedRow, gcBefore);
> }
> {code}
> For the SerializingCache, this means it calls removeDeleted on a temporary, deserialized copy, which leaves the cache contents unaffected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3921) Compaction doesn't clear out
expired tombstones from SerializingCache
Posted by "Peter Schuller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209211#comment-13209211 ]
Peter Schuller commented on CASSANDRA-3921:
-------------------------------------------
If we invalidate on every put, I'm +1 on just ignoring the problem. Sure, it's possible to have a constant subset of a hotset repeatedly read, and someone making an attempt to make it take less space in cache by deleting data and waiting for tombstones... but that's so obscure/extreme that we can probably fix 500 other JIRA:s before this is a priority :) Definitely +1 on NOOP:ing the method though; or more importantly, documenting why it's a NOOP.
(Btw the incoherence of my previous comment is what happens when you split the posting of a comment in two pieces with a meeting in between...)
> Compaction doesn't clear out expired tombstones from SerializingCache
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-3921
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3921
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Jonathan Ellis
> Priority: Minor
> Fix For: 1.1.0
>
>
> Compaction calls removeDeletedInCache, which looks like this:
> {code}
> . public void removeDeletedInCache(DecoratedKey key)
> {
> ColumnFamily cachedRow = cfs.getRawCachedRow(key);
> if (cachedRow != null)
> ColumnFamilyStore.removeDeleted(cachedRow, gcBefore);
> }
> {code}
> For the SerializingCache, this means it calls removeDeleted on a temporary, deserialized copy, which leaves the cache contents unaffected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3921) Compaction doesn't clear out
expired tombstones from SerializingCache
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208986#comment-13208986 ]
Jonathan Ellis commented on CASSANDRA-3921:
-------------------------------------------
I'm not sure what the right fix is. We could invalidate the row, but that means a big compaction could wipe out the row cache entirely, which is worse. Should we just wontfix this?
> Compaction doesn't clear out expired tombstones from SerializingCache
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-3921
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3921
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Jonathan Ellis
> Priority: Minor
> Fix For: 1.1.0
>
>
> Compaction calls removeDeletedInCache, which looks like this:
> {code}
> . public void removeDeletedInCache(DecoratedKey key)
> {
> ColumnFamily cachedRow = cfs.getRawCachedRow(key);
> if (cachedRow != null)
> ColumnFamilyStore.removeDeleted(cachedRow, gcBefore);
> }
> {code}
> For the SerializingCache, this means it calls removeDeleted on a temporary, deserialized copy, which leaves the cache contents unaffected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3921) Compaction doesn't clear out
expired tombstones from SerializingCache
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209202#comment-13209202 ]
Sylvain Lebresne commented on CASSANDRA-3921:
---------------------------------------------
Yeah, it's the serializing cache, we invalidate on each put, so the chance of tons of tombstone building up overtime is anecdotal imo. We could probably fix this by cloning the cachedRow after the get, apply removeDeleted on the clone and do a conditional replace (if it's still the value we got). This involve a bunch of serialization/deserialization so it's unclear that doing that is a better optimization than leaving the tombstone. So I'm good leaving it the way it is. Except that we may want to make removeDeletedInCache be a noop for copying caches just to avoid the useless deserialization.
> Compaction doesn't clear out expired tombstones from SerializingCache
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-3921
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3921
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Jonathan Ellis
> Priority: Minor
> Fix For: 1.1.0
>
>
> Compaction calls removeDeletedInCache, which looks like this:
> {code}
> . public void removeDeletedInCache(DecoratedKey key)
> {
> ColumnFamily cachedRow = cfs.getRawCachedRow(key);
> if (cachedRow != null)
> ColumnFamilyStore.removeDeleted(cachedRow, gcBefore);
> }
> {code}
> For the SerializingCache, this means it calls removeDeleted on a temporary, deserialized copy, which leaves the cache contents unaffected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira