You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Ramzi Rabah (JIRA)" <ji...@apache.org> on 2009/12/05 01:13:20 UTC
[jira] Created: (CASSANDRA-604) Compactions might remove tombstones
without removing the actual data
Compactions might remove tombstones without removing the actual data
--------------------------------------------------------------------
Key: CASSANDRA-604
URL: https://issues.apache.org/jira/browse/CASSANDRA-604
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.5
Environment: Cent-OS
Reporter: Ramzi Rabah
Fix For: 0.5
I was looking at the code for compaction, and noticed that when we are doing compactions during the normal course of
Cassandra, we call:
for (List<SSTableReader> sstables :
getCompactionBuckets(ssTables_, 50L * 1024L * 1024L))
{
if (sstables.size() < minThreshold)
{
continue;
}
other wise docompactions...
where getCompactionBuckets puts in buckets very small files, or files
that are 0.5-1.5 of each other's sizes. It will only compact those if
they are >= minimum threshold which is 4 by default.
So far so good. Now how about this scenario, I have an old entry that
I inserted long time ago and that was compacted into a 75MB file.
There are fewer 75MB files than 4. I do many deletes, and I end with 4
extra sstable files filled with tombstones, each about 300 MB large.
These 4 files are compacted together and in the compaction code, if
the tombstone is there we don't copy it over to the new file. Now
since we did not compact the 75MB files, but we compacted the
tombstone files, that leaves us with the tombstone gone, but
the data still intact in the 75MB file. If we compacted all the
files together I don't think that would be a problem, but since we
only compact 4, this potentially leaves data not cleaned.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-604) Compactions might remove
tombstones without removing the actual data
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786711#action_12786711 ]
Jonathan Ellis commented on CASSANDRA-604:
------------------------------------------
I don't follow.
All the tombstone does is suppress earlier versions. And all compaction does is write out a new sstable containing only the newest version. So compacting data w/o the tombstone will either yield (1) a single version that is newer than the tombstone, in which case the tombstone will be ignored on reads, or (2) a single version older than the tombstone, in which case it will still be supressed. Either way correctness is preserved.
> Compactions might remove tombstones without removing the actual data
> --------------------------------------------------------------------
>
> Key: CASSANDRA-604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-604
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Environment: Cent-OS
> Reporter: Ramzi Rabah
> Fix For: 0.5
>
>
> I was looking at the code for compaction, and noticed that when we are doing compactions during the normal course of
> Cassandra, we call:
> for (List<SSTableReader> sstables :
> getCompactionBuckets(ssTables_, 50L * 1024L * 1024L))
> {
> if (sstables.size() < minThreshold)
> {
> continue;
> }
> other wise docompactions...
> where getCompactionBuckets puts in buckets very small files, or files
> that are 0.5-1.5 of each other's sizes. It will only compact those if
> they are >= minimum threshold which is 4 by default.
> So far so good. Now how about this scenario, I have an old entry that
> I inserted long time ago and that was compacted into a 75MB file.
> There are fewer 75MB files than 4. I do many deletes, and I end with 4
> extra sstable files filled with tombstones, each about 300 MB large.
> These 4 files are compacted together and in the compaction code, if
> the tombstone is there we don't copy it over to the new file. Now
> since we did not compact the 75MB files, but we compacted the
> tombstone files, that leaves us with the tombstone gone, but
> the data still intact in the 75MB file. If we compacted all the
> files together I don't think that would be a problem, but since we
> only compact 4, this potentially leaves data not cleaned.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-604) Compactions might remove
tombstones without removing the actual data
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-604.
--------------------------------------
Resolution: Fixed
committed. let us know if your testing uncovers any problems.
> Compactions might remove tombstones without removing the actual data
> --------------------------------------------------------------------
>
> Key: CASSANDRA-604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-604
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cent-OS
> Reporter: Ramzi Rabah
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 604.patch
>
>
> I was looking at the code for compaction, and noticed that when we are doing compactions during the normal course of
> Cassandra, we call:
> for (List<SSTableReader> sstables :
> getCompactionBuckets(ssTables_, 50L * 1024L * 1024L))
> {
> if (sstables.size() < minThreshold)
> {
> continue;
> }
> other wise docompactions...
> where getCompactionBuckets puts in buckets very small files, or files
> that are 0.5-1.5 of each other's sizes. It will only compact those if
> they are >= minimum threshold which is 4 by default.
> So far so good. Now how about this scenario, I have an old entry that
> I inserted long time ago and that was compacted into a 75MB file.
> There are fewer 75MB files than 4. I do many deletes, and I end with 4
> extra sstable files filled with tombstones, each about 300 MB large.
> These 4 files are compacted together and in the compaction code, if
> the tombstone is there we don't copy it over to the new file. Now
> since we did not compact the 75MB files, but we compacted the
> tombstone files, that leaves us with the tombstone gone, but
> the data still intact in the 75MB file. If we compacted all the
> files together I don't think that would be a problem, but since we
> only compact 4, this potentially leaves data not cleaned.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-604) Compactions might remove tombstones
without removing the actual data
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-604:
-------------------------------------
Assignee: Jonathan Ellis
Priority: Minor (was: Major)
Affects Version/s: (was: 0.5)
> Compactions might remove tombstones without removing the actual data
> --------------------------------------------------------------------
>
> Key: CASSANDRA-604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-604
> Project: Cassandra
> Issue Type: Bug
> Environment: Cent-OS
> Reporter: Ramzi Rabah
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
> Attachments: 604.patch
>
>
> I was looking at the code for compaction, and noticed that when we are doing compactions during the normal course of
> Cassandra, we call:
> for (List<SSTableReader> sstables :
> getCompactionBuckets(ssTables_, 50L * 1024L * 1024L))
> {
> if (sstables.size() < minThreshold)
> {
> continue;
> }
> other wise docompactions...
> where getCompactionBuckets puts in buckets very small files, or files
> that are 0.5-1.5 of each other's sizes. It will only compact those if
> they are >= minimum threshold which is 4 by default.
> So far so good. Now how about this scenario, I have an old entry that
> I inserted long time ago and that was compacted into a 75MB file.
> There are fewer 75MB files than 4. I do many deletes, and I end with 4
> extra sstable files filled with tombstones, each about 300 MB large.
> These 4 files are compacted together and in the compaction code, if
> the tombstone is there we don't copy it over to the new file. Now
> since we did not compact the 75MB files, but we compacted the
> tombstone files, that leaves us with the tombstone gone, but
> the data still intact in the 75MB file. If we compacted all the
> files together I don't think that would be a problem, but since we
> only compact 4, this potentially leaves data not cleaned.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-604) Compactions might remove tombstones
without removing the actual data
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-604:
-------------------------------------
Attachment: 604.patch
patch to only GC when compacting all sstables, as outlined above.
Ramzi to review?
> Compactions might remove tombstones without removing the actual data
> --------------------------------------------------------------------
>
> Key: CASSANDRA-604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-604
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Environment: Cent-OS
> Reporter: Ramzi Rabah
> Fix For: 0.5
>
> Attachments: 604.patch
>
>
> I was looking at the code for compaction, and noticed that when we are doing compactions during the normal course of
> Cassandra, we call:
> for (List<SSTableReader> sstables :
> getCompactionBuckets(ssTables_, 50L * 1024L * 1024L))
> {
> if (sstables.size() < minThreshold)
> {
> continue;
> }
> other wise docompactions...
> where getCompactionBuckets puts in buckets very small files, or files
> that are 0.5-1.5 of each other's sizes. It will only compact those if
> they are >= minimum threshold which is 4 by default.
> So far so good. Now how about this scenario, I have an old entry that
> I inserted long time ago and that was compacted into a 75MB file.
> There are fewer 75MB files than 4. I do many deletes, and I end with 4
> extra sstable files filled with tombstones, each about 300 MB large.
> These 4 files are compacted together and in the compaction code, if
> the tombstone is there we don't copy it over to the new file. Now
> since we did not compact the 75MB files, but we compacted the
> tombstone files, that leaves us with the tombstone gone, but
> the data still intact in the 75MB file. If we compacted all the
> files together I don't think that would be a problem, but since we
> only compact 4, this potentially leaves data not cleaned.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-604) Compactions might remove tombstones
without removing the actual data
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-604:
-------------------------------------
Component/s: Core
Priority: Major (was: Minor)
> Compactions might remove tombstones without removing the actual data
> --------------------------------------------------------------------
>
> Key: CASSANDRA-604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-604
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cent-OS
> Reporter: Ramzi Rabah
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 604.patch
>
>
> I was looking at the code for compaction, and noticed that when we are doing compactions during the normal course of
> Cassandra, we call:
> for (List<SSTableReader> sstables :
> getCompactionBuckets(ssTables_, 50L * 1024L * 1024L))
> {
> if (sstables.size() < minThreshold)
> {
> continue;
> }
> other wise docompactions...
> where getCompactionBuckets puts in buckets very small files, or files
> that are 0.5-1.5 of each other's sizes. It will only compact those if
> they are >= minimum threshold which is 4 by default.
> So far so good. Now how about this scenario, I have an old entry that
> I inserted long time ago and that was compacted into a 75MB file.
> There are fewer 75MB files than 4. I do many deletes, and I end with 4
> extra sstable files filled with tombstones, each about 300 MB large.
> These 4 files are compacted together and in the compaction code, if
> the tombstone is there we don't copy it over to the new file. Now
> since we did not compact the 75MB files, but we compacted the
> tombstone files, that leaves us with the tombstone gone, but
> the data still intact in the 75MB file. If we compacted all the
> files together I don't think that would be a problem, but since we
> only compact 4, this potentially leaves data not cleaned.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-604) Compactions might remove
tombstones without removing the actual data
Posted by "Ramzi Rabah (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786662#action_12786662 ]
Ramzi Rabah commented on CASSANDRA-604:
---------------------------------------
Thinking about it some more, there is another case when data can be lost. In the above case the file containing the tombstone was compacted by itself before the data file.
The second case is that the file containing the data is compacted by itself before the tombstone is compacted.
So in both cases, it seems like the only viable solution I can think of, is to only remove the tombstones when every single SSTable file for the column family is compacted (I.E. major compaction). Otherwise, the tombstone should stick around.
Does that make sense?
> Compactions might remove tombstones without removing the actual data
> --------------------------------------------------------------------
>
> Key: CASSANDRA-604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-604
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Environment: Cent-OS
> Reporter: Ramzi Rabah
> Fix For: 0.5
>
>
> I was looking at the code for compaction, and noticed that when we are doing compactions during the normal course of
> Cassandra, we call:
> for (List<SSTableReader> sstables :
> getCompactionBuckets(ssTables_, 50L * 1024L * 1024L))
> {
> if (sstables.size() < minThreshold)
> {
> continue;
> }
> other wise docompactions...
> where getCompactionBuckets puts in buckets very small files, or files
> that are 0.5-1.5 of each other's sizes. It will only compact those if
> they are >= minimum threshold which is 4 by default.
> So far so good. Now how about this scenario, I have an old entry that
> I inserted long time ago and that was compacted into a 75MB file.
> There are fewer 75MB files than 4. I do many deletes, and I end with 4
> extra sstable files filled with tombstones, each about 300 MB large.
> These 4 files are compacted together and in the compaction code, if
> the tombstone is there we don't copy it over to the new file. Now
> since we did not compact the 75MB files, but we compacted the
> tombstone files, that leaves us with the tombstone gone, but
> the data still intact in the 75MB file. If we compacted all the
> files together I don't think that would be a problem, but since we
> only compact 4, this potentially leaves data not cleaned.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-604) Compactions might remove
tombstones without removing the actual data
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786748#action_12786748 ]
Jonathan Ellis commented on CASSANDRA-604:
------------------------------------------
That makes sense. I agree that only GCing tombstones during major compactions (or, other compactions that happen to include all sstables) is the easiest fix.
> Compactions might remove tombstones without removing the actual data
> --------------------------------------------------------------------
>
> Key: CASSANDRA-604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-604
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Environment: Cent-OS
> Reporter: Ramzi Rabah
> Fix For: 0.5
>
>
> I was looking at the code for compaction, and noticed that when we are doing compactions during the normal course of
> Cassandra, we call:
> for (List<SSTableReader> sstables :
> getCompactionBuckets(ssTables_, 50L * 1024L * 1024L))
> {
> if (sstables.size() < minThreshold)
> {
> continue;
> }
> other wise docompactions...
> where getCompactionBuckets puts in buckets very small files, or files
> that are 0.5-1.5 of each other's sizes. It will only compact those if
> they are >= minimum threshold which is 4 by default.
> So far so good. Now how about this scenario, I have an old entry that
> I inserted long time ago and that was compacted into a 75MB file.
> There are fewer 75MB files than 4. I do many deletes, and I end with 4
> extra sstable files filled with tombstones, each about 300 MB large.
> These 4 files are compacted together and in the compaction code, if
> the tombstone is there we don't copy it over to the new file. Now
> since we did not compact the 75MB files, but we compacted the
> tombstone files, that leaves us with the tombstone gone, but
> the data still intact in the 75MB file. If we compacted all the
> files together I don't think that would be a problem, but since we
> only compact 4, this potentially leaves data not cleaned.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-604) Compactions might remove
tombstones without removing the actual data
Posted by "Ramzi Rabah (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787629#action_12787629 ]
Ramzi Rabah commented on CASSANDRA-604:
---------------------------------------
Looks great. I will test the patch with a higher volume of inserts and deletes and let you know how the test goes.
> Compactions might remove tombstones without removing the actual data
> --------------------------------------------------------------------
>
> Key: CASSANDRA-604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-604
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cent-OS
> Reporter: Ramzi Rabah
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 604.patch
>
>
> I was looking at the code for compaction, and noticed that when we are doing compactions during the normal course of
> Cassandra, we call:
> for (List<SSTableReader> sstables :
> getCompactionBuckets(ssTables_, 50L * 1024L * 1024L))
> {
> if (sstables.size() < minThreshold)
> {
> continue;
> }
> other wise docompactions...
> where getCompactionBuckets puts in buckets very small files, or files
> that are 0.5-1.5 of each other's sizes. It will only compact those if
> they are >= minimum threshold which is 4 by default.
> So far so good. Now how about this scenario, I have an old entry that
> I inserted long time ago and that was compacted into a 75MB file.
> There are fewer 75MB files than 4. I do many deletes, and I end with 4
> extra sstable files filled with tombstones, each about 300 MB large.
> These 4 files are compacted together and in the compaction code, if
> the tombstone is there we don't copy it over to the new file. Now
> since we did not compact the 75MB files, but we compacted the
> tombstone files, that leaves us with the tombstone gone, but
> the data still intact in the 75MB file. If we compacted all the
> files together I don't think that would be a problem, but since we
> only compact 4, this potentially leaves data not cleaned.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-604) Compactions might remove
tombstones without removing the actual data
Posted by "Ramzi Rabah (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786743#action_12786743 ]
Ramzi Rabah commented on CASSANDRA-604:
---------------------------------------
What I meant was there are 2 orders of compactions that might happen that would lead to the clean up of tombstones but not data:
Case 1) I described first, data is in sstable 1, tombstone in sstable 2. sstables 2-5 are compacted producing sstable 6, and sstable 6 has no tombstone. That leaves us with sstable 1 data, sstable 6 no tombstone --> Probably bad. In this case tombstones are compacted before the data is ever compacted.
Case 2) data is in sstable 1, tombstone is in sstable 5. sstable 1-4 get compacted to sstable 6 that has data, so now we have sstable 5 tombstone, sstable 6 data. sstable 5,7,8,9 are compacted producing sstable 10. sstable 6 data, sstable 10 no tombstone --> Probably bad. In this case, data was compacted first, then tombstones.
Both cases can probably be fixed in the same manner, but just wanted to point out all scenarios which I can think of that can cause this problem.
> Compactions might remove tombstones without removing the actual data
> --------------------------------------------------------------------
>
> Key: CASSANDRA-604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-604
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Environment: Cent-OS
> Reporter: Ramzi Rabah
> Fix For: 0.5
>
>
> I was looking at the code for compaction, and noticed that when we are doing compactions during the normal course of
> Cassandra, we call:
> for (List<SSTableReader> sstables :
> getCompactionBuckets(ssTables_, 50L * 1024L * 1024L))
> {
> if (sstables.size() < minThreshold)
> {
> continue;
> }
> other wise docompactions...
> where getCompactionBuckets puts in buckets very small files, or files
> that are 0.5-1.5 of each other's sizes. It will only compact those if
> they are >= minimum threshold which is 4 by default.
> So far so good. Now how about this scenario, I have an old entry that
> I inserted long time ago and that was compacted into a 75MB file.
> There are fewer 75MB files than 4. I do many deletes, and I end with 4
> extra sstable files filled with tombstones, each about 300 MB large.
> These 4 files are compacted together and in the compaction code, if
> the tombstone is there we don't copy it over to the new file. Now
> since we did not compact the 75MB files, but we compacted the
> tombstone files, that leaves us with the tombstone gone, but
> the data still intact in the 75MB file. If we compacted all the
> files together I don't think that would be a problem, but since we
> only compact 4, this potentially leaves data not cleaned.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-604) Compactions might remove
tombstones without removing the actual data
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788069#action_12788069 ]
Hudson commented on CASSANDRA-604:
----------------------------------
Integrated in Cassandra #282 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/282/])
only GC when compacting all sstables, to avoid situations where the data a tombstone is intended to supress is in an sstable that is not part of the compaction set.
patch by jbellis; reviewed by Ramzi Rabah for
> Compactions might remove tombstones without removing the actual data
> --------------------------------------------------------------------
>
> Key: CASSANDRA-604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-604
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cent-OS
> Reporter: Ramzi Rabah
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 604.patch
>
>
> I was looking at the code for compaction, and noticed that when we are doing compactions during the normal course of
> Cassandra, we call:
> for (List<SSTableReader> sstables :
> getCompactionBuckets(ssTables_, 50L * 1024L * 1024L))
> {
> if (sstables.size() < minThreshold)
> {
> continue;
> }
> other wise docompactions...
> where getCompactionBuckets puts in buckets very small files, or files
> that are 0.5-1.5 of each other's sizes. It will only compact those if
> they are >= minimum threshold which is 4 by default.
> So far so good. Now how about this scenario, I have an old entry that
> I inserted long time ago and that was compacted into a 75MB file.
> There are fewer 75MB files than 4. I do many deletes, and I end with 4
> extra sstable files filled with tombstones, each about 300 MB large.
> These 4 files are compacted together and in the compaction code, if
> the tombstone is there we don't copy it over to the new file. Now
> since we did not compact the 75MB files, but we compacted the
> tombstone files, that leaves us with the tombstone gone, but
> the data still intact in the 75MB file. If we compacted all the
> files together I don't think that would be a problem, but since we
> only compact 4, this potentially leaves data not cleaned.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.