You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Joaquin Casares (JIRA)" <ji...@apache.org> on 2011/03/07 18:43:59 UTC
[jira] Created: (CASSANDRA-2279) Tombstones turn into empty arrays,
then won't stay dead when running sstables2json
Tombstones turn into empty arrays, then won't stay dead when running sstables2json
----------------------------------------------------------------------------------
Key: CASSANDRA-2279
URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.6.10
Reporter: Joaquin Casares
Priority: Minor
The keys would only show up in sstables2json and look like this:
(root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
{
"waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
}
(root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
===>
The steps that I took to reproduce it were:
Create a keyspace, column family, and a key
Delete the key on Node 1
Flush
Repair on a cluster with more than 1 node
Wait GCSeconds
Compact
And the empty row would appear on Node 2
However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
Create a keyspace, column family, and a key
Delete the key
Flush
Sample write (writing to some temporary key)
Deleting the attribute to that temporary key (not the entire key)
Flush
Compact
or these steps:
Create a keyspace, column family, and a key
Delete the key
Flush
Wait GCseconds
Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004613#comment-13004613 ]
Sylvain Lebresne commented on CASSANDRA-2279:
---------------------------------------------
I'm not able to reproduce. And I don't see why repair would screw up with tombstones collection. Only compaction should remove them and repair shouldn't have anything to do with this. If sstable2json is the only one to show it, it's even weirder (that is, listing with the cli should also show an empty row if there was problem with tombstone collection).
Are you sure you compacted node 2 (on which the empty row showed up) ? Are you sure there wasn't a Queue-2527-Compacted file alongside Queue-2527-Data.db ? Are you sure node 2 local time wasn't highly out of sync with node 1 ?
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2279:
----------------------------------------
Attachment: 0001-RowIteration-unit-test.patch
Attaching the unit test.
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
> Attachments: 0001-Handle-localDeletionTime-in-RowIteratorFactory.patch, 0001-RowIteration-unit-test.patch
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2279:
----------------------------------------
Attachment: 0001-Handle-localDeletionTime-in-RowIteratorFactory.patch
It's a bit of a shot in the dark since I'm not sure exactly how was produced and how to reproduce this, but I found a place in RowIteratorFactor where we don't handle a CF localDeletionTime correctly. Since RowIterator is used for repair and for sstable2json, it sounds like a promising candidate for this (and it's a legit problem even if not the one at hand here).
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
> Attachments: 0001-Handle-localDeletionTime-in-RowIteratorFactory.patch
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006519#comment-13006519 ]
Jonathan Ellis commented on CASSANDRA-2279:
-------------------------------------------
what's the significance of gc_grace_seconds: 2 in the test CF? I don't see any sleeps in the test.
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
> Attachments: 0001-Handle-localDeletionTime-in-RowIteratorFactory.patch, 0001-RowIteration-unit-test.patch
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2279:
--------------------------------------
Fix Version/s: (was: 0.7.4)
0.7.5
that's a big improvement over the existing factory code! committed.
we'll need a separate patch for 0.6 -- RowIteratorFactory doesn't exist (the analogous code is inline in getRangeSlice).
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.5
>
> Attachments: RowIteration-unit-tests.patch, fix-RowIteratorFactory.patch
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008450#comment-13008450 ]
Sylvain Lebresne commented on CASSANDRA-2279:
---------------------------------------------
bq. we'll need a separate patch for 0.6
I don't think 0.6 suffers from the problem fixed by the attached patch. It uses CFStore.getFamily() for range slices to do its bidding which handles correctly the column family deletion times as far as I can tell.
Which make me think that there could be something else at hand here. I'll have a look at Joaquin instructions to try to reproduce what he is seeing.
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.5
>
> Attachments: RowIteration-unit-tests.patch, fix-RowIteratorFactory.patch, nodeA.txt, nodeB.txt
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2279:
----------------------------------------
Attachment: (was: 0001-RowIteration-unit-test.patch)
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2279:
--------------------------------------
Summary: Tombstones not collected post-repair (was: Tombstones turn into empty arrays, then won't stay dead when running sstables2json)
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2279) Tombstones turn into empty arrays,
then won't stay dead when running sstables2json
Posted by "Joaquin Casares (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joaquin Casares updated CASSANDRA-2279:
---------------------------------------
Description:
The keys would only show up in sstables2json and look like this:
(root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
{
"waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
}
(root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
===>
The steps that I took to reproduce it were:
Create a keyspace, column family, and a key
Delete the key on Node 1 using the cli (del cf['key'];)
Flush
Repair on a cluster with more than 1 node
Wait GCSeconds
Compact
And the empty row would appear on Node 2
However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
Create a keyspace, column family, and a key
Delete the key
Flush
Sample write (writing to some temporary key)
Deleting the attribute to that temporary key (not the entire key)
Flush
Compact
or these steps:
Create a keyspace, column family, and a key
Delete the key
Flush
Wait GCseconds
Compact
was:
The keys would only show up in sstables2json and look like this:
(root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
{
"waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
}
(root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
===>
The steps that I took to reproduce it were:
Create a keyspace, column family, and a key
Delete the key on Node 1
Flush
Repair on a cluster with more than 1 node
Wait GCSeconds
Compact
And the empty row would appear on Node 2
However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
Create a keyspace, column family, and a key
Delete the key
Flush
Sample write (writing to some temporary key)
Deleting the attribute to that temporary key (not the entire key)
Flush
Compact
or these steps:
Create a keyspace, column family, and a key
Delete the key
Flush
Wait GCseconds
Compact
> Tombstones turn into empty arrays, then won't stay dead when running sstables2json
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017694#comment-13017694 ]
Sylvain Lebresne commented on CASSANDRA-2279:
---------------------------------------------
For the record, we did try to reproduce with Joaquin and weren't able to find anything wrong in there. In particular, in one instance when we though we had a column coming back from the dead, we were actually looking at a compacted sstable (which sstable2json won't avoid).
So I do not know if there is an actual problem here.
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.5
>
> Attachments: RowIteration-unit-tests.patch, fix-RowIteratorFactory.patch, nodeA.txt, nodeB.txt
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006429#comment-13006429 ]
Jonathan Ellis commented on CASSANDRA-2279:
-------------------------------------------
how hard would it be to create a unit test that catches this?
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
> Attachments: 0001-Handle-localDeletionTime-in-RowIteratorFactory.patch
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2279) Tombstones turn into empty arrays,
then won't stay dead when running sstables2json
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2279:
--------------------------------------
Component/s: Tools
Affects Version/s: (was: 0.6.10)
0.6
Fix Version/s: 0.7.4
0.6.13
Assignee: Sylvain Lebresne
> Tombstones turn into empty arrays, then won't stay dead when running sstables2json
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2279:
----------------------------------------
Attachment: 0001-RowIteration-unit-test.patch
Sorry, that was me trying stuff and forgetting to remove it. I update the test to use an existing column with standard gc_grace
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
> Attachments: 0001-Handle-localDeletionTime-in-RowIteratorFactory.patch, 0001-RowIteration-unit-test.patch
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017689#comment-13017689 ]
Jonathan Ellis commented on CASSANDRA-2279:
-------------------------------------------
compaction should be runnable any time, as long as you do repair before gcgrace expires.
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.5
>
> Attachments: RowIteration-unit-tests.patch, fix-RowIteratorFactory.patch, nodeA.txt, nodeB.txt
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Joaquin Casares (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013702#comment-13013702 ]
Joaquin Casares commented on CASSANDRA-2279:
--------------------------------------------
I just read this and was wondering if this may be the case:
http://wiki.apache.org/cassandra/Operations#Dealing_with_the_consequences_of_nodetool_repair_not_running_within_GCGraceSeconds
GCGraceSeconds was set to 30 minutes to allow the repair to finish and I waited 35 before running the compaction on the above steps.
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.5
>
> Attachments: RowIteration-unit-tests.patch, fix-RowIteratorFactory.patch, nodeA.txt, nodeB.txt
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2279) Tombstones not
collected post-repair
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006407#comment-13006407 ]
Sylvain Lebresne edited comment on CASSANDRA-2279 at 3/14/11 1:45 PM:
----------------------------------------------------------------------
It's a bit of a shot in the dark since I'm not sure exactly how was produced and how to reproduce this, but I found a place in RowIteratorFactor where we don't handle a CF localDeletionTime correctly. Since RowIterator is used for repair and for sstable2json, it sounds like a promising candidate for this (and it's a legit problem even if not the one at hand here).
Patch is against 0.7.
was (Author: slebresne):
It's a bit of a shot in the dark since I'm not sure exactly how was produced and how to reproduce this, but I found a place in RowIteratorFactor where we don't handle a CF localDeletionTime correctly. Since RowIterator is used for repair and for sstable2json, it sounds like a promising candidate for this (and it's a legit problem even if not the one at hand here).
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
> Attachments: 0001-Handle-localDeletionTime-in-RowIteratorFactory.patch
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Joaquin Casares (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joaquin Casares updated CASSANDRA-2279:
---------------------------------------
Attachment: nodeB.txt
nodeA.txt
Here are all the commands that were run. I was messing around with T2 references and the bug didn't show up the first time, so I tried it again.
The second time it worked following the exact steps that I listed. I set the GCSeconds to be 30 minutes to wait for the repair to finish 100%.
This time however, the SSTables actually have the old values as well.
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.5
>
> Attachments: RowIteration-unit-tests.patch, fix-RowIteratorFactory.patch, nodeA.txt, nodeB.txt
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2279:
----------------------------------------
Attachment: (was: 0001-Handle-localDeletionTime-in-RowIteratorFactory.patch)
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2279:
----------------------------------------
Attachment: fix-RowIteratorFactory.patch
RowIteration-unit-tests.patch
There was actually 2 related bugs in RowIteratorFactory. Re-attaching new patch with the 2 unit tests and the fix (fixing both problem and simplifying, I think, RowIteratorFactory).
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
> Attachments: RowIteration-unit-tests.patch, fix-RowIteratorFactory.patch
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006597#comment-13006597 ]
Hudson commented on CASSANDRA-2279:
-----------------------------------
Integrated in Cassandra-0.7 #378 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/378/])
fix tombstone handling in repair andsstable2json
patch by slebresne; reviewed by jbellis for CASSANDRA-2279
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.5
>
> Attachments: RowIteration-unit-tests.patch, fix-RowIteratorFactory.patch
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2279:
--------------------------------------
Affects Version/s: (was: 0.6)
0.7.0
Fix Version/s: (was: 0.6.13)
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.7.0
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.7.5
>
> Attachments: RowIteration-unit-tests.patch, fix-RowIteratorFactory.patch, nodeA.txt, nodeB.txt
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2279:
----------------------------------------
Attachment: (was: 0001-RowIteration-unit-test.patch)
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
> Attachments: 0001-Handle-localDeletionTime-in-RowIteratorFactory.patch, 0001-RowIteration-unit-test.patch
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006433#comment-13006433 ]
Sylvain Lebresne commented on CASSANDRA-2279:
---------------------------------------------
Should be doable. I'll do that.
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
> Attachments: 0001-Handle-localDeletionTime-in-RowIteratorFactory.patch
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005582#comment-13005582 ]
Sylvain Lebresne commented on CASSANDRA-2279:
---------------------------------------------
Ok, strongest hypothesis: this is a sign of CASSANDRA-2305.
Joaquim: was some row cache enabled on the incriminated CF ?
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2279) Tombstones not collected
post-repair
Posted by "Joaquin Casares (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005735#comment-13005735 ]
Joaquin Casares commented on CASSANDRA-2279:
--------------------------------------------
The entire thing was reproduced on a fresh install and I was just using the standard Keyspace that already exists, I believe. I'll go through and follow my steps again and give a more detailed account.
> Tombstones not collected post-repair
> ------------------------------------
>
> Key: CASSANDRA-2279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.6
> Reporter: Joaquin Casares
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.6.13, 0.7.4
>
>
> The keys would only show up in sstables2json and look like this:
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
> ===> /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
> 2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor] - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> 2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] - Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
> {
> "waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b": []
> }
> (root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
> ===>
> The steps that I took to reproduce it were:
> Create a keyspace, column family, and a key
> Delete the key on Node 1 using the cli (del cf['key'];)
> Flush
> Repair on a cluster with more than 1 node
> Wait GCSeconds
> Compact
> And the empty row would appear on Node 2
> However, when I was able to get rid of the empty rows, I was following these steps on a single machine:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Sample write (writing to some temporary key)
> Deleting the attribute to that temporary key (not the entire key)
> Flush
> Compact
> or these steps:
> Create a keyspace, column family, and a key
> Delete the key
> Flush
> Wait GCseconds
> Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira