You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sylvain Lebresne (JIRA)" <ji...@apache.org> on 2011/03/17 15:37:29 UTC
[jira] Created: (CASSANDRA-2349) Expring columns can expire between
the two phase of LazilyCompactedRow.
Expring columns can expire between the two phase of LazilyCompactedRow.
-----------------------------------------------------------------------
Key: CASSANDRA-2349
URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.0
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
Priority: Critical
Fix For: 0.7.5
LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2349) Expring columns can expire
between the two phase of LazilyCompactedRow.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007971#comment-13007971 ]
Jonathan Ellis commented on CASSANDRA-2349:
-------------------------------------------
don't we need forceKeepExpired for both passes or we have the opposite problem? If so, would prefer fKE a constructor arg for the iterator instead of setter.
> Expring columns can expire between the two phase of LazilyCompactedRow.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 0.7.5
>
> Attachments: 0001-Don-t-transform-expired-tombstone-in-LazilyCompacted.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2349) Expring columns can expire
between the two phase of LazilyCompactedRow.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007972#comment-13007972 ]
Jonathan Ellis commented on CASSANDRA-2349:
-------------------------------------------
Or: should we have an expireBefore parameter like gcBefore so it can stay constant during the compaction?
> Expring columns can expire between the two phase of LazilyCompactedRow.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 0.7.5
>
> Attachments: 0001-Don-t-transform-expired-tombstone-in-LazilyCompacted.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2349) Expring columns can expire
between the two phase of LazilyCompactedRow.
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007945#comment-13007945 ]
Sylvain Lebresne commented on CASSANDRA-2349:
---------------------------------------------
Actually, this won't work because we cannot anticipate the index size.
> Expring columns can expire between the two phase of LazilyCompactedRow.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 0.7.5
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2349) Expring columns can expire
between the two phase of LazilyCompactedRow.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009442#comment-13009442 ]
Hudson commented on CASSANDRA-2349:
-----------------------------------
Integrated in Cassandra-0.7 #397 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/397/])
> Expring columns can expire between the two phase of LazilyCompactedRow.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 0.7.5
>
> Attachments: 0001-Introduce-expireBefore-and-keep-it-constant-all-thro.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2349) Expring columns can expire
between the two phase of LazilyCompactedRow.
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007932#comment-13007932 ]
Sylvain Lebresne commented on CASSANDRA-2349:
---------------------------------------------
This could be fixed by making LazilyCompactedRow do only one phase. We could write a place holder header, write the column, and seek back at the end to write the header. The tricky part is that this won't fit well with AbstractCompactedRow, since it is assume that the column count is known before write is called.
> Expring columns can expire between the two phase of LazilyCompactedRow.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 0.7.5
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2349) Expring columns can
expire between the two phase of LazilyCompactedRow.
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007982#comment-13007982 ]
Sylvain Lebresne edited comment on CASSANDRA-2349 at 3/17/11 4:36 PM:
----------------------------------------------------------------------
bq. don't we need forceKeepExpired for both passes or we have the opposite problem? If so, would prefer fKE a constructor arg for the iterator instead of setter.
There is no opposite problem, since you can't have a column expired for the first phase but not for the second one.
Note that the point of transforming expired columns to tombstone is to re-gain quickly the disk space used by the column value (without having to wait for gcGrace). So using fKE in phase 1 would prevent that for LazilyCompactedRow, which would be a pity.
was (Author: slebresne):
bq. don't we need forceKeepExpired for both passes or we have the opposite problem? If so, would prefer fKE a constructor arg for the iterator instead of setter.
There is no opposite problem, since you can't have a column expired for the first phase but not for the second one.
> Expring columns can expire between the two phase of LazilyCompactedRow.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 0.7.5
>
> Attachments: 0001-Don-t-transform-expired-tombstone-in-LazilyCompacted.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2349) Expring columns can expire
between the two phase of LazilyCompactedRow.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007992#comment-13007992 ]
Jonathan Ellis commented on CASSANDRA-2349:
-------------------------------------------
bq. There is no opposite problem, since you can't have a column expired for the first phase but not for the second one.
But you can have an expired column counted as a tombstone for the first but not in the second since you have enabled forceKE. No?
bq. the point of transforming expired columns to tombstone is to re-gain quickly the disk space used by the column value
Right, which is why using a constant expireBefore value that we pass to the serializer instead of computing it locally each time seems like a better solution.
> Expring columns can expire between the two phase of LazilyCompactedRow.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 0.7.5
>
> Attachments: 0001-Don-t-transform-expired-tombstone-in-LazilyCompacted.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2349) Expring columns can expire between
the two phase of LazilyCompactedRow.
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2349:
----------------------------------------
Attachment: 0001-Don-t-transform-expired-tombstone-in-LazilyCompacted.patch
Attaching simpler solution consisting in not transforming expired columns into tombstones, so that the second phase of LazilyCompactedRow sees the same columns than the first phase.
I would be happy to come up with a unit test for this, but this is a subtle race condition and I'm really not sure how to write a test that capture it.
> Expring columns can expire between the two phase of LazilyCompactedRow.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 0.7.5
>
> Attachments: 0001-Don-t-transform-expired-tombstone-in-LazilyCompacted.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2349) Expring columns can expire between
the two phase of LazilyCompactedRow.
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2349:
----------------------------------------
Remaining Estimate: 2h (was: 8h)
Original Estimate: 2h (was: 8h)
> Expring columns can expire between the two phase of LazilyCompactedRow.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 0.7.5
>
> Attachments: 0001-Don-t-transform-expired-tombstone-in-LazilyCompacted.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2349) Expring columns can expire between
the two phase of LazilyCompactedRow.
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2349:
----------------------------------------
Attachment: 0001-Introduce-expireBefore-and-keep-it-constant-all-thro.patch
Attached patch using the expireBefore idea.
> Expring columns can expire between the two phase of LazilyCompactedRow.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 0.7.5
>
> Attachments: 0001-Introduce-expireBefore-and-keep-it-constant-all-thro.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2349) Expring columns can expire
between the two phase of LazilyCompactedRow.
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007982#comment-13007982 ]
Sylvain Lebresne commented on CASSANDRA-2349:
---------------------------------------------
bq. don't we need forceKeepExpired for both passes or we have the opposite problem? If so, would prefer fKE a constructor arg for the iterator instead of setter.
There is no opposite problem, since you can't have a column expired for the first phase but not for the second one.
> Expring columns can expire between the two phase of LazilyCompactedRow.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 0.7.5
>
> Attachments: 0001-Don-t-transform-expired-tombstone-in-LazilyCompacted.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2349) Expring columns can expire
between the two phase of LazilyCompactedRow.
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007999#comment-13007999 ]
Sylvain Lebresne commented on CASSANDRA-2349:
---------------------------------------------
bq. But you can have an expired column counted as a tombstone for the first but not in the second since you have enabled forceKE. No?
You are right. Oups.
You are right, having an expireBefore is a better solution. I'll work out the patch.
> Expring columns can expire between the two phase of LazilyCompactedRow.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 0.7.5
>
> Attachments: 0001-Don-t-transform-expired-tombstone-in-LazilyCompacted.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2349) Expring columns can expire between
the two phase of LazilyCompactedRow.
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-2349:
----------------------------------------
Attachment: (was: 0001-Don-t-transform-expired-tombstone-in-LazilyCompacted.patch)
> Expring columns can expire between the two phase of LazilyCompactedRow.
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2349
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Sylvain Lebresne
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 0.7.5
>
> Attachments: 0001-Introduce-expireBefore-and-keep-it-constant-all-thro.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LazilyCompactedRow reads the columns to compact twice. First to create the index, bloom filter and calculate the data size, and then another phase to actually write the columns. But a column can expire between those two phase, which will result in a bad data size in the sstable (and a possibly corrupted row index).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira