You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Richard Low (JIRA)" <ji...@apache.org> on 2011/01/24 11:59:43 UTC
[jira] Created: (CASSANDRA-2039) LazilyCompactedRow doesn't add
CFInfo to digest
LazilyCompactedRow doesn't add CFInfo to digest
-----------------------------------------------
Key: CASSANDRA-2039
URL: https://issues.apache.org/jira/browse/CASSANDRA-2039
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.0
Reporter: Richard Low
Priority: Minor
LazilyCompactedRow.update doesn't add the CFInfo or columnCount to the digest, so the hash value in the Merkle tree does not include this data. However, PrecompactedRow does include this. Two consequences of this are:
* Row-level tombstones are not compared when using LazilyCompactedRow so could remain inconsistent
* LazilyCompactedRow and PrecompactedRow produce different hashes of the same row, so if two nodes have differing in_memory_compaction_limit_in_mb values, rows of size in between the two limits will have different hashes so will always be repaired even when they are the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-2039) LazilyCompactedRow doesn't add
CFInfo to digest
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991066#comment-12991066 ]
Hudson commented on CASSANDRA-2039:
-----------------------------------
Integrated in Cassandra #710 (See [https://hudson.apache.org/hudson/job/Cassandra/710/])
make PreCompactedRow and LazyCompactedRow digest computations match
patch by Richard Low and jbellis for CASSANDRA-2039
> LazilyCompactedRow doesn't add CFInfo to digest
> -----------------------------------------------
>
> Key: CASSANDRA-2039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2039
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Richard Low
> Priority: Minor
> Fix For: 0.8
>
> Attachments: trunk-2038-LazilyCompactedRowTest.txt, trunk-2038-v2.txt, trunk-2038.txt
>
>
> LazilyCompactedRow.update doesn't add the CFInfo or columnCount to the digest, so the hash value in the Merkle tree does not include this data. However, PrecompactedRow does include this. Two consequences of this are:
> * Row-level tombstones are not compared when using LazilyCompactedRow so could remain inconsistent
> * LazilyCompactedRow and PrecompactedRow produce different hashes of the same row, so if two nodes have differing in_memory_compaction_limit_in_mb values, rows of size in between the two limits will have different hashes so will always be repaired even when they are the same.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2039) LazilyCompactedRow doesn't add
CFInfo to digest
Posted by "Richard Low (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Low updated CASSANDRA-2039:
-----------------------------------
Attachment: trunk-2038.txt
> LazilyCompactedRow doesn't add CFInfo to digest
> -----------------------------------------------
>
> Key: CASSANDRA-2039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2039
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Richard Low
> Priority: Minor
> Attachments: trunk-2038.txt
>
>
> LazilyCompactedRow.update doesn't add the CFInfo or columnCount to the digest, so the hash value in the Merkle tree does not include this data. However, PrecompactedRow does include this. Two consequences of this are:
> * Row-level tombstones are not compared when using LazilyCompactedRow so could remain inconsistent
> * LazilyCompactedRow and PrecompactedRow produce different hashes of the same row, so if two nodes have differing in_memory_compaction_limit_in_mb values, rows of size in between the two limits will have different hashes so will always be repaired even when they are the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-2039) LazilyCompactedRow doesn't add
CFInfo to digest
Posted by "Richard Low (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Low updated CASSANDRA-2039:
-----------------------------------
Attachment: trunk-2038-LazilyCompactedRowTest.txt
> LazilyCompactedRow doesn't add CFInfo to digest
> -----------------------------------------------
>
> Key: CASSANDRA-2039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2039
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Richard Low
> Priority: Minor
> Fix For: 0.7.2
>
> Attachments: trunk-2038-LazilyCompactedRowTest.txt, trunk-2038.txt
>
>
> LazilyCompactedRow.update doesn't add the CFInfo or columnCount to the digest, so the hash value in the Merkle tree does not include this data. However, PrecompactedRow does include this. Two consequences of this are:
> * Row-level tombstones are not compared when using LazilyCompactedRow so could remain inconsistent
> * LazilyCompactedRow and PrecompactedRow produce different hashes of the same row, so if two nodes have differing in_memory_compaction_limit_in_mb values, rows of size in between the two limits will have different hashes so will always be repaired even when they are the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-2039) LazilyCompactedRow doesn't add
CFInfo to digest
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991050#comment-12991050 ]
Jonathan Ellis commented on CASSANDRA-2039:
-------------------------------------------
Committed v2. Thanks!
bq. what would be the impact of using estimatedColumnCount here instead
It would break the part of CompactionIterator that leaves out rows with no columns from the new SSTable. "estimated" is the maximum possible number of columns in the new row, so it's ok to use it in the bloom filter, but not in the "is this row empty post-merge" check.
> LazilyCompactedRow doesn't add CFInfo to digest
> -----------------------------------------------
>
> Key: CASSANDRA-2039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2039
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Richard Low
> Priority: Minor
> Fix For: 0.8
>
> Attachments: trunk-2038-LazilyCompactedRowTest.txt, trunk-2038-v2.txt, trunk-2038.txt
>
>
> LazilyCompactedRow.update doesn't add the CFInfo or columnCount to the digest, so the hash value in the Merkle tree does not include this data. However, PrecompactedRow does include this. Two consequences of this are:
> * Row-level tombstones are not compared when using LazilyCompactedRow so could remain inconsistent
> * LazilyCompactedRow and PrecompactedRow produce different hashes of the same row, so if two nodes have differing in_memory_compaction_limit_in_mb values, rows of size in between the two limits will have different hashes so will always be repaired even when they are the same.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2039) LazilyCompactedRow doesn't add
CFInfo to digest
Posted by "Richard Low (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987149#action_12987149 ]
Richard Low commented on CASSANDRA-2039:
----------------------------------------
The v2 patch does this (and supersedes the other two patches). Now the updated LazilyCompactedRowTest passes, as do the other tests.
It now might be possible to compute the digest for LazilyCompactedRow in just one pass through the data - the first pass is now just used to calculate columnCount. columnCount could be excluded from the digest (I added it just because it was there in PrecompactedRow). However, it's used by isEmpty - what would be the impact of using estimatedColumnCount here instead? Is estimatedColumnCount zero if and only if columnCount is zero? Or does it matter is isEmpty sometimes returns false when it's true?
> LazilyCompactedRow doesn't add CFInfo to digest
> -----------------------------------------------
>
> Key: CASSANDRA-2039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2039
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Richard Low
> Priority: Minor
> Fix For: 0.8
>
> Attachments: trunk-2038-LazilyCompactedRowTest.txt, trunk-2038-v2.txt, trunk-2038.txt
>
>
> LazilyCompactedRow.update doesn't add the CFInfo or columnCount to the digest, so the hash value in the Merkle tree does not include this data. However, PrecompactedRow does include this. Two consequences of this are:
> * Row-level tombstones are not compared when using LazilyCompactedRow so could remain inconsistent
> * LazilyCompactedRow and PrecompactedRow produce different hashes of the same row, so if two nodes have differing in_memory_compaction_limit_in_mb values, rows of size in between the two limits will have different hashes so will always be repaired even when they are the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Assigned] (CASSANDRA-2039) LazilyCompactedRow doesn't add
CFInfo to digest
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-2039:
-----------------------------------------
Assignee: Richard Low
> LazilyCompactedRow doesn't add CFInfo to digest
> -----------------------------------------------
>
> Key: CASSANDRA-2039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2039
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Richard Low
> Assignee: Richard Low
> Priority: Minor
> Fix For: 0.8
>
> Attachments: trunk-2038-LazilyCompactedRowTest.txt, trunk-2038-v2.txt, trunk-2038.txt
>
>
> LazilyCompactedRow.update doesn't add the CFInfo or columnCount to the digest, so the hash value in the Merkle tree does not include this data. However, PrecompactedRow does include this. Two consequences of this are:
> * Row-level tombstones are not compared when using LazilyCompactedRow so could remain inconsistent
> * LazilyCompactedRow and PrecompactedRow produce different hashes of the same row, so if two nodes have differing in_memory_compaction_limit_in_mb values, rows of size in between the two limits will have different hashes so will always be repaired even when they are the same.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2039) LazilyCompactedRow doesn't add
CFInfo to digest
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2039:
--------------------------------------
Fix Version/s: (was: 0.7.2)
0.8
That sounds good. Let's do that in trunk to avoid breaking compatibility in 0.7.
> LazilyCompactedRow doesn't add CFInfo to digest
> -----------------------------------------------
>
> Key: CASSANDRA-2039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2039
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Richard Low
> Priority: Minor
> Fix For: 0.8
>
> Attachments: trunk-2038-LazilyCompactedRowTest.txt, trunk-2038.txt
>
>
> LazilyCompactedRow.update doesn't add the CFInfo or columnCount to the digest, so the hash value in the Merkle tree does not include this data. However, PrecompactedRow does include this. Two consequences of this are:
> * Row-level tombstones are not compared when using LazilyCompactedRow so could remain inconsistent
> * LazilyCompactedRow and PrecompactedRow produce different hashes of the same row, so if two nodes have differing in_memory_compaction_limit_in_mb values, rows of size in between the two limits will have different hashes so will always be repaired even when they are the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-2039) LazilyCompactedRow doesn't add
CFInfo to digest
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2039:
--------------------------------------
Reviewer: jbellis
Fix Version/s: 0.7.2
this looks okay eyeballing it, but can you add a check to LazilyCompactedRowTest similar to assertBytes to make sure this stays fixed?
> LazilyCompactedRow doesn't add CFInfo to digest
> -----------------------------------------------
>
> Key: CASSANDRA-2039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2039
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Richard Low
> Priority: Minor
> Fix For: 0.7.2
>
> Attachments: trunk-2038.txt
>
>
> LazilyCompactedRow.update doesn't add the CFInfo or columnCount to the digest, so the hash value in the Merkle tree does not include this data. However, PrecompactedRow does include this. Two consequences of this are:
> * Row-level tombstones are not compared when using LazilyCompactedRow so could remain inconsistent
> * LazilyCompactedRow and PrecompactedRow produce different hashes of the same row, so if two nodes have differing in_memory_compaction_limit_in_mb values, rows of size in between the two limits will have different hashes so will always be repaired even when they are the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-2039) LazilyCompactedRow doesn't add
CFInfo to digest
Posted by "Richard Low (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Low updated CASSANDRA-2039:
-----------------------------------
Attachment: trunk-2038-v2.txt
> LazilyCompactedRow doesn't add CFInfo to digest
> -----------------------------------------------
>
> Key: CASSANDRA-2039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2039
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Richard Low
> Priority: Minor
> Fix For: 0.8
>
> Attachments: trunk-2038-LazilyCompactedRowTest.txt, trunk-2038-v2.txt, trunk-2038.txt
>
>
> LazilyCompactedRow.update doesn't add the CFInfo or columnCount to the digest, so the hash value in the Merkle tree does not include this data. However, PrecompactedRow does include this. Two consequences of this are:
> * Row-level tombstones are not compared when using LazilyCompactedRow so could remain inconsistent
> * LazilyCompactedRow and PrecompactedRow produce different hashes of the same row, so if two nodes have differing in_memory_compaction_limit_in_mb values, rows of size in between the two limits will have different hashes so will always be repaired even when they are the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-2039) LazilyCompactedRow doesn't add
CFInfo to digest
Posted by "Richard Low (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986984#action_12986984 ]
Richard Low commented on CASSANDRA-2039:
----------------------------------------
I added assertDigest (see the patch trunk-2038-LazilyCompactedRowTest.txt), but it currently fails in testManyRows. The reason is that the Bloom filters have different sizes, because getEstimatedColumnCount returns a value too large in LazilyCompactedRow. There is probably no way round this without doing another pass on the data.
However, it isn't necessary to add the Bloom filter or indeed the index to the digest - they are determined by the data that comes later. So could the header be excluded from the digest?
> LazilyCompactedRow doesn't add CFInfo to digest
> -----------------------------------------------
>
> Key: CASSANDRA-2039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2039
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Richard Low
> Priority: Minor
> Fix For: 0.7.2
>
> Attachments: trunk-2038-LazilyCompactedRowTest.txt, trunk-2038.txt
>
>
> LazilyCompactedRow.update doesn't add the CFInfo or columnCount to the digest, so the hash value in the Merkle tree does not include this data. However, PrecompactedRow does include this. Two consequences of this are:
> * Row-level tombstones are not compared when using LazilyCompactedRow so could remain inconsistent
> * LazilyCompactedRow and PrecompactedRow produce different hashes of the same row, so if two nodes have differing in_memory_compaction_limit_in_mb values, rows of size in between the two limits will have different hashes so will always be repaired even when they are the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.