You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2012/10/31 17:45:12 UTC
[jira] [Created] (CASSANDRA-4885) Remove or rework per-row bloom
filters
Jonathan Ellis created CASSANDRA-4885:
-----------------------------------------
Summary: Remove or rework per-row bloom filters
Key: CASSANDRA-4885
URL: https://issues.apache.org/jira/browse/CASSANDRA-4885
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Jonathan Ellis
Fix For: 1.3
Per-row bloom filters may be a misfeature.
On small rows we don't create them.
On large rows we essentially only do slice queries that can't take advantage of it.
And on very large rows if we ever did deserialize it, the performance hit of doing so would outweigh the benefit of skipping the actual read.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4885) Remove or rework per-row
bloom filters
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507573#comment-13507573 ]
Jonathan Ellis edited comment on CASSANDRA-4885 at 11/30/12 7:18 PM:
---------------------------------------------------------------------
We coulde use our knowledge of the cql3 schema to only generate BF for wide rows...
But I still think it should default to off, cql3 wide rows are almost always used to support denormalized, clustered resultsets, which are still going to correspond to slices.
was (Author: jbellis):
What if we use our knowledge of the cql3 schema to only generate BF for wide rows?
> Remove or rework per-row bloom filters
> --------------------------------------
>
> Key: CASSANDRA-4885
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4885
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jason Brown
> Fix For: 1.3
>
> Attachments: 0001-CASSANRDA-4885-Remove-per-row-bloom-filter.patch, 0002-CASSANRDA-4885-update-test.patch
>
>
> Per-row bloom filters may be a misfeature.
> On small rows we don't create them.
> On large rows we essentially only do slice queries that can't take advantage of it.
> And on very large rows if we ever did deserialize it, the performance hit of doing so would outweigh the benefit of skipping the actual read.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4885) Remove or rework per-row
bloom filters
Posted by "Jason Brown (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507709#comment-13507709 ]
Jason Brown edited comment on CASSANDRA-4885 at 11/30/12 10:21 PM:
-------------------------------------------------------------------
This seems reasonable to me. Let me do some more reading/research, and I think this should be doable. Let me rework my patch, then.
was (Author: jasobrown):
This seems reasonable to me. Let me do some more reading/research, and I think this should be doable.
> Remove or rework per-row bloom filters
> --------------------------------------
>
> Key: CASSANDRA-4885
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4885
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jason Brown
> Fix For: 1.3
>
> Attachments: 0001-CASSANRDA-4885-Remove-per-row-bloom-filter.patch, 0002-CASSANRDA-4885-update-test.patch
>
>
> Per-row bloom filters may be a misfeature.
> On small rows we don't create them.
> On large rows we essentially only do slice queries that can't take advantage of it.
> And on very large rows if we ever did deserialize it, the performance hit of doing so would outweigh the benefit of skipping the actual read.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4885) Remove or rework per-row bloom
filters
Posted by "Jason Brown (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Brown updated CASSANDRA-4885:
-----------------------------------
Attachment: 0001-CASSANRDA-4885-Remove-per-row-bloom-filter.patch
OK, so I've removed the bloom filter from the row header and the key index. For the row header, on reads we just skip over the BF section that's on disk (if the sstable is any version lesser than the new 'ja'), and never write it out when serializing. For the index, more or less the same solution. I think I've caught the places in the logic to make sure it does the right thing (don't misread existing files, never write out the BF in new files).
However, I did run into a problem with one of the unit tests. ScrubTest.testScrubFile() expects a file that is corrupted so it can then attempt to fix it. The existing corrupt file (test/data/corrupt-sstables/Keyspace1-Super5-f-2-Data.db) has a corrupted BF that throws an exception the unit test is expecting. Now that we're skipping over the BF in the row header, however, the exception never gets thrown and the test fails because the file is 'no longer corrupt' :). I hacked up the code to trip a failure (by changing the columns count), but can somebody recommend a good way to create a corrupt file?
> Remove or rework per-row bloom filters
> --------------------------------------
>
> Key: CASSANDRA-4885
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4885
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jason Brown
> Fix For: 1.3
>
> Attachments: 0001-CASSANRDA-4885-Remove-per-row-bloom-filter.patch
>
>
> Per-row bloom filters may be a misfeature.
> On small rows we don't create them.
> On large rows we essentially only do slice queries that can't take advantage of it.
> And on very large rows if we ever did deserialize it, the performance hit of doing so would outweigh the benefit of skipping the actual read.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4885) Remove or rework per-row bloom
filters
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-4885:
-----------------------------------------
Assignee: Jason Brown
> Remove or rework per-row bloom filters
> --------------------------------------
>
> Key: CASSANDRA-4885
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4885
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jason Brown
> Fix For: 1.3
>
>
> Per-row bloom filters may be a misfeature.
> On small rows we don't create them.
> On large rows we essentially only do slice queries that can't take advantage of it.
> And on very large rows if we ever did deserialize it, the performance hit of doing so would outweigh the benefit of skipping the actual read.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4885) Remove or rework per-row bloom
filters
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507573#comment-13507573 ]
Jonathan Ellis commented on CASSANDRA-4885:
-------------------------------------------
What if we use our knowledge of the cql3 schema to only generate BF for wide rows?
> Remove or rework per-row bloom filters
> --------------------------------------
>
> Key: CASSANDRA-4885
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4885
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jason Brown
> Fix For: 1.3
>
> Attachments: 0001-CASSANRDA-4885-Remove-per-row-bloom-filter.patch, 0002-CASSANRDA-4885-update-test.patch
>
>
> Per-row bloom filters may be a misfeature.
> On small rows we don't create them.
> On large rows we essentially only do slice queries that can't take advantage of it.
> And on very large rows if we ever did deserialize it, the performance hit of doing so would outweigh the benefit of skipping the actual read.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4885) Remove or rework per-row bloom
filters
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496706#comment-13496706 ]
Jonathan Ellis commented on CASSANDRA-4885:
-------------------------------------------
Maybe reopen it at the halfway point after writing a healthy one, and scribble over 10% of it with zeros?
> Remove or rework per-row bloom filters
> --------------------------------------
>
> Key: CASSANDRA-4885
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4885
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jason Brown
> Fix For: 1.3
>
> Attachments: 0001-CASSANRDA-4885-Remove-per-row-bloom-filter.patch
>
>
> Per-row bloom filters may be a misfeature.
> On small rows we don't create them.
> On large rows we essentially only do slice queries that can't take advantage of it.
> And on very large rows if we ever did deserialize it, the performance hit of doing so would outweigh the benefit of skipping the actual read.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4885) Remove or rework per-row bloom
filters
Posted by "Jason Brown (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Brown updated CASSANDRA-4885:
-----------------------------------
Attachment: 0002-CASSANRDA-4885-update-test.patch
Ok, took some work, but (re-)corrupted the target data file to trigger the scrub. I tried deleting large blocks, scribbling over vast swathes, and even minor surgical changes (changing length of col names), but, in the end, the best was to just overwrite the first 64 bytes with 0x00.
> Remove or rework per-row bloom filters
> --------------------------------------
>
> Key: CASSANDRA-4885
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4885
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jason Brown
> Fix For: 1.3
>
> Attachments: 0001-CASSANRDA-4885-Remove-per-row-bloom-filter.patch, 0002-CASSANRDA-4885-update-test.patch
>
>
> Per-row bloom filters may be a misfeature.
> On small rows we don't create them.
> On large rows we essentially only do slice queries that can't take advantage of it.
> And on very large rows if we ever did deserialize it, the performance hit of doing so would outweigh the benefit of skipping the actual read.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4885) Remove or rework per-row bloom
filters
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507260#comment-13507260 ]
Sylvain Lebresne commented on CASSANDRA-4885:
---------------------------------------------
bq. On large rows we essentially only do slice queries
Now that I think about that, with CQL3, we do potentially do name queries on wide rows fairly often (basically anytime you request just one CQL3 row in a table that have a composite PK, and unless there is a collection). So maybe it could be worth being a bit prudent with completely removing this too quickly.
For instance, maybe we could have a columnBF_fp_chance like for table-level BF and recognize a special value of that (say, 1) as don't generate any BF at all? That could even be the default, but at least someone that say often require to test the existence of a CQL3 row in a table with composite PK could activate the column BF and probably have some benefit from it.
> Remove or rework per-row bloom filters
> --------------------------------------
>
> Key: CASSANDRA-4885
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4885
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jason Brown
> Fix For: 1.3
>
> Attachments: 0001-CASSANRDA-4885-Remove-per-row-bloom-filter.patch, 0002-CASSANRDA-4885-update-test.patch
>
>
> Per-row bloom filters may be a misfeature.
> On small rows we don't create them.
> On large rows we essentially only do slice queries that can't take advantage of it.
> And on very large rows if we ever did deserialize it, the performance hit of doing so would outweigh the benefit of skipping the actual read.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4885) Remove or rework per-row bloom
filters
Posted by "Jason Brown (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496901#comment-13496901 ]
Jason Brown commented on CASSANDRA-4885:
----------------------------------------
Yeah, I figured the 'whack it in the knee caps with a baseball bat' technique would work :), wasn't sure if there would be a more controlled way. Digging out the sports equipment now...
> Remove or rework per-row bloom filters
> --------------------------------------
>
> Key: CASSANDRA-4885
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4885
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jason Brown
> Fix For: 1.3
>
> Attachments: 0001-CASSANRDA-4885-Remove-per-row-bloom-filter.patch
>
>
> Per-row bloom filters may be a misfeature.
> On small rows we don't create them.
> On large rows we essentially only do slice queries that can't take advantage of it.
> And on very large rows if we ever did deserialize it, the performance hit of doing so would outweigh the benefit of skipping the actual read.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4885) Remove or rework per-row bloom
filters
Posted by "Jason Brown (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507709#comment-13507709 ]
Jason Brown commented on CASSANDRA-4885:
----------------------------------------
This seems reasonable to me. Let me do some more reading/research, and I think this should be doable.
> Remove or rework per-row bloom filters
> --------------------------------------
>
> Key: CASSANDRA-4885
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4885
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jason Brown
> Fix For: 1.3
>
> Attachments: 0001-CASSANRDA-4885-Remove-per-row-bloom-filter.patch, 0002-CASSANRDA-4885-update-test.patch
>
>
> Per-row bloom filters may be a misfeature.
> On small rows we don't create them.
> On large rows we essentially only do slice queries that can't take advantage of it.
> And on very large rows if we ever did deserialize it, the performance hit of doing so would outweigh the benefit of skipping the actual read.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4885) Remove or rework per-row
bloom filters
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507573#comment-13507573 ]
Jonathan Ellis edited comment on CASSANDRA-4885 at 11/30/12 7:19 PM:
---------------------------------------------------------------------
We could use our knowledge of the cql3 schema to only generate BF for wide rows...
But I still think it should default to off, cql3 wide rows are almost always used to support denormalized, clustered resultsets, which are still going to correspond to slices.
was (Author: jbellis):
We coulde use our knowledge of the cql3 schema to only generate BF for wide rows...
But I still think it should default to off, cql3 wide rows are almost always used to support denormalized, clustered resultsets, which are still going to correspond to slices.
> Remove or rework per-row bloom filters
> --------------------------------------
>
> Key: CASSANDRA-4885
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4885
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jason Brown
> Fix For: 1.3
>
> Attachments: 0001-CASSANRDA-4885-Remove-per-row-bloom-filter.patch, 0002-CASSANRDA-4885-update-test.patch
>
>
> Per-row bloom filters may be a misfeature.
> On small rows we don't create them.
> On large rows we essentially only do slice queries that can't take advantage of it.
> And on very large rows if we ever did deserialize it, the performance hit of doing so would outweigh the benefit of skipping the actual read.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4885) Remove or rework per-row bloom
filters
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507585#comment-13507585 ]
Sylvain Lebresne commented on CASSANDRA-4885:
---------------------------------------------
bq. But I still think it should default to off
I agree. And that's also why I'm not sure we should try to be too smart here because the usefulness of the BF depends more on how the table is used than of how the schema look like. As said above, I'd be in favor of a columnBF_FP_chance with a default to off since 1) it similar to our current row BF tunable so it will be familiar to users and 2) it's fine-grained which cannot hurt. But open to discussion, just seem the best option so far to me.
> Remove or rework per-row bloom filters
> --------------------------------------
>
> Key: CASSANDRA-4885
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4885
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jason Brown
> Fix For: 1.3
>
> Attachments: 0001-CASSANRDA-4885-Remove-per-row-bloom-filter.patch, 0002-CASSANRDA-4885-update-test.patch
>
>
> Per-row bloom filters may be a misfeature.
> On small rows we don't create them.
> On large rows we essentially only do slice queries that can't take advantage of it.
> And on very large rows if we ever did deserialize it, the performance hit of doing so would outweigh the benefit of skipping the actual read.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira