You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Joaquin Casares (Created) (JIRA)" <ji...@apache.org> on 2012/03/08 19:57:57 UTC
[jira] [Created] (CASSANDRA-4023) Batch reading BloomFilters on
startup
Batch reading BloomFilters on startup
-------------------------------------
Key: CASSANDRA-4023
URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
Project: Cassandra
Issue Type: Improvement
Reporter: Joaquin Casares
The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Batch reading BloomFilters on
startup
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225621#comment-13225621 ]
Brandon Williams commented on CASSANDRA-4023:
---------------------------------------------
bq. (Maybe it's time to add random vs sequential speed ratio as a setting, which at least is general enough to be useful in other places.)
This sounds like a good idea, we're never going to strike a balance that's sufficient between SSD and rotational media without a knob to turn.
> Batch reading BloomFilters on startup
> -------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Joaquin Casares
> Labels: datastax_qa
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Yuki Morishita (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-4023:
--------------------------------------
Attachment: (was: 0001-fix-loading-promoted-row-index.patch)
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238818#comment-13238818 ]
Jonathan Ellis commented on CASSANDRA-4023:
-------------------------------------------
Does v3 include a fix for the NPE Dave found?
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt, trunk-4023.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Batch reading BloomFilters on
startup
Posted by "Michael Harris (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225777#comment-13225777 ]
Michael Harris commented on CASSANDRA-4023:
-------------------------------------------
My $0.02 is that it may be helpful to batch reads. Not sure if the underlying stream used in reading the bloom filters reads a large chunk and caches it, but if not, it could help to instead of just calling ois.readLong(), you read 64K or 1M or whatever you feel is appropriate (maybe configurable?) into a buffer and grab the longs out of those. This doesn't completely fix the problem of disk contention, but it might cause larger sequential reads to be submitted to the disk, which then might behave nicer?
> Batch reading BloomFilters on startup
> -------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Joaquin Casares
> Labels: datastax_qa
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237240#comment-13237240 ]
Jonathan Ellis commented on CASSANDRA-4023:
-------------------------------------------
Could we rename lastKey to lastKeyForUnpromoted, and then take the hasPromotedIndexes out of that assignment? Since you also added it to the big if statement.
I guess deserializePositionOnly is only for unpromoted indexes? Because for promoted ones decoratedKey will never be null.
+1 otherwise, I guess we'll have to rely on CASSANDRA-2392 for startup speed in 1.2.
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 0001-fix-loading-promoted-row-index.patch, 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Jonathan Ellis (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-4023.
---------------------------------------
Resolution: Fixed
Reviewer: jbellis (was: j.casares)
committed, thanks!
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Dave Brosius (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238867#comment-13238867 ]
Dave Brosius commented on CASSANDRA-4023:
-----------------------------------------
no v3 has this bug
the problem is in the general case where RowIndexEntry.serializer.deserialize() is using the hasPromotedIndexes code, the amount of data read is variable (assuming in this case the first int read in deserialize can be non 0). Therefore in SSTableReader the code can no longer statically predict when a particular key will be the last key.
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt, trunk-4023.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Jonathan Ellis (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-4023:
-----------------------------------------
Assignee: Yuki Morishita (was: Jonathan Ellis)
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4023:
--------------------------------------
Attachment: 4023.txt
patch to do buffered bloom filter deserialization
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237226#comment-13237226 ]
Jonathan Ellis commented on CASSANDRA-4023:
-------------------------------------------
Is this io-reducing something that could also apply to 1.0?
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 0001-fix-loading-promoted-row-index.patch, 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Yuki Morishita (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238860#comment-13238860 ]
Yuki Morishita commented on CASSANDRA-4023:
-------------------------------------------
NPE only happens on trunk and that is why I reopened this issue.
Fix is attached as trunk-4023.txt.
v3 attached above is for 1.0 branch and already committed to 1.0 and 1.1 without any problem.
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt, trunk-4023.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4023:
--------------------------------------
Reviewer: j.casares
Component/s: Core
Priority: Minor (was: Major)
Affects Version/s: 1.0.1
Fix Version/s: 1.1.0
1.0.9
Assignee: Jonathan Ellis
Summary: Improve BloomFilter deserialization performance (was: Batch reading BloomFilters on startup)
Sorry, the description threw me off. So really we're talking about our paged bloom filter (CASSANDRA-2466), not the multithreaded startup (CASSANDRA-2988).
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Yuki Morishita (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-4023:
--------------------------------------
Attachment: cassandra-1.0-4023-v3.txt
Attaching v3, which is slightly modified from v2 and comes with test.
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Michael Harris (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225878#comment-13225878 ]
Michael Harris commented on CASSANDRA-4023:
-------------------------------------------
+1 on the patch. When we get a chance, we'll try out the patch on the cluster that demonstrated the issue.
The overall intent of the ticket filed is that startup is slower overall, and this was just one example in the code that I found that could be part of the problem. Any other ideas as to things that could cause slower startup? We noticed the logs had a lot of SSTables being opened during startup, and we do have some 100+GB SSTables on disk, so general SSTable reading was my first instinct to check.
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Michael Harris (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225891#comment-13225891 ]
Michael Harris commented on CASSANDRA-4023:
-------------------------------------------
Also, FWIW, the observed slowness is in 1.0.7, not 1.0.1 as mentioned in the affects version (though it certainly may affect 1.0.1 as well). I don't seem to have permission to update this.
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Batch reading BloomFilters on
startup
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225614#comment-13225614 ]
Jonathan Ellis commented on CASSANDRA-4023:
-------------------------------------------
We added the multithreadedness specifically because it *improves* startup time for people with multiple spindles or SSDs...
Any ideas how to get the best of both worlds besides falling back to a config options?
(Maybe it's time to add random vs sequential speed ratio as a setting, which at least is general enough to be useful in other places.)
> Batch reading BloomFilters on startup
> -------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Joaquin Casares
> Labels: datastax_qa
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Dave Brosius (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237632#comment-13237632 ]
Dave Brosius commented on CASSANDRA-4023:
-----------------------------------------
Seems like this is the fix to me
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableReader.java b/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
index 9e92220..959a66d 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
@@ -361,7 +361,7 @@ public class SSTableReader extends SSTable
int len = ByteBufferUtil.readShortLength(input);
boolean firstKey = left == null;
- boolean lastKey = indexPosition + DBConstants.SHORT_SIZE + len + DBConstants.LONG_SIZE == indexSize;
+ boolean lastKey = indexPosition + DBConstants.SHORT_SIZE + len + DBConstants.LONG_SIZE + (descriptor.hasPromotedIndexes ? DBConstants.INT_SIZE : 0) == indexSize;
boolean shouldAddEntry = indexSummary.shouldAddEntry();
if (shouldAddEntry || cacheLoading || recreatebloom || firstKey || lastKey)
{
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 0001-fix-loading-promoted-row-index.patch, 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Dave Brosius (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238867#comment-13238867 ]
Dave Brosius edited comment on CASSANDRA-4023 at 3/26/12 9:34 PM:
------------------------------------------------------------------
no v3 has this bug
the problem is in the general case where RowIndexEntry.serializer.deserialize() is using the hasPromotedIndexes code, the amount of data read is variable (assuming in this case the first int read in deserialize can be non 0). Therefore in SSTableReader the code can no longer statically predict when a particular key will be the last key.
Ah I see.. yes this happens on trunk... so if it is fixed elsewhere, then good... never mind me.
was (Author: dbrosius@apache.org):
no v3 has this bug
the problem is in the general case where RowIndexEntry.serializer.deserialize() is using the hasPromotedIndexes code, the amount of data read is variable (assuming in this case the first int read in deserialize can be non 0). Therefore in SSTableReader the code can no longer statically predict when a particular key will be the last key.
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt, trunk-4023.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Michael Harris (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226742#comment-13226742 ]
Michael Harris commented on CASSANDRA-4023:
-------------------------------------------
Hi Jonathan. We patched our cluster and startup time didn't decrease much if at all. I still think the patch is probably a positive help, but it looks like there are bigger problems elsewhere. Let me know if any additional testing or logs could help out here.
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Dave Brosius (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237628#comment-13237628 ]
Dave Brosius commented on CASSANDRA-4023:
-----------------------------------------
Getting NPE around this code (trunk).
SSTableReader.open on system-schema_columnfamilies-ib-2
SSTableReader.load bails out of the while (true) loop because
if (indexPosition == indexSize)
causing the right variable to remain null, which then gets passed to
this.last = getMinimalKey(right);
which NPEs
boolean lastKey = indexPosition + DBConstants.SHORT_SIZE + len + DBConstants.LONG_SIZE == indexSize; is false on the last item
indexPosition = 0
len = 9
indexSize = 23
19 != 23
public static DecoratedKey getMinimalKey(DecoratedKey key)
{
return key.key.position() > 0 || key.key.hasRemaining()
? new DecoratedKey(key.token, HeapAllocator.instance.clone(key.key))
: key;
}
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 0001-fix-loading-promoted-row-index.patch, 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Yuki Morishita (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237231#comment-13237231 ]
Yuki Morishita commented on CASSANDRA-4023:
-------------------------------------------
bq. Is this io-reducing something that could also apply to 1.0?
It's for RowIndexEntry which is introduced for v1.2 (CASSANDRA-2319), so it only relates to trunk.
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 0001-fix-loading-promoted-row-index.patch, 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Jonathan Ellis (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-4023.
---------------------------------------
Resolution: Fixed
committed.
(I note, for the record, that my original 4023.txt patch is unnecessary because we're already using a buffered inputstream in loadBloomFilter.)
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt, trunk-4023.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Dave Brosius (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237632#comment-13237632 ]
Dave Brosius edited comment on CASSANDRA-4023 at 3/24/12 6:58 PM:
------------------------------------------------------------------
Seems like this is the fix to me (at least in the simple case where the int read in deserialize is 0) more would need to be done if not.
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableReader.java b/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
index 9e92220..959a66d 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
@@ -361,7 +361,7 @@ public class SSTableReader extends SSTable
int len = ByteBufferUtil.readShortLength(input);
boolean firstKey = left == null;
- boolean lastKey = indexPosition + DBConstants.SHORT_SIZE + len + DBConstants.LONG_SIZE == indexSize;
+ boolean lastKey = indexPosition + DBConstants.SHORT_SIZE + len + DBConstants.LONG_SIZE + (descriptor.hasPromotedIndexes ? DBConstants.INT_SIZE : 0) == indexSize;
boolean shouldAddEntry = indexSummary.shouldAddEntry();
if (shouldAddEntry || cacheLoading || recreatebloom || firstKey || lastKey)
{
was (Author: dbrosius@apache.org):
Seems like this is the fix to me
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableReader.java b/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
index 9e92220..959a66d 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
@@ -361,7 +361,7 @@ public class SSTableReader extends SSTable
int len = ByteBufferUtil.readShortLength(input);
boolean firstKey = left == null;
- boolean lastKey = indexPosition + DBConstants.SHORT_SIZE + len + DBConstants.LONG_SIZE == indexSize;
+ boolean lastKey = indexPosition + DBConstants.SHORT_SIZE + len + DBConstants.LONG_SIZE + (descriptor.hasPromotedIndexes ? DBConstants.INT_SIZE : 0) == indexSize;
boolean shouldAddEntry = indexSummary.shouldAddEntry();
if (shouldAddEntry || cacheLoading || recreatebloom || firstKey || lastKey)
{
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 0001-fix-loading-promoted-row-index.patch, 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Yuki Morishita (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-4023:
--------------------------------------
Attachment: 0001-fix-loading-promoted-row-index.patch
Attaching patch to current trunk.
Unfortunately, there is no way to determine serialized size of promoted row index without actually deserializing some of it, so we cannot determine the last key.
Patch deserializes every key when sstable has promoted index, but instead it tries to reduce the amount of io when deserialize index itself.
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 0001-fix-loading-promoted-row-index.patch, 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Yuki Morishita (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-4023:
--------------------------------------
Attachment: cassandra-1.0-4023-v2.txt
I measured how long does it take to load each sstable component(Data, Index, Filter) and found out that loading from index file takes longer in 1.0 than in 0.8.
By looking code for difference between 0.8 and 1.0, I noticed that in 1.0, every keys stored in index file get deserialized, while in 0.8, only those keys that should be added to index summary get deserialized.
The reason we deserialize all keys in 1.0 is to obtain first and last keys stored in sstable. Attached patch tries to skip deserializing keys when possible.
Patch is against 1.0 branch.
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt, cassandra-1.0-4023-v2.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Yuki Morishita (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita reopened CASSANDRA-4023:
---------------------------------------
In trunk, CASSANDRA-2319 changed the content in primary index file, so we need to calculate its serialized size differently to determine if the key is the last one.
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Yuki Morishita (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-4023:
--------------------------------------
Attachment: trunk-4023.txt
bq. Could we rename lastKey to lastKeyForUnpromoted, and then take the hasPromotedIndexes out of that assignment?
Done.
bq. I guess deserializePositionOnly is only for unpromoted indexes?
It deserializes whole row index only when decoratedKey is not null AND cache key. Otherwise only deserialize position.
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt, cassandra-1.0-4023-v2.txt, cassandra-1.0-4023-v3.txt, trunk-4023.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233122#comment-13233122 ]
Jonathan Ellis commented on CASSANDRA-4023:
-------------------------------------------
LGTM, but can you add a test for indexSummary, SSTR.first, and SSTR.last? (Adding to one of the existing SSTR or CF tests is fine of course.)
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Yuki Morishita
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt, cassandra-1.0-4023-v2.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-4023) Batch reading
BloomFilters on startup
Posted by "Michael Harris (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225777#comment-13225777 ]
Michael Harris edited comment on CASSANDRA-4023 at 3/9/12 3:37 AM:
-------------------------------------------------------------------
My $0.02 is that it may be helpful to batch reads. Not sure if the underlying stream used in reading the bloom filters reads a large chunk and caches it, but if not, it could help to instead of just calling ois.readLong(), you read 64K or 1M or whatever you feel is appropriate (maybe configurable?) into a buffer and grab the longs out of those. This doesn't completely fix the problem of disk contention, but it might cause larger sequential reads to be submitted to the disk, which then might behave nicer?
The specific example I'm thinking of here is: it looks like the deserialization of LegacyBloomFilter (perhaps what 0.8 uses?) is just a ois.readObject() for a BitSet. And that's like, it. Whereas for BloomFilter (the new version?), deserialization is a tight loop of readLong() calls. Same with serialization FWIW. Not that using Java serialization for LTS is necessarily a good idea, but it may be happier for the disk.
was (Author: mharris):
My $0.02 is that it may be helpful to batch reads. Not sure if the underlying stream used in reading the bloom filters reads a large chunk and caches it, but if not, it could help to instead of just calling ois.readLong(), you read 64K or 1M or whatever you feel is appropriate (maybe configurable?) into a buffer and grab the longs out of those. This doesn't completely fix the problem of disk contention, but it might cause larger sequential reads to be submitted to the disk, which then might behave nicer?
> Batch reading BloomFilters on startup
> -------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Joaquin Casares
> Labels: datastax_qa
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Batch reading BloomFilters on
startup
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225393#comment-13225393 ]
Brandon Williams commented on CASSANDRA-4023:
---------------------------------------------
The theory here being that the multithreadedness is causing seek contention when loading the sstables.
> Batch reading BloomFilters on startup
> -------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Joaquin Casares
> Labels: datastax_qa
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4023) Improve BloomFilter
deserialization performance
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226123#comment-13226123 ]
Jonathan Ellis commented on CASSANDRA-4023:
-------------------------------------------
Right, actual versions affected are {{[affects-version .. fix-version)}}
> Improve BloomFilter deserialization performance
> -----------------------------------------------
>
> Key: CASSANDRA-4023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4023
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.1
> Reporter: Joaquin Casares
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 4023.txt
>
>
> The difference of startup times between a 0.8.7 cluster and 1.0.7 cluster with the same amount of data is 4x greater in 1.0.7.
> It seems as though 1.0.7 loads the BloomFilter through a series of reading longs out in a multithreaded process while 0.8.7 reads the entire object.
> Perhaps we should update the new BloomFilter to do reading in batch as well?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira