You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Filipe Manana (JIRA)" <ji...@apache.org> on 2010/10/09 22:01:37 UTC
[jira] Created: (COUCHDB-913) Caching of BTree nodes and documents
Caching of BTree nodes and documents
------------------------------------
Key: COUCHDB-913
URL: https://issues.apache.org/jira/browse/COUCHDB-913
Project: CouchDB
Issue Type: Improvement
Reporter: Filipe Manana
Assignee: Filipe Manana
The following patch adds optional caching of BTree nodes (for both DBs and view indexes) and documents:
http://github.com/fdmanana/couchdb/commit/c6361d1e0d86942fc86350bea9ca1e8d7539c1b3
Look at the new parameters introduced in default.ini.tpl.in for more details.
This is based on the work initially done by Randall Leeds, found at:
http://github.com/tilgovi/couchdb/commits/forevercache
I'll not commit it before 1.1 nor without any feedback.
cheers
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-913) Caching of BTree nodes and
documents
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923747#action_12923747 ]
Adam Kocoloski commented on COUCHDB-913:
----------------------------------------
Regarding #3: I take it back - I'm wrong about DB compaction. The cache is linked to the transient process that does the compaction, so it disappears automatically. I do still think there's a leak in view group compaction, though its init_group(), not reset_group() which does the work.
> Caching of BTree nodes and documents
> ------------------------------------
>
> Key: COUCHDB-913
> URL: https://issues.apache.org/jira/browse/COUCHDB-913
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> The following patch adds optional caching of BTree nodes (for both DBs and view indexes) and documents:
> http://github.com/fdmanana/couchdb/commit/c6361d1e0d86942fc86350bea9ca1e8d7539c1b3
> Look at the new parameters introduced in default.ini.tpl.in for more details.
> This is based on the work initially done by Randall Leeds, found at:
> http://github.com/tilgovi/couchdb/commits/forevercache
> I'll not commit it before 1.1 nor without any feedback.
> cheers
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-913) Caching of BTree nodes and
documents
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923745#action_12923745 ]
Adam Kocoloski commented on COUCHDB-913:
----------------------------------------
Hi Filipe, I read through the term_cache_trees code and the patches to couch. I have a couple of comments:
1) The free_cache_entry code could save one get call by using take_smallest/1 and take_largest/1 instead of smallest/1 and largest/1 on the access time tree, right?
2) Similarly, I think you could skip the gb_trees:delete(Key, Items) call in the put handler and use gb_trees:enter instead of insert at the end of the handler.
3) Unless I'm missing something I believe you're leaking a cache process during DB and view group compaction. A new_btree_cache() is created during couch_db_updater:init_db() and couch_view_group:reset_group(), but the old cache is not destroyed when either compaction finishes.
Cheers, Adam
> Caching of BTree nodes and documents
> ------------------------------------
>
> Key: COUCHDB-913
> URL: https://issues.apache.org/jira/browse/COUCHDB-913
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> The following patch adds optional caching of BTree nodes (for both DBs and view indexes) and documents:
> http://github.com/fdmanana/couchdb/commit/c6361d1e0d86942fc86350bea9ca1e8d7539c1b3
> Look at the new parameters introduced in default.ini.tpl.in for more details.
> This is based on the work initially done by Randall Leeds, found at:
> http://github.com/tilgovi/couchdb/commits/forevercache
> I'll not commit it before 1.1 nor without any feedback.
> cheers
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-913) Caching of BTree nodes and
documents
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919593#action_12919593 ]
Filipe Manana commented on COUCHDB-913:
---------------------------------------
An alternative scheme for the .ini configuration, would be along the lines:
[database_btree_cache]
somedb1 = [{size, 50}, {policy, lru}]
somedb2 = [{size, 200}, {policy, mru}]
_default = [{size, 100}, {policy, lru}]
[doc_cache]
somedb1 = [{size, 10}, {policy, lru}]
somedb2 = [{size, 100}, {policy, mru}]
_default = [{size, 0}] ; disabled for all other databases
[view_group_btree_cache]
_design/foobar = [{size, 300}, {policy, lru}]
_design/focus = [{size, 50}, {policy, lru}]
_default = [{size, 100}, {policy, lru}]
Instead of the scheme in patch:
[couchdb]
database_btree_cache_size = 100 ; max number of cached BTree nodes per database (set to 0 to disable)
database_btree_cache_policy = lru ; must be lru or mru
view_btree_cache_size = 100 ; max number of cached BTree nodes per view group (set to 0 to disable)
view_btree_cache_policy = lru ; must be lru or mru
doc_cache_size = 0 ; max number of cached documents per database (set to 0 to disable)
> Caching of BTree nodes and documents
> ------------------------------------
>
> Key: COUCHDB-913
> URL: https://issues.apache.org/jira/browse/COUCHDB-913
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> The following patch adds optional caching of BTree nodes (for both DBs and view indexes) and documents:
> http://github.com/fdmanana/couchdb/commit/c6361d1e0d86942fc86350bea9ca1e8d7539c1b3
> Look at the new parameters introduced in default.ini.tpl.in for more details.
> This is based on the work initially done by Randall Leeds, found at:
> http://github.com/tilgovi/couchdb/commits/forevercache
> I'll not commit it before 1.1 nor without any feedback.
> cheers
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-913) Caching of BTree nodes and
documents
Posted by "Paul Joseph Davis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923697#action_12923697 ]
Paul Joseph Davis commented on COUCHDB-913:
-------------------------------------------
is there a way to look at the difference in CPU in RAM usage due to the different setups. I'm wondering how much of a tradeoff this would be.
> Caching of BTree nodes and documents
> ------------------------------------
>
> Key: COUCHDB-913
> URL: https://issues.apache.org/jira/browse/COUCHDB-913
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> The following patch adds optional caching of BTree nodes (for both DBs and view indexes) and documents:
> http://github.com/fdmanana/couchdb/commit/c6361d1e0d86942fc86350bea9ca1e8d7539c1b3
> Look at the new parameters introduced in default.ini.tpl.in for more details.
> This is based on the work initially done by Randall Leeds, found at:
> http://github.com/tilgovi/couchdb/commits/forevercache
> I'll not commit it before 1.1 nor without any feedback.
> cheers
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-913) Caching of BTree nodes and
documents
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923700#action_12923700 ]
Filipe Manana commented on COUCHDB-913:
---------------------------------------
For a few (2, 3) DBs opened, with a cache size of 100, it doesn't seem noticeable. Measured by looking at 'top' on Linux.
Unfortunately relaximation doesn't measure CPU and memory usage (or some other CouchDB perfomance test tool).
Nevertheless, any of the caches can be enabled or disabled at a DB/View group level or globally. Just wondering if the current defaults are ok.
> Caching of BTree nodes and documents
> ------------------------------------
>
> Key: COUCHDB-913
> URL: https://issues.apache.org/jira/browse/COUCHDB-913
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> The following patch adds optional caching of BTree nodes (for both DBs and view indexes) and documents:
> http://github.com/fdmanana/couchdb/commit/c6361d1e0d86942fc86350bea9ca1e8d7539c1b3
> Look at the new parameters introduced in default.ini.tpl.in for more details.
> This is based on the work initially done by Randall Leeds, found at:
> http://github.com/tilgovi/couchdb/commits/forevercache
> I'll not commit it before 1.1 nor without any feedback.
> cheers
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-913) Caching of BTree nodes and
documents
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923695#action_12923695 ]
Filipe Manana commented on COUCHDB-913:
---------------------------------------
I ran a relaximation test, with the default settings (BTree caches with size 100 and no doc cache):
$ node tests/compare_write_and_read.js --wclients 300 --rclients 150 \
-name1 btree_caching -name2 trunk \
-url1 http://localhost:5984/ -url2 http://localhost:5985/ \
--duration 120
Results are very significant, an improvement in both reads and writes:
http://graphs.mikeal.couchone.com/#/graph/6a174582ead9d38a9200967573028e16
> Caching of BTree nodes and documents
> ------------------------------------
>
> Key: COUCHDB-913
> URL: https://issues.apache.org/jira/browse/COUCHDB-913
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> The following patch adds optional caching of BTree nodes (for both DBs and view indexes) and documents:
> http://github.com/fdmanana/couchdb/commit/c6361d1e0d86942fc86350bea9ca1e8d7539c1b3
> Look at the new parameters introduced in default.ini.tpl.in for more details.
> This is based on the work initially done by Randall Leeds, found at:
> http://github.com/tilgovi/couchdb/commits/forevercache
> I'll not commit it before 1.1 nor without any feedback.
> cheers
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-913) Caching of BTree nodes and
documents
Posted by "Paul Joseph Davis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919612#action_12919612 ]
Paul Joseph Davis commented on COUCHDB-913:
-------------------------------------------
I much prefer the second version for both its clarity and extra flexibility.
> Caching of BTree nodes and documents
> ------------------------------------
>
> Key: COUCHDB-913
> URL: https://issues.apache.org/jira/browse/COUCHDB-913
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> The following patch adds optional caching of BTree nodes (for both DBs and view indexes) and documents:
> http://github.com/fdmanana/couchdb/commit/c6361d1e0d86942fc86350bea9ca1e8d7539c1b3
> Look at the new parameters introduced in default.ini.tpl.in for more details.
> This is based on the work initially done by Randall Leeds, found at:
> http://github.com/tilgovi/couchdb/commits/forevercache
> I'll not commit it before 1.1 nor without any feedback.
> cheers
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-913) Caching of BTree nodes and
documents
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919833#action_12919833 ]
Filipe Manana commented on COUCHDB-913:
---------------------------------------
Updated the branch to use the 2nd (more flexible) configuration.
Full patch diff at:
http://github.com/fdmanana/couchdb/compare/btree_node_and_doc_caching
> Caching of BTree nodes and documents
> ------------------------------------
>
> Key: COUCHDB-913
> URL: https://issues.apache.org/jira/browse/COUCHDB-913
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> The following patch adds optional caching of BTree nodes (for both DBs and view indexes) and documents:
> http://github.com/fdmanana/couchdb/commit/c6361d1e0d86942fc86350bea9ca1e8d7539c1b3
> Look at the new parameters introduced in default.ini.tpl.in for more details.
> This is based on the work initially done by Randall Leeds, found at:
> http://github.com/tilgovi/couchdb/commits/forevercache
> I'll not commit it before 1.1 nor without any feedback.
> cheers
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-913) Caching of BTree nodes and
documents
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923821#action_12923821 ]
Filipe Manana commented on COUCHDB-913:
---------------------------------------
Adam,
Thanks for the review.
I updated the term_cache_trees module: http://github.com/fdmanana/couchdb/commit/e53d37f65f5984bdcf239315a6394f1e8dbd2adb
(it is based on an ets implementation, so it might have still more room for optimizations)
As for the leak after view compaction, I think just the following line solves it: http://github.com/fdmanana/couchdb/commit/4c92720a5b1fb6607a09bef592345fd5d80d40fa
> Caching of BTree nodes and documents
> ------------------------------------
>
> Key: COUCHDB-913
> URL: https://issues.apache.org/jira/browse/COUCHDB-913
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> The following patch adds optional caching of BTree nodes (for both DBs and view indexes) and documents:
> http://github.com/fdmanana/couchdb/commit/c6361d1e0d86942fc86350bea9ca1e8d7539c1b3
> Look at the new parameters introduced in default.ini.tpl.in for more details.
> This is based on the work initially done by Randall Leeds, found at:
> http://github.com/tilgovi/couchdb/commits/forevercache
> I'll not commit it before 1.1 nor without any feedback.
> cheers
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.