You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "nickva (via GitHub)" <gi...@apache.org> on 2025/08/06 21:09:02 UTC
[PR] Store frequently accessed header metadata terms in a cache [couchdb]
nickva opened a new pull request, #5619:
URL: https://github.com/apache/couchdb/pull/5619
Previously, on every write and _all_docs and other call the props term was always loaded from disk, decompressed and run through binary_to_term. The term is small, and likely will be in the page cache, but still we'd have to go back to the disk to fetch it, in the worst case.
To fix it, load the terms once at startup and cache them into memory. This way we get the best of both worlds: a small header record, since we're using a pointer to a term, and fast interactive operations, since we don't have to do extra reads and deserializations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Store frequently accessed header metadata terms in a cache [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on PR #5619:
URL: https://github.com/apache/couchdb/pull/5619#issuecomment-3164330884
the security object and props are stored in the #db header when we open a db and kept in memory (in couch_db_updater's gen_server state).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Store frequently accessed header metadata terms in a cache [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on PR #5619:
URL: https://github.com/apache/couchdb/pull/5619#issuecomment-3168156962
I think it's a mistake for caching to occur both in front of and behind the PSE abstraction. caching the Props (and using that cache) in the #db{} record where we already cache the SecObj is the better way imo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Store frequently accessed header metadata terms in a cache [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on PR #5619:
URL: https://github.com/apache/couchdb/pull/5619#issuecomment-3161881531
To notice the difference between main and PR can also try this trace pattern:
#### MAIN
`http $DB/db/_all_docs'?limit=0'`
=>
```
> recon_trace:calls({couch_file, pread_term, fun([_, _]) -> message(caller_line()) end}, 1000).
1
19:4:44.127475 <0.5659.0> couch_file:pread_term(<0.1778.0>, 0) {couch_bt_engine,get_header_term,3,{"src/couch_bt_engine.erl",1198}}
19:4:44.128057 <0.5659.0> couch_file:pread_term(<0.1778.0>, 0) {couch_bt_engine,get_header_term,3,{"src/couch_bt_engine.erl",1198}}
19:4:44.128553 <0.5659.0> couch_file:pread_term(<0.1778.0>, 94860) {couch_btree,get_node,2,{"src/couch_btree.erl",474}}
19:4:44.129022 <0.5659.0> couch_file:pread_term(<0.1778.0>, 49484) {couch_btree,get_node,2,{"src/couch_btree.erl",474}}
```
`couch_btree:get_node/2` are expected but `get_header_terms` should be cached...
#### WITH PR
```
> recon_trace:calls({couch_file, pread_term, fun([_, _]) -> message(caller_line()) end}, 1000).
1
19:8:11.302801 <0.4886.0> couch_file:pread_term(<0.949.0>, 86369) {couch_btree,get_node,2,{"src/couch_btree.erl",474}}
19:8:11.303378 <0.4886.0> couch_file:pread_term(<0.949.0>, 82175) {couch_btree,get_node,2,{"src/couch_btree.erl",474}}
```
Just `couch_btree:get_node/2` calls, as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Store frequently accessed header metadata terms in a cache [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on PR #5619:
URL: https://github.com/apache/couchdb/pull/5619#issuecomment-3164538226
> the security object and props are stored in the #db header when we open a db and kept in memory (in couch_db_updater's gen_server state).
I removed the security_ptr for now as it is cached in the couch_db_updater, but left a note in the comments, that will require a larger change, especially that we read that one from db records across nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Store frequently accessed header metadata terms in a cache [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on PR #5619:
URL: https://github.com/apache/couchdb/pull/5619#issuecomment-3168359990
> I think it's a mistake for caching to occur both in front of and behind the PSE abstraction. caching the Props (and using that cache) in the #db{} record where we already cache the SecObj is the better way imo.
This is meant a cache layer just above couch_file. The lower layer (bt_engine) knows better how to cache these things and keep them in sync. Based on the internals of the engine it can be easier to do (check file pointers, etc). This is also the place potentially put a btree roots cache (I might try that next, it might not be too bad to add).
In general I'd remove the security caching in the db record, except that we use that for get_db fabric and we'd have to do a two step release or a config toggle for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Store frequently accessed header metadata terms in a cache [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5619:
URL: https://github.com/apache/couchdb/pull/5619#discussion_r2260847904
##########
src/couch/src/couch_bt_engine.hrl:
##########
@@ -14,8 +14,7 @@
filepath,
fd,
fd_monitor,
- % deprecated but keeping it here to avoid altering the record size
- fsync_options_deprecated,
+ cache = #{},
Review Comment:
yeah, exactly, that one is from when we removed the fsync options (fsync before header, after header, or both, now we always sync both)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Store frequently accessed header metadata terms in a cache [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva closed pull request #5619: Store frequently accessed header metadata terms in a cache
URL: https://github.com/apache/couchdb/pull/5619
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Store frequently accessed header metadata terms in a cache [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on PR #5619:
URL: https://github.com/apache/couchdb/pull/5619#issuecomment-3198698444
I tried another approach in https://github.com/apache/couchdb/pull/5625, there we cache both the terms and header terms, so closing this PR in favor of that one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Store frequently accessed header metadata terms in a cache [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on PR #5619:
URL: https://github.com/apache/couchdb/pull/5619#issuecomment-3161887238
Random thought for a future PR, maybe cache the root nodes from the b-trees as well (with a limit on total size)...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Store frequently accessed header metadata terms in a cache [couchdb]
Posted by "jiahuili430 (via GitHub)" <gi...@apache.org>.
jiahuili430 commented on code in PR #5619:
URL: https://github.com/apache/couchdb/pull/5619#discussion_r2260634165
##########
src/couch/src/couch_bt_engine.hrl:
##########
@@ -14,8 +14,7 @@
filepath,
fd,
fd_monitor,
- % deprecated but keeping it here to avoid altering the record size
- fsync_options_deprecated,
+ cache = #{},
Review Comment:
`fsync_options_deprecated` is a deprecated field and I haven't found any other
source code related to it, so I guess replacing it with `cache` should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org