You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "nickva (via GitHub)" <gi...@apache.org> on 2025/09/20 05:17:58 UTC
[PR] Cache and store mem3 shard properties in one place only [couchdb]
nickva opened a new pull request, #5672:
URL: https://github.com/apache/couchdb/pull/5672
Previously, shard properties were duplicated across all Q*N shards in the cache. If we needed to access them we loaded all the shards (from ets or disk), and then immediately threw them all away except the first one.
To optimise and clean up properties put them in their own ?OPTS ets table. Item lookup, updating, and cleanup mirrors the behavior of ?SHARDS.
There are a few other related optimisations and cleanups:
* In the `for_docid` function we calculated the hash twice: once, when we calculated the `HashKey` for the ets selector, then again, in the `load_shards_from_disk(DbName, DocId)` if we loaded shards from disk. To optimise it, calculate the `HashKey` once and pass it on as `load_shards_from_disk(DbName, HashKey)`.
* Previously, we didn't cache the properties for the shards dbs itself, so add a way to do that. If shards db changes the changes feed will restart ,and then the shards dbs properties will update again. These properties may be used used in the `_all_docs` call for instance, so having it cached would help not having to load it from disk.
* Remove functions which were not used anywhere, and stop exporting functions which are used locally only: `mem3:engine/1`, `find_dirty_shards/0`, `get_engine_opt/1`, `get_props_opt/1`, `get_shard_props/1`.
* For mem3 shards and opts ets tables, since there could be multiple pending writers in different processes trying to update different entries, it makes sense to also enable `{write_concurrency, auto}` for those public tables. See: https://www.erlang.org/doc/apps/stdlib/ets#new_2_write_concurrency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Cache and store mem3 shard properties in one place only [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5672:
URL: https://github.com/apache/couchdb/pull/5672#discussion_r2365674596
##########
src/mem3/src/mem3_shards.erl:
##########
@@ -577,6 +599,14 @@ filter_shards_by_range(Range, Shards) ->
Shards
).
+cache_shards_db_props() ->
+ DbName = mem3_sync:shards_db(),
+ {ok, Db} = mem3_util:ensure_exists(DbName),
+ DbProps = couch_db:get_props(Db),
+ DbOpts = [{props, DbProps}],
+ ets:insert(?OPTS, {DbName, DbOpts}),
+ couch_db:close(Db).
Review Comment:
Good idea, it wouldn't hurt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Cache and store mem3 shard properties in one place only [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva merged PR #5672:
URL: https://github.com/apache/couchdb/pull/5672
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Cache and store mem3 shard properties in one place only [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on code in PR #5672:
URL: https://github.com/apache/couchdb/pull/5672#discussion_r2365565195
##########
src/mem3/src/mem3_shards.erl:
##########
@@ -577,6 +599,14 @@ filter_shards_by_range(Range, Shards) ->
Shards
).
+cache_shards_db_props() ->
+ DbName = mem3_sync:shards_db(),
+ {ok, Db} = mem3_util:ensure_exists(DbName),
+ DbProps = couch_db:get_props(Db),
+ DbOpts = [{props, DbProps}],
+ ets:insert(?OPTS, {DbName, DbOpts}),
+ couch_db:close(Db).
Review Comment:
try/after here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org