You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2021/11/26 19:12:24 UTC
[GitHub] [couchdb-khash] kocolosk opened a new pull request #11: Update toolchain
kocolosk opened a new pull request #11:
URL: https://github.com/apache/couchdb-khash/pull/11
Following the pattern from b64url
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb-khash] kocolosk commented on pull request #11: Update toolchain
Posted by GitBox <gi...@apache.org>.
kocolosk commented on pull request #11:
URL: https://github.com/apache/couchdb-khash/pull/11#issuecomment-980643822
I wrote a short Escript to benchmark khash against Erlang maps and attached it to this PR. It's not my finest demonstration of scientific rigor, but it does provide some guidance. General observations:
- Maps are nearly always faster than khash for lookups, sometimes significantly so
- Maps are typically faster at inserts when the total dataset is ~100 MB or smaller
Larger datasets exhibit a fair amount of noise; I expect this is largely due to garbage collection. Using maps instead of khash instances inside e.g. the `ddoc_cache_lru` gen_server might introduce some undesirable GC pauses for environments where the cache size is quite large. I'm not sure how a map-of-maps would behave from a GC perspective.
On the topic of LRUs, I noticed that we have different implements for the authentication cache (`ets_lru`) and the ddoc cache (which replaced `ets_lru` with the current implementation in apache/couchdb@939761b7183). Meanwhile, the Erlang Solutions folks seem to have started a new project to create a high-performance caching library at [esl/segmented_cache](https://github.com/esl/segmented_cache) that plays tricks with new OTP features like atomics. It seems like it'd be nice to arrive at a common solution over time there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb-khash] kocolosk commented on pull request #11: Update toolchain
Posted by GitBox <gi...@apache.org>.
kocolosk commented on pull request #11:
URL: https://github.com/apache/couchdb-khash/pull/11#issuecomment-980823833
I was thinking about this a bit more in between various college football games today and remembered the obvious answer for large maps: just use ETS.
I also went back and reviewed the two places where we are using `khash` in the CouchDB codebase today. The first is the `couch_event` application, which is responsible for notifying processes about updates that happen to DBs of interest. It mantains Listener -> Db and Db -> Listener mappings in separate khash instances. This would seem to be an ideal case to replace with maps: I believe the churn rate for listeners is relatively low, while lookups can be quite frequent.
The other current use case is in the `ddoc_cache` application. This one is a bit more subtle. There are two mappings:
1. Pids: khash(Cache Entry Manager Pid -> Cache Key)
2. Dbs: khash(DbName -> khash(DDocId -> khash(Cache Key -> Cache Entry Manager Pid)))
That second mapping showcases a nice property of the khash NIF, namely that nesting the hashes allows one to update a "leaf" hash without needing to update the parent hash. Nested maps would have rather different runtime characteristics. Given that I wouldn't feel confident about refactoring `ddoc_cache` to use maps internally today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb-khash] kocolosk commented on pull request #11: Update toolchain
Posted by GitBox <gi...@apache.org>.
kocolosk commented on pull request #11:
URL: https://github.com/apache/couchdb-khash/pull/11#issuecomment-980365239
Oh, good idea! Better to not need the extra code at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb-khash] nickva commented on pull request #11: Update toolchain
Posted by GitBox <gi...@apache.org>.
nickva commented on pull request #11:
URL: https://github.com/apache/couchdb-khash/pull/11#issuecomment-980326555
A thought I had is that we had created khash before maps were available. With optimized maps available since OTP 18, does khash still perform better than maps? Perhaps the easiest here might be to switch to maps and have one less C dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org