You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "nickva (via GitHub)" <gi...@apache.org> on 2026/06/16 06:13:47 UTC
[PR] Speed up bulk doc updates [couchdb]
nickva opened a new pull request, #6039:
URL: https://github.com/apache/couchdb/pull/6039
Previously in `fabric_doc_update` we used a dict keyed by the whole `#doc{}` record. Updating a large batch meant spening a decent amount of CPU time repeatedly hashing the document bodies. To avoid that we use small and lightweight integers as key and while at it, also switch to use maps instead of dicts.
A few more details on the two optimizations
1) Use a lightweight integer tag to match and index documents instead of using their whole `#doc{}` records Using `#doc{}` as a key means applying the `phash/2` function to it, which, in turn, traverses the whole term to produce the hash slot number for the dict implementation
https://github.com/erlang/otp/blob/OTP-29.0.2/lib/stdlib/src/dict.erl#L676. A seemingly obvious choice to use might be a the DocId, however in a batch we can have duplicate DocIds so that won't work. So we borrow an already used pattern in couch_db local doc updater which tags docs in a batch with the `tag` field in the `meta` proplist. This coordinator tag lives only in the coordinator we strip away before sending the data to workers or returning it.
2) Use a map instead of a dict. This one is a no-brainer. A map scales better and uses an internal C based structures. Dict is as just phashes of tuples essentially
I benchmarked lazily with the built-in fabric bench and at last on my laptop the difference is noticeable for the default 1000 doc batches. I ran the benchmarks 3 times alternating between main and PR code.
Environment
----
```
fabric_bench:go().
*** Parameters
* batch_size : 1000
* doc_size : medium
* docs : 100000
* individual_docs : 1000
* n : default
* q : default
*** Environment
* Nodes : 1
* Bench ver. : 1
* N : 1
* Q : 2
* OS : unix/darwin
* Couch ver. : 3.5.2-25b1b9a
* Couch git sha: 25b1b9a
* VM details : Erlang/OTP 27 [erts-15.2.7.9] [source] [64-bit] [smp:12:12]
* [ds:12:12:16] [async-threads:1]
```
Main
----
```
*** Inserting 100000 docs
* Add 100000 docs, ok:100/accepted:0 (Hz): 440
* Get random doc 100000X (Hz): 1600
* All docs (Hz): 49000
* All docs w/ include_docs (Hz): 3900
* Changes (Hz): 24000
* Single doc updates 1000X (Hz): 280
* Time to run all benchmarks (sec): 300
*** Inserting 100000 docs
* Add 100000 docs, ok:100/accepted:0 (Hz): 440
* Get random doc 100000X (Hz): 1600
* All docs (Hz): 46000
* All docs w/ include_docs (Hz): 3900
* Changes (Hz): 16000
* Single doc updates 1000X (Hz): 290
* Time to run all benchmarks (sec): 301
*** Inserting 100000 docs
* Add 100000 docs, ok:100/accepted:0 (Hz): 460
* Get random doc 100000X (Hz): 1700
* All docs (Hz): 48000
* All docs w/ include_docs (Hz): 3900
* Changes (Hz): 21000
* Single doc updates 1000X (Hz): 300
* Time to run all benchmarks (sec): 288
```
PR
---
```
*** Inserting 100000 docs
* Add 100000 docs, ok:100/accepted:0 (Hz): 540
* Get random doc 100000X (Hz): 1600
* All docs (Hz): 49000
* All docs w/ include_docs (Hz): 3900
* Changes (Hz): 25000
* Single doc updates 1000X (Hz): 290
* Time to run all benchmarks (sec): 257
*** Inserting 100000 docs
* Add 100000 docs, ok:100/accepted:0 (Hz): 550
* Get random doc 100000X (Hz): 1700
* All docs (Hz): 45000
* All docs w/ include_docs (Hz): 3900
* Changes (Hz): 22000
* Single doc updates 1000X (Hz): 300
* Time to run all benchmarks (sec): 253
*** Inserting 100000 docs
* Add 100000 docs, ok:100/accepted:0 (Hz): 560
* Get random doc 100000X (Hz): 1600
* All docs (Hz): 46000
* All docs w/ include_docs (Hz): 4000
* Changes (Hz): 29000
* Single doc updates 1000X (Hz): 300
* Time to run all benchmarks (sec): 249
```
If we approximate it's about 20% or so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up bulk doc updates [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva merged PR #6039:
URL: https://github.com/apache/couchdb/pull/6039
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org