You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2022/12/15 22:57:11 UTC
[GitHub] [couchdb] nickva opened a new issue, #4308: Benchmark Replicator Optimizations in CouchDB 3.3
nickva opened a new issue, #4308:
URL: https://github.com/apache/couchdb/issues/4308
3.3 had a few replicator optimization improvements this issue to perform some representative benchmark to quantify them.
Previously we ran some local tests on dev laptops but it's interesting to have results from actual servers, hopefully something non-local so we can compare.
The list of PRs in questions would be:
* https://github.com/apache/couchdb/pull/4238
* https://github.com/apache/couchdb/pull/4201
* https://github.com/apache/couchdb/pull/4193
* https://github.com/apache/couchdb/pull/4185
* https://github.com/apache/couchdb/pull/4179
* https://github.com/apache/couchdb/pull/4163
* https://github.com/apache/couchdb/pull/4144
* https://github.com/apache/couchdb/pull/4059
* https://github.com/apache/couchdb/pull/4056
* https://github.com/apache/couchdb/pull/4052
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #4308: Benchmark Replicator Optimizations in CouchDB 3.3
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #4308:
URL: https://github.com/apache/couchdb/issues/4308#issuecomment-1353820482
#### Benchmark Setup:
Couchdyno: https://github.com/cloudant-labs/couchdyno
Document count: **1000000**
Document size: **10Kb**
Document data: https://gist.github.com/nickva/9c0223bedd6f225efd81df51819ae76e
```python
> fh = open("generated10k.json")
> jsonstr = fh.read()
> json10kb = json.loads(jsonstr)
> len(jsonstr)
11487
```
**Source** (UK): 8 vCPUs, Q=8
**Target** (US East): 80 vCPUs, 64GBs memory (replication jobs run here), Q=8
```
couchdyno config:
> - connection_timeout = 30000
> - create_target = False
> - cycle_timeout = 28800
> - delete_before_updating = False
> - http_connections = 20
> - num_branches = 1
> - num_docs = 100000
> - num_revs = 1
> - reset_source = True
> - reset_target = True
> - skip_rev_check = True
> - worker_processes = 4
```
Using these settings:
```ini
[chttpd]
bulk_get_use_batches = true
[replicator]
use_bulk_get = true
[ioq]
concurrency = 20
[ioq.bypass]
interactive = true
```
#### Before (Cloudant release 8310)
```
r.replicate_1_to_n_and_compare(1, num=1000000, revs=1, branches=1, normal=True, reset_target=True, reset_source=True, src_params=json10kb)
> replicated in 4578 sec
> replicated in 4583 sec
> replicated in 4553 sec
```
#### After (Cloudant release 8341)
```
r.replicate_1_to_n_and_compare(1, num=1000000, revs=1, branches=1, normal=True, reset_target=True, reset_source=True, src_params=json10kb)
> replicated in 1518 sec
> replicated in 1548 sec
> replicated in 1518 sec
```
### Results:
```
In [1]: with_optimizations = (1518 + 1548 + 1518) / 3
In [2]: before_optimizations = (4578 + 4553 + 4553) / 3
In [3]: round(before_optimizations / with_optimizations, 1)
Out[3]: 3.0
```
We got a **3x** speed-up for 1 million 10kb docs replicated across the Atlantic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva closed issue #4308: Benchmark Replicator Optimizations in CouchDB 3.3
Posted by GitBox <gi...@apache.org>.
nickva closed issue #4308: Benchmark Replicator Optimizations in CouchDB 3.3
URL: https://github.com/apache/couchdb/issues/4308
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org