You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2021/04/19 15:13:06 UTC
[GitHub] [couchdb] konrad-ohms opened a new issue #3517: Performance regression on CouchDB v3 while using custom rereduce function
konrad-ohms opened a new issue #3517:
URL: https://github.com/apache/couchdb/issues/3517
I was trying to upgrade my database installation from CouchDB v2.3.1 to v3.1.1 and experienced about 6 time slower performance on particular view invocations.
## Description
[NOTE]: # ( Describe the problem you're encountering. )
[TIP]: # ( Do NOT give us access or passwords to your actual CouchDB! )
## Steps to Reproduce
To make it easier to reproduce, I created a script which sets up CouchDB 2.3.1 and CouchDB 3.1.1 in docker/podman, generates example data and imports the problematic view.
The test script generates 12000 documents and a single design document including the slow view.
The script can be used to measure response times against the local system.
[2021_04_19-couchdb-test.zip](https://github.com/apache/couchdb/files/6337054/2021_04_19-couchdb-test.zip)
## Show usage
```
$ ./couchdb-test.sh
Usage: couchdb-test.sh setup|query|cleanup
```
## Initial setup of scenario to reproduce
The following step takes about 13 min to run, the data generation is not very efficient with curl, but I did not want to force to much requirements on the host except of having bash, curl and docker/podman installed.
```
$ ./couchdb-test.sh setup
CouchDB performance regression test script
Initializing clean CouchDB setups
Cleanup containers if present
Error: failed to evict container: "": failed to find container "couchdb2" in state: no container with name or ID couchdb2 found: no such container
Create container
e0077b6673c9751053d4c9613c38c5de0015a283315a3dd4f125a6242de835af
Cleanup containers if present
Error: failed to evict container: "": failed to find container "couchdb3" in state: no container with name or ID couchdb3 found: no such container
Create container
d114d4a2e6eb28f8f322e85083e46aa033701206a13c1789c6beccc8d457f656
Setup CouchDB2
Waiting for CouchDB2 to become available at http://localhost:3002...
CouchDB is up and running
Creating _users
{"ok":true}
Creating _replicator
{"ok":true}
Creating _global_changes
{"ok":true}
Creating demo
{"ok":true}
Setup CouchDB3
Waiting for CouchDB2 to become available at http://localhost:3003...
CouchDB is up and running
Creating _users
{"ok":true}
Creating _replicator
{"ok":true}
Creating _global_changes
{"ok":true}
Creating demo
{"ok":true}
Creating design doc
{"ok":true,"id":"_design/demo","rev":"1-6732411a147da115d79425ba6aeb71de"}
Creating design doc
{"ok":true,"id":"_design/demo","rev":"1-6732411a147da115d79425ba6aeb71de"}
Generating data
inserting docs cycle 1/2000
{"ok":true,"id":"5d933844b700c58fc1b422deb0000687","rev":"1-8353a76e296fb10a71bfb93b7b0ae73f"}
{"ok":true,"id":"5d933844b700c58fc1b422deb0000cf3","rev":"1-60f7912abd240b2433b1ea010f0699c4"}
{"ok":true,"id":"5d933844b700c58fc1b422deb0001801","rev":"1-1a83628882bcb89401df77e7fcc42fd1"}
{"ok":true,"id":"5d933844b700c58fc1b422deb0002000","rev":"1-8a91cdc871ffb7623369ab81d2b6e980"}
{"ok":true,"id":"5d933844b700c58fc1b422deb00023ce","rev":"1-66ac9947820d8f32ca321a91d11b8794"}
...
inserting docs cycle 1999/2000
{"ok":true,"id":"01594d00a265804fd144a3e55575806e","rev":"1-6620f7ce5b499af7486ba404b43fcf6a"}
{"ok":true,"id":"01594d00a265804fd144a3e555758337","rev":"1-6595ca907b8e04dd56f83c56a1ccb5ee"}
{"ok":true,"id":"01594d00a265804fd144a3e5557584f1","rev":"1-31bfdc81ae9ab7c8d5eb16007f479309"}
{"ok":true,"id":"01594d00a265804fd144a3e555758f9e","rev":"1-b3109111048a9a14bce96b1a6a5df642"}
{"ok":true,"id":"01594d00a265804fd144a3e555759188","rev":"1-37fc4bd5012c2e90c13324cc7911d053"}
{"ok":true,"id":"01594d00a265804fd144a3e555759880","rev":"1-7457236eacd2fc5742491a431bccabdb"}
inserting docs cycle 2000/2000
{"ok":true,"id":"01594d00a265804fd144a3e55575a0a7","rev":"1-f7fac9a3f6ce538d67747271336211c4"}
{"ok":true,"id":"01594d00a265804fd144a3e55575ab65","rev":"1-7e3d2b01acbc416b60e6e5be050dd790"}
{"ok":true,"id":"01594d00a265804fd144a3e55575b4de","rev":"1-c959b425a724ad6346f646d8a63da6cd"}
{"ok":true,"id":"01594d00a265804fd144a3e55575c456","rev":"1-d8bd0d00d0b04b52796cfe73d9a80760"}
{"ok":true,"id":"01594d00a265804fd144a3e55575cbcb","rev":"1-24f6bc4fcfdceae7ff2fae47a68fbf42"}
{"ok":true,"id":"01594d00a265804fd144a3e55575d758","rev":"1-b0dfc903b7f27dc98130c8d51931f5d4"}
```
## Query performance
```
$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"9772359d7167d42054df2c9bca1388c9","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3002
Round: 1/10
real 0m6.258s
user 0m0.013s
sys 0m0.012s
Round: 2/10
real 0m6.526s
user 0m0.014s
sys 0m0.011s
Round: 3/10
real 0m6.477s
user 0m0.010s
sys 0m0.015s
Round: 4/10
real 0m6.156s
user 0m0.011s
sys 0m0.014s
Round: 5/10
real 0m6.185s
user 0m0.015s
sys 0m0.011s
Round: 6/10
real 0m6.082s
user 0m0.015s
sys 0m0.015s
Round: 7/10
real 0m5.932s
user 0m0.012s
sys 0m0.014s
Round: 8/10
real 0m5.594s
user 0m0.013s
sys 0m0.013s
Round: 9/10
real 0m6.258s
user 0m0.012s
sys 0m0.014s
Round: 10/10
real 0m5.796s
user 0m0.009s
sys 0m0.017s
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"5f8a19fe389af901b8044f541b0d3c9f","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10
real 0m42.177s
user 0m0.012s
sys 0m0.016s
Round: 2/10
real 0m39.650s
user 0m0.013s
sys 0m0.016s
Round: 3/10
real 0m40.073s
user 0m0.013s
sys 0m0.014s
Round: 4/10
real 0m40.407s
user 0m0.011s
sys 0m0.017s
Round: 5/10
real 0m39.927s
user 0m0.013s
sys 0m0.015s
Round: 6/10
real 0m38.747s
user 0m0.012s
sys 0m0.016s
Round: 7/10
real 0m38.286s
user 0m0.014s
sys 0m0.013s
Round: 8/10
real 0m38.962s
user 0m0.015s
sys 0m0.014s
Round: 9/10
real 0m37.821s
user 0m0.017s
sys 0m0.012s
Round: 10/10
real 0m37.505s
user 0m0.015s
sys 0m0.013s
```
## Cleanup
```
$ ./couchdb-test.sh cleanup
CouchDB performance regression test script
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b885253d64ea docker.io/library/couchdb:3.1.1 /opt/couchdb/bin/... 2 minutes ago Up 2 minutes ago 0.0.0.0:3003->5984/tcp couchdb3
908b29168c7e docker.io/library/couchdb:2.3.1 /opt/couchdb/bin/... 2 minutes ago Up 2 minutes ago 0.0.0.0:3002->5984/tcp couchdb2
Removing containers couchdb2 and couchdb3 if present
908b29168c7e35b6a0dda9d2723c54ff972e44b8467ccd05b28421d033c48a9d
b885253d64ea11a40cbb2401d4f9e04d6a34cdffcf375c37eb989d19a4f9db3f
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ podman rmi docker.io/library/couchdb:2.3.1 docker.io/library/couchdb:3.1.1
```
## Expected Behaviour
The performance of the view should not decrease, or at least not at that magnitude.
## Your Environment
* CouchDB version used: v2.3.1 and v3.1.1
* Browser name and version: curl
* Operating system and version: I was running the script on RHEL 8.3 with podman version 2.2.1.
## Additional Context
Used map function:
```
function (doc) {
if(doc.object === "ainst" && doc.result) {
var subscription = doc.subscription || doc.tenant;
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 1,
execTimeSum : 0,
execTimeCount : 0,
execTimeMin : Infinity,
execTimeMax : 0
}
if (doc.result.status === "successful") {
stats.successfulCount = 1;
if (doc.result.executionTime) {
stats.execTimeSum = doc.result.executionTime;
stats.execTimeCount = 1;
stats.execTimeMin = doc.result.executionTime;
stats.execTimeMax = doc.result.executionTime;
}
} else if (doc.result.status === "failed") {
stats.failedCount = 1;
} else if (doc.result.status === "canceled") {
stats.canceledCount = 1;
} else if (doc.result.status === "executing") {
stats.executingCount = 1;
} else if (doc.result.status === "unsuccessful") {
stats.unsuccessfulCount = 1;
} else if (doc.result.status === "unknown") {
stats.unknownCount = 1;
}
emit([subscription, doc.aDefId], stats);
}
}
```
Custom reduce function:
```
function (keys, values, rereduce) {
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 0,
execTimeSum : 0,
execTimeCount : 0,
execTimeMin : Infinity,
execTimeMax : 0
}
for (var index in values) {
stats.successfulCount += values[index].successfulCount;
stats.failedCount += values[index].failedCount;
stats.canceledCount += values[index].canceledCount;
stats.executingCount += values[index].executingCount;
stats.unsuccessfulCount += values[index].unsuccessfulCount;
stats.unknownCount += values[index].unknownCount;
stats.totalCount += values[index].totalCount;
if (values[index].execTimeCount) {
stats.execTimeSum += values[index].execTimeSum;
stats.execTimeCount += values[index].execTimeCount;
stats.execTimeMin = Math.min(stats.execTimeMin, values[index].execTimeMin);
stats.execTimeMax = Math.max(stats.execTimeMax, values[index].execTimeMax);
}
}
return stats;
}
```
It might be related to the change in the JavaScript engine in CouchDB 3.0, but I am just guessing https://blog.couchdb.org/2020/02/26/the-road-to-couchdb-3-0-update-to-javascript-engine/
Thanks in advance for any help
Konrad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-909124414
Okay, I validated your fix, please find the outcome below:
First, I wanted to create a patch build which is as close as possible to what I have tested before, so I used the same OS as in the container image I used initially:
```bash
$ podman run --rm -it docker.io/library/couchdb:3.1.1 cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
```
To get a build env matching the OS and its required libraries, I had a look into <https://github.com/apache/couchdb-ci> and built the following image on an 18.04.5 LTS amd64 VM with Docker version 20.10.8, build 3967b7d:
```bash
$ git clone https://github.com/apache/couchdb-ci.git
Cloning into 'couchdb-ci'...
remote: Enumerating objects: 1742, done.
remote: Counting objects: 100% (87/87), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 1742 (delta 55), reused 56 (delta 29), pack-reused 1655
Receiving objects: 100% (1742/1742), 10.24 MiB | 16.11 MiB/s, done.
Resolving deltas: 100% (998/998), done.
$ cd couchdb-ci/
$ ./build.sh platform debian-buster
...
Successfully built d1c29b763a2c
Successfully tagged apache/couchdbci-debian:buster-erlang-20.3.8.26-1
```
From there I used CouchDB v3.1.1 and applied your patch within the container:
```bash
$ docker run --name couchdb-3.1.1-fix-validation -it apache/couchdbci-debian:buster-erlang-20.3.8.26-1
jenkins@2665e52069d0:/$ cd ~
jenkins@2665e52069d0:~$ whoami
jenkins
jenkins@2665e52069d0:~$ pwd
/home/jenkins
jenkins@2665e52069d0:~$ git clone --depth=1 -b 3.1.1 https://github.com/apache/couchdb.git
Cloning into 'couchdb'...
remote: Enumerating objects: 1207, done.
remote: Counting objects: 100% (1207/1207), done.
remote: Compressing objects: 100% (1099/1099), done.
remote: Total 1207 (delta 245), reused 380 (delta 55), pack-reused 0
Receiving objects: 100% (1207/1207), 1.80 MiB | 6.14 MiB/s, done.
Resolving deltas: 100% (245/245), done.
Note: checking out 'ce596c65d9d7f0bc5d9937bcaf6253b343015690'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
jenkins@2665e52069d0:~$ # Apply Jan's patch
jenkins@2665e52069d0:~$ cat <<'EOF' > jan.patch
> diff --git a/share/server/60/rewrite_fun.js b/share/server/60/rewrite_fun.js
> index 1b27a9d14..9b85f953f 100644
> --- a/share/server/60/rewrite_fun.js
> +++ b/share/server/60/rewrite_fun.js
> @@ -15,30 +15,39 @@
> //
> // https://github.com/dmunch/couch-chakra/blob/master/js/normalizeFunction.js
>
> +
> +const cache = {}
> function rewriteFunInt(fun) {
> - const ast = esprima.parse(fun);
> - let idx = ast.body.length - 1;
> - let decl = {};
> -
> - // Search for the first FunctionDeclaration beginning from the end
> - do {
> - decl = ast.body[idx--];
> - } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> - idx++;
> -
> - // If we have a function declaration without an Id, wrap it
> - // in an ExpressionStatement and change it into
> - // a FuntionExpression
> - if (decl.type == "FunctionDeclaration" && decl.id == null) {
> - decl.type = "FunctionExpression";
> - ast.body[idx] = {
> - type: "ExpressionStatement",
> - expression: decl
> - };
> - }
> + const crc = crc32_str(fun)
> + if (cache[crc]) {
> + return cache[crc];
> + } else {
> + const ast = esprima.parse(fun);
> + let idx = ast.body.length - 1;
> + let decl = {};
> +
> + // Search for the first FunctionDeclaration beginning from the end
> + do {
> + decl = ast.body[idx--];
> + } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> + idx++;
> +
> + // If we have a function declaration without an Id, wrap it
> + // in an ExpressionStatement and change it into
> + // a FuntionExpression
> + if (decl.type == "FunctionDeclaration" && decl.id == null) {
> + decl.type = "FunctionExpression";
> + ast.body[idx] = {
> + type: "ExpressionStatement",
> + expression: decl
> + };
> + }
>
> - // Generate source from the rewritten AST
> - return escodegen.generate(ast);
> + // Generate source from the rewritten AST
> + const gen = escodegen.generate(ast);
> + cache[crc] = gen;
> + return gen;
> + }
> }
>
>
> @@ -53,4 +62,51 @@ function rewriteFuns(funsJSON) {
> return rewriteFunInt(fun);
> });
> return JSON.stringify(results);
> -}
> \ No newline at end of file
> +}
> +
> +// nicked from https://github.com/SheetJS/js-crc32/blob/master/crc32.js
> +
> +function signed_crc_table() {
> +var c = 0, table = new Array(256);
> +
> +for(var n =0; n != 256; ++n){
> +c = n;
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +table[n] = c;
> +}
> +
> +return typeof Int32Array !== 'undefined' ? new Int32Array(table) : table;
> +}
> +
> +var T = signed_crc_table();
> +
> +function crc32_str(str, seed) {
> +var C = seed ^ -1;
> +for(var i = 0, L=str.length, c, d; i < L;) {
> +c = str.charCodeAt(i++);
> +if(c < 0x80) {
> +C = (C>>>8) ^ T[(C ^ c)&0xFF];
> +} else if(c < 0x800) {
> +C = (C>>>8) ^ T[(C ^ (192|((c>>6)&31)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +} else if(c >= 0xD800 && c < 0xE000) {
> +c = (c&1023)+64; d = str.charCodeAt(i++)&1023;
> +C = (C>>>8) ^ T[(C ^ (240|((c>>8)&7)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>2)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((d>>6)&15)|((c&3)<<4)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(d&63)))&0xFF];
> +} else {
> +C = (C>>>8) ^ T[(C ^ (224|((c>>12)&15)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>6)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +}
> +}
> +return C ^ -1;
> +}
> EOF
jenkins@2665e52069d0:~$ cd couchdb/
jenkins@2665e52069d0:~/couchdb$ git status
Not currently on any branch.
nothing to commit, working tree clean
jenkins@2665e52069d0:~/couchdb$ git checkout -b 3.1.1-patch-jan
Switched to a new branch '3.1.1-patch-jan'
jenkins@2665e52069d0:~/couchdb$ git apply ../jan.patch
jenkins@2665e52069d0:~/couchdb$ git status
On branch 3.1.1-patch-jan
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: share/server/60/rewrite_fun.js
no changes added to commit (use "git add" and/or "git commit -a")
```
Afterwards I compiled the source:
```bash
jenkins@2665e52069d0:~/couchdb$ ./configure
...
==> couchdb (update-deps)
Updating config from {git,"https://github.com/apache/couchdb-config.git",
{tag,"2.1.7"}}
Updating b64url from {git,"https://github.com/apache/couchdb-b64url.git",
{tag,"1.0.2"}}
Updating ets_lru from {git,"https://github.com/apache/couchdb-ets-lru.git",
{tag,"1.1.0"}}
Updating khash from {git,"https://github.com/apache/couchdb-khash.git",
{tag,"1.1.0"}}
Updating snappy from {git,"https://github.com/apache/couchdb-snappy.git",
{tag,"CouchDB-1.0.4"}}
Updating docs from {git,"https://github.com/apache/couchdb-documentation",
{tag,"3.1.1-RC2"}}
Updating fauxton from {git,"https://github.com/apache/couchdb-fauxton",
{tag,"v1.2.6"}}
Updating folsom from {git,"https://github.com/apache/couchdb-folsom.git",
{tag,"CouchDB-0.8.3"}}
Updating hyper from {git,"https://github.com/apache/couchdb-hyper.git",
{tag,"CouchDB-2.2.0-6"}}
Updating ibrowse from {git,"https://github.com/apache/couchdb-ibrowse.git",
{tag,"CouchDB-4.0.1-1"}}
Updating jiffy from {git,"https://github.com/apache/couchdb-jiffy.git",
{tag,"CouchDB-1.0.4-1"}}
Updating mochiweb from {git,"https://github.com/apache/couchdb-mochiweb.git",
{tag,"v2.20.0"}}
Updating meck from {git,"https://github.com/apache/couchdb-meck.git",
{tag,"0.8.8"}}
Updating recon from {git,"https://github.com/apache/couchdb-recon.git",
{tag,"2.5.0"}}
Updating proper from {git,"https://github.com/proper-testing/proper",
{tag,"v1.3"}}
Updating bear from {git,"https://github.com/apache/couchdb-bear.git",
"008f48aff819126e281d5ccae80a258bf9bf9c30"}
You have configured Apache CouchDB, time to relax. Relax.
# Compile
jenkins@2665e52069d0:~/couchdb$ make release
...
The HTML pages are in build/html.
sphinx-build -b man -a -W -n -A local=1 -D latex_elements.papersize=a4 -d build/doctree src/ build/man
Running Sphinx v1.8.4
making output directory...
loading pickled environment... done
building [mo]: all of 0 po files
building [man]: all source files
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
writing... apachecouchdb.1 { intro/index intro/overview intro/why intro/consistency intro/curl intro/security intro/tour intro/api replication/index replication/intro replication/replicator replication/conflicts replication/protocol ddocs/index ddocs/ddocs ddocs/views/index ddocs/views/intro ddocs/views/collation ddocs/views/joins ddocs/views/nosql ddocs/views/pagination ddocs/search best-practices/index best-practices/documents best-practices/forms best-practices/iso-date best-practices/jsdevel best-practices/views best-practices/reverse-proxies install/index install/unix install/windows install/mac install/freebsd install/docker install/snap install/kubernetes install/search install/upgrading install/troubleshooting setup/index setup/single-node setup/cluster config/index config/intro config/couchdb config/cluster config/couch-peruser config/http config/auth config/compaction config/indexbuilds config/ioq config/logging config/replicator config/query-servers config/misc config/r
esharding cluster/index cluster/theory cluster/nodes cluster/databases cluster/sharding cluster/purging maintenance/index maintenance/compaction maintenance/performance maintenance/backups fauxton/index fauxton/install experimental api/index api/basics api/server/index api/server/common api/server/authn api/server/configuration api/database/index api/database/common api/database/bulk-api api/database/find api/database/shard api/database/changes api/database/compact api/database/security api/database/misc api/document/index api/document/common api/document/attachments api/ddoc/index api/ddoc/common api/ddoc/views api/ddoc/search api/ddoc/render api/ddoc/rewrites api/partitioned-dbs api/local json-structure query-server/index query-server/protocol query-server/javascript query-server/erlang partitioned-dbs/index whatsnew/index whatsnew/3.1 whatsnew/3.0 whatsnew/2.3 whatsnew/2.2 whatsnew/2.1 whatsnew/2.0 whatsnew/1.7 whatsnew/1.6 whatsnew/1.5 whatsnew/1.4 whatsnew/1.3 whatsnew/1.2 what
snew/1.1 whatsnew/1.0 whatsnew/0.11 whatsnew/0.10 whatsnew/0.9 whatsnew/0.8 cve/index cve/2010-0009 cve/2010-2234 cve/2010-3854 cve/2012-5641 cve/2012-5649 cve/2012-5650 cve/2014-2668 cve/2017-12635 cve/2017-12636 cve/2018-11769 cve/2018-17188 cve/2018-8007 cve/2020-1955 about contributing }
build succeeded.
The manual pages are in build/man.
make[1]: Leaving directory '/home/jenkins/couchdb/src/docs'
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN: 'generate' command does not apply to directory /home/jenkins/couchdb
... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.
jenkins@2665e52069d0:~/couchdb$ echo $?
0
```
The compilation seem to have worked, so I created deb packages based on that. The `make dist` seemed not to like uncommited changes, so I added your patch as a commit:
```bash
jenkins@2665e52069d0:~/couchdb$ git config --global user.email "couchdb-validation@no-reply.com"
jenkins@2665e52069d0:~/couchdb$ git config --global user.name "CouchDB validation"
jenkins@2665e52069d0:~/couchdb$ git commit -am "fix: Apply caching"
[3.1.1-patch-jan 6c2d01a] fix: Apply caching
1 file changed, 79 insertions(+), 23 deletions(-)
jenkins@2665e52069d0:~/couchdb$ make dist
==> config (compile)
==> b64url (compile)
==> ets_lru (compile)
==> khash (compile)
==> snappy (compile)
==> bear (compile)
==> meck (compile)
==> folsom (compile)
==> hyper (compile)
==> ibrowse (compile)
==> jiffy (compile)
==> mochiweb (compile)
==> recon (compile)
==> proper (compile)
==> couch_epi (compile)
==> couch_log (compile)
==> chttpd (compile)
==> couch (compile)
Compiling priv/couch_js/1.8.5/http.c
Compiling priv/couch_js/1.8.5/main.c
Compiling priv/couch_js/1.8.5/utf8.c
Compiling priv/couch_js/1.8.5/util.c
==> couch_event (compile)
==> mem3 (compile)
==> couch_index (compile)
==> couch_mrview (compile)
==> couch_replicator (compile)
==> couch_plugins (compile)
==> couch_pse_tests (compile)
==> couch_stats (compile)
==> couch_peruser (compile)
==> couch_tests (compile)
==> ddoc_cache (compile)
==> dreyfus (compile)
==> fabric (compile)
==> global_changes (compile)
==> ioq (compile)
==> jwtf (compile)
==> ken (compile)
==> mango (compile)
==> rexi (compile)
==> setup (compile)
==> smoosh (compile)
==> rel (compile)
==> couchdb (compile)
COUCHDB_GIT_SHA: 6c2d01a
COUCHDB_VERSION: 3.1.1-6c2d01a
COUCHDB_VERSION_SUFFIX: 6c2d01a
DIRTY:
IN_RC:
IN_RELEASE:
ON_TAG:
REL_TAG: 3.1.1
SUB_VSN:
Building Apache CouchDB 3.1.1-6c2d01a
fatal: ambiguous argument '6c976bd..HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Done: apache-couchdb-3.1.1-6c2d01a.tar.gz
```
I copied that tar back to the host OS:
```bash
$ docker cp couchdb-3.1.1-fix-validation:/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a.tar.gz .
$ git clone https://github.com/apache/couchdb-pkg.git
Cloning into 'couchdb-pkg'...
remote: Enumerating objects: 723, done.
remote: Counting objects: 100% (94/94), done.
remote: Compressing objects: 100% (70/70), done.
remote: Total 723 (delta 40), reused 57 (delta 20), pack-reused 629
Receiving objects: 100% (723/723), 25.31 MiB | 23.49 MiB/s, done.
Resolving deltas: 100% (335/335), done.
$ cd couchdb-pkg/
```
The assumed image name was `couchdbdev/<osname>-<codename>-erlang-<erlang-version>`, so I retagged my created one before running the packaging job:
```bash
$ docker tag apache/couchdbci-debian:buster-erlang-20.3.8.26-1 couchdbdev/debian-buster-erlang-20.3.8.26-1
$ ERLANGVERSION=20.3.8.26-1 ./build.sh couch debian-buster /root/apache-couchdb-3.1.1-6c2d01a.tar.gz
Using apache-couchdb-3.1.1-6c2d01a.tar.gz to build packages...
mkdir -p ../couchdb
cp apache-couchdb-3.1.1-6c2d01a.tar.gz ../couchdb
cd ../couchdb && tar xfz *.tar.gz
cp debian/control.in debian/control
sed -i 's/%SPIDERMONKEY%/libmozjs-60-0/g' debian/control
sed -i 's/%SPIDERMONKEY_DEV%/libmozjs-60-dev/g' debian/control
echo 'SM_VER = 60' > debian/sm_ver.mk
rm -rf /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/debian
cp -R debian /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dch -v 3.1.1-6c2d01a~buster "Automatically generated package from upstream."
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dpkg-buildpackage -b -us -uc
dpkg-buildpackage: info: source package couchdb
dpkg-buildpackage: info: source version 3.1.1-6c2d01a~buster
dpkg-buildpackage: info: source distribution UNRELEASED
dpkg-buildpackage: info: source changed by "CouchDB Developers" <"dev@couchdb.apache.org">
dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
debian/rules clean
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh clean --with=systemd
debian/rules override_dh_auto_clean
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_clean
make -j1 distclean
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (clean)
==> b64url (clean)
==> ets_lru (clean)
==> khash (clean)
==> snappy (clean)
==> bear (clean)
==> meck (clean)
==> folsom (clean)
==> hyper (clean)
==> ibrowse (clean)
==> jiffy (clean)
==> mochiweb (clean)
==> recon (clean)
==> couch_epi (clean)
==> couch_log (clean)
==> chttpd (clean)
==> couch (clean)
==> couch_event (clean)
==> mem3 (clean)
==> couch_index (clean)
==> couch_mrview (clean)
==> couch_replicator (clean)
==> couch_plugins (clean)
==> couch_pse_tests (clean)
==> couch_stats (clean)
==> couch_peruser (clean)
==> couch_tests (clean)
==> ddoc_cache (clean)
==> dreyfus (clean)
==> fabric (clean)
==> global_changes (clean)
==> ioq (clean)
==> jwtf (clean)
==> ken (clean)
==> mango (clean)
==> rexi (clean)
==> setup (clean)
==> smoosh (clean)
==> rel (clean)
==> apache-couchdb-3.1.1-6c2d01a (clean)
make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
mv src/mango/src/mango_cursor_text.erl.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.erl.nocompile': No such file or directory
make[2]: [debian/rules:21: override_dh_auto_clean] Error 1 (ignored)
mv src/mango/src/mango_cursor_text.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.nocompile': No such file or directory
make[2]: [debian/rules:22: override_dh_auto_clean] Error 1 (ignored)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_clean
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules build
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh build --with=systemd
dh_update_autotools_config
debian/rules override_dh_auto_configure
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
./configure --spidermonkey-version 60
==> configuring couchdb in rel/couchdb.config
You have configured Apache CouchDB, time to relax. Relax.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules override_dh_auto_build
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_build -- release
make -j1 release
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (compile)
Compiled src/config_util.erl
Compiled src/config_writer.erl
Compiled src/config_sup.erl
Compiled src/config_notifier.erl
Compiled src/config_listener_mon.erl
Compiled src/config_listener.erl
Compiled src/config_app.erl
Compiled src/config.erl
==> b64url (compile)
Compiled src/b64url.erl
Compiling c_src/b64url.c
==> ets_lru (compile)
Compiled src/ets_lru.erl
==> khash (compile)
Compiled src/khash.erl
Compiling c_src/hash.c
Compiling c_src/khash.c
==> snappy (compile)
Compiled src/snappy.erl
Compiling c_src/snappy_nif.cc
c_src/snappy_nif.cc: In function 'ERL_NIF_TERM snappy_compress(ErlNifEnv*, int, const ERL_NIF_TERM*)':
c_src/snappy_nif.cc:156:28: warning: catching polymorphic type 'class std::bad_alloc' by value [-Wcatch-value=]
} catch(std::bad_alloc e) {
^
Compiling c_src/snappy/snappy-sinksource.cc
Compiling c_src/snappy/snappy-stubs-internal.cc
Compiling c_src/snappy/snappy.cc
c_src/snappy/snappy.cc: In instantiation of 'bool snappy::SnappyScatteredWriter<Allocator>::AppendFromSelf(size_t, size_t) [with Allocator = snappy::SnappySinkAllocator; size_t = long unsigned int]':
c_src/snappy/snappy.cc:779:13: required from 'void snappy::SnappyDecompressor::DecompressAllTags(Writer*) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>]'
c_src/snappy/snappy.cc:865:3: required from 'bool snappy::InternalUncompressAllTags(snappy::SnappyDecompressor*, Writer*, snappy::uint32) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>; snappy::uint32 = unsigned int]'
c_src/snappy/snappy.cc:1549:78: required from here
c_src/snappy/snappy.cc:1401:21: warning: comparison of integer expressions of different signedness: 'size_t' {aka 'long unsigned int'} and 'long int' [-Wsign-compare]
if (offset - 1u < op_ptr_ - op_base_) {
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
==> bear (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/bear/src/bear.erl:28: Warning: export_all flag enabled - all functions will be exported
Compiled src/bear.erl
==> meck (compile)
Compiled src/meck_util.erl
Compiled src/meck_matcher.erl
Compiled src/meck_ret_spec.erl
Compiled src/meck_history.erl
Compiled src/meck_expect.erl
Compiled src/meck_cover.erl
Compiled src/meck_code.erl
Compiled src/meck_code_gen.erl
Compiled src/meck_proc.erl
Compiled src/meck_args_matcher.erl
Compiled src/meck.erl
==> folsom (compile)
Compiled src/folsom_utils.erl
Compiled src/folsom_sup.erl
Compiled src/folsom_sample_slide_uniform.erl
Compiled src/folsom_sample_uniform.erl
Compiled src/folsom_sample_slide_sup.erl
Compiled src/folsom_sample_slide_server.erl
Compiled src/folsom_sample_slide_sorted.erl
Compiled src/folsom_vm_metrics.erl
Compiled src/folsom_sample_none.erl
Compiled src/folsom_sample_slide.erl
Compiled src/folsom_metrics_spiral.erl
Compiled src/folsom_sample_exdec.erl
Compiled src/folsom_sample.erl
Compiled src/folsom_metrics_meter.erl
Compiled src/folsom_metrics_history.erl
Compiled src/folsom_metrics_meter_reader.erl
Compiled src/folsom_metrics_histogram_ets.erl
Compiled src/folsom_metrics_gauge.erl
Compiled src/folsom_metrics_histogram.erl
Compiled src/folsom_metrics_duration.erl
Compiled src/folsom_metrics_counter.erl
Compiled src/folsom_ewma.erl
Compiled src/folsom_meter_timer_server.erl
Compiled src/folsom.erl
Compiled src/folsom_metrics.erl
Compiled src/folsom_ets.erl
==> hyper (compile)
Compiled src/hyper_register.erl
Compiled src/hyper_carray.erl
Compiled src/hyper_gb.erl
Compiled src/hyper_const.erl
Compiled src/hyper_array.erl
Compiled src/hyper_binary_rle.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:245: Warning: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:288: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:333: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
Compiled src/hyper.erl
Compiled src/hyper_binary.erl
Compiling c_src/hyper_carray.c
==> ibrowse (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_lib.erl:372: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_lib.erl
Compiled src/ibrowse_sup.erl
Compiled src/ibrowse_socks5.erl
Compiled src/ibrowse_lb.erl
Compiled src/ibrowse_app.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:83: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:93: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:171: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:182: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_test.erl
Compiled src/ibrowse.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:428: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:1831: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_http_client.erl
==> jiffy (compile)
Compiling c_src/decoder.c
Compiling c_src/encoder.c
Compiling c_src/jiffy.c
Compiling c_src/termstack.c
Compiling c_src/utf8.c
Compiling c_src/util.c
Compiling c_src/doubles.cc
Compiling c_src/objects.cc
Compiling c_src/double-conversion/bignum-dtoa.cc
Compiling c_src/double-conversion/bignum.cc
Compiling c_src/double-conversion/cached-powers.cc
Compiling c_src/double-conversion/diy-fp.cc
Compiling c_src/double-conversion/double-conversion.cc
Compiling c_src/double-conversion/fast-dtoa.cc
Compiling c_src/double-conversion/fixed-dtoa.cc
Compiling c_src/double-conversion/strtod.cc
Compiled src/jiffy_utf8.erl
Compiled src/jiffy.erl
==> mochiweb (compile)
Compiled src/reloader.erl
Compiled src/mochiweb_websocket.erl
Compiled src/mochiweb_socket.erl
Compiled src/mochiweb_util.erl
Compiled src/mochiweb_session.erl
Compiled src/mochiweb_socket_server.erl
Compiled src/mochiweb_response.erl
Compiled src/mochiweb_mime.erl
Compiled src/mochiweb_multipart.erl
Compiled src/mochiweb_io.erl
Compiled src/mochiweb_http.erl
Compiled src/mochiweb_request.erl
Compiled src/mochiweb_echo.erl
Compiled src/mochiweb_headers.erl
Compiled src/mochiweb_cover.erl
Compiled src/mochiweb_cookies.erl
Compiled src/mochiweb_clock.erl
Compiled src/mochiweb_base64url.erl
Compiled src/mochiweb_acceptor.erl
Compiled src/mochiweb.erl
Compiled src/mochiweb_html.erl
Compiled src/mochiutf8.erl
Compiled src/mochitemp.erl
Compiled src/mochinum.erl
Compiled src/mochilogfile2.erl
Compiled src/mochilists.erl
Compiled src/mochijson.erl
Compiled src/mochihex.erl
Compiled src/mochiglobal.erl
Compiled src/mochifmt_std.erl
Compiled src/mochifmt_records.erl
Compiled src/mochijson2.erl
Compiled src/mochifmt.erl
Compiled src/mochiweb_charref.erl
==> recon (compile)
Compiled src/recon_map.erl
Compiled src/recon_rec.erl
Compiled src/recon_trace.erl
Compiled src/recon_alloc.erl
Compiled src/recon_lib.erl
Compiled src/recon.erl
==> proper (compile)
Compiled src/vararg.erl
Compiled src/proper_target.erl
Compiled src/proper_types.erl
Compiled src/proper_unicode.erl
Compiled src/proper_unused_imports_remover.erl
Compiled src/proper_symb.erl
Compiled src/proper_transformer.erl
Compiled src/proper_shrink.erl
Compiled src/proper_statem.erl
Compiled src/proper_sets.erl
Compiled src/proper_queue.erl
Compiled src/proper_sa.erl
Compiled src/proper_prop_remover.erl
Compiled src/proper_orddict.erl
Compiled src/proper_ordsets.erl
Compiled src/proper_gen.erl
Compiled src/proper_typeserver.erl
Compiled src/proper_gb_trees.erl
Compiled src/proper_gb_sets.erl
Compiled src/proper_gen_next.erl
Compiled src/proper_dict.erl
Compiled src/proper_array.erl
Compiled src/proper_fsm.erl
Compiled src/proper_arith.erl
Compiled src/proper.erl
==> couch_epi (compile)
Compiled src/couch_epi_util.erl
Compiled src/couch_epi_sup.erl
Compiled src/couch_epi_plugin.erl
Compiled src/couch_epi_module_keeper.erl
Compiled src/couch_epi_functions.erl
Compiled src/couch_epi_functions_gen.erl
Compiled src/couch_epi_data_gen.erl
Compiled src/couch_epi_data.erl
Compiled src/couch_epi_codegen.erl
Compiled src/couch_epi_app.erl
Compiled src/couch_epi_codechange_monitor.erl
Compiled src/couch_epi.erl
==> couch_log (compile)
Compiled src/couch_log_writer.erl
Compiled src/couch_log_writer_journald.erl
Compiled src/couch_log_writer_stderr.erl
Compiled src/couch_log_writer_syslog.erl
Compiled src/couch_log_writer_file.erl
Compiled src/couch_log_util.erl
Compiled src/couch_log_sup.erl
Compiled src/couch_log_server.erl
Compiled src/couch_log_monitor.erl
Compiled src/couch_log_trunc_io_fmt.erl
Compiled src/couch_log_error_logger_h.erl
Compiled src/couch_log_config_dyn.erl
Compiled src/couch_log_trunc_io.erl
Compiled src/couch_log_app.erl
Compiled src/couch_log_config.erl
Compiled src/couch_log.erl
Compiled src/couch_log_formatter.erl
==> chttpd (compile)
Compiled src/chttpd.erl
Compiled src/chttpd_xframe_options.erl
Compiled src/chttpd_sup.erl
Compiled src/chttpd_view.erl
Compiled src/chttpd_stats.erl
Compiled src/chttpd_test_util.erl
Compiled src/chttpd_prefer_header.erl
Compiled src/chttpd_rewrite.erl
Compiled src/chttpd_show.erl
Compiled src/chttpd_plugin.erl
Compiled src/chttpd_httpd_handlers.erl
Compiled src/chttpd_handlers.erl
Compiled src/chttpd_node.erl
Compiled src/chttpd_misc.erl
Compiled src/chttpd_epi.erl
Compiled src/chttpd_external.erl
Compiled src/chttpd_cors.erl
Compiled src/chttpd_auth_request.erl
Compiled src/chttpd_auth.erl
Compiled src/chttpd_app.erl
Compiled src/chttpd_auth_cache.erl
Compiled src/chttpd_db.erl
==> couch (compile)
Compiled src/couch_httpd.erl
Compiled src/test_request.erl
Compiled src/couch_work_queue.erl
Compiled src/couch_uuids.erl
Compiled src/couch_users_db.erl
Compiled src/couch_totp.erl
Compiled src/test_util.erl
Compiled src/couch_util.erl
Compiled src/couch_task_status.erl
Compiled src/couch_sup.erl
Compiled src/couch_secondary_sup.erl
Compiled src/couch_rand.erl
Compiled src/couch_stream.erl
Compiled src/couch_proc_manager.erl
Compiled src/couch_query_servers.erl
Compiled src/couch_server.erl
Compiled src/couch_primary_sup.erl
Compiled src/couch_partition.erl
Compiled src/couch_passwords.erl
Compiled src/couch_os_process.erl
Compiled src/couch_lru.erl
Compiled src/couch_multidb_changes.erl
Compiled src/couch_io_logger.erl
Compiled src/couch_native_process.erl
Compiled src/couch_httpd_vhost.erl
Compiled src/couch_key_tree.erl
Compiled src/couch_httpd_rewrite.erl
Compiled src/couch_httpd_handlers.erl
Compiled src/couch_httpd_multipart.erl
Compiled src/couch_httpd_misc_handlers.erl
Compiled src/couch_httpd_external.erl
Compiled src/couch_hotp.erl
Compiled src/couch_hash.erl
Compiled src/couch_flags_config.erl
Compiled src/couch_httpd_auth.erl
Compiled src/couch_flags.erl
Compiled src/couch_event_sup.erl
Compiled src/couch_emsort.erl
Compiled src/couch_ejson_size.erl
Compiled src/couch_file.erl
Compiled src/couch_httpd_db.erl
Compiled src/couch_ejson_compare.erl
Compiled src/couch_drv.erl
Compiled src/couch_debug.erl
Compiled src/couch_doc.erl
Compiled src/couch_db_split.erl
Compiled src/couch_db_plugin.erl
Compiled src/couch_db_epi.erl
Compiled src/couch_db_header.erl
Compiled src/couch_db_updater.erl
Compiled src/couch_compress.erl
Compiled src/couch_db_engine.erl
Compiled src/couch_changes.erl
Compiled src/couch_bt_engine_stream.erl
Compiled src/couch_bt_engine_header.erl
Compiled src/couch_db.erl
Compiled src/couch_btree.erl
Compiled src/couch_base32.erl
Compiled src/couch_bt_engine_compactor.erl
Compiled src/couch_auth_cache.erl
Compiled src/couch_att.erl
Compiled src/couch_bt_engine.erl
Compiled src/couch.erl
Compiled src/couch_app.erl
Compiling priv/couch_js/60/http.cpp
Compiling priv/couch_js/60/main.cpp
Compiling priv/couch_js/60/util.cpp
Compiling priv/icu_driver/couch_icu_driver.c
Compiling priv/couch_ejson_compare/couch_ejson_compare.c
==> couch_event (compile)
Compiled src/couch_event_listener.erl
Compiled src/couch_event_sup2.erl
Compiled src/couch_event_os_listener.erl
Compiled src/couch_event_app.erl
Compiled src/couch_event.erl
Compiled src/couch_event_server.erl
Compiled src/couch_event_listener_mfa.erl
==> mem3 (compile)
Compiled src/mem3_sync_nodes.erl
Compiled src/mem3_sync_security.erl
Compiled src/mem3_sync_event.erl
Compiled src/mem3_sync_event_listener.erl
Compiled src/mem3_sup.erl
Compiled src/mem3_sync.erl
Compiled src/mem3_util.erl
Compiled src/mem3_seeds.erl
Compiled src/mem3_reshard_validate.erl
Compiled src/mem3_reshard_sup.erl
Compiled src/mem3_shards.erl
Compiled src/mem3_rpc.erl
Compiled src/mem3_reshard_job_sup.erl
Compiled src/mem3_reshard_store.erl
Compiled src/mem3_reshard_index.erl
Compiled src/mem3_reshard_httpd.erl
Compiled src/mem3_reshard_job.erl
Compiled src/mem3_reshard_api.erl
Compiled src/mem3_reshard_dbdoc.erl
Compiled src/mem3_plugin_couch_db.erl
Compiled src/mem3_nodes.erl
Compiled src/mem3_httpd_handlers.erl
Compiled src/mem3_reshard.erl
Compiled src/mem3_hash.erl
Compiled src/mem3_epi.erl
Compiled src/mem3_httpd.erl
Compiled src/mem3_rep.erl
Compiled src/mem3_app.erl
Compiled src/mem3_cluster.erl
Compiled src/mem3.erl
==> couch_index (compile)
Compiled src/couch_index_sup.erl
Compiled src/couch_index_util.erl
Compiled src/couch_index_plugin_couch_db.erl
Compiled src/couch_index_updater.erl
Compiled src/couch_index_epi.erl
Compiled src/couch_index_plugin.erl
Compiled src/couch_index_app.erl
Compiled src/couch_index_server.erl
Compiled src/couch_index_compactor.erl
Compiled src/couch_index.erl
==> couch_mrview (compile)
Compiled src/couch_mrview_update_notifier.erl
Compiled src/couch_mrview_updater.erl
Compiled src/couch_mrview_test_util.erl
Compiled src/couch_mrview_show.erl
Compiled src/couch_mrview_util.erl
Compiled src/couch_mrview_index.erl
Compiled src/couch_mrview_compactor.erl
Compiled src/couch_mrview_cleanup.erl
Compiled src/couch_mrview_http.erl
Compiled src/couch_mrview.erl
==> couch_replicator (compile)
Compiled src/couch_replicator_httpc.erl
Compiled src/couch_replicator_auth.erl
Compiled src/couch_replicator_utils.erl
Compiled src/couch_replicator_sup.erl
Compiled src/couch_replicator_stats.erl
Compiled src/couch_replicator_scheduler_sup.erl
Compiled src/json_stream_parse.erl
Compiled src/couch_replicator_worker.erl
Compiled src/couch_replicator_rate_limiter_tables.erl
Compiled src/couch_replicator_rate_limiter.erl
Compiled src/couch_replicator_notifier.erl
Compiled src/couch_replicator_job_sup.erl
Compiled src/couch_replicator_scheduler_job.erl
Compiled src/couch_replicator_scheduler.erl
Compiled src/couch_replicator_ids.erl
Compiled src/couch_replicator_httpd_util.erl
Compiled src/couch_replicator_httpd.erl
Compiled src/couch_replicator_httpc_pool.erl
Compiled src/couch_replicator_filters.erl
Compiled src/couch_replicator_fabric_rpc.erl
Compiled src/couch_replicator_doc_processor_worker.erl
Compiled src/couch_replicator_fabric.erl
Compiled src/couch_replicator_db_changes.erl
Compiled src/couch_replicator_connection.erl
Compiled src/couch_replicator_docs.erl
Compiled src/couch_replicator_changes_reader.erl
Compiled src/couch_replicator_doc_processor.erl
Compiled src/couch_replicator_clustering.erl
Compiled src/couch_replicator_app.erl
Compiled src/couch_replicator_auth_noop.erl
Compiled src/couch_replicator_auth_session.erl
Compiled src/couch_replicator.erl
Compiled src/couch_replicator_api_wrap.erl
==> couch_plugins (compile)
Compiled src/couch_plugins_httpd.erl
Compiled src/couch_plugins.erl
==> couch_pse_tests (compile)
Compiled src/cpse_test_ref_counting.erl
Compiled src/cpse_util.erl
Compiled src/cpse_test_read_write_docs.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/couch_pse_tests/src/cpse_test_purge_seqs.erl:14: Warning: export_all flag enabled - all functions will be exported
Compiled src/cpse_test_purge_seqs.erl
Compiled src/cpse_test_purge_replication.erl
Compiled src/cpse_test_purge_docs.erl
Compiled src/cpse_test_open_close_delete.erl
Compiled src/cpse_test_purge_bad_checkpoints.erl
Compiled src/cpse_test_get_set_props.erl
Compiled src/cpse_test_fold_purge_infos.erl
Compiled src/cpse_test_fold_changes.erl
Compiled src/cpse_test_fold_docs.erl
Compiled src/cpse_test_copy_purge_infos.erl
Compiled src/cpse_gather.erl
Compiled src/cpse_test_attachments.erl
Compiled src/cpse_test_compaction.erl
==> couch_stats (compile)
Compiled src/couch_stats_sup.erl
Compiled src/couch_stats_app.erl
Compiled src/couch_stats_process_tracker.erl
Compiled src/couch_stats.erl
Compiled src/couch_stats_httpd.erl
Compiled src/couch_stats_aggregator.erl
==> couch_peruser (compile)
Compiled src/couch_peruser_app.erl
Compiled src/couch_peruser_sup.erl
Compiled src/couch_peruser.erl
==> couch_tests (compile)
Compiled setups/couch_epi_dispatch.erl
Compiled src/couch_tests_combinatorics.erl
Compiled src/couch_tests.erl
==> ddoc_cache (compile)
Compiled src/ddoc_cache_value.erl
Compiled src/ddoc_cache_sup.erl
Compiled src/ddoc_cache_opener.erl
Compiled src/ddoc_cache_entry_validation_funs.erl
Compiled src/ddoc_cache_entry_ddocid_rev.erl
Compiled src/ddoc_cache_entry_ddocid.erl
Compiled src/ddoc_cache_entry_custom.erl
Compiled src/ddoc_cache_app.erl
Compiled src/ddoc_cache_lru.erl
Compiled src/ddoc_cache.erl
Compiled src/ddoc_cache_entry.erl
==> dreyfus (compile)
Compiled src/dreyfus_sup.erl
Compiled src/dreyfus_plugin_couch_db.erl
Compiled src/dreyfus_rpc.erl
Compiled src/dreyfus_index_updater.erl
Compiled src/dreyfus_util.erl
Compiled src/dreyfus_index_manager.erl
Compiled src/dreyfus_httpd_handlers.erl
Compiled src/dreyfus_index.erl
Compiled src/dreyfus_fabric_search.erl
Compiled src/dreyfus_fabric_info.erl
Compiled src/dreyfus_httpd.erl
Compiled src/dreyfus_fabric_group2.erl
Compiled src/dreyfus_fabric_cleanup.erl
Compiled src/dreyfus_fabric_group1.erl
Compiled src/dreyfus_epi.erl
Compiled src/dreyfus_config.erl
Compiled src/dreyfus_app.erl
Compiled src/dreyfus_fabric.erl
Compiled src/dreyfus_bookmark.erl
Compiled src/clouseau_rpc.erl
==> fabric (compile)
Compiled src/fabric_db_update_listener.erl
Compiled src/fabric_view_reduce.erl
Compiled src/fabric_view_map.erl
Compiled src/fabric_view_all_docs.erl
Compiled src/fabric_view_changes.erl
Compiled src/fabric_view.erl
Compiled src/fabric_streams.erl
Compiled src/fabric_util.erl
Compiled src/fabric_rpc.erl
Compiled src/fabric_ring.erl
Compiled src/fabric_group_info.erl
Compiled src/fabric_doc_update.erl
Compiled src/fabric_doc_purge.erl
Compiled src/fabric_doc_missing_revs.erl
Compiled src/fabric_doc_open_revs.erl
Compiled src/fabric_doc_atts.erl
Compiled src/fabric_doc_open.erl
Compiled src/fabric_dict.erl
Compiled src/fabric_doc_attachments.erl
Compiled src/fabric_design_doc_count.erl
Compiled src/fabric_db_partition_info.erl
Compiled src/fabric_db_meta.erl
Compiled src/fabric_db_doc_count.erl
Compiled src/fabric_db_info.erl
Compiled src/fabric_db_delete.erl
Compiled src/fabric_db_create.erl
Compiled src/fabric.erl
==> global_changes (compile)
Compiled src/global_changes_sup.erl
Compiled src/global_changes_util.erl
Compiled src/global_changes_plugin.erl
Compiled src/global_changes_httpd_handlers.erl
Compiled src/global_changes_listener.erl
Compiled src/global_changes_server.erl
Compiled src/global_changes_epi.erl
Compiled src/global_changes_app.erl
Compiled src/global_changes_httpd.erl
==> ioq (compile)
Compiled src/ioq_app.erl
Compiled src/ioq_sup.erl
Compiled src/ioq.erl
==> jwtf (compile)
Compiled src/jwtf_app.erl
Compiled src/jwtf_sup.erl
Compiled src/jwtf.erl
Compiled src/jwtf_keystore.erl
==> ken (compile)
Compiled src/ken_sup.erl
Compiled src/ken_event_handler.erl
Compiled src/ken_app.erl
Compiled src/ken.erl
Compiled src/ken_server.erl
==> mango (compile)
Compiled src/mango_sup.erl
Compiled src/mango_sort.erl
Compiled src/mango_util.erl
Compiled src/mango_selector_text.erl
Compiled src/mango_selector.erl
Compiled src/mango_opts.erl
Compiled src/mango_json.erl
Compiled src/mango_json_bookmark.erl
Compiled src/mango_native_proc.erl
Compiled src/mango_idx_special.erl
Compiled src/mango_idx_text.erl
Compiled src/mango_httpd_handlers.erl
Compiled src/mango_idx_view.erl
Compiled src/mango_fields.erl
Compiled src/mango_execution_stats.erl
Compiled src/mango_idx.erl
Compiled src/mango_epi.erl
Compiled src/mango_httpd.erl
Compiled src/mango_error.erl
Compiled src/mango_doc.erl
Compiled src/mango_cursor_text.erl
Compiled src/mango_cursor_special.erl
Compiled src/mango_cursor.erl
Compiled src/mango_app.erl
Compiled src/mango_crud.erl
Compiled src/mango_cursor_view.erl
==> rexi (compile)
Compiled src/rexi_server_sup.erl
Compiled src/rexi_sup.erl
Compiled src/rexi_utils.erl
Compiled src/rexi_server_mon.erl
Compiled src/rexi_monitor.erl
Compiled src/rexi_app.erl
Compiled src/rexi_server.erl
Compiled src/rexi_buffer.erl
Compiled src/rexi.erl
==> setup (compile)
Compiled src/setup_sup.erl
Compiled src/setup_httpd_handlers.erl
Compiled src/setup_app.erl
Compiled src/setup_epi.erl
Compiled src/setup_httpd.erl
Compiled src/setup.erl
==> smoosh (compile)
Compiled src/smoosh_sup.erl
Compiled src/smoosh_utils.erl
Compiled src/smoosh_priority_queue.erl
Compiled src/smoosh_app.erl
Compiled src/smoosh_channel.erl
Compiled src/smoosh_server.erl
Compiled src/smoosh.erl
==> rel (compile)
==> apache-couchdb-3.1.1-6c2d01a (compile)
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN: 'generate' command does not apply to directory /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.
make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
rm -rf rel/couchdb/var/log
rm -rf rel/couchdb/data
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules binary
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh binary --with=systemd
dh_testroot
dh_prep
dh_installdirs
dh_install
dh_installdocs
dh_installchangelogs
dh_installman
dh_installdebconf
dh_systemd_enable
dh_installinit
dh_systemd_start
dh_installlogrotate
dh_lintian
dh_perl
dh_link
dh_strip_nondeterminism
dh_compress
dh_fixperms
dh_missing
dh_strip
dh_makeshlibs
debian/rules override_dh_shlibdeps
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_shlibdeps -- --ignore-missing-info -xlibmozjs185-1.0
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_ejson_compare.so debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_icu_driver.so were not linked against libicudata.so.63 (they use none of the library's symbols)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_installdeb
dh_gencontrol
dh_md5sums
debian/rules override_dh_builddeb
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_builddeb -- -Zgzip
dpkg-deb: building package 'couchdb-dbgsym' in '../couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb'.
dpkg-deb: building package 'couchdb' in '../couchdb_3.1.1-6c2d01a~buster_amd64.deb'.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dpkg-genbuildinfo --build=binary
dpkg-genchanges --build=binary >../couchdb_3.1.1-6c2d01a~buster_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
dpkg-source --after-build .
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
dpkg-buildpackage: info: binary-only upload (no source included)
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/.. && lintian --profile couchdb couch*.deb || true
warning: the authors of lintian do not recommend running it with root privileges!
W: couchdb: maintainer-script-should-not-use-recursive-chown-or-chmod postinst:207
N: 3 tags overridden (2 errors, 1 info); 2 unused overrides
chmod a+rwx ../couchdb/couchdb*
mkdir -p pkgs/couch/debian-buster && chmod 777 pkgs/couch/debian-buster
cp ../couchdb/couchdb* pkgs/couch/debian-buster
if [ -f debian/control.bak ]; then mv -f debian/control.bak debian/control; fi
if [ -f rpm/SPECS/couchdb.spec.bak ]; then mv -f rpm/SPECS/couchdb.spec.bak rpm/SPECS/couchdb.spec; fi
rm -rf parts prime stage js/build debian/sm_ver.mk
# packages were created
$ ls -lah pkgs/couch/debian-buster/
total 30M
drwxrwxrwx 2 root root 207 Aug 31 01:58 .
drwxrwxrwx 3 root root 45 Aug 31 01:58 ..
-rwxr-xr-x 1 root root 11K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo
-rwxr-xr-x 1 root root 1.5K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.changes
-rwxr-xr-x 1 root root 30M Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.deb
-rwxr-xr-x 1 root root 474K Aug 31 01:58 couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb
```
Next use https://github.com/apache/couchdb-docker/tree/main/3.1.1 to produce the container images, but adjust them to include the custom couchdb version.
```bash
$ git clone https://github.com/apache/couchdb-docker.git
Cloning into 'couchdb-docker'...
remote: Enumerating objects: 1010, done.
remote: Counting objects: 100% (118/118), done.
remote: Compressing objects: 100% (82/82), done.
remote: Total 1010 (delta 64), reused 64 (delta 32), pack-reused 892
Receiving objects: 100% (1010/1010), 248.11 KiB | 1.19 MiB/s, done.
Resolving deltas: 100% (490/490), done.
$ cd ~/couchdb-docker/3.1.1
$ cp -r ~/couchdb-pkg/pkgs/couch/debian-buster/ .
$ ls debian-buster/
couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo couchdb_3.1.1-6c2d01a~buster_amd64.changes couchdb_3.1.1-6c2d01a~buster_amd64.deb couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb
$ git diff
diff --git a/3.1.1/Dockerfile b/3.1.1/Dockerfile
index 75b4b29..59a90d9 100644
--- a/3.1.1/Dockerfile
+++ b/3.1.1/Dockerfile
@@ -60,6 +60,10 @@ RUN . /etc/os-release; \
echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | \
tee /etc/apt/sources.list.d/couchdb.list >/dev/null
+# I know, we should not include deb packages in images, but it should be good enough to validate the fix
+COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+
# https://github.com/apache/couchdb-pkg/blob/master/debian/README.Debian
RUN set -eux; \
apt-get update; \
@@ -67,7 +71,7 @@ RUN set -eux; \
echo "couchdb couchdb/mode select none" | debconf-set-selections; \
# we DO want recommends this time
DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages \
- couchdb="$COUCHDB_VERSION"~buster \
+ /tmp/packages/* \
; \
# Undo symlinks to /var/log and /var/lib
rmdir /var/lib/couchdb /var/log/couchdb; \
docker build -t "couchdb:3.1.1-patch-jan" .
Sending build context to Docker daemon 31.45MB
Step 1/20 : FROM debian:buster-slim
---> 5c3d57f3e47a
Step 2/20 : LABEL maintainer="CouchDB Developers dev@couchdb.apache.org"
---> Using cache
---> 60f37c14a0e1
Step 3/20 : RUN groupadd -g 5984 -r couchdb && useradd -u 5984 -d /opt/couchdb -g couchdb couchdb
---> Using cache
---> 78202346b500
Step 4/20 : RUN set -ex; apt-get update; apt-get install -y --no-install-recommends apt-transport-https ca-certificates dirmngr gnupg ; rm -rf /var/lib/apt/lists/*
---> Using cache
---> fad0f8a736e0
Step 5/20 : RUN set -eux; apt-get update; apt-get install -y --no-install-recommends gosu tini; rm -rf /var/lib/apt/lists/*; gosu nobody true; tini --version
---> Using cache
---> aca1db0792a3
Step 6/20 : ENV GPG_COUCH_KEY 390EF70BB1EA12B2773962950EE62FB37A00258D
---> Using cache
---> d28f3c244c10
Step 7/20 : RUN set -eux; apt-get update; apt-get install -y curl; export GNUPGHOME="$(mktemp -d)"; curl -fL -o keys.asc https://couchdb.apache.org/repo/keys.asc; gpg --batch --import keys.asc; gpg --batch --export "${GPG_COUCH_KEY}" > /usr/share/keyrings/couchdb-archive-keyring.gpg; command -v gpgconf && gpgconf --kill all || :; rm -rf "$GNUPGHOME"; apt-key list; apt purge -y --autoremove curl; rm -rf /var/lib/apt/lists/*
---> Using cache
---> e2949d6cc031
Step 8/20 : ENV COUCHDB_VERSION 3.1.1
---> Using cache
---> baa8089d1bea
Step 9/20 : RUN . /etc/os-release; echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | tee /etc/apt/sources.list.d/couchdb.list >/dev/null
---> Using cache
---> a7933b88e802
Step 10/20 : COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
---> Using cache
---> ded52d34b729
Step 11/20 : COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
---> Using cache
---> eb9b215cb430
Step 12/20 : RUN set -eux; apt-get update; echo "couchdb couchdb/mode select none" | debconf-set-selections; DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/* ; rmdir /var/lib/couchdb /var/log/couchdb; rm /opt/couchdb/data /opt/couchdb/var/log; mkdir -p /opt/couchdb/data /opt/couchdb/var/log; chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log; chmod 777 /opt/couchdb/data /opt/couchdb/var/log; rm /opt/couchdb/etc/default.d/10-filelog.ini; find /opt/couchdb \! \( -user couchdb -group couchdb \) -exec chown -f couchdb:couchdb '{}' +; find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 '{}' +; find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 '{}' +; chmod -f 0777 /opt/couchdb/etc/local.d; rm -rf /var/lib/apt/lists/*;
---> Running in 6f530f886258
+ apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [302 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 https://apache.jfrog.io/artifactory/couchdb-deb buster InRelease [5151 B]
Get:7 https://apache.jfrog.io/artifactory/couchdb-deb buster/main amd64 Packages [3889 B]
Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages [15.2 kB]
Fetched 8473 kB in 3s (3017 kB/s)
Reading package lists...
+ echo couchdb couchdb/mode select none
+ debconf-set-selections
+ DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2 libicu63 libk5crypto3
libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0 libncurses6
libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps psmisc
publicsuffix
Suggested packages:
gpm krb5-doc krb5-user
The following NEW packages will be installed:
couchdb couchdb-dbgsym curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2
libicu63 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0
libncurses6 libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps
psmisc publicsuffix
0 upgraded, 22 newly installed, 0 to remove and 0 not upgraded.
Need to get 13.5 MB/45.0 MB of archives.
After this operation, 104 MB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security buster/updates/main amd64 krb5-locales all 1.17-3+deb10u2 [95.5 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libncurses6 amd64 6.1+20181013-2+deb10u2 [102 kB]
Get:3 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5support0 amd64 1.17-3+deb10u2 [65.7 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 libk5crypto3 amd64 1.17-3+deb10u2 [122 kB]
Get:5 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5-3 amd64 1.17-3+deb10u2 [369 kB]
Get:6 http://security.debian.org/debian-security buster/updates/main amd64 libgssapi-krb5-2 amd64 1.17-3+deb10u2 [158 kB]
Get:7 http://deb.debian.org/debian buster/main amd64 libprocps7 amd64 2:3.3.15-2 [61.7 kB]
Get:8 http://deb.debian.org/debian buster/main amd64 procps amd64 2:3.3.15-2 [259 kB]
Get:9 http://deb.debian.org/debian buster/main amd64 libkeyutils1 amd64 1.6-6 [15.0 kB]
Get:10 http://deb.debian.org/debian buster/main amd64 libnghttp2-14 amd64 1.36.0-2+deb10u1 [85.0 kB]
Get:11 /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb couchdb amd64 3.1.1-6c2d01a~buster [30.9 MB]
Get:12 /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb couchdb-dbgsym amd64 3.1.1-6c2d01a~buster [485 kB]
Get:13 http://deb.debian.org/debian buster/main amd64 libpsl5 amd64 0.20.2-2 [53.7 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2 [60.5 kB]
Get:15 http://deb.debian.org/debian buster/main amd64 libssh2-1 amd64 1.8.0-2.1 [140 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 libcurl4 amd64 7.64.0-4+deb10u2 [332 kB]
Get:17 http://deb.debian.org/debian buster/main amd64 curl amd64 7.64.0-4+deb10u2 [265 kB]
Get:18 http://deb.debian.org/debian buster/main amd64 libicu63 amd64 63.1-6+deb10u1 [8300 kB]
Get:19 http://deb.debian.org/debian buster/main amd64 libmozjs-60-0 amd64 60.2.3-3 [2785 kB]
Get:20 http://deb.debian.org/debian buster/main amd64 libgpm2 amd64 1.20.7-5 [35.1 kB]
Get:21 http://deb.debian.org/debian buster/main amd64 psmisc amd64 23.2-1 [126 kB]
Get:22 http://deb.debian.org/debian buster/main amd64 publicsuffix all 20190415.1030-1 [116 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 13.5 MB in 1s (12.8 MB/s)
Selecting previously unselected package libncurses6:amd64.
(Reading database ... 7118 files and directories currently installed.)
Preparing to unpack .../00-libncurses6_6.1+20181013-2+deb10u2_amd64.deb ...
Unpacking libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Selecting previously unselected package libprocps7:amd64.
Preparing to unpack .../01-libprocps7_2%3a3.3.15-2_amd64.deb ...
Unpacking libprocps7:amd64 (2:3.3.15-2) ...
Selecting previously unselected package procps.
Preparing to unpack .../02-procps_2%3a3.3.15-2_amd64.deb ...
Unpacking procps (2:3.3.15-2) ...
Selecting previously unselected package krb5-locales.
Preparing to unpack .../03-krb5-locales_1.17-3+deb10u2_all.deb ...
Unpacking krb5-locales (1.17-3+deb10u2) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../04-libkeyutils1_1.6-6_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.6-6) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../05-libkrb5support0_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../06-libk5crypto3_1.17-3+deb10u2_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../07-libkrb5-3_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../08-libgssapi-krb5-2_1.17-3+deb10u2_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../09-libnghttp2-14_1.36.0-2+deb10u1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../10-libpsl5_0.20.2-2_amd64.deb ...
Unpacking libpsl5:amd64 (0.20.2-2) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../11-librtmp1_2.4+20151223.gitfa8646d.1-2_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Selecting previously unselected package libssh2-1:amd64.
Preparing to unpack .../12-libssh2-1_1.8.0-2.1_amd64.deb ...
Unpacking libssh2-1:amd64 (1.8.0-2.1) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../13-libcurl4_7.64.0-4+deb10u2_amd64.deb ...
Unpacking libcurl4:amd64 (7.64.0-4+deb10u2) ...
Selecting previously unselected package curl.
Preparing to unpack .../14-curl_7.64.0-4+deb10u2_amd64.deb ...
Unpacking curl (7.64.0-4+deb10u2) ...
Selecting previously unselected package libicu63:amd64.
Preparing to unpack .../15-libicu63_63.1-6+deb10u1_amd64.deb ...
Unpacking libicu63:amd64 (63.1-6+deb10u1) ...
Selecting previously unselected package libmozjs-60-0:amd64.
Preparing to unpack .../16-libmozjs-60-0_60.2.3-3_amd64.deb ...
Unpacking libmozjs-60-0:amd64 (60.2.3-3) ...
Selecting previously unselected package couchdb.
Preparing to unpack .../17-couchdb_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package couchdb-dbgsym.
Preparing to unpack .../18-couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package libgpm2:amd64.
Preparing to unpack .../19-libgpm2_1.20.7-5_amd64.deb ...
Unpacking libgpm2:amd64 (1.20.7-5) ...
Selecting previously unselected package psmisc.
Preparing to unpack .../20-psmisc_23.2-1_amd64.deb ...
Unpacking psmisc (23.2-1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../21-publicsuffix_20190415.1030-1_all.deb ...
Unpacking publicsuffix (20190415.1030-1) ...
Setting up libkeyutils1:amd64 (1.6-6) ...
Setting up libpsl5:amd64 (0.20.2-2) ...
Setting up libgpm2:amd64 (1.20.7-5) ...
Setting up psmisc (23.2-1) ...
Setting up libprocps7:amd64 (2:3.3.15-2) ...
Setting up libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Setting up krb5-locales (1.17-3+deb10u2) ...
Setting up libicu63:amd64 (63.1-6+deb10u1) ...
Setting up libkrb5support0:amd64 (1.17-3+deb10u2) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Setting up libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Setting up libk5crypto3:amd64 (1.17-3+deb10u2) ...
Setting up procps (2:3.3.15-2) ...
update-alternatives: using /usr/bin/w.procps to provide /usr/bin/w (w) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/w.1.gz because associated file /usr/share/man/man1/w.procps.1.gz (of link group w) doesn't exist
Setting up libssh2-1:amd64 (1.8.0-2.1) ...
Setting up libkrb5-3:amd64 (1.17-3+deb10u2) ...
Setting up publicsuffix (20190415.1030-1) ...
Setting up libmozjs-60-0:amd64 (60.2.3-3) ...
Setting up libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Setting up libcurl4:amd64 (7.64.0-4+deb10u2) ...
Setting up curl (7.64.0-4+deb10u2) ...
Setting up couchdb (3.1.1-6c2d01a~buster) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Processing triggers for libc-bin (2.28-10) ...
+ rmdir /var/lib/couchdb /var/log/couchdb
+ rm /opt/couchdb/data /opt/couchdb/var/log
+ mkdir -p /opt/couchdb/data /opt/couchdb/var/log
+ chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log
+ chmod 777 /opt/couchdb/data /opt/couchdb/var/log
+ rm /opt/couchdb/etc/default.d/10-filelog.ini
+ find /opt/couchdb ! ( -user couchdb -group couchdb ) -exec chown -f couchdb:couchdb {} +
+ find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 {} +
+ find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 {} +
+ chmod -f 0777 /opt/couchdb/etc/local.d
+ rm -rf /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_InRelease /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/auxfiles /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/deb.debian.org_debian_dists_buster_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_main_binary-amd64_Packages.lz4
Removing intermediate container 6f530f886258
---> eaea07a7886e
Step 13/20 : COPY --chown=couchdb:couchdb 10-docker-default.ini /opt/couchdb/etc/default.d/
---> f1c9d8e86732
Step 14/20 : COPY --chown=couchdb:couchdb vm.args /opt/couchdb/etc/
---> a984087a4ee3
Step 15/20 : COPY docker-entrypoint.sh /usr/local/bin
---> 66cd3ececce1
Step 16/20 : RUN ln -s usr/local/bin/docker-entrypoint.sh /docker-entrypoint.sh # backwards compat
---> Running in ae10e0e02906
Removing intermediate container ae10e0e02906
---> bf8817264923
Step 17/20 : ENTRYPOINT ["tini", "--", "/docker-entrypoint.sh"]
---> Running in 1485a79c319f
Removing intermediate container 1485a79c319f
---> 737d9f7a9b97
Step 18/20 : VOLUME /opt/couchdb/data
---> Running in 6aa4eafbc054
Removing intermediate container 6aa4eafbc054
---> 3f111d557168
Step 19/20 : EXPOSE 5984 4369 9100
---> Running in 07b6a3204c82
Removing intermediate container 07b6a3204c82
---> c70917bea3fd
Step 20/20 : CMD ["/opt/couchdb/bin/couchdb"]
---> Running in d61719ef6dea
Removing intermediate container d61719ef6dea
---> ca3940266314
Successfully built ca3940266314
Successfully tagged couchdb:3.1.1-patch-jan
```
I pushed the image and pulled it to my workstation again to have the same underlying hardware as in the previous test.
I also adjusted the couchdb-test.sh [couchdb-test-validate-patch-3517.zip](https://github.com/apache/couchdb/files/7083304/couchdb-test-validate-patch-3517.zip) script to use the patched image and compare it against the vanilla CouchDB 3.1.1 image.
```bash
$ ./couchdb-test.sh setup
...
inserting docs cycle 2000/2000
{"ok":true,"id":"a479f05c01bf22c5168ee73617783eb7","rev":"1-f7fac9a3f6ce538d67747271336211c4"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784bb6","rev":"1-7e3d2b01acbc416b60e6e5be050dd790"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784e38","rev":"1-c959b425a724ad6346f646d8a63da6cd"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617785d5b","rev":"1-d8bd0d00d0b04b52796cfe73d9a80760"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617786743","rev":"1-24f6bc4fcfdceae7ff2fae47a68fbf42"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617787715","rev":"1-b0dfc903b7f27dc98130c8d51931f5d4"}
$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"84f2eae60ea04fe192284667cb0e4640","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10
real 0m36.391s
user 0m0.014s
sys 0m0.010s
Round: 2/10
real 0m36.426s
user 0m0.009s
sys 0m0.015s
Round: 3/10
real 0m37.448s
user 0m0.015s
sys 0m0.010s
Round: 4/10
real 0m37.182s
user 0m0.010s
sys 0m0.016s
Round: 5/10
real 0m34.802s
user 0m0.013s
sys 0m0.013s
Round: 6/10
real 0m34.877s
user 0m0.009s
sys 0m0.016s
Round: 7/10
real 0m34.204s
user 0m0.007s
sys 0m0.017s
Round: 8/10
real 0m34.092s
user 0m0.009s
sys 0m0.015s
Round: 9/10
real 0m35.571s
user 0m0.014s
sys 0m0.011s
Round: 10/10
real 0m35.790s
user 0m0.010s
sys 0m0.015s
================ Query CouchDB 3.1.1 patch jan ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"6c2d01a","uuid":"7df6b96ce9f321de839f65a9a6f264f3","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3004
Round: 1/10
real 0m11.194s
user 0m0.012s
sys 0m0.010s
Round: 2/10
real 0m11.224s
user 0m0.014s
sys 0m0.011s
Round: 3/10
real 0m10.966s
user 0m0.012s
sys 0m0.012s
Round: 4/10
real 0m11.212s
user 0m0.013s
sys 0m0.012s
Round: 5/10
real 0m10.696s
user 0m0.016s
sys 0m0.009s
Round: 6/10
real 0m10.859s
user 0m0.013s
sys 0m0.012s
Round: 7/10
real 0m11.342s
user 0m0.012s
sys 0m0.013s
Round: 8/10
real 0m11.276s
user 0m0.011s
sys 0m0.013s
Round: 9/10
real 0m11.357s
user 0m0.015s
sys 0m0.009s
Round: 10/10
real 0m11.542s
user 0m0.014s
sys 0m0.010s
```
That's actually great news, we have an average response time from about 11.2s now compared to 35.7s on the vanilla image :-)
If possible, I would vote for including your patch in a new 3.x build, as it would be more than 3 times faster for the same queries with unchanged client code (at least for the given dataset).
Still, nothing seem to beat build-in reducers, but at least it would not be way slower to run customer reducers on CouchDB 3 vs. CouchDB 2 installations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-909124414
Okay, I validated your fix, please find the outcome below:
## Creating a custom build
First, I wanted to create a patch build which is as close as possible to what I have tested before, so I used the same OS as in the container image I used initially:
```bash
$ podman run --rm -it docker.io/library/couchdb:3.1.1 cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
```
To get a build env matching the OS and its required libraries, I had a look into <https://github.com/apache/couchdb-ci> and built the following image on an 18.04.5 LTS amd64 VM with Docker version 20.10.8, build 3967b7d:
```bash
$ git clone https://github.com/apache/couchdb-ci.git
Cloning into 'couchdb-ci'...
remote: Enumerating objects: 1742, done.
remote: Counting objects: 100% (87/87), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 1742 (delta 55), reused 56 (delta 29), pack-reused 1655
Receiving objects: 100% (1742/1742), 10.24 MiB | 16.11 MiB/s, done.
Resolving deltas: 100% (998/998), done.
$ cd couchdb-ci/
$ ./build.sh platform debian-buster
...
Successfully built d1c29b763a2c
Successfully tagged apache/couchdbci-debian:buster-erlang-20.3.8.26-1
```
From there I used CouchDB v3.1.1 and applied your patch within the container:
```bash
$ docker run --name couchdb-3.1.1-fix-validation -it apache/couchdbci-debian:buster-erlang-20.3.8.26-1
jenkins@2665e52069d0:/$ cd ~
jenkins@2665e52069d0:~$ whoami
jenkins
jenkins@2665e52069d0:~$ pwd
/home/jenkins
jenkins@2665e52069d0:~$ git clone --depth=1 -b 3.1.1 https://github.com/apache/couchdb.git
Cloning into 'couchdb'...
remote: Enumerating objects: 1207, done.
remote: Counting objects: 100% (1207/1207), done.
remote: Compressing objects: 100% (1099/1099), done.
remote: Total 1207 (delta 245), reused 380 (delta 55), pack-reused 0
Receiving objects: 100% (1207/1207), 1.80 MiB | 6.14 MiB/s, done.
Resolving deltas: 100% (245/245), done.
Note: checking out 'ce596c65d9d7f0bc5d9937bcaf6253b343015690'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
jenkins@2665e52069d0:~$ # Apply Jan's patch
jenkins@2665e52069d0:~$ cat <<'EOF' > jan.patch
> diff --git a/share/server/60/rewrite_fun.js b/share/server/60/rewrite_fun.js
> index 1b27a9d14..9b85f953f 100644
> --- a/share/server/60/rewrite_fun.js
> +++ b/share/server/60/rewrite_fun.js
> @@ -15,30 +15,39 @@
> //
> // https://github.com/dmunch/couch-chakra/blob/master/js/normalizeFunction.js
>
> +
> +const cache = {}
> function rewriteFunInt(fun) {
> - const ast = esprima.parse(fun);
> - let idx = ast.body.length - 1;
> - let decl = {};
> -
> - // Search for the first FunctionDeclaration beginning from the end
> - do {
> - decl = ast.body[idx--];
> - } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> - idx++;
> -
> - // If we have a function declaration without an Id, wrap it
> - // in an ExpressionStatement and change it into
> - // a FuntionExpression
> - if (decl.type == "FunctionDeclaration" && decl.id == null) {
> - decl.type = "FunctionExpression";
> - ast.body[idx] = {
> - type: "ExpressionStatement",
> - expression: decl
> - };
> - }
> + const crc = crc32_str(fun)
> + if (cache[crc]) {
> + return cache[crc];
> + } else {
> + const ast = esprima.parse(fun);
> + let idx = ast.body.length - 1;
> + let decl = {};
> +
> + // Search for the first FunctionDeclaration beginning from the end
> + do {
> + decl = ast.body[idx--];
> + } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> + idx++;
> +
> + // If we have a function declaration without an Id, wrap it
> + // in an ExpressionStatement and change it into
> + // a FuntionExpression
> + if (decl.type == "FunctionDeclaration" && decl.id == null) {
> + decl.type = "FunctionExpression";
> + ast.body[idx] = {
> + type: "ExpressionStatement",
> + expression: decl
> + };
> + }
>
> - // Generate source from the rewritten AST
> - return escodegen.generate(ast);
> + // Generate source from the rewritten AST
> + const gen = escodegen.generate(ast);
> + cache[crc] = gen;
> + return gen;
> + }
> }
>
>
> @@ -53,4 +62,51 @@ function rewriteFuns(funsJSON) {
> return rewriteFunInt(fun);
> });
> return JSON.stringify(results);
> -}
> \ No newline at end of file
> +}
> +
> +// nicked from https://github.com/SheetJS/js-crc32/blob/master/crc32.js
> +
> +function signed_crc_table() {
> +var c = 0, table = new Array(256);
> +
> +for(var n =0; n != 256; ++n){
> +c = n;
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +table[n] = c;
> +}
> +
> +return typeof Int32Array !== 'undefined' ? new Int32Array(table) : table;
> +}
> +
> +var T = signed_crc_table();
> +
> +function crc32_str(str, seed) {
> +var C = seed ^ -1;
> +for(var i = 0, L=str.length, c, d; i < L;) {
> +c = str.charCodeAt(i++);
> +if(c < 0x80) {
> +C = (C>>>8) ^ T[(C ^ c)&0xFF];
> +} else if(c < 0x800) {
> +C = (C>>>8) ^ T[(C ^ (192|((c>>6)&31)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +} else if(c >= 0xD800 && c < 0xE000) {
> +c = (c&1023)+64; d = str.charCodeAt(i++)&1023;
> +C = (C>>>8) ^ T[(C ^ (240|((c>>8)&7)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>2)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((d>>6)&15)|((c&3)<<4)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(d&63)))&0xFF];
> +} else {
> +C = (C>>>8) ^ T[(C ^ (224|((c>>12)&15)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>6)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +}
> +}
> +return C ^ -1;
> +}
> EOF
jenkins@2665e52069d0:~$ cd couchdb/
jenkins@2665e52069d0:~/couchdb$ git status
Not currently on any branch.
nothing to commit, working tree clean
jenkins@2665e52069d0:~/couchdb$ git checkout -b 3.1.1-patch-jan
Switched to a new branch '3.1.1-patch-jan'
jenkins@2665e52069d0:~/couchdb$ git apply ../jan.patch
jenkins@2665e52069d0:~/couchdb$ git status
On branch 3.1.1-patch-jan
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: share/server/60/rewrite_fun.js
no changes added to commit (use "git add" and/or "git commit -a")
```
Afterwards I compiled the source:
```bash
jenkins@2665e52069d0:~/couchdb$ ./configure
...
==> couchdb (update-deps)
Updating config from {git,"https://github.com/apache/couchdb-config.git",
{tag,"2.1.7"}}
Updating b64url from {git,"https://github.com/apache/couchdb-b64url.git",
{tag,"1.0.2"}}
Updating ets_lru from {git,"https://github.com/apache/couchdb-ets-lru.git",
{tag,"1.1.0"}}
Updating khash from {git,"https://github.com/apache/couchdb-khash.git",
{tag,"1.1.0"}}
Updating snappy from {git,"https://github.com/apache/couchdb-snappy.git",
{tag,"CouchDB-1.0.4"}}
Updating docs from {git,"https://github.com/apache/couchdb-documentation",
{tag,"3.1.1-RC2"}}
Updating fauxton from {git,"https://github.com/apache/couchdb-fauxton",
{tag,"v1.2.6"}}
Updating folsom from {git,"https://github.com/apache/couchdb-folsom.git",
{tag,"CouchDB-0.8.3"}}
Updating hyper from {git,"https://github.com/apache/couchdb-hyper.git",
{tag,"CouchDB-2.2.0-6"}}
Updating ibrowse from {git,"https://github.com/apache/couchdb-ibrowse.git",
{tag,"CouchDB-4.0.1-1"}}
Updating jiffy from {git,"https://github.com/apache/couchdb-jiffy.git",
{tag,"CouchDB-1.0.4-1"}}
Updating mochiweb from {git,"https://github.com/apache/couchdb-mochiweb.git",
{tag,"v2.20.0"}}
Updating meck from {git,"https://github.com/apache/couchdb-meck.git",
{tag,"0.8.8"}}
Updating recon from {git,"https://github.com/apache/couchdb-recon.git",
{tag,"2.5.0"}}
Updating proper from {git,"https://github.com/proper-testing/proper",
{tag,"v1.3"}}
Updating bear from {git,"https://github.com/apache/couchdb-bear.git",
"008f48aff819126e281d5ccae80a258bf9bf9c30"}
You have configured Apache CouchDB, time to relax. Relax.
# Compile
jenkins@2665e52069d0:~/couchdb$ make release
...
The HTML pages are in build/html.
sphinx-build -b man -a -W -n -A local=1 -D latex_elements.papersize=a4 -d build/doctree src/ build/man
Running Sphinx v1.8.4
making output directory...
loading pickled environment... done
building [mo]: all of 0 po files
building [man]: all source files
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
writing... apachecouchdb.1 { intro/index intro/overview intro/why intro/consistency intro/curl intro/security intro/tour intro/api replication/index replication/intro replication/replicator replication/conflicts replication/protocol ddocs/index ddocs/ddocs ddocs/views/index ddocs/views/intro ddocs/views/collation ddocs/views/joins ddocs/views/nosql ddocs/views/pagination ddocs/search best-practices/index best-practices/documents best-practices/forms best-practices/iso-date best-practices/jsdevel best-practices/views best-practices/reverse-proxies install/index install/unix install/windows install/mac install/freebsd install/docker install/snap install/kubernetes install/search install/upgrading install/troubleshooting setup/index setup/single-node setup/cluster config/index config/intro config/couchdb config/cluster config/couch-peruser config/http config/auth config/compaction config/indexbuilds config/ioq config/logging config/replicator config/query-servers config/misc config/r
esharding cluster/index cluster/theory cluster/nodes cluster/databases cluster/sharding cluster/purging maintenance/index maintenance/compaction maintenance/performance maintenance/backups fauxton/index fauxton/install experimental api/index api/basics api/server/index api/server/common api/server/authn api/server/configuration api/database/index api/database/common api/database/bulk-api api/database/find api/database/shard api/database/changes api/database/compact api/database/security api/database/misc api/document/index api/document/common api/document/attachments api/ddoc/index api/ddoc/common api/ddoc/views api/ddoc/search api/ddoc/render api/ddoc/rewrites api/partitioned-dbs api/local json-structure query-server/index query-server/protocol query-server/javascript query-server/erlang partitioned-dbs/index whatsnew/index whatsnew/3.1 whatsnew/3.0 whatsnew/2.3 whatsnew/2.2 whatsnew/2.1 whatsnew/2.0 whatsnew/1.7 whatsnew/1.6 whatsnew/1.5 whatsnew/1.4 whatsnew/1.3 whatsnew/1.2 what
snew/1.1 whatsnew/1.0 whatsnew/0.11 whatsnew/0.10 whatsnew/0.9 whatsnew/0.8 cve/index cve/2010-0009 cve/2010-2234 cve/2010-3854 cve/2012-5641 cve/2012-5649 cve/2012-5650 cve/2014-2668 cve/2017-12635 cve/2017-12636 cve/2018-11769 cve/2018-17188 cve/2018-8007 cve/2020-1955 about contributing }
build succeeded.
The manual pages are in build/man.
make[1]: Leaving directory '/home/jenkins/couchdb/src/docs'
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN: 'generate' command does not apply to directory /home/jenkins/couchdb
... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.
jenkins@2665e52069d0:~/couchdb$ echo $?
0
```
The compilation seem to have worked, so I created deb packages based on that. The `make dist` seemed not to like uncommited changes, so I added your patch as a commit:
```bash
jenkins@2665e52069d0:~/couchdb$ git config --global user.email "couchdb-validation@no-reply.com"
jenkins@2665e52069d0:~/couchdb$ git config --global user.name "CouchDB validation"
jenkins@2665e52069d0:~/couchdb$ git commit -am "fix: Apply caching"
[3.1.1-patch-jan 6c2d01a] fix: Apply caching
1 file changed, 79 insertions(+), 23 deletions(-)
jenkins@2665e52069d0:~/couchdb$ make dist
==> config (compile)
==> b64url (compile)
==> ets_lru (compile)
==> khash (compile)
==> snappy (compile)
==> bear (compile)
==> meck (compile)
==> folsom (compile)
==> hyper (compile)
==> ibrowse (compile)
==> jiffy (compile)
==> mochiweb (compile)
==> recon (compile)
==> proper (compile)
==> couch_epi (compile)
==> couch_log (compile)
==> chttpd (compile)
==> couch (compile)
Compiling priv/couch_js/1.8.5/http.c
Compiling priv/couch_js/1.8.5/main.c
Compiling priv/couch_js/1.8.5/utf8.c
Compiling priv/couch_js/1.8.5/util.c
==> couch_event (compile)
==> mem3 (compile)
==> couch_index (compile)
==> couch_mrview (compile)
==> couch_replicator (compile)
==> couch_plugins (compile)
==> couch_pse_tests (compile)
==> couch_stats (compile)
==> couch_peruser (compile)
==> couch_tests (compile)
==> ddoc_cache (compile)
==> dreyfus (compile)
==> fabric (compile)
==> global_changes (compile)
==> ioq (compile)
==> jwtf (compile)
==> ken (compile)
==> mango (compile)
==> rexi (compile)
==> setup (compile)
==> smoosh (compile)
==> rel (compile)
==> couchdb (compile)
COUCHDB_GIT_SHA: 6c2d01a
COUCHDB_VERSION: 3.1.1-6c2d01a
COUCHDB_VERSION_SUFFIX: 6c2d01a
DIRTY:
IN_RC:
IN_RELEASE:
ON_TAG:
REL_TAG: 3.1.1
SUB_VSN:
Building Apache CouchDB 3.1.1-6c2d01a
fatal: ambiguous argument '6c976bd..HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Done: apache-couchdb-3.1.1-6c2d01a.tar.gz
```
I copied that tar back to the host OS:
```bash
$ docker cp couchdb-3.1.1-fix-validation:/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a.tar.gz .
$ git clone https://github.com/apache/couchdb-pkg.git
Cloning into 'couchdb-pkg'...
remote: Enumerating objects: 723, done.
remote: Counting objects: 100% (94/94), done.
remote: Compressing objects: 100% (70/70), done.
remote: Total 723 (delta 40), reused 57 (delta 20), pack-reused 629
Receiving objects: 100% (723/723), 25.31 MiB | 23.49 MiB/s, done.
Resolving deltas: 100% (335/335), done.
$ cd couchdb-pkg/
```
The assumed image name was `couchdbdev/<osname>-<codename>-erlang-<erlang-version>`, so I retagged my created one before running the packaging job:
```bash
$ docker tag apache/couchdbci-debian:buster-erlang-20.3.8.26-1 couchdbdev/debian-buster-erlang-20.3.8.26-1
$ ERLANGVERSION=20.3.8.26-1 ./build.sh couch debian-buster /root/apache-couchdb-3.1.1-6c2d01a.tar.gz
Using apache-couchdb-3.1.1-6c2d01a.tar.gz to build packages...
mkdir -p ../couchdb
cp apache-couchdb-3.1.1-6c2d01a.tar.gz ../couchdb
cd ../couchdb && tar xfz *.tar.gz
cp debian/control.in debian/control
sed -i 's/%SPIDERMONKEY%/libmozjs-60-0/g' debian/control
sed -i 's/%SPIDERMONKEY_DEV%/libmozjs-60-dev/g' debian/control
echo 'SM_VER = 60' > debian/sm_ver.mk
rm -rf /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/debian
cp -R debian /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dch -v 3.1.1-6c2d01a~buster "Automatically generated package from upstream."
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dpkg-buildpackage -b -us -uc
dpkg-buildpackage: info: source package couchdb
dpkg-buildpackage: info: source version 3.1.1-6c2d01a~buster
dpkg-buildpackage: info: source distribution UNRELEASED
dpkg-buildpackage: info: source changed by "CouchDB Developers" <"dev@couchdb.apache.org">
dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
debian/rules clean
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh clean --with=systemd
debian/rules override_dh_auto_clean
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_clean
make -j1 distclean
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (clean)
==> b64url (clean)
==> ets_lru (clean)
==> khash (clean)
==> snappy (clean)
==> bear (clean)
==> meck (clean)
==> folsom (clean)
==> hyper (clean)
==> ibrowse (clean)
==> jiffy (clean)
==> mochiweb (clean)
==> recon (clean)
==> couch_epi (clean)
==> couch_log (clean)
==> chttpd (clean)
==> couch (clean)
==> couch_event (clean)
==> mem3 (clean)
==> couch_index (clean)
==> couch_mrview (clean)
==> couch_replicator (clean)
==> couch_plugins (clean)
==> couch_pse_tests (clean)
==> couch_stats (clean)
==> couch_peruser (clean)
==> couch_tests (clean)
==> ddoc_cache (clean)
==> dreyfus (clean)
==> fabric (clean)
==> global_changes (clean)
==> ioq (clean)
==> jwtf (clean)
==> ken (clean)
==> mango (clean)
==> rexi (clean)
==> setup (clean)
==> smoosh (clean)
==> rel (clean)
==> apache-couchdb-3.1.1-6c2d01a (clean)
make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
mv src/mango/src/mango_cursor_text.erl.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.erl.nocompile': No such file or directory
make[2]: [debian/rules:21: override_dh_auto_clean] Error 1 (ignored)
mv src/mango/src/mango_cursor_text.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.nocompile': No such file or directory
make[2]: [debian/rules:22: override_dh_auto_clean] Error 1 (ignored)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_clean
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules build
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh build --with=systemd
dh_update_autotools_config
debian/rules override_dh_auto_configure
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
./configure --spidermonkey-version 60
==> configuring couchdb in rel/couchdb.config
You have configured Apache CouchDB, time to relax. Relax.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules override_dh_auto_build
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_build -- release
make -j1 release
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (compile)
Compiled src/config_util.erl
Compiled src/config_writer.erl
Compiled src/config_sup.erl
Compiled src/config_notifier.erl
Compiled src/config_listener_mon.erl
Compiled src/config_listener.erl
Compiled src/config_app.erl
Compiled src/config.erl
==> b64url (compile)
Compiled src/b64url.erl
Compiling c_src/b64url.c
==> ets_lru (compile)
Compiled src/ets_lru.erl
==> khash (compile)
Compiled src/khash.erl
Compiling c_src/hash.c
Compiling c_src/khash.c
==> snappy (compile)
Compiled src/snappy.erl
Compiling c_src/snappy_nif.cc
c_src/snappy_nif.cc: In function 'ERL_NIF_TERM snappy_compress(ErlNifEnv*, int, const ERL_NIF_TERM*)':
c_src/snappy_nif.cc:156:28: warning: catching polymorphic type 'class std::bad_alloc' by value [-Wcatch-value=]
} catch(std::bad_alloc e) {
^
Compiling c_src/snappy/snappy-sinksource.cc
Compiling c_src/snappy/snappy-stubs-internal.cc
Compiling c_src/snappy/snappy.cc
c_src/snappy/snappy.cc: In instantiation of 'bool snappy::SnappyScatteredWriter<Allocator>::AppendFromSelf(size_t, size_t) [with Allocator = snappy::SnappySinkAllocator; size_t = long unsigned int]':
c_src/snappy/snappy.cc:779:13: required from 'void snappy::SnappyDecompressor::DecompressAllTags(Writer*) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>]'
c_src/snappy/snappy.cc:865:3: required from 'bool snappy::InternalUncompressAllTags(snappy::SnappyDecompressor*, Writer*, snappy::uint32) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>; snappy::uint32 = unsigned int]'
c_src/snappy/snappy.cc:1549:78: required from here
c_src/snappy/snappy.cc:1401:21: warning: comparison of integer expressions of different signedness: 'size_t' {aka 'long unsigned int'} and 'long int' [-Wsign-compare]
if (offset - 1u < op_ptr_ - op_base_) {
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
==> bear (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/bear/src/bear.erl:28: Warning: export_all flag enabled - all functions will be exported
Compiled src/bear.erl
==> meck (compile)
Compiled src/meck_util.erl
Compiled src/meck_matcher.erl
Compiled src/meck_ret_spec.erl
Compiled src/meck_history.erl
Compiled src/meck_expect.erl
Compiled src/meck_cover.erl
Compiled src/meck_code.erl
Compiled src/meck_code_gen.erl
Compiled src/meck_proc.erl
Compiled src/meck_args_matcher.erl
Compiled src/meck.erl
==> folsom (compile)
Compiled src/folsom_utils.erl
Compiled src/folsom_sup.erl
Compiled src/folsom_sample_slide_uniform.erl
Compiled src/folsom_sample_uniform.erl
Compiled src/folsom_sample_slide_sup.erl
Compiled src/folsom_sample_slide_server.erl
Compiled src/folsom_sample_slide_sorted.erl
Compiled src/folsom_vm_metrics.erl
Compiled src/folsom_sample_none.erl
Compiled src/folsom_sample_slide.erl
Compiled src/folsom_metrics_spiral.erl
Compiled src/folsom_sample_exdec.erl
Compiled src/folsom_sample.erl
Compiled src/folsom_metrics_meter.erl
Compiled src/folsom_metrics_history.erl
Compiled src/folsom_metrics_meter_reader.erl
Compiled src/folsom_metrics_histogram_ets.erl
Compiled src/folsom_metrics_gauge.erl
Compiled src/folsom_metrics_histogram.erl
Compiled src/folsom_metrics_duration.erl
Compiled src/folsom_metrics_counter.erl
Compiled src/folsom_ewma.erl
Compiled src/folsom_meter_timer_server.erl
Compiled src/folsom.erl
Compiled src/folsom_metrics.erl
Compiled src/folsom_ets.erl
==> hyper (compile)
Compiled src/hyper_register.erl
Compiled src/hyper_carray.erl
Compiled src/hyper_gb.erl
Compiled src/hyper_const.erl
Compiled src/hyper_array.erl
Compiled src/hyper_binary_rle.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:245: Warning: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:288: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:333: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
Compiled src/hyper.erl
Compiled src/hyper_binary.erl
Compiling c_src/hyper_carray.c
==> ibrowse (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_lib.erl:372: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_lib.erl
Compiled src/ibrowse_sup.erl
Compiled src/ibrowse_socks5.erl
Compiled src/ibrowse_lb.erl
Compiled src/ibrowse_app.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:83: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:93: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:171: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:182: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_test.erl
Compiled src/ibrowse.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:428: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:1831: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_http_client.erl
==> jiffy (compile)
Compiling c_src/decoder.c
Compiling c_src/encoder.c
Compiling c_src/jiffy.c
Compiling c_src/termstack.c
Compiling c_src/utf8.c
Compiling c_src/util.c
Compiling c_src/doubles.cc
Compiling c_src/objects.cc
Compiling c_src/double-conversion/bignum-dtoa.cc
Compiling c_src/double-conversion/bignum.cc
Compiling c_src/double-conversion/cached-powers.cc
Compiling c_src/double-conversion/diy-fp.cc
Compiling c_src/double-conversion/double-conversion.cc
Compiling c_src/double-conversion/fast-dtoa.cc
Compiling c_src/double-conversion/fixed-dtoa.cc
Compiling c_src/double-conversion/strtod.cc
Compiled src/jiffy_utf8.erl
Compiled src/jiffy.erl
==> mochiweb (compile)
Compiled src/reloader.erl
Compiled src/mochiweb_websocket.erl
Compiled src/mochiweb_socket.erl
Compiled src/mochiweb_util.erl
Compiled src/mochiweb_session.erl
Compiled src/mochiweb_socket_server.erl
Compiled src/mochiweb_response.erl
Compiled src/mochiweb_mime.erl
Compiled src/mochiweb_multipart.erl
Compiled src/mochiweb_io.erl
Compiled src/mochiweb_http.erl
Compiled src/mochiweb_request.erl
Compiled src/mochiweb_echo.erl
Compiled src/mochiweb_headers.erl
Compiled src/mochiweb_cover.erl
Compiled src/mochiweb_cookies.erl
Compiled src/mochiweb_clock.erl
Compiled src/mochiweb_base64url.erl
Compiled src/mochiweb_acceptor.erl
Compiled src/mochiweb.erl
Compiled src/mochiweb_html.erl
Compiled src/mochiutf8.erl
Compiled src/mochitemp.erl
Compiled src/mochinum.erl
Compiled src/mochilogfile2.erl
Compiled src/mochilists.erl
Compiled src/mochijson.erl
Compiled src/mochihex.erl
Compiled src/mochiglobal.erl
Compiled src/mochifmt_std.erl
Compiled src/mochifmt_records.erl
Compiled src/mochijson2.erl
Compiled src/mochifmt.erl
Compiled src/mochiweb_charref.erl
==> recon (compile)
Compiled src/recon_map.erl
Compiled src/recon_rec.erl
Compiled src/recon_trace.erl
Compiled src/recon_alloc.erl
Compiled src/recon_lib.erl
Compiled src/recon.erl
==> proper (compile)
Compiled src/vararg.erl
Compiled src/proper_target.erl
Compiled src/proper_types.erl
Compiled src/proper_unicode.erl
Compiled src/proper_unused_imports_remover.erl
Compiled src/proper_symb.erl
Compiled src/proper_transformer.erl
Compiled src/proper_shrink.erl
Compiled src/proper_statem.erl
Compiled src/proper_sets.erl
Compiled src/proper_queue.erl
Compiled src/proper_sa.erl
Compiled src/proper_prop_remover.erl
Compiled src/proper_orddict.erl
Compiled src/proper_ordsets.erl
Compiled src/proper_gen.erl
Compiled src/proper_typeserver.erl
Compiled src/proper_gb_trees.erl
Compiled src/proper_gb_sets.erl
Compiled src/proper_gen_next.erl
Compiled src/proper_dict.erl
Compiled src/proper_array.erl
Compiled src/proper_fsm.erl
Compiled src/proper_arith.erl
Compiled src/proper.erl
==> couch_epi (compile)
Compiled src/couch_epi_util.erl
Compiled src/couch_epi_sup.erl
Compiled src/couch_epi_plugin.erl
Compiled src/couch_epi_module_keeper.erl
Compiled src/couch_epi_functions.erl
Compiled src/couch_epi_functions_gen.erl
Compiled src/couch_epi_data_gen.erl
Compiled src/couch_epi_data.erl
Compiled src/couch_epi_codegen.erl
Compiled src/couch_epi_app.erl
Compiled src/couch_epi_codechange_monitor.erl
Compiled src/couch_epi.erl
==> couch_log (compile)
Compiled src/couch_log_writer.erl
Compiled src/couch_log_writer_journald.erl
Compiled src/couch_log_writer_stderr.erl
Compiled src/couch_log_writer_syslog.erl
Compiled src/couch_log_writer_file.erl
Compiled src/couch_log_util.erl
Compiled src/couch_log_sup.erl
Compiled src/couch_log_server.erl
Compiled src/couch_log_monitor.erl
Compiled src/couch_log_trunc_io_fmt.erl
Compiled src/couch_log_error_logger_h.erl
Compiled src/couch_log_config_dyn.erl
Compiled src/couch_log_trunc_io.erl
Compiled src/couch_log_app.erl
Compiled src/couch_log_config.erl
Compiled src/couch_log.erl
Compiled src/couch_log_formatter.erl
==> chttpd (compile)
Compiled src/chttpd.erl
Compiled src/chttpd_xframe_options.erl
Compiled src/chttpd_sup.erl
Compiled src/chttpd_view.erl
Compiled src/chttpd_stats.erl
Compiled src/chttpd_test_util.erl
Compiled src/chttpd_prefer_header.erl
Compiled src/chttpd_rewrite.erl
Compiled src/chttpd_show.erl
Compiled src/chttpd_plugin.erl
Compiled src/chttpd_httpd_handlers.erl
Compiled src/chttpd_handlers.erl
Compiled src/chttpd_node.erl
Compiled src/chttpd_misc.erl
Compiled src/chttpd_epi.erl
Compiled src/chttpd_external.erl
Compiled src/chttpd_cors.erl
Compiled src/chttpd_auth_request.erl
Compiled src/chttpd_auth.erl
Compiled src/chttpd_app.erl
Compiled src/chttpd_auth_cache.erl
Compiled src/chttpd_db.erl
==> couch (compile)
Compiled src/couch_httpd.erl
Compiled src/test_request.erl
Compiled src/couch_work_queue.erl
Compiled src/couch_uuids.erl
Compiled src/couch_users_db.erl
Compiled src/couch_totp.erl
Compiled src/test_util.erl
Compiled src/couch_util.erl
Compiled src/couch_task_status.erl
Compiled src/couch_sup.erl
Compiled src/couch_secondary_sup.erl
Compiled src/couch_rand.erl
Compiled src/couch_stream.erl
Compiled src/couch_proc_manager.erl
Compiled src/couch_query_servers.erl
Compiled src/couch_server.erl
Compiled src/couch_primary_sup.erl
Compiled src/couch_partition.erl
Compiled src/couch_passwords.erl
Compiled src/couch_os_process.erl
Compiled src/couch_lru.erl
Compiled src/couch_multidb_changes.erl
Compiled src/couch_io_logger.erl
Compiled src/couch_native_process.erl
Compiled src/couch_httpd_vhost.erl
Compiled src/couch_key_tree.erl
Compiled src/couch_httpd_rewrite.erl
Compiled src/couch_httpd_handlers.erl
Compiled src/couch_httpd_multipart.erl
Compiled src/couch_httpd_misc_handlers.erl
Compiled src/couch_httpd_external.erl
Compiled src/couch_hotp.erl
Compiled src/couch_hash.erl
Compiled src/couch_flags_config.erl
Compiled src/couch_httpd_auth.erl
Compiled src/couch_flags.erl
Compiled src/couch_event_sup.erl
Compiled src/couch_emsort.erl
Compiled src/couch_ejson_size.erl
Compiled src/couch_file.erl
Compiled src/couch_httpd_db.erl
Compiled src/couch_ejson_compare.erl
Compiled src/couch_drv.erl
Compiled src/couch_debug.erl
Compiled src/couch_doc.erl
Compiled src/couch_db_split.erl
Compiled src/couch_db_plugin.erl
Compiled src/couch_db_epi.erl
Compiled src/couch_db_header.erl
Compiled src/couch_db_updater.erl
Compiled src/couch_compress.erl
Compiled src/couch_db_engine.erl
Compiled src/couch_changes.erl
Compiled src/couch_bt_engine_stream.erl
Compiled src/couch_bt_engine_header.erl
Compiled src/couch_db.erl
Compiled src/couch_btree.erl
Compiled src/couch_base32.erl
Compiled src/couch_bt_engine_compactor.erl
Compiled src/couch_auth_cache.erl
Compiled src/couch_att.erl
Compiled src/couch_bt_engine.erl
Compiled src/couch.erl
Compiled src/couch_app.erl
Compiling priv/couch_js/60/http.cpp
Compiling priv/couch_js/60/main.cpp
Compiling priv/couch_js/60/util.cpp
Compiling priv/icu_driver/couch_icu_driver.c
Compiling priv/couch_ejson_compare/couch_ejson_compare.c
==> couch_event (compile)
Compiled src/couch_event_listener.erl
Compiled src/couch_event_sup2.erl
Compiled src/couch_event_os_listener.erl
Compiled src/couch_event_app.erl
Compiled src/couch_event.erl
Compiled src/couch_event_server.erl
Compiled src/couch_event_listener_mfa.erl
==> mem3 (compile)
Compiled src/mem3_sync_nodes.erl
Compiled src/mem3_sync_security.erl
Compiled src/mem3_sync_event.erl
Compiled src/mem3_sync_event_listener.erl
Compiled src/mem3_sup.erl
Compiled src/mem3_sync.erl
Compiled src/mem3_util.erl
Compiled src/mem3_seeds.erl
Compiled src/mem3_reshard_validate.erl
Compiled src/mem3_reshard_sup.erl
Compiled src/mem3_shards.erl
Compiled src/mem3_rpc.erl
Compiled src/mem3_reshard_job_sup.erl
Compiled src/mem3_reshard_store.erl
Compiled src/mem3_reshard_index.erl
Compiled src/mem3_reshard_httpd.erl
Compiled src/mem3_reshard_job.erl
Compiled src/mem3_reshard_api.erl
Compiled src/mem3_reshard_dbdoc.erl
Compiled src/mem3_plugin_couch_db.erl
Compiled src/mem3_nodes.erl
Compiled src/mem3_httpd_handlers.erl
Compiled src/mem3_reshard.erl
Compiled src/mem3_hash.erl
Compiled src/mem3_epi.erl
Compiled src/mem3_httpd.erl
Compiled src/mem3_rep.erl
Compiled src/mem3_app.erl
Compiled src/mem3_cluster.erl
Compiled src/mem3.erl
==> couch_index (compile)
Compiled src/couch_index_sup.erl
Compiled src/couch_index_util.erl
Compiled src/couch_index_plugin_couch_db.erl
Compiled src/couch_index_updater.erl
Compiled src/couch_index_epi.erl
Compiled src/couch_index_plugin.erl
Compiled src/couch_index_app.erl
Compiled src/couch_index_server.erl
Compiled src/couch_index_compactor.erl
Compiled src/couch_index.erl
==> couch_mrview (compile)
Compiled src/couch_mrview_update_notifier.erl
Compiled src/couch_mrview_updater.erl
Compiled src/couch_mrview_test_util.erl
Compiled src/couch_mrview_show.erl
Compiled src/couch_mrview_util.erl
Compiled src/couch_mrview_index.erl
Compiled src/couch_mrview_compactor.erl
Compiled src/couch_mrview_cleanup.erl
Compiled src/couch_mrview_http.erl
Compiled src/couch_mrview.erl
==> couch_replicator (compile)
Compiled src/couch_replicator_httpc.erl
Compiled src/couch_replicator_auth.erl
Compiled src/couch_replicator_utils.erl
Compiled src/couch_replicator_sup.erl
Compiled src/couch_replicator_stats.erl
Compiled src/couch_replicator_scheduler_sup.erl
Compiled src/json_stream_parse.erl
Compiled src/couch_replicator_worker.erl
Compiled src/couch_replicator_rate_limiter_tables.erl
Compiled src/couch_replicator_rate_limiter.erl
Compiled src/couch_replicator_notifier.erl
Compiled src/couch_replicator_job_sup.erl
Compiled src/couch_replicator_scheduler_job.erl
Compiled src/couch_replicator_scheduler.erl
Compiled src/couch_replicator_ids.erl
Compiled src/couch_replicator_httpd_util.erl
Compiled src/couch_replicator_httpd.erl
Compiled src/couch_replicator_httpc_pool.erl
Compiled src/couch_replicator_filters.erl
Compiled src/couch_replicator_fabric_rpc.erl
Compiled src/couch_replicator_doc_processor_worker.erl
Compiled src/couch_replicator_fabric.erl
Compiled src/couch_replicator_db_changes.erl
Compiled src/couch_replicator_connection.erl
Compiled src/couch_replicator_docs.erl
Compiled src/couch_replicator_changes_reader.erl
Compiled src/couch_replicator_doc_processor.erl
Compiled src/couch_replicator_clustering.erl
Compiled src/couch_replicator_app.erl
Compiled src/couch_replicator_auth_noop.erl
Compiled src/couch_replicator_auth_session.erl
Compiled src/couch_replicator.erl
Compiled src/couch_replicator_api_wrap.erl
==> couch_plugins (compile)
Compiled src/couch_plugins_httpd.erl
Compiled src/couch_plugins.erl
==> couch_pse_tests (compile)
Compiled src/cpse_test_ref_counting.erl
Compiled src/cpse_util.erl
Compiled src/cpse_test_read_write_docs.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/couch_pse_tests/src/cpse_test_purge_seqs.erl:14: Warning: export_all flag enabled - all functions will be exported
Compiled src/cpse_test_purge_seqs.erl
Compiled src/cpse_test_purge_replication.erl
Compiled src/cpse_test_purge_docs.erl
Compiled src/cpse_test_open_close_delete.erl
Compiled src/cpse_test_purge_bad_checkpoints.erl
Compiled src/cpse_test_get_set_props.erl
Compiled src/cpse_test_fold_purge_infos.erl
Compiled src/cpse_test_fold_changes.erl
Compiled src/cpse_test_fold_docs.erl
Compiled src/cpse_test_copy_purge_infos.erl
Compiled src/cpse_gather.erl
Compiled src/cpse_test_attachments.erl
Compiled src/cpse_test_compaction.erl
==> couch_stats (compile)
Compiled src/couch_stats_sup.erl
Compiled src/couch_stats_app.erl
Compiled src/couch_stats_process_tracker.erl
Compiled src/couch_stats.erl
Compiled src/couch_stats_httpd.erl
Compiled src/couch_stats_aggregator.erl
==> couch_peruser (compile)
Compiled src/couch_peruser_app.erl
Compiled src/couch_peruser_sup.erl
Compiled src/couch_peruser.erl
==> couch_tests (compile)
Compiled setups/couch_epi_dispatch.erl
Compiled src/couch_tests_combinatorics.erl
Compiled src/couch_tests.erl
==> ddoc_cache (compile)
Compiled src/ddoc_cache_value.erl
Compiled src/ddoc_cache_sup.erl
Compiled src/ddoc_cache_opener.erl
Compiled src/ddoc_cache_entry_validation_funs.erl
Compiled src/ddoc_cache_entry_ddocid_rev.erl
Compiled src/ddoc_cache_entry_ddocid.erl
Compiled src/ddoc_cache_entry_custom.erl
Compiled src/ddoc_cache_app.erl
Compiled src/ddoc_cache_lru.erl
Compiled src/ddoc_cache.erl
Compiled src/ddoc_cache_entry.erl
==> dreyfus (compile)
Compiled src/dreyfus_sup.erl
Compiled src/dreyfus_plugin_couch_db.erl
Compiled src/dreyfus_rpc.erl
Compiled src/dreyfus_index_updater.erl
Compiled src/dreyfus_util.erl
Compiled src/dreyfus_index_manager.erl
Compiled src/dreyfus_httpd_handlers.erl
Compiled src/dreyfus_index.erl
Compiled src/dreyfus_fabric_search.erl
Compiled src/dreyfus_fabric_info.erl
Compiled src/dreyfus_httpd.erl
Compiled src/dreyfus_fabric_group2.erl
Compiled src/dreyfus_fabric_cleanup.erl
Compiled src/dreyfus_fabric_group1.erl
Compiled src/dreyfus_epi.erl
Compiled src/dreyfus_config.erl
Compiled src/dreyfus_app.erl
Compiled src/dreyfus_fabric.erl
Compiled src/dreyfus_bookmark.erl
Compiled src/clouseau_rpc.erl
==> fabric (compile)
Compiled src/fabric_db_update_listener.erl
Compiled src/fabric_view_reduce.erl
Compiled src/fabric_view_map.erl
Compiled src/fabric_view_all_docs.erl
Compiled src/fabric_view_changes.erl
Compiled src/fabric_view.erl
Compiled src/fabric_streams.erl
Compiled src/fabric_util.erl
Compiled src/fabric_rpc.erl
Compiled src/fabric_ring.erl
Compiled src/fabric_group_info.erl
Compiled src/fabric_doc_update.erl
Compiled src/fabric_doc_purge.erl
Compiled src/fabric_doc_missing_revs.erl
Compiled src/fabric_doc_open_revs.erl
Compiled src/fabric_doc_atts.erl
Compiled src/fabric_doc_open.erl
Compiled src/fabric_dict.erl
Compiled src/fabric_doc_attachments.erl
Compiled src/fabric_design_doc_count.erl
Compiled src/fabric_db_partition_info.erl
Compiled src/fabric_db_meta.erl
Compiled src/fabric_db_doc_count.erl
Compiled src/fabric_db_info.erl
Compiled src/fabric_db_delete.erl
Compiled src/fabric_db_create.erl
Compiled src/fabric.erl
==> global_changes (compile)
Compiled src/global_changes_sup.erl
Compiled src/global_changes_util.erl
Compiled src/global_changes_plugin.erl
Compiled src/global_changes_httpd_handlers.erl
Compiled src/global_changes_listener.erl
Compiled src/global_changes_server.erl
Compiled src/global_changes_epi.erl
Compiled src/global_changes_app.erl
Compiled src/global_changes_httpd.erl
==> ioq (compile)
Compiled src/ioq_app.erl
Compiled src/ioq_sup.erl
Compiled src/ioq.erl
==> jwtf (compile)
Compiled src/jwtf_app.erl
Compiled src/jwtf_sup.erl
Compiled src/jwtf.erl
Compiled src/jwtf_keystore.erl
==> ken (compile)
Compiled src/ken_sup.erl
Compiled src/ken_event_handler.erl
Compiled src/ken_app.erl
Compiled src/ken.erl
Compiled src/ken_server.erl
==> mango (compile)
Compiled src/mango_sup.erl
Compiled src/mango_sort.erl
Compiled src/mango_util.erl
Compiled src/mango_selector_text.erl
Compiled src/mango_selector.erl
Compiled src/mango_opts.erl
Compiled src/mango_json.erl
Compiled src/mango_json_bookmark.erl
Compiled src/mango_native_proc.erl
Compiled src/mango_idx_special.erl
Compiled src/mango_idx_text.erl
Compiled src/mango_httpd_handlers.erl
Compiled src/mango_idx_view.erl
Compiled src/mango_fields.erl
Compiled src/mango_execution_stats.erl
Compiled src/mango_idx.erl
Compiled src/mango_epi.erl
Compiled src/mango_httpd.erl
Compiled src/mango_error.erl
Compiled src/mango_doc.erl
Compiled src/mango_cursor_text.erl
Compiled src/mango_cursor_special.erl
Compiled src/mango_cursor.erl
Compiled src/mango_app.erl
Compiled src/mango_crud.erl
Compiled src/mango_cursor_view.erl
==> rexi (compile)
Compiled src/rexi_server_sup.erl
Compiled src/rexi_sup.erl
Compiled src/rexi_utils.erl
Compiled src/rexi_server_mon.erl
Compiled src/rexi_monitor.erl
Compiled src/rexi_app.erl
Compiled src/rexi_server.erl
Compiled src/rexi_buffer.erl
Compiled src/rexi.erl
==> setup (compile)
Compiled src/setup_sup.erl
Compiled src/setup_httpd_handlers.erl
Compiled src/setup_app.erl
Compiled src/setup_epi.erl
Compiled src/setup_httpd.erl
Compiled src/setup.erl
==> smoosh (compile)
Compiled src/smoosh_sup.erl
Compiled src/smoosh_utils.erl
Compiled src/smoosh_priority_queue.erl
Compiled src/smoosh_app.erl
Compiled src/smoosh_channel.erl
Compiled src/smoosh_server.erl
Compiled src/smoosh.erl
==> rel (compile)
==> apache-couchdb-3.1.1-6c2d01a (compile)
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN: 'generate' command does not apply to directory /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.
make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
rm -rf rel/couchdb/var/log
rm -rf rel/couchdb/data
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules binary
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh binary --with=systemd
dh_testroot
dh_prep
dh_installdirs
dh_install
dh_installdocs
dh_installchangelogs
dh_installman
dh_installdebconf
dh_systemd_enable
dh_installinit
dh_systemd_start
dh_installlogrotate
dh_lintian
dh_perl
dh_link
dh_strip_nondeterminism
dh_compress
dh_fixperms
dh_missing
dh_strip
dh_makeshlibs
debian/rules override_dh_shlibdeps
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_shlibdeps -- --ignore-missing-info -xlibmozjs185-1.0
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_ejson_compare.so debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_icu_driver.so were not linked against libicudata.so.63 (they use none of the library's symbols)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_installdeb
dh_gencontrol
dh_md5sums
debian/rules override_dh_builddeb
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_builddeb -- -Zgzip
dpkg-deb: building package 'couchdb-dbgsym' in '../couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb'.
dpkg-deb: building package 'couchdb' in '../couchdb_3.1.1-6c2d01a~buster_amd64.deb'.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dpkg-genbuildinfo --build=binary
dpkg-genchanges --build=binary >../couchdb_3.1.1-6c2d01a~buster_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
dpkg-source --after-build .
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
dpkg-buildpackage: info: binary-only upload (no source included)
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/.. && lintian --profile couchdb couch*.deb || true
warning: the authors of lintian do not recommend running it with root privileges!
W: couchdb: maintainer-script-should-not-use-recursive-chown-or-chmod postinst:207
N: 3 tags overridden (2 errors, 1 info); 2 unused overrides
chmod a+rwx ../couchdb/couchdb*
mkdir -p pkgs/couch/debian-buster && chmod 777 pkgs/couch/debian-buster
cp ../couchdb/couchdb* pkgs/couch/debian-buster
if [ -f debian/control.bak ]; then mv -f debian/control.bak debian/control; fi
if [ -f rpm/SPECS/couchdb.spec.bak ]; then mv -f rpm/SPECS/couchdb.spec.bak rpm/SPECS/couchdb.spec; fi
rm -rf parts prime stage js/build debian/sm_ver.mk
# packages were created
$ ls -lah pkgs/couch/debian-buster/
total 30M
drwxrwxrwx 2 root root 207 Aug 31 01:58 .
drwxrwxrwx 3 root root 45 Aug 31 01:58 ..
-rwxr-xr-x 1 root root 11K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo
-rwxr-xr-x 1 root root 1.5K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.changes
-rwxr-xr-x 1 root root 30M Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.deb
-rwxr-xr-x 1 root root 474K Aug 31 01:58 couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb
```
Next use https://github.com/apache/couchdb-docker/tree/main/3.1.1 to produce the container images, but adjust them to include the custom couchdb version.
```bash
$ git clone https://github.com/apache/couchdb-docker.git
Cloning into 'couchdb-docker'...
remote: Enumerating objects: 1010, done.
remote: Counting objects: 100% (118/118), done.
remote: Compressing objects: 100% (82/82), done.
remote: Total 1010 (delta 64), reused 64 (delta 32), pack-reused 892
Receiving objects: 100% (1010/1010), 248.11 KiB | 1.19 MiB/s, done.
Resolving deltas: 100% (490/490), done.
$ cd ~/couchdb-docker/3.1.1
$ cp -r ~/couchdb-pkg/pkgs/couch/debian-buster/ .
$ ls debian-buster/
couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo couchdb_3.1.1-6c2d01a~buster_amd64.changes couchdb_3.1.1-6c2d01a~buster_amd64.deb couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb
$ git diff
diff --git a/3.1.1/Dockerfile b/3.1.1/Dockerfile
index 75b4b29..59a90d9 100644
--- a/3.1.1/Dockerfile
+++ b/3.1.1/Dockerfile
@@ -60,6 +60,10 @@ RUN . /etc/os-release; \
echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | \
tee /etc/apt/sources.list.d/couchdb.list >/dev/null
+# I know, we should not include deb packages in images, but it should be good enough to validate the fix
+COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+
# https://github.com/apache/couchdb-pkg/blob/master/debian/README.Debian
RUN set -eux; \
apt-get update; \
@@ -67,7 +71,7 @@ RUN set -eux; \
echo "couchdb couchdb/mode select none" | debconf-set-selections; \
# we DO want recommends this time
DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages \
- couchdb="$COUCHDB_VERSION"~buster \
+ /tmp/packages/* \
; \
# Undo symlinks to /var/log and /var/lib
rmdir /var/lib/couchdb /var/log/couchdb; \
docker build -t "couchdb:3.1.1-patch-jan" .
Sending build context to Docker daemon 31.45MB
Step 1/20 : FROM debian:buster-slim
---> 5c3d57f3e47a
Step 2/20 : LABEL maintainer="CouchDB Developers dev@couchdb.apache.org"
---> Using cache
---> 60f37c14a0e1
Step 3/20 : RUN groupadd -g 5984 -r couchdb && useradd -u 5984 -d /opt/couchdb -g couchdb couchdb
---> Using cache
---> 78202346b500
Step 4/20 : RUN set -ex; apt-get update; apt-get install -y --no-install-recommends apt-transport-https ca-certificates dirmngr gnupg ; rm -rf /var/lib/apt/lists/*
---> Using cache
---> fad0f8a736e0
Step 5/20 : RUN set -eux; apt-get update; apt-get install -y --no-install-recommends gosu tini; rm -rf /var/lib/apt/lists/*; gosu nobody true; tini --version
---> Using cache
---> aca1db0792a3
Step 6/20 : ENV GPG_COUCH_KEY 390EF70BB1EA12B2773962950EE62FB37A00258D
---> Using cache
---> d28f3c244c10
Step 7/20 : RUN set -eux; apt-get update; apt-get install -y curl; export GNUPGHOME="$(mktemp -d)"; curl -fL -o keys.asc https://couchdb.apache.org/repo/keys.asc; gpg --batch --import keys.asc; gpg --batch --export "${GPG_COUCH_KEY}" > /usr/share/keyrings/couchdb-archive-keyring.gpg; command -v gpgconf && gpgconf --kill all || :; rm -rf "$GNUPGHOME"; apt-key list; apt purge -y --autoremove curl; rm -rf /var/lib/apt/lists/*
---> Using cache
---> e2949d6cc031
Step 8/20 : ENV COUCHDB_VERSION 3.1.1
---> Using cache
---> baa8089d1bea
Step 9/20 : RUN . /etc/os-release; echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | tee /etc/apt/sources.list.d/couchdb.list >/dev/null
---> Using cache
---> a7933b88e802
Step 10/20 : COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
---> Using cache
---> ded52d34b729
Step 11/20 : COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
---> Using cache
---> eb9b215cb430
Step 12/20 : RUN set -eux; apt-get update; echo "couchdb couchdb/mode select none" | debconf-set-selections; DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/* ; rmdir /var/lib/couchdb /var/log/couchdb; rm /opt/couchdb/data /opt/couchdb/var/log; mkdir -p /opt/couchdb/data /opt/couchdb/var/log; chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log; chmod 777 /opt/couchdb/data /opt/couchdb/var/log; rm /opt/couchdb/etc/default.d/10-filelog.ini; find /opt/couchdb \! \( -user couchdb -group couchdb \) -exec chown -f couchdb:couchdb '{}' +; find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 '{}' +; find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 '{}' +; chmod -f 0777 /opt/couchdb/etc/local.d; rm -rf /var/lib/apt/lists/*;
---> Running in 6f530f886258
+ apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [302 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 https://apache.jfrog.io/artifactory/couchdb-deb buster InRelease [5151 B]
Get:7 https://apache.jfrog.io/artifactory/couchdb-deb buster/main amd64 Packages [3889 B]
Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages [15.2 kB]
Fetched 8473 kB in 3s (3017 kB/s)
Reading package lists...
+ echo couchdb couchdb/mode select none
+ debconf-set-selections
+ DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2 libicu63 libk5crypto3
libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0 libncurses6
libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps psmisc
publicsuffix
Suggested packages:
gpm krb5-doc krb5-user
The following NEW packages will be installed:
couchdb couchdb-dbgsym curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2
libicu63 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0
libncurses6 libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps
psmisc publicsuffix
0 upgraded, 22 newly installed, 0 to remove and 0 not upgraded.
Need to get 13.5 MB/45.0 MB of archives.
After this operation, 104 MB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security buster/updates/main amd64 krb5-locales all 1.17-3+deb10u2 [95.5 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libncurses6 amd64 6.1+20181013-2+deb10u2 [102 kB]
Get:3 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5support0 amd64 1.17-3+deb10u2 [65.7 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 libk5crypto3 amd64 1.17-3+deb10u2 [122 kB]
Get:5 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5-3 amd64 1.17-3+deb10u2 [369 kB]
Get:6 http://security.debian.org/debian-security buster/updates/main amd64 libgssapi-krb5-2 amd64 1.17-3+deb10u2 [158 kB]
Get:7 http://deb.debian.org/debian buster/main amd64 libprocps7 amd64 2:3.3.15-2 [61.7 kB]
Get:8 http://deb.debian.org/debian buster/main amd64 procps amd64 2:3.3.15-2 [259 kB]
Get:9 http://deb.debian.org/debian buster/main amd64 libkeyutils1 amd64 1.6-6 [15.0 kB]
Get:10 http://deb.debian.org/debian buster/main amd64 libnghttp2-14 amd64 1.36.0-2+deb10u1 [85.0 kB]
Get:11 /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb couchdb amd64 3.1.1-6c2d01a~buster [30.9 MB]
Get:12 /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb couchdb-dbgsym amd64 3.1.1-6c2d01a~buster [485 kB]
Get:13 http://deb.debian.org/debian buster/main amd64 libpsl5 amd64 0.20.2-2 [53.7 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2 [60.5 kB]
Get:15 http://deb.debian.org/debian buster/main amd64 libssh2-1 amd64 1.8.0-2.1 [140 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 libcurl4 amd64 7.64.0-4+deb10u2 [332 kB]
Get:17 http://deb.debian.org/debian buster/main amd64 curl amd64 7.64.0-4+deb10u2 [265 kB]
Get:18 http://deb.debian.org/debian buster/main amd64 libicu63 amd64 63.1-6+deb10u1 [8300 kB]
Get:19 http://deb.debian.org/debian buster/main amd64 libmozjs-60-0 amd64 60.2.3-3 [2785 kB]
Get:20 http://deb.debian.org/debian buster/main amd64 libgpm2 amd64 1.20.7-5 [35.1 kB]
Get:21 http://deb.debian.org/debian buster/main amd64 psmisc amd64 23.2-1 [126 kB]
Get:22 http://deb.debian.org/debian buster/main amd64 publicsuffix all 20190415.1030-1 [116 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 13.5 MB in 1s (12.8 MB/s)
Selecting previously unselected package libncurses6:amd64.
(Reading database ... 7118 files and directories currently installed.)
Preparing to unpack .../00-libncurses6_6.1+20181013-2+deb10u2_amd64.deb ...
Unpacking libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Selecting previously unselected package libprocps7:amd64.
Preparing to unpack .../01-libprocps7_2%3a3.3.15-2_amd64.deb ...
Unpacking libprocps7:amd64 (2:3.3.15-2) ...
Selecting previously unselected package procps.
Preparing to unpack .../02-procps_2%3a3.3.15-2_amd64.deb ...
Unpacking procps (2:3.3.15-2) ...
Selecting previously unselected package krb5-locales.
Preparing to unpack .../03-krb5-locales_1.17-3+deb10u2_all.deb ...
Unpacking krb5-locales (1.17-3+deb10u2) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../04-libkeyutils1_1.6-6_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.6-6) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../05-libkrb5support0_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../06-libk5crypto3_1.17-3+deb10u2_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../07-libkrb5-3_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../08-libgssapi-krb5-2_1.17-3+deb10u2_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../09-libnghttp2-14_1.36.0-2+deb10u1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../10-libpsl5_0.20.2-2_amd64.deb ...
Unpacking libpsl5:amd64 (0.20.2-2) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../11-librtmp1_2.4+20151223.gitfa8646d.1-2_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Selecting previously unselected package libssh2-1:amd64.
Preparing to unpack .../12-libssh2-1_1.8.0-2.1_amd64.deb ...
Unpacking libssh2-1:amd64 (1.8.0-2.1) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../13-libcurl4_7.64.0-4+deb10u2_amd64.deb ...
Unpacking libcurl4:amd64 (7.64.0-4+deb10u2) ...
Selecting previously unselected package curl.
Preparing to unpack .../14-curl_7.64.0-4+deb10u2_amd64.deb ...
Unpacking curl (7.64.0-4+deb10u2) ...
Selecting previously unselected package libicu63:amd64.
Preparing to unpack .../15-libicu63_63.1-6+deb10u1_amd64.deb ...
Unpacking libicu63:amd64 (63.1-6+deb10u1) ...
Selecting previously unselected package libmozjs-60-0:amd64.
Preparing to unpack .../16-libmozjs-60-0_60.2.3-3_amd64.deb ...
Unpacking libmozjs-60-0:amd64 (60.2.3-3) ...
Selecting previously unselected package couchdb.
Preparing to unpack .../17-couchdb_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package couchdb-dbgsym.
Preparing to unpack .../18-couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package libgpm2:amd64.
Preparing to unpack .../19-libgpm2_1.20.7-5_amd64.deb ...
Unpacking libgpm2:amd64 (1.20.7-5) ...
Selecting previously unselected package psmisc.
Preparing to unpack .../20-psmisc_23.2-1_amd64.deb ...
Unpacking psmisc (23.2-1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../21-publicsuffix_20190415.1030-1_all.deb ...
Unpacking publicsuffix (20190415.1030-1) ...
Setting up libkeyutils1:amd64 (1.6-6) ...
Setting up libpsl5:amd64 (0.20.2-2) ...
Setting up libgpm2:amd64 (1.20.7-5) ...
Setting up psmisc (23.2-1) ...
Setting up libprocps7:amd64 (2:3.3.15-2) ...
Setting up libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Setting up krb5-locales (1.17-3+deb10u2) ...
Setting up libicu63:amd64 (63.1-6+deb10u1) ...
Setting up libkrb5support0:amd64 (1.17-3+deb10u2) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Setting up libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Setting up libk5crypto3:amd64 (1.17-3+deb10u2) ...
Setting up procps (2:3.3.15-2) ...
update-alternatives: using /usr/bin/w.procps to provide /usr/bin/w (w) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/w.1.gz because associated file /usr/share/man/man1/w.procps.1.gz (of link group w) doesn't exist
Setting up libssh2-1:amd64 (1.8.0-2.1) ...
Setting up libkrb5-3:amd64 (1.17-3+deb10u2) ...
Setting up publicsuffix (20190415.1030-1) ...
Setting up libmozjs-60-0:amd64 (60.2.3-3) ...
Setting up libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Setting up libcurl4:amd64 (7.64.0-4+deb10u2) ...
Setting up curl (7.64.0-4+deb10u2) ...
Setting up couchdb (3.1.1-6c2d01a~buster) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Processing triggers for libc-bin (2.28-10) ...
+ rmdir /var/lib/couchdb /var/log/couchdb
+ rm /opt/couchdb/data /opt/couchdb/var/log
+ mkdir -p /opt/couchdb/data /opt/couchdb/var/log
+ chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log
+ chmod 777 /opt/couchdb/data /opt/couchdb/var/log
+ rm /opt/couchdb/etc/default.d/10-filelog.ini
+ find /opt/couchdb ! ( -user couchdb -group couchdb ) -exec chown -f couchdb:couchdb {} +
+ find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 {} +
+ find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 {} +
+ chmod -f 0777 /opt/couchdb/etc/local.d
+ rm -rf /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_InRelease /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/auxfiles /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/deb.debian.org_debian_dists_buster_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_main_binary-amd64_Packages.lz4
Removing intermediate container 6f530f886258
---> eaea07a7886e
Step 13/20 : COPY --chown=couchdb:couchdb 10-docker-default.ini /opt/couchdb/etc/default.d/
---> f1c9d8e86732
Step 14/20 : COPY --chown=couchdb:couchdb vm.args /opt/couchdb/etc/
---> a984087a4ee3
Step 15/20 : COPY docker-entrypoint.sh /usr/local/bin
---> 66cd3ececce1
Step 16/20 : RUN ln -s usr/local/bin/docker-entrypoint.sh /docker-entrypoint.sh # backwards compat
---> Running in ae10e0e02906
Removing intermediate container ae10e0e02906
---> bf8817264923
Step 17/20 : ENTRYPOINT ["tini", "--", "/docker-entrypoint.sh"]
---> Running in 1485a79c319f
Removing intermediate container 1485a79c319f
---> 737d9f7a9b97
Step 18/20 : VOLUME /opt/couchdb/data
---> Running in 6aa4eafbc054
Removing intermediate container 6aa4eafbc054
---> 3f111d557168
Step 19/20 : EXPOSE 5984 4369 9100
---> Running in 07b6a3204c82
Removing intermediate container 07b6a3204c82
---> c70917bea3fd
Step 20/20 : CMD ["/opt/couchdb/bin/couchdb"]
---> Running in d61719ef6dea
Removing intermediate container d61719ef6dea
---> ca3940266314
Successfully built ca3940266314
Successfully tagged couchdb:3.1.1-patch-jan
```
I pushed the image and pulled it to my workstation again to have the same underlying hardware as in the previous test.
## Results
I adjusted the couchdb-test.sh [couchdb-test-validate-patch-3517.zip](https://github.com/apache/couchdb/files/7083304/couchdb-test-validate-patch-3517.zip) script to use the patched image and compare it against the vanilla CouchDB 3.1.1 image.
```bash
$ ./couchdb-test.sh setup
...
inserting docs cycle 2000/2000
{"ok":true,"id":"a479f05c01bf22c5168ee73617783eb7","rev":"1-f7fac9a3f6ce538d67747271336211c4"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784bb6","rev":"1-7e3d2b01acbc416b60e6e5be050dd790"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784e38","rev":"1-c959b425a724ad6346f646d8a63da6cd"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617785d5b","rev":"1-d8bd0d00d0b04b52796cfe73d9a80760"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617786743","rev":"1-24f6bc4fcfdceae7ff2fae47a68fbf42"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617787715","rev":"1-b0dfc903b7f27dc98130c8d51931f5d4"}
$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"84f2eae60ea04fe192284667cb0e4640","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10
real 0m36.391s
user 0m0.014s
sys 0m0.010s
Round: 2/10
real 0m36.426s
user 0m0.009s
sys 0m0.015s
Round: 3/10
real 0m37.448s
user 0m0.015s
sys 0m0.010s
Round: 4/10
real 0m37.182s
user 0m0.010s
sys 0m0.016s
Round: 5/10
real 0m34.802s
user 0m0.013s
sys 0m0.013s
Round: 6/10
real 0m34.877s
user 0m0.009s
sys 0m0.016s
Round: 7/10
real 0m34.204s
user 0m0.007s
sys 0m0.017s
Round: 8/10
real 0m34.092s
user 0m0.009s
sys 0m0.015s
Round: 9/10
real 0m35.571s
user 0m0.014s
sys 0m0.011s
Round: 10/10
real 0m35.790s
user 0m0.010s
sys 0m0.015s
================ Query CouchDB 3.1.1 patch jan ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"6c2d01a","uuid":"7df6b96ce9f321de839f65a9a6f264f3","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3004
Round: 1/10
real 0m11.194s
user 0m0.012s
sys 0m0.010s
Round: 2/10
real 0m11.224s
user 0m0.014s
sys 0m0.011s
Round: 3/10
real 0m10.966s
user 0m0.012s
sys 0m0.012s
Round: 4/10
real 0m11.212s
user 0m0.013s
sys 0m0.012s
Round: 5/10
real 0m10.696s
user 0m0.016s
sys 0m0.009s
Round: 6/10
real 0m10.859s
user 0m0.013s
sys 0m0.012s
Round: 7/10
real 0m11.342s
user 0m0.012s
sys 0m0.013s
Round: 8/10
real 0m11.276s
user 0m0.011s
sys 0m0.013s
Round: 9/10
real 0m11.357s
user 0m0.015s
sys 0m0.009s
Round: 10/10
real 0m11.542s
user 0m0.014s
sys 0m0.010s
```
That's actually great news, we have an average response time from about 11.2s now compared to 35.7s on the vanilla image :-)
If possible, I would vote for including your patch in a new 3.x build, as it would be more than 3 times faster for the same queries with unchanged client code (at least for the given dataset).
Still, nothing seem to beat build-in reducers, but at least it would not be way slower to run customer reducers on CouchDB 3 vs. CouchDB 2 installations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-874308851
Thanks for trying with q=8, @konrad-ohms. The JS engine could be responsible as you indicated then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-891758583
Thank you @nickva for your investigation. Good to know that it is really located in the Javascript engine and not elsewhere within the CouchDB code base.
I am not too familiar with CouchDB's dependencies, are you stating, that the Problem is basically a performance regression in Mozilla’s SpiderMonkey implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] janl commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
janl commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-905412556
I’ve attempted a quick demo that caches the transpilation inside the JS process, this would still do one transpile per couchjs process, but should still be a lot better.
@konrad-ohms or @nickva would you be able to retry this with the original setup?
```
diff --git a/share/server/60/rewrite_fun.js b/share/server/60/rewrite_fun.js
index 1b27a9d14..9b85f953f 100644
--- a/share/server/60/rewrite_fun.js
+++ b/share/server/60/rewrite_fun.js
@@ -15,30 +15,39 @@
//
// https://github.com/dmunch/couch-chakra/blob/master/js/normalizeFunction.js
+
+const cache = {}
function rewriteFunInt(fun) {
- const ast = esprima.parse(fun);
- let idx = ast.body.length - 1;
- let decl = {};
-
- // Search for the first FunctionDeclaration beginning from the end
- do {
- decl = ast.body[idx--];
- } while (idx >= 0 && decl.type !== "FunctionDeclaration");
- idx++;
-
- // If we have a function declaration without an Id, wrap it
- // in an ExpressionStatement and change it into
- // a FuntionExpression
- if (decl.type == "FunctionDeclaration" && decl.id == null) {
- decl.type = "FunctionExpression";
- ast.body[idx] = {
- type: "ExpressionStatement",
- expression: decl
- };
- }
+ const crc = crc32_str(fun)
+ if (cache[crc]) {
+ return cache[crc];
+ } else {
+ const ast = esprima.parse(fun);
+ let idx = ast.body.length - 1;
+ let decl = {};
+
+ // Search for the first FunctionDeclaration beginning from the end
+ do {
+ decl = ast.body[idx--];
+ } while (idx >= 0 && decl.type !== "FunctionDeclaration");
+ idx++;
+
+ // If we have a function declaration without an Id, wrap it
+ // in an ExpressionStatement and change it into
+ // a FuntionExpression
+ if (decl.type == "FunctionDeclaration" && decl.id == null) {
+ decl.type = "FunctionExpression";
+ ast.body[idx] = {
+ type: "ExpressionStatement",
+ expression: decl
+ };
+ }
- // Generate source from the rewritten AST
- return escodegen.generate(ast);
+ // Generate source from the rewritten AST
+ const gen = escodegen.generate(ast);
+ cache[crc] = gen;
+ return gen;
+ }
}
@@ -53,4 +62,51 @@ function rewriteFuns(funsJSON) {
return rewriteFunInt(fun);
});
return JSON.stringify(results);
-}
\ No newline at end of file
+}
+
+// nicked from https://github.com/SheetJS/js-crc32/blob/master/crc32.js
+
+function signed_crc_table() {
+ var c = 0, table = new Array(256);
+
+ for(var n =0; n != 256; ++n){
+ c = n;
+ c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+ c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+ c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+ c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+ c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+ c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+ c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+ c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
+ table[n] = c;
+ }
+
+ return typeof Int32Array !== 'undefined' ? new Int32Array(table) : table;
+}
+
+var T = signed_crc_table();
+
+function crc32_str(str, seed) {
+ var C = seed ^ -1;
+ for(var i = 0, L=str.length, c, d; i < L;) {
+ c = str.charCodeAt(i++);
+ if(c < 0x80) {
+ C = (C>>>8) ^ T[(C ^ c)&0xFF];
+ } else if(c < 0x800) {
+ C = (C>>>8) ^ T[(C ^ (192|((c>>6)&31)))&0xFF];
+ C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
+ } else if(c >= 0xD800 && c < 0xE000) {
+ c = (c&1023)+64; d = str.charCodeAt(i++)&1023;
+ C = (C>>>8) ^ T[(C ^ (240|((c>>8)&7)))&0xFF];
+ C = (C>>>8) ^ T[(C ^ (128|((c>>2)&63)))&0xFF];
+ C = (C>>>8) ^ T[(C ^ (128|((d>>6)&15)|((c&3)<<4)))&0xFF];
+ C = (C>>>8) ^ T[(C ^ (128|(d&63)))&0xFF];
+ } else {
+ C = (C>>>8) ^ T[(C ^ (224|((c>>12)&15)))&0xFF];
+ C = (C>>>8) ^ T[(C ^ (128|((c>>6)&63)))&0xFF];
+ C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
+ }
+ }
+ return C ^ -1;
+}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
nickva edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-890372099
I confirmed that the JS engine is implicated. I ran 2.3.1 and 3.x latest and 2.3.1 on Ubuntu 20.04. For 3.x I installed the libmozjs185 couchdb-build package for Ubuntu 18.04 and ran with q=8 to minimize difference and similar performance numbers as 2.3.1.
Packages I downloaded and installed manually:
```
couch-libmozjs185-1.0_1.8.5-1.0.0+couch-2_bionic_amd64.deb
couch-libmozjs185-dev_1.8.5-1.0.0+couch-2_bionic_amd64.deb
libffi6_3.2.1-8_amd64.deb
libffi-dev_3.2.1-8_amd64.deb
```
Configure 3.x to build with version 1.8.5
```
./configure --spidermonkey-version 1.8.5 --dev
```
Test was modified to not set up docker container.
```
./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"fake_uuid_for_dev","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15994
Round: 1/10
real 0m13.022s
user 0m0.023s
sys 0m0.023s
Round: 2/10
real 0m7.234s
user 0m0.020s
sys 0m0.008s
Round: 3/10
real 0m6.715s
user 0m0.026s
sys 0m0.000s
Round: 4/10
real 0m7.021s
user 0m0.018s
sys 0m0.009s
Round: 5/10
real 0m7.159s
user 0m0.005s
sys 0m0.021s
Round: 6/10
real 0m7.318s
user 0m0.014s
sys 0m0.014s
Round: 7/10
real 0m7.464s
user 0m0.007s
sys 0m0.021s
Round: 8/10
real 0m7.175s
user 0m0.016s
sys 0m0.011s
Round: 9/10
real 0m7.096s
user 0m0.004s
sys 0m0.023s
Round: 10/10
real 0m7.235s
user 0m0.006s
sys 0m0.022s
```
```
q=8
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1-ba63878","git_sha":"ba63878","uuid":"fake_uuid_for_dev","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15984
Round: 1/10
real 0m5.979s
user 0m0.017s
sys 0m0.010s
Round: 2/10
real 0m6.757s
user 0m0.013s
sys 0m0.013s
Round: 3/10
real 0m7.112s
user 0m0.012s
sys 0m0.016s
Round: 4/10
real 0m7.359s
user 0m0.017s
sys 0m0.011s
Round: 5/10
real 0m7.210s
user 0m0.017s
sys 0m0.013s
Round: 6/10
real 0m7.233s
user 0m0.008s
sys 0m0.021s
Round: 7/10
real 0m6.284s
user 0m0.028s
sys 0m0.000s
Round: 8/10
real 0m7.101s
user 0m0.020s
sys 0m0.008s
Round: 9/10
real 0m6.943s
user 0m0.022s
sys 0m0.006s
Round: 10/10
real 0m6.736s
user 0m0.016s
sys 0m0.011s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] rkulkarni commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
rkulkarni commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-888923713
Hello folks, is there an update on this issue? I'm also facing this issue when I migrate from 2.3.1 to 3.1.1. View queries with reduce are significantly slower.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-870453565
Friendly reminder, if anyone could have a look, please?
It is blocking us to upgrade to CouchDB v3 at the moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-915121820
Thanks @janl, I compiled your PR and ran the test suite again.
Unfortunately, the second fix in the PR is not performing as good as the prior patch, but it is still better than the original 3.1.1 performance.
| Version | Average Performance |
|--------------------------|---------------------|
| 3.1.1 | 36.43s |
| 3.1.1 patch 1 | 11.45s |
| 3.1.1 patch 2 (PR #3728) | 25.25s |
Please find my full result output below:
```bash
$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"ae7c1c70f02a0b873d0a4f414e1a4d7e","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10
real 0m35.155s
user 0m0.019s
sys 0m0.014s
Round: 2/10
real 0m35.645s
user 0m0.011s
sys 0m0.015s
Round: 3/10
real 0m37.915s
user 0m0.008s
sys 0m0.018s
Round: 4/10
real 0m36.850s
user 0m0.008s
sys 0m0.018s
Round: 5/10
real 0m35.977s
user 0m0.016s
sys 0m0.012s
Round: 6/10
real 0m36.915s
user 0m0.010s
sys 0m0.016s
Round: 7/10
real 0m35.969s
user 0m0.010s
sys 0m0.017s
Round: 8/10
real 0m37.248s
user 0m0.014s
sys 0m0.013s
Round: 9/10
real 0m36.506s
user 0m0.013s
sys 0m0.012s
Round: 10/10
real 0m36.101s
user 0m0.012s
sys 0m0.014s
================ Query CouchDB 3.1.1 patch jan ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"6c2d01a","uuid":"54f521952cafc33bdb70987c4fc6cd69","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3004
Round: 1/10
real 0m11.213s
user 0m0.015s
sys 0m0.010s
Round: 2/10
real 0m11.737s
user 0m0.008s
sys 0m0.017s
Round: 3/10
real 0m11.524s
user 0m0.013s
sys 0m0.012s
Round: 4/10
real 0m11.387s
user 0m0.011s
sys 0m0.014s
Round: 5/10
real 0m11.257s
user 0m0.013s
sys 0m0.011s
Round: 6/10
real 0m11.325s
user 0m0.012s
sys 0m0.013s
Round: 7/10
real 0m11.846s
user 0m0.012s
sys 0m0.013s
Round: 8/10
real 0m11.652s
user 0m0.009s
sys 0m0.015s
Round: 9/10
real 0m11.508s
user 0m0.013s
sys 0m0.012s
Round: 10/10
real 0m11.351s
user 0m0.015s
sys 0m0.010s
================ Query CouchDB 3.1.1 patch2 jan ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"65fd7ec","uuid":"73454c85dfb63b0826f42e25310b6dec","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3005
Round: 1/10
real 0m25.617s
user 0m0.017s
sys 0m0.011s
Round: 2/10
real 0m25.098s
user 0m0.014s
sys 0m0.015s
Round: 3/10
real 0m24.765s
user 0m0.010s
sys 0m0.015s
Round: 4/10
real 0m26.233s
user 0m0.010s
sys 0m0.016s
Round: 5/10
real 0m25.144s
user 0m0.011s
sys 0m0.016s
Round: 6/10
real 0m25.586s
user 0m0.015s
sys 0m0.012s
Round: 7/10
real 0m24.918s
user 0m0.013s
sys 0m0.014s
Round: 8/10
real 0m24.787s
user 0m0.011s
sys 0m0.015s
Round: 9/10
real 0m25.519s
user 0m0.015s
sys 0m0.011s
Round: 10/10
real 0m24.812s
user 0m0.014s
sys 0m0.011s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] janl commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
janl commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-910303881
@konrad-ohms thank you for confirming this, we’ll see about getting this into 3.x ✌️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-909124414
Okay, I validated your fix, please find the outcome below:
First, I wanted to create a patch build which is as close as possible to what I have tested before, so I used the same OS as in the container image I used initially:
```bash
$ podman run --rm -it docker.io/library/couchdb:3.1.1 cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
```
To get a build env matching the OS and its required libraries, I had a look into <https://github.com/apache/couchdb-ci> and built the following image on an 18.04.5 LTS amd64 VM with Docker version 20.10.8, build 3967b7d:
```bash
$ git clone https://github.com/apache/couchdb-ci.git
Cloning into 'couchdb-ci'...
remote: Enumerating objects: 1742, done.
remote: Counting objects: 100% (87/87), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 1742 (delta 55), reused 56 (delta 29), pack-reused 1655
Receiving objects: 100% (1742/1742), 10.24 MiB | 16.11 MiB/s, done.
Resolving deltas: 100% (998/998), done.
$ cd couchdb-ci/
$ ./build.sh platform debian-buster
...
Successfully built d1c29b763a2c
Successfully tagged apache/couchdbci-debian:buster-erlang-20.3.8.26-1
```
From there I used CouchDB v3.1.1 and applied your patch within the container:
```bash
$ docker run --name couchdb-3.1.1-fix-validation -it apache/couchdbci-debian:buster-erlang-20.3.8.26-1
jenkins@2665e52069d0:/$ cd ~
jenkins@2665e52069d0:~$ whoami
jenkins
jenkins@2665e52069d0:~$ pwd
/home/jenkins
jenkins@2665e52069d0:~$ git clone --depth=1 -b 3.1.1 https://github.com/apache/couchdb.git
Cloning into 'couchdb'...
remote: Enumerating objects: 1207, done.
remote: Counting objects: 100% (1207/1207), done.
remote: Compressing objects: 100% (1099/1099), done.
remote: Total 1207 (delta 245), reused 380 (delta 55), pack-reused 0
Receiving objects: 100% (1207/1207), 1.80 MiB | 6.14 MiB/s, done.
Resolving deltas: 100% (245/245), done.
Note: checking out 'ce596c65d9d7f0bc5d9937bcaf6253b343015690'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
jenkins@2665e52069d0:~$ # Apply Jan's patch
jenkins@2665e52069d0:~$ cat <<'EOF' > jan.patch
> diff --git a/share/server/60/rewrite_fun.js b/share/server/60/rewrite_fun.js
> index 1b27a9d14..9b85f953f 100644
> --- a/share/server/60/rewrite_fun.js
> +++ b/share/server/60/rewrite_fun.js
> @@ -15,30 +15,39 @@
> //
> // https://github.com/dmunch/couch-chakra/blob/master/js/normalizeFunction.js
>
> +
> +const cache = {}
> function rewriteFunInt(fun) {
> - const ast = esprima.parse(fun);
> - let idx = ast.body.length - 1;
> - let decl = {};
> -
> - // Search for the first FunctionDeclaration beginning from the end
> - do {
> - decl = ast.body[idx--];
> - } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> - idx++;
> -
> - // If we have a function declaration without an Id, wrap it
> - // in an ExpressionStatement and change it into
> - // a FuntionExpression
> - if (decl.type == "FunctionDeclaration" && decl.id == null) {
> - decl.type = "FunctionExpression";
> - ast.body[idx] = {
> - type: "ExpressionStatement",
> - expression: decl
> - };
> - }
> + const crc = crc32_str(fun)
> + if (cache[crc]) {
> + return cache[crc];
> + } else {
> + const ast = esprima.parse(fun);
> + let idx = ast.body.length - 1;
> + let decl = {};
> +
> + // Search for the first FunctionDeclaration beginning from the end
> + do {
> + decl = ast.body[idx--];
> + } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> + idx++;
> +
> + // If we have a function declaration without an Id, wrap it
> + // in an ExpressionStatement and change it into
> + // a FuntionExpression
> + if (decl.type == "FunctionDeclaration" && decl.id == null) {
> + decl.type = "FunctionExpression";
> + ast.body[idx] = {
> + type: "ExpressionStatement",
> + expression: decl
> + };
> + }
>
> - // Generate source from the rewritten AST
> - return escodegen.generate(ast);
> + // Generate source from the rewritten AST
> + const gen = escodegen.generate(ast);
> + cache[crc] = gen;
> + return gen;
> + }
> }
>
>
> @@ -53,4 +62,51 @@ function rewriteFuns(funsJSON) {
> return rewriteFunInt(fun);
> });
> return JSON.stringify(results);
> -}
> \ No newline at end of file
> +}
> +
> +// nicked from https://github.com/SheetJS/js-crc32/blob/master/crc32.js
> +
> +function signed_crc_table() {
> +var c = 0, table = new Array(256);
> +
> +for(var n =0; n != 256; ++n){
> +c = n;
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +table[n] = c;
> +}
> +
> +return typeof Int32Array !== 'undefined' ? new Int32Array(table) : table;
> +}
> +
> +var T = signed_crc_table();
> +
> +function crc32_str(str, seed) {
> +var C = seed ^ -1;
> +for(var i = 0, L=str.length, c, d; i < L;) {
> +c = str.charCodeAt(i++);
> +if(c < 0x80) {
> +C = (C>>>8) ^ T[(C ^ c)&0xFF];
> +} else if(c < 0x800) {
> +C = (C>>>8) ^ T[(C ^ (192|((c>>6)&31)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +} else if(c >= 0xD800 && c < 0xE000) {
> +c = (c&1023)+64; d = str.charCodeAt(i++)&1023;
> +C = (C>>>8) ^ T[(C ^ (240|((c>>8)&7)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>2)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((d>>6)&15)|((c&3)<<4)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(d&63)))&0xFF];
> +} else {
> +C = (C>>>8) ^ T[(C ^ (224|((c>>12)&15)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>6)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +}
> +}
> +return C ^ -1;
> +}
> EOF
jenkins@2665e52069d0:~$ cd couchdb/
jenkins@2665e52069d0:~/couchdb$ git status
Not currently on any branch.
nothing to commit, working tree clean
jenkins@2665e52069d0:~/couchdb$ git checkout -b 3.1.1-patch-jan
Switched to a new branch '3.1.1-patch-jan'
jenkins@2665e52069d0:~/couchdb$ git apply ../jan.patch
jenkins@2665e52069d0:~/couchdb$ git status
On branch 3.1.1-patch-jan
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: share/server/60/rewrite_fun.js
no changes added to commit (use "git add" and/or "git commit -a")
```
Afterwards I compiled the source:
```bash
jenkins@2665e52069d0:~/couchdb$ ./configure
...
==> couchdb (update-deps)
Updating config from {git,"https://github.com/apache/couchdb-config.git",
{tag,"2.1.7"}}
Updating b64url from {git,"https://github.com/apache/couchdb-b64url.git",
{tag,"1.0.2"}}
Updating ets_lru from {git,"https://github.com/apache/couchdb-ets-lru.git",
{tag,"1.1.0"}}
Updating khash from {git,"https://github.com/apache/couchdb-khash.git",
{tag,"1.1.0"}}
Updating snappy from {git,"https://github.com/apache/couchdb-snappy.git",
{tag,"CouchDB-1.0.4"}}
Updating docs from {git,"https://github.com/apache/couchdb-documentation",
{tag,"3.1.1-RC2"}}
Updating fauxton from {git,"https://github.com/apache/couchdb-fauxton",
{tag,"v1.2.6"}}
Updating folsom from {git,"https://github.com/apache/couchdb-folsom.git",
{tag,"CouchDB-0.8.3"}}
Updating hyper from {git,"https://github.com/apache/couchdb-hyper.git",
{tag,"CouchDB-2.2.0-6"}}
Updating ibrowse from {git,"https://github.com/apache/couchdb-ibrowse.git",
{tag,"CouchDB-4.0.1-1"}}
Updating jiffy from {git,"https://github.com/apache/couchdb-jiffy.git",
{tag,"CouchDB-1.0.4-1"}}
Updating mochiweb from {git,"https://github.com/apache/couchdb-mochiweb.git",
{tag,"v2.20.0"}}
Updating meck from {git,"https://github.com/apache/couchdb-meck.git",
{tag,"0.8.8"}}
Updating recon from {git,"https://github.com/apache/couchdb-recon.git",
{tag,"2.5.0"}}
Updating proper from {git,"https://github.com/proper-testing/proper",
{tag,"v1.3"}}
Updating bear from {git,"https://github.com/apache/couchdb-bear.git",
"008f48aff819126e281d5ccae80a258bf9bf9c30"}
You have configured Apache CouchDB, time to relax. Relax.
# Compile
jenkins@2665e52069d0:~/couchdb$ make release
...
The HTML pages are in build/html.
sphinx-build -b man -a -W -n -A local=1 -D latex_elements.papersize=a4 -d build/doctree src/ build/man
Running Sphinx v1.8.4
making output directory...
loading pickled environment... done
building [mo]: all of 0 po files
building [man]: all source files
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
writing... apachecouchdb.1 { intro/index intro/overview intro/why intro/consistency intro/curl intro/security intro/tour intro/api replication/index replication/intro replication/replicator replication/conflicts replication/protocol ddocs/index ddocs/ddocs ddocs/views/index ddocs/views/intro ddocs/views/collation ddocs/views/joins ddocs/views/nosql ddocs/views/pagination ddocs/search best-practices/index best-practices/documents best-practices/forms best-practices/iso-date best-practices/jsdevel best-practices/views best-practices/reverse-proxies install/index install/unix install/windows install/mac install/freebsd install/docker install/snap install/kubernetes install/search install/upgrading install/troubleshooting setup/index setup/single-node setup/cluster config/index config/intro config/couchdb config/cluster config/couch-peruser config/http config/auth config/compaction config/indexbuilds config/ioq config/logging config/replicator config/query-servers config/misc config/r
esharding cluster/index cluster/theory cluster/nodes cluster/databases cluster/sharding cluster/purging maintenance/index maintenance/compaction maintenance/performance maintenance/backups fauxton/index fauxton/install experimental api/index api/basics api/server/index api/server/common api/server/authn api/server/configuration api/database/index api/database/common api/database/bulk-api api/database/find api/database/shard api/database/changes api/database/compact api/database/security api/database/misc api/document/index api/document/common api/document/attachments api/ddoc/index api/ddoc/common api/ddoc/views api/ddoc/search api/ddoc/render api/ddoc/rewrites api/partitioned-dbs api/local json-structure query-server/index query-server/protocol query-server/javascript query-server/erlang partitioned-dbs/index whatsnew/index whatsnew/3.1 whatsnew/3.0 whatsnew/2.3 whatsnew/2.2 whatsnew/2.1 whatsnew/2.0 whatsnew/1.7 whatsnew/1.6 whatsnew/1.5 whatsnew/1.4 whatsnew/1.3 whatsnew/1.2 what
snew/1.1 whatsnew/1.0 whatsnew/0.11 whatsnew/0.10 whatsnew/0.9 whatsnew/0.8 cve/index cve/2010-0009 cve/2010-2234 cve/2010-3854 cve/2012-5641 cve/2012-5649 cve/2012-5650 cve/2014-2668 cve/2017-12635 cve/2017-12636 cve/2018-11769 cve/2018-17188 cve/2018-8007 cve/2020-1955 about contributing }
build succeeded.
The manual pages are in build/man.
make[1]: Leaving directory '/home/jenkins/couchdb/src/docs'
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN: 'generate' command does not apply to directory /home/jenkins/couchdb
... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.
jenkins@2665e52069d0:~/couchdb$ echo $?
0
```
The compilation seem to have worked, so I created deb packages based on that. The `make dist` seemed not to like uncommited changes, so I added your patch as a commit:
```bash
jenkins@2665e52069d0:~/couchdb$ git config --global user.email "couchdb-validation@no-reply.com"
jenkins@2665e52069d0:~/couchdb$ git config --global user.name "CouchDB validation"
jenkins@2665e52069d0:~/couchdb$ git commit -am "fix: Apply caching"
[3.1.1-patch-jan 6c2d01a] fix: Apply caching
1 file changed, 79 insertions(+), 23 deletions(-)
jenkins@2665e52069d0:~/couchdb$ make dist
==> config (compile)
==> b64url (compile)
==> ets_lru (compile)
==> khash (compile)
==> snappy (compile)
==> bear (compile)
==> meck (compile)
==> folsom (compile)
==> hyper (compile)
==> ibrowse (compile)
==> jiffy (compile)
==> mochiweb (compile)
==> recon (compile)
==> proper (compile)
==> couch_epi (compile)
==> couch_log (compile)
==> chttpd (compile)
==> couch (compile)
Compiling priv/couch_js/1.8.5/http.c
Compiling priv/couch_js/1.8.5/main.c
Compiling priv/couch_js/1.8.5/utf8.c
Compiling priv/couch_js/1.8.5/util.c
==> couch_event (compile)
==> mem3 (compile)
==> couch_index (compile)
==> couch_mrview (compile)
==> couch_replicator (compile)
==> couch_plugins (compile)
==> couch_pse_tests (compile)
==> couch_stats (compile)
==> couch_peruser (compile)
==> couch_tests (compile)
==> ddoc_cache (compile)
==> dreyfus (compile)
==> fabric (compile)
==> global_changes (compile)
==> ioq (compile)
==> jwtf (compile)
==> ken (compile)
==> mango (compile)
==> rexi (compile)
==> setup (compile)
==> smoosh (compile)
==> rel (compile)
==> couchdb (compile)
COUCHDB_GIT_SHA: 6c2d01a
COUCHDB_VERSION: 3.1.1-6c2d01a
COUCHDB_VERSION_SUFFIX: 6c2d01a
DIRTY:
IN_RC:
IN_RELEASE:
ON_TAG:
REL_TAG: 3.1.1
SUB_VSN:
Building Apache CouchDB 3.1.1-6c2d01a
fatal: ambiguous argument '6c976bd..HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Done: apache-couchdb-3.1.1-6c2d01a.tar.gz
```
I copied that tar back to the host OS:
```bash
$ docker cp couchdb-3.1.1-fix-validation:/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a.tar.gz .
$ git clone https://github.com/apache/couchdb-pkg.git
Cloning into 'couchdb-pkg'...
remote: Enumerating objects: 723, done.
remote: Counting objects: 100% (94/94), done.
remote: Compressing objects: 100% (70/70), done.
remote: Total 723 (delta 40), reused 57 (delta 20), pack-reused 629
Receiving objects: 100% (723/723), 25.31 MiB | 23.49 MiB/s, done.
Resolving deltas: 100% (335/335), done.
$ cd couchdb-pkg/
```
The assumed image name was `couchdbdev/<osname>-<codename>-erlang-<erlang-version>`, so I retagged my created one before running the packaging job:
```bash
$ docker tag apache/couchdbci-debian:buster-erlang-20.3.8.26-1 couchdbdev/debian-buster-erlang-20.3.8.26-1
$ ERLANGVERSION=20.3.8.26-1 ./build.sh couch debian-buster /root/apache-couchdb-3.1.1-6c2d01a.tar.gz
Using apache-couchdb-3.1.1-6c2d01a.tar.gz to build packages...
mkdir -p ../couchdb
cp apache-couchdb-3.1.1-6c2d01a.tar.gz ../couchdb
cd ../couchdb && tar xfz *.tar.gz
cp debian/control.in debian/control
sed -i 's/%SPIDERMONKEY%/libmozjs-60-0/g' debian/control
sed -i 's/%SPIDERMONKEY_DEV%/libmozjs-60-dev/g' debian/control
echo 'SM_VER = 60' > debian/sm_ver.mk
rm -rf /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/debian
cp -R debian /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dch -v 3.1.1-6c2d01a~buster "Automatically generated package from upstream."
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dpkg-buildpackage -b -us -uc
dpkg-buildpackage: info: source package couchdb
dpkg-buildpackage: info: source version 3.1.1-6c2d01a~buster
dpkg-buildpackage: info: source distribution UNRELEASED
dpkg-buildpackage: info: source changed by "CouchDB Developers" <"dev@couchdb.apache.org">
dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
debian/rules clean
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh clean --with=systemd
debian/rules override_dh_auto_clean
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_clean
make -j1 distclean
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (clean)
==> b64url (clean)
==> ets_lru (clean)
==> khash (clean)
==> snappy (clean)
==> bear (clean)
==> meck (clean)
==> folsom (clean)
==> hyper (clean)
==> ibrowse (clean)
==> jiffy (clean)
==> mochiweb (clean)
==> recon (clean)
==> couch_epi (clean)
==> couch_log (clean)
==> chttpd (clean)
==> couch (clean)
==> couch_event (clean)
==> mem3 (clean)
==> couch_index (clean)
==> couch_mrview (clean)
==> couch_replicator (clean)
==> couch_plugins (clean)
==> couch_pse_tests (clean)
==> couch_stats (clean)
==> couch_peruser (clean)
==> couch_tests (clean)
==> ddoc_cache (clean)
==> dreyfus (clean)
==> fabric (clean)
==> global_changes (clean)
==> ioq (clean)
==> jwtf (clean)
==> ken (clean)
==> mango (clean)
==> rexi (clean)
==> setup (clean)
==> smoosh (clean)
==> rel (clean)
==> apache-couchdb-3.1.1-6c2d01a (clean)
make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
mv src/mango/src/mango_cursor_text.erl.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.erl.nocompile': No such file or directory
make[2]: [debian/rules:21: override_dh_auto_clean] Error 1 (ignored)
mv src/mango/src/mango_cursor_text.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.nocompile': No such file or directory
make[2]: [debian/rules:22: override_dh_auto_clean] Error 1 (ignored)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_clean
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules build
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh build --with=systemd
dh_update_autotools_config
debian/rules override_dh_auto_configure
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
./configure --spidermonkey-version 60
==> configuring couchdb in rel/couchdb.config
You have configured Apache CouchDB, time to relax. Relax.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules override_dh_auto_build
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_build -- release
make -j1 release
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (compile)
Compiled src/config_util.erl
Compiled src/config_writer.erl
Compiled src/config_sup.erl
Compiled src/config_notifier.erl
Compiled src/config_listener_mon.erl
Compiled src/config_listener.erl
Compiled src/config_app.erl
Compiled src/config.erl
==> b64url (compile)
Compiled src/b64url.erl
Compiling c_src/b64url.c
==> ets_lru (compile)
Compiled src/ets_lru.erl
==> khash (compile)
Compiled src/khash.erl
Compiling c_src/hash.c
Compiling c_src/khash.c
==> snappy (compile)
Compiled src/snappy.erl
Compiling c_src/snappy_nif.cc
c_src/snappy_nif.cc: In function 'ERL_NIF_TERM snappy_compress(ErlNifEnv*, int, const ERL_NIF_TERM*)':
c_src/snappy_nif.cc:156:28: warning: catching polymorphic type 'class std::bad_alloc' by value [-Wcatch-value=]
} catch(std::bad_alloc e) {
^
Compiling c_src/snappy/snappy-sinksource.cc
Compiling c_src/snappy/snappy-stubs-internal.cc
Compiling c_src/snappy/snappy.cc
c_src/snappy/snappy.cc: In instantiation of 'bool snappy::SnappyScatteredWriter<Allocator>::AppendFromSelf(size_t, size_t) [with Allocator = snappy::SnappySinkAllocator; size_t = long unsigned int]':
c_src/snappy/snappy.cc:779:13: required from 'void snappy::SnappyDecompressor::DecompressAllTags(Writer*) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>]'
c_src/snappy/snappy.cc:865:3: required from 'bool snappy::InternalUncompressAllTags(snappy::SnappyDecompressor*, Writer*, snappy::uint32) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>; snappy::uint32 = unsigned int]'
c_src/snappy/snappy.cc:1549:78: required from here
c_src/snappy/snappy.cc:1401:21: warning: comparison of integer expressions of different signedness: 'size_t' {aka 'long unsigned int'} and 'long int' [-Wsign-compare]
if (offset - 1u < op_ptr_ - op_base_) {
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
==> bear (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/bear/src/bear.erl:28: Warning: export_all flag enabled - all functions will be exported
Compiled src/bear.erl
==> meck (compile)
Compiled src/meck_util.erl
Compiled src/meck_matcher.erl
Compiled src/meck_ret_spec.erl
Compiled src/meck_history.erl
Compiled src/meck_expect.erl
Compiled src/meck_cover.erl
Compiled src/meck_code.erl
Compiled src/meck_code_gen.erl
Compiled src/meck_proc.erl
Compiled src/meck_args_matcher.erl
Compiled src/meck.erl
==> folsom (compile)
Compiled src/folsom_utils.erl
Compiled src/folsom_sup.erl
Compiled src/folsom_sample_slide_uniform.erl
Compiled src/folsom_sample_uniform.erl
Compiled src/folsom_sample_slide_sup.erl
Compiled src/folsom_sample_slide_server.erl
Compiled src/folsom_sample_slide_sorted.erl
Compiled src/folsom_vm_metrics.erl
Compiled src/folsom_sample_none.erl
Compiled src/folsom_sample_slide.erl
Compiled src/folsom_metrics_spiral.erl
Compiled src/folsom_sample_exdec.erl
Compiled src/folsom_sample.erl
Compiled src/folsom_metrics_meter.erl
Compiled src/folsom_metrics_history.erl
Compiled src/folsom_metrics_meter_reader.erl
Compiled src/folsom_metrics_histogram_ets.erl
Compiled src/folsom_metrics_gauge.erl
Compiled src/folsom_metrics_histogram.erl
Compiled src/folsom_metrics_duration.erl
Compiled src/folsom_metrics_counter.erl
Compiled src/folsom_ewma.erl
Compiled src/folsom_meter_timer_server.erl
Compiled src/folsom.erl
Compiled src/folsom_metrics.erl
Compiled src/folsom_ets.erl
==> hyper (compile)
Compiled src/hyper_register.erl
Compiled src/hyper_carray.erl
Compiled src/hyper_gb.erl
Compiled src/hyper_const.erl
Compiled src/hyper_array.erl
Compiled src/hyper_binary_rle.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:245: Warning: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:288: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:333: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
Compiled src/hyper.erl
Compiled src/hyper_binary.erl
Compiling c_src/hyper_carray.c
==> ibrowse (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_lib.erl:372: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_lib.erl
Compiled src/ibrowse_sup.erl
Compiled src/ibrowse_socks5.erl
Compiled src/ibrowse_lb.erl
Compiled src/ibrowse_app.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:83: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:93: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:171: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:182: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_test.erl
Compiled src/ibrowse.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:428: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:1831: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_http_client.erl
==> jiffy (compile)
Compiling c_src/decoder.c
Compiling c_src/encoder.c
Compiling c_src/jiffy.c
Compiling c_src/termstack.c
Compiling c_src/utf8.c
Compiling c_src/util.c
Compiling c_src/doubles.cc
Compiling c_src/objects.cc
Compiling c_src/double-conversion/bignum-dtoa.cc
Compiling c_src/double-conversion/bignum.cc
Compiling c_src/double-conversion/cached-powers.cc
Compiling c_src/double-conversion/diy-fp.cc
Compiling c_src/double-conversion/double-conversion.cc
Compiling c_src/double-conversion/fast-dtoa.cc
Compiling c_src/double-conversion/fixed-dtoa.cc
Compiling c_src/double-conversion/strtod.cc
Compiled src/jiffy_utf8.erl
Compiled src/jiffy.erl
==> mochiweb (compile)
Compiled src/reloader.erl
Compiled src/mochiweb_websocket.erl
Compiled src/mochiweb_socket.erl
Compiled src/mochiweb_util.erl
Compiled src/mochiweb_session.erl
Compiled src/mochiweb_socket_server.erl
Compiled src/mochiweb_response.erl
Compiled src/mochiweb_mime.erl
Compiled src/mochiweb_multipart.erl
Compiled src/mochiweb_io.erl
Compiled src/mochiweb_http.erl
Compiled src/mochiweb_request.erl
Compiled src/mochiweb_echo.erl
Compiled src/mochiweb_headers.erl
Compiled src/mochiweb_cover.erl
Compiled src/mochiweb_cookies.erl
Compiled src/mochiweb_clock.erl
Compiled src/mochiweb_base64url.erl
Compiled src/mochiweb_acceptor.erl
Compiled src/mochiweb.erl
Compiled src/mochiweb_html.erl
Compiled src/mochiutf8.erl
Compiled src/mochitemp.erl
Compiled src/mochinum.erl
Compiled src/mochilogfile2.erl
Compiled src/mochilists.erl
Compiled src/mochijson.erl
Compiled src/mochihex.erl
Compiled src/mochiglobal.erl
Compiled src/mochifmt_std.erl
Compiled src/mochifmt_records.erl
Compiled src/mochijson2.erl
Compiled src/mochifmt.erl
Compiled src/mochiweb_charref.erl
==> recon (compile)
Compiled src/recon_map.erl
Compiled src/recon_rec.erl
Compiled src/recon_trace.erl
Compiled src/recon_alloc.erl
Compiled src/recon_lib.erl
Compiled src/recon.erl
==> proper (compile)
Compiled src/vararg.erl
Compiled src/proper_target.erl
Compiled src/proper_types.erl
Compiled src/proper_unicode.erl
Compiled src/proper_unused_imports_remover.erl
Compiled src/proper_symb.erl
Compiled src/proper_transformer.erl
Compiled src/proper_shrink.erl
Compiled src/proper_statem.erl
Compiled src/proper_sets.erl
Compiled src/proper_queue.erl
Compiled src/proper_sa.erl
Compiled src/proper_prop_remover.erl
Compiled src/proper_orddict.erl
Compiled src/proper_ordsets.erl
Compiled src/proper_gen.erl
Compiled src/proper_typeserver.erl
Compiled src/proper_gb_trees.erl
Compiled src/proper_gb_sets.erl
Compiled src/proper_gen_next.erl
Compiled src/proper_dict.erl
Compiled src/proper_array.erl
Compiled src/proper_fsm.erl
Compiled src/proper_arith.erl
Compiled src/proper.erl
==> couch_epi (compile)
Compiled src/couch_epi_util.erl
Compiled src/couch_epi_sup.erl
Compiled src/couch_epi_plugin.erl
Compiled src/couch_epi_module_keeper.erl
Compiled src/couch_epi_functions.erl
Compiled src/couch_epi_functions_gen.erl
Compiled src/couch_epi_data_gen.erl
Compiled src/couch_epi_data.erl
Compiled src/couch_epi_codegen.erl
Compiled src/couch_epi_app.erl
Compiled src/couch_epi_codechange_monitor.erl
Compiled src/couch_epi.erl
==> couch_log (compile)
Compiled src/couch_log_writer.erl
Compiled src/couch_log_writer_journald.erl
Compiled src/couch_log_writer_stderr.erl
Compiled src/couch_log_writer_syslog.erl
Compiled src/couch_log_writer_file.erl
Compiled src/couch_log_util.erl
Compiled src/couch_log_sup.erl
Compiled src/couch_log_server.erl
Compiled src/couch_log_monitor.erl
Compiled src/couch_log_trunc_io_fmt.erl
Compiled src/couch_log_error_logger_h.erl
Compiled src/couch_log_config_dyn.erl
Compiled src/couch_log_trunc_io.erl
Compiled src/couch_log_app.erl
Compiled src/couch_log_config.erl
Compiled src/couch_log.erl
Compiled src/couch_log_formatter.erl
==> chttpd (compile)
Compiled src/chttpd.erl
Compiled src/chttpd_xframe_options.erl
Compiled src/chttpd_sup.erl
Compiled src/chttpd_view.erl
Compiled src/chttpd_stats.erl
Compiled src/chttpd_test_util.erl
Compiled src/chttpd_prefer_header.erl
Compiled src/chttpd_rewrite.erl
Compiled src/chttpd_show.erl
Compiled src/chttpd_plugin.erl
Compiled src/chttpd_httpd_handlers.erl
Compiled src/chttpd_handlers.erl
Compiled src/chttpd_node.erl
Compiled src/chttpd_misc.erl
Compiled src/chttpd_epi.erl
Compiled src/chttpd_external.erl
Compiled src/chttpd_cors.erl
Compiled src/chttpd_auth_request.erl
Compiled src/chttpd_auth.erl
Compiled src/chttpd_app.erl
Compiled src/chttpd_auth_cache.erl
Compiled src/chttpd_db.erl
==> couch (compile)
Compiled src/couch_httpd.erl
Compiled src/test_request.erl
Compiled src/couch_work_queue.erl
Compiled src/couch_uuids.erl
Compiled src/couch_users_db.erl
Compiled src/couch_totp.erl
Compiled src/test_util.erl
Compiled src/couch_util.erl
Compiled src/couch_task_status.erl
Compiled src/couch_sup.erl
Compiled src/couch_secondary_sup.erl
Compiled src/couch_rand.erl
Compiled src/couch_stream.erl
Compiled src/couch_proc_manager.erl
Compiled src/couch_query_servers.erl
Compiled src/couch_server.erl
Compiled src/couch_primary_sup.erl
Compiled src/couch_partition.erl
Compiled src/couch_passwords.erl
Compiled src/couch_os_process.erl
Compiled src/couch_lru.erl
Compiled src/couch_multidb_changes.erl
Compiled src/couch_io_logger.erl
Compiled src/couch_native_process.erl
Compiled src/couch_httpd_vhost.erl
Compiled src/couch_key_tree.erl
Compiled src/couch_httpd_rewrite.erl
Compiled src/couch_httpd_handlers.erl
Compiled src/couch_httpd_multipart.erl
Compiled src/couch_httpd_misc_handlers.erl
Compiled src/couch_httpd_external.erl
Compiled src/couch_hotp.erl
Compiled src/couch_hash.erl
Compiled src/couch_flags_config.erl
Compiled src/couch_httpd_auth.erl
Compiled src/couch_flags.erl
Compiled src/couch_event_sup.erl
Compiled src/couch_emsort.erl
Compiled src/couch_ejson_size.erl
Compiled src/couch_file.erl
Compiled src/couch_httpd_db.erl
Compiled src/couch_ejson_compare.erl
Compiled src/couch_drv.erl
Compiled src/couch_debug.erl
Compiled src/couch_doc.erl
Compiled src/couch_db_split.erl
Compiled src/couch_db_plugin.erl
Compiled src/couch_db_epi.erl
Compiled src/couch_db_header.erl
Compiled src/couch_db_updater.erl
Compiled src/couch_compress.erl
Compiled src/couch_db_engine.erl
Compiled src/couch_changes.erl
Compiled src/couch_bt_engine_stream.erl
Compiled src/couch_bt_engine_header.erl
Compiled src/couch_db.erl
Compiled src/couch_btree.erl
Compiled src/couch_base32.erl
Compiled src/couch_bt_engine_compactor.erl
Compiled src/couch_auth_cache.erl
Compiled src/couch_att.erl
Compiled src/couch_bt_engine.erl
Compiled src/couch.erl
Compiled src/couch_app.erl
Compiling priv/couch_js/60/http.cpp
Compiling priv/couch_js/60/main.cpp
Compiling priv/couch_js/60/util.cpp
Compiling priv/icu_driver/couch_icu_driver.c
Compiling priv/couch_ejson_compare/couch_ejson_compare.c
==> couch_event (compile)
Compiled src/couch_event_listener.erl
Compiled src/couch_event_sup2.erl
Compiled src/couch_event_os_listener.erl
Compiled src/couch_event_app.erl
Compiled src/couch_event.erl
Compiled src/couch_event_server.erl
Compiled src/couch_event_listener_mfa.erl
==> mem3 (compile)
Compiled src/mem3_sync_nodes.erl
Compiled src/mem3_sync_security.erl
Compiled src/mem3_sync_event.erl
Compiled src/mem3_sync_event_listener.erl
Compiled src/mem3_sup.erl
Compiled src/mem3_sync.erl
Compiled src/mem3_util.erl
Compiled src/mem3_seeds.erl
Compiled src/mem3_reshard_validate.erl
Compiled src/mem3_reshard_sup.erl
Compiled src/mem3_shards.erl
Compiled src/mem3_rpc.erl
Compiled src/mem3_reshard_job_sup.erl
Compiled src/mem3_reshard_store.erl
Compiled src/mem3_reshard_index.erl
Compiled src/mem3_reshard_httpd.erl
Compiled src/mem3_reshard_job.erl
Compiled src/mem3_reshard_api.erl
Compiled src/mem3_reshard_dbdoc.erl
Compiled src/mem3_plugin_couch_db.erl
Compiled src/mem3_nodes.erl
Compiled src/mem3_httpd_handlers.erl
Compiled src/mem3_reshard.erl
Compiled src/mem3_hash.erl
Compiled src/mem3_epi.erl
Compiled src/mem3_httpd.erl
Compiled src/mem3_rep.erl
Compiled src/mem3_app.erl
Compiled src/mem3_cluster.erl
Compiled src/mem3.erl
==> couch_index (compile)
Compiled src/couch_index_sup.erl
Compiled src/couch_index_util.erl
Compiled src/couch_index_plugin_couch_db.erl
Compiled src/couch_index_updater.erl
Compiled src/couch_index_epi.erl
Compiled src/couch_index_plugin.erl
Compiled src/couch_index_app.erl
Compiled src/couch_index_server.erl
Compiled src/couch_index_compactor.erl
Compiled src/couch_index.erl
==> couch_mrview (compile)
Compiled src/couch_mrview_update_notifier.erl
Compiled src/couch_mrview_updater.erl
Compiled src/couch_mrview_test_util.erl
Compiled src/couch_mrview_show.erl
Compiled src/couch_mrview_util.erl
Compiled src/couch_mrview_index.erl
Compiled src/couch_mrview_compactor.erl
Compiled src/couch_mrview_cleanup.erl
Compiled src/couch_mrview_http.erl
Compiled src/couch_mrview.erl
==> couch_replicator (compile)
Compiled src/couch_replicator_httpc.erl
Compiled src/couch_replicator_auth.erl
Compiled src/couch_replicator_utils.erl
Compiled src/couch_replicator_sup.erl
Compiled src/couch_replicator_stats.erl
Compiled src/couch_replicator_scheduler_sup.erl
Compiled src/json_stream_parse.erl
Compiled src/couch_replicator_worker.erl
Compiled src/couch_replicator_rate_limiter_tables.erl
Compiled src/couch_replicator_rate_limiter.erl
Compiled src/couch_replicator_notifier.erl
Compiled src/couch_replicator_job_sup.erl
Compiled src/couch_replicator_scheduler_job.erl
Compiled src/couch_replicator_scheduler.erl
Compiled src/couch_replicator_ids.erl
Compiled src/couch_replicator_httpd_util.erl
Compiled src/couch_replicator_httpd.erl
Compiled src/couch_replicator_httpc_pool.erl
Compiled src/couch_replicator_filters.erl
Compiled src/couch_replicator_fabric_rpc.erl
Compiled src/couch_replicator_doc_processor_worker.erl
Compiled src/couch_replicator_fabric.erl
Compiled src/couch_replicator_db_changes.erl
Compiled src/couch_replicator_connection.erl
Compiled src/couch_replicator_docs.erl
Compiled src/couch_replicator_changes_reader.erl
Compiled src/couch_replicator_doc_processor.erl
Compiled src/couch_replicator_clustering.erl
Compiled src/couch_replicator_app.erl
Compiled src/couch_replicator_auth_noop.erl
Compiled src/couch_replicator_auth_session.erl
Compiled src/couch_replicator.erl
Compiled src/couch_replicator_api_wrap.erl
==> couch_plugins (compile)
Compiled src/couch_plugins_httpd.erl
Compiled src/couch_plugins.erl
==> couch_pse_tests (compile)
Compiled src/cpse_test_ref_counting.erl
Compiled src/cpse_util.erl
Compiled src/cpse_test_read_write_docs.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/couch_pse_tests/src/cpse_test_purge_seqs.erl:14: Warning: export_all flag enabled - all functions will be exported
Compiled src/cpse_test_purge_seqs.erl
Compiled src/cpse_test_purge_replication.erl
Compiled src/cpse_test_purge_docs.erl
Compiled src/cpse_test_open_close_delete.erl
Compiled src/cpse_test_purge_bad_checkpoints.erl
Compiled src/cpse_test_get_set_props.erl
Compiled src/cpse_test_fold_purge_infos.erl
Compiled src/cpse_test_fold_changes.erl
Compiled src/cpse_test_fold_docs.erl
Compiled src/cpse_test_copy_purge_infos.erl
Compiled src/cpse_gather.erl
Compiled src/cpse_test_attachments.erl
Compiled src/cpse_test_compaction.erl
==> couch_stats (compile)
Compiled src/couch_stats_sup.erl
Compiled src/couch_stats_app.erl
Compiled src/couch_stats_process_tracker.erl
Compiled src/couch_stats.erl
Compiled src/couch_stats_httpd.erl
Compiled src/couch_stats_aggregator.erl
==> couch_peruser (compile)
Compiled src/couch_peruser_app.erl
Compiled src/couch_peruser_sup.erl
Compiled src/couch_peruser.erl
==> couch_tests (compile)
Compiled setups/couch_epi_dispatch.erl
Compiled src/couch_tests_combinatorics.erl
Compiled src/couch_tests.erl
==> ddoc_cache (compile)
Compiled src/ddoc_cache_value.erl
Compiled src/ddoc_cache_sup.erl
Compiled src/ddoc_cache_opener.erl
Compiled src/ddoc_cache_entry_validation_funs.erl
Compiled src/ddoc_cache_entry_ddocid_rev.erl
Compiled src/ddoc_cache_entry_ddocid.erl
Compiled src/ddoc_cache_entry_custom.erl
Compiled src/ddoc_cache_app.erl
Compiled src/ddoc_cache_lru.erl
Compiled src/ddoc_cache.erl
Compiled src/ddoc_cache_entry.erl
==> dreyfus (compile)
Compiled src/dreyfus_sup.erl
Compiled src/dreyfus_plugin_couch_db.erl
Compiled src/dreyfus_rpc.erl
Compiled src/dreyfus_index_updater.erl
Compiled src/dreyfus_util.erl
Compiled src/dreyfus_index_manager.erl
Compiled src/dreyfus_httpd_handlers.erl
Compiled src/dreyfus_index.erl
Compiled src/dreyfus_fabric_search.erl
Compiled src/dreyfus_fabric_info.erl
Compiled src/dreyfus_httpd.erl
Compiled src/dreyfus_fabric_group2.erl
Compiled src/dreyfus_fabric_cleanup.erl
Compiled src/dreyfus_fabric_group1.erl
Compiled src/dreyfus_epi.erl
Compiled src/dreyfus_config.erl
Compiled src/dreyfus_app.erl
Compiled src/dreyfus_fabric.erl
Compiled src/dreyfus_bookmark.erl
Compiled src/clouseau_rpc.erl
==> fabric (compile)
Compiled src/fabric_db_update_listener.erl
Compiled src/fabric_view_reduce.erl
Compiled src/fabric_view_map.erl
Compiled src/fabric_view_all_docs.erl
Compiled src/fabric_view_changes.erl
Compiled src/fabric_view.erl
Compiled src/fabric_streams.erl
Compiled src/fabric_util.erl
Compiled src/fabric_rpc.erl
Compiled src/fabric_ring.erl
Compiled src/fabric_group_info.erl
Compiled src/fabric_doc_update.erl
Compiled src/fabric_doc_purge.erl
Compiled src/fabric_doc_missing_revs.erl
Compiled src/fabric_doc_open_revs.erl
Compiled src/fabric_doc_atts.erl
Compiled src/fabric_doc_open.erl
Compiled src/fabric_dict.erl
Compiled src/fabric_doc_attachments.erl
Compiled src/fabric_design_doc_count.erl
Compiled src/fabric_db_partition_info.erl
Compiled src/fabric_db_meta.erl
Compiled src/fabric_db_doc_count.erl
Compiled src/fabric_db_info.erl
Compiled src/fabric_db_delete.erl
Compiled src/fabric_db_create.erl
Compiled src/fabric.erl
==> global_changes (compile)
Compiled src/global_changes_sup.erl
Compiled src/global_changes_util.erl
Compiled src/global_changes_plugin.erl
Compiled src/global_changes_httpd_handlers.erl
Compiled src/global_changes_listener.erl
Compiled src/global_changes_server.erl
Compiled src/global_changes_epi.erl
Compiled src/global_changes_app.erl
Compiled src/global_changes_httpd.erl
==> ioq (compile)
Compiled src/ioq_app.erl
Compiled src/ioq_sup.erl
Compiled src/ioq.erl
==> jwtf (compile)
Compiled src/jwtf_app.erl
Compiled src/jwtf_sup.erl
Compiled src/jwtf.erl
Compiled src/jwtf_keystore.erl
==> ken (compile)
Compiled src/ken_sup.erl
Compiled src/ken_event_handler.erl
Compiled src/ken_app.erl
Compiled src/ken.erl
Compiled src/ken_server.erl
==> mango (compile)
Compiled src/mango_sup.erl
Compiled src/mango_sort.erl
Compiled src/mango_util.erl
Compiled src/mango_selector_text.erl
Compiled src/mango_selector.erl
Compiled src/mango_opts.erl
Compiled src/mango_json.erl
Compiled src/mango_json_bookmark.erl
Compiled src/mango_native_proc.erl
Compiled src/mango_idx_special.erl
Compiled src/mango_idx_text.erl
Compiled src/mango_httpd_handlers.erl
Compiled src/mango_idx_view.erl
Compiled src/mango_fields.erl
Compiled src/mango_execution_stats.erl
Compiled src/mango_idx.erl
Compiled src/mango_epi.erl
Compiled src/mango_httpd.erl
Compiled src/mango_error.erl
Compiled src/mango_doc.erl
Compiled src/mango_cursor_text.erl
Compiled src/mango_cursor_special.erl
Compiled src/mango_cursor.erl
Compiled src/mango_app.erl
Compiled src/mango_crud.erl
Compiled src/mango_cursor_view.erl
==> rexi (compile)
Compiled src/rexi_server_sup.erl
Compiled src/rexi_sup.erl
Compiled src/rexi_utils.erl
Compiled src/rexi_server_mon.erl
Compiled src/rexi_monitor.erl
Compiled src/rexi_app.erl
Compiled src/rexi_server.erl
Compiled src/rexi_buffer.erl
Compiled src/rexi.erl
==> setup (compile)
Compiled src/setup_sup.erl
Compiled src/setup_httpd_handlers.erl
Compiled src/setup_app.erl
Compiled src/setup_epi.erl
Compiled src/setup_httpd.erl
Compiled src/setup.erl
==> smoosh (compile)
Compiled src/smoosh_sup.erl
Compiled src/smoosh_utils.erl
Compiled src/smoosh_priority_queue.erl
Compiled src/smoosh_app.erl
Compiled src/smoosh_channel.erl
Compiled src/smoosh_server.erl
Compiled src/smoosh.erl
==> rel (compile)
==> apache-couchdb-3.1.1-6c2d01a (compile)
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN: 'generate' command does not apply to directory /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.
make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
rm -rf rel/couchdb/var/log
rm -rf rel/couchdb/data
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules binary
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh binary --with=systemd
dh_testroot
dh_prep
dh_installdirs
dh_install
dh_installdocs
dh_installchangelogs
dh_installman
dh_installdebconf
dh_systemd_enable
dh_installinit
dh_systemd_start
dh_installlogrotate
dh_lintian
dh_perl
dh_link
dh_strip_nondeterminism
dh_compress
dh_fixperms
dh_missing
dh_strip
dh_makeshlibs
debian/rules override_dh_shlibdeps
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_shlibdeps -- --ignore-missing-info -xlibmozjs185-1.0
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_ejson_compare.so debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_icu_driver.so were not linked against libicudata.so.63 (they use none of the library's symbols)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_installdeb
dh_gencontrol
dh_md5sums
debian/rules override_dh_builddeb
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_builddeb -- -Zgzip
dpkg-deb: building package 'couchdb-dbgsym' in '../couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb'.
dpkg-deb: building package 'couchdb' in '../couchdb_3.1.1-6c2d01a~buster_amd64.deb'.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dpkg-genbuildinfo --build=binary
dpkg-genchanges --build=binary >../couchdb_3.1.1-6c2d01a~buster_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
dpkg-source --after-build .
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
dpkg-buildpackage: info: binary-only upload (no source included)
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/.. && lintian --profile couchdb couch*.deb || true
warning: the authors of lintian do not recommend running it with root privileges!
W: couchdb: maintainer-script-should-not-use-recursive-chown-or-chmod postinst:207
N: 3 tags overridden (2 errors, 1 info); 2 unused overrides
chmod a+rwx ../couchdb/couchdb*
mkdir -p pkgs/couch/debian-buster && chmod 777 pkgs/couch/debian-buster
cp ../couchdb/couchdb* pkgs/couch/debian-buster
if [ -f debian/control.bak ]; then mv -f debian/control.bak debian/control; fi
if [ -f rpm/SPECS/couchdb.spec.bak ]; then mv -f rpm/SPECS/couchdb.spec.bak rpm/SPECS/couchdb.spec; fi
rm -rf parts prime stage js/build debian/sm_ver.mk
# packages were created
$ ls -lah pkgs/couch/debian-buster/
total 30M
drwxrwxrwx 2 root root 207 Aug 31 01:58 .
drwxrwxrwx 3 root root 45 Aug 31 01:58 ..
-rwxr-xr-x 1 root root 11K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo
-rwxr-xr-x 1 root root 1.5K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.changes
-rwxr-xr-x 1 root root 30M Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.deb
-rwxr-xr-x 1 root root 474K Aug 31 01:58 couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb
```
Next use https://github.com/apache/couchdb-docker/tree/main/3.1.1 to produce the container images, but adjust them to include the custom couchdb version.
```bash
$ git clone https://github.com/apache/couchdb-docker.git
Cloning into 'couchdb-docker'...
remote: Enumerating objects: 1010, done.
remote: Counting objects: 100% (118/118), done.
remote: Compressing objects: 100% (82/82), done.
remote: Total 1010 (delta 64), reused 64 (delta 32), pack-reused 892
Receiving objects: 100% (1010/1010), 248.11 KiB | 1.19 MiB/s, done.
Resolving deltas: 100% (490/490), done.
$ cd ~/couchdb-docker/3.1.1
$ cp -r ~/couchdb-pkg/pkgs/couch/debian-buster/ .
$ ls debian-buster/
couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo couchdb_3.1.1-6c2d01a~buster_amd64.changes couchdb_3.1.1-6c2d01a~buster_amd64.deb couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb
$ git diff
diff --git a/3.1.1/Dockerfile b/3.1.1/Dockerfile
index 75b4b29..59a90d9 100644
--- a/3.1.1/Dockerfile
+++ b/3.1.1/Dockerfile
@@ -60,6 +60,10 @@ RUN . /etc/os-release; \
echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | \
tee /etc/apt/sources.list.d/couchdb.list >/dev/null
+# I know, we should not include deb packages in images, but it should be good enough to validate the fix
+COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+
# https://github.com/apache/couchdb-pkg/blob/master/debian/README.Debian
RUN set -eux; \
apt-get update; \
@@ -67,7 +71,7 @@ RUN set -eux; \
echo "couchdb couchdb/mode select none" | debconf-set-selections; \
# we DO want recommends this time
DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages \
- couchdb="$COUCHDB_VERSION"~buster \
+ /tmp/packages/* \
; \
# Undo symlinks to /var/log and /var/lib
rmdir /var/lib/couchdb /var/log/couchdb; \
docker build -t "couchdb:3.1.1-patch-jan" .
Sending build context to Docker daemon 31.45MB
Step 1/20 : FROM debian:buster-slim
---> 5c3d57f3e47a
Step 2/20 : LABEL maintainer="CouchDB Developers dev@couchdb.apache.org"
---> Using cache
---> 60f37c14a0e1
Step 3/20 : RUN groupadd -g 5984 -r couchdb && useradd -u 5984 -d /opt/couchdb -g couchdb couchdb
---> Using cache
---> 78202346b500
Step 4/20 : RUN set -ex; apt-get update; apt-get install -y --no-install-recommends apt-transport-https ca-certificates dirmngr gnupg ; rm -rf /var/lib/apt/lists/*
---> Using cache
---> fad0f8a736e0
Step 5/20 : RUN set -eux; apt-get update; apt-get install -y --no-install-recommends gosu tini; rm -rf /var/lib/apt/lists/*; gosu nobody true; tini --version
---> Using cache
---> aca1db0792a3
Step 6/20 : ENV GPG_COUCH_KEY 390EF70BB1EA12B2773962950EE62FB37A00258D
---> Using cache
---> d28f3c244c10
Step 7/20 : RUN set -eux; apt-get update; apt-get install -y curl; export GNUPGHOME="$(mktemp -d)"; curl -fL -o keys.asc https://couchdb.apache.org/repo/keys.asc; gpg --batch --import keys.asc; gpg --batch --export "${GPG_COUCH_KEY}" > /usr/share/keyrings/couchdb-archive-keyring.gpg; command -v gpgconf && gpgconf --kill all || :; rm -rf "$GNUPGHOME"; apt-key list; apt purge -y --autoremove curl; rm -rf /var/lib/apt/lists/*
---> Using cache
---> e2949d6cc031
Step 8/20 : ENV COUCHDB_VERSION 3.1.1
---> Using cache
---> baa8089d1bea
Step 9/20 : RUN . /etc/os-release; echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | tee /etc/apt/sources.list.d/couchdb.list >/dev/null
---> Using cache
---> a7933b88e802
Step 10/20 : COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
---> Using cache
---> ded52d34b729
Step 11/20 : COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
---> Using cache
---> eb9b215cb430
Step 12/20 : RUN set -eux; apt-get update; echo "couchdb couchdb/mode select none" | debconf-set-selections; DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/* ; rmdir /var/lib/couchdb /var/log/couchdb; rm /opt/couchdb/data /opt/couchdb/var/log; mkdir -p /opt/couchdb/data /opt/couchdb/var/log; chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log; chmod 777 /opt/couchdb/data /opt/couchdb/var/log; rm /opt/couchdb/etc/default.d/10-filelog.ini; find /opt/couchdb \! \( -user couchdb -group couchdb \) -exec chown -f couchdb:couchdb '{}' +; find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 '{}' +; find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 '{}' +; chmod -f 0777 /opt/couchdb/etc/local.d; rm -rf /var/lib/apt/lists/*;
---> Running in 6f530f886258
+ apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [302 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 https://apache.jfrog.io/artifactory/couchdb-deb buster InRelease [5151 B]
Get:7 https://apache.jfrog.io/artifactory/couchdb-deb buster/main amd64 Packages [3889 B]
Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages [15.2 kB]
Fetched 8473 kB in 3s (3017 kB/s)
Reading package lists...
+ echo couchdb couchdb/mode select none
+ debconf-set-selections
+ DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2 libicu63 libk5crypto3
libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0 libncurses6
libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps psmisc
publicsuffix
Suggested packages:
gpm krb5-doc krb5-user
The following NEW packages will be installed:
couchdb couchdb-dbgsym curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2
libicu63 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0
libncurses6 libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps
psmisc publicsuffix
0 upgraded, 22 newly installed, 0 to remove and 0 not upgraded.
Need to get 13.5 MB/45.0 MB of archives.
After this operation, 104 MB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security buster/updates/main amd64 krb5-locales all 1.17-3+deb10u2 [95.5 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libncurses6 amd64 6.1+20181013-2+deb10u2 [102 kB]
Get:3 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5support0 amd64 1.17-3+deb10u2 [65.7 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 libk5crypto3 amd64 1.17-3+deb10u2 [122 kB]
Get:5 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5-3 amd64 1.17-3+deb10u2 [369 kB]
Get:6 http://security.debian.org/debian-security buster/updates/main amd64 libgssapi-krb5-2 amd64 1.17-3+deb10u2 [158 kB]
Get:7 http://deb.debian.org/debian buster/main amd64 libprocps7 amd64 2:3.3.15-2 [61.7 kB]
Get:8 http://deb.debian.org/debian buster/main amd64 procps amd64 2:3.3.15-2 [259 kB]
Get:9 http://deb.debian.org/debian buster/main amd64 libkeyutils1 amd64 1.6-6 [15.0 kB]
Get:10 http://deb.debian.org/debian buster/main amd64 libnghttp2-14 amd64 1.36.0-2+deb10u1 [85.0 kB]
Get:11 /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb couchdb amd64 3.1.1-6c2d01a~buster [30.9 MB]
Get:12 /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb couchdb-dbgsym amd64 3.1.1-6c2d01a~buster [485 kB]
Get:13 http://deb.debian.org/debian buster/main amd64 libpsl5 amd64 0.20.2-2 [53.7 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2 [60.5 kB]
Get:15 http://deb.debian.org/debian buster/main amd64 libssh2-1 amd64 1.8.0-2.1 [140 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 libcurl4 amd64 7.64.0-4+deb10u2 [332 kB]
Get:17 http://deb.debian.org/debian buster/main amd64 curl amd64 7.64.0-4+deb10u2 [265 kB]
Get:18 http://deb.debian.org/debian buster/main amd64 libicu63 amd64 63.1-6+deb10u1 [8300 kB]
Get:19 http://deb.debian.org/debian buster/main amd64 libmozjs-60-0 amd64 60.2.3-3 [2785 kB]
Get:20 http://deb.debian.org/debian buster/main amd64 libgpm2 amd64 1.20.7-5 [35.1 kB]
Get:21 http://deb.debian.org/debian buster/main amd64 psmisc amd64 23.2-1 [126 kB]
Get:22 http://deb.debian.org/debian buster/main amd64 publicsuffix all 20190415.1030-1 [116 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 13.5 MB in 1s (12.8 MB/s)
Selecting previously unselected package libncurses6:amd64.
(Reading database ... 7118 files and directories currently installed.)
Preparing to unpack .../00-libncurses6_6.1+20181013-2+deb10u2_amd64.deb ...
Unpacking libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Selecting previously unselected package libprocps7:amd64.
Preparing to unpack .../01-libprocps7_2%3a3.3.15-2_amd64.deb ...
Unpacking libprocps7:amd64 (2:3.3.15-2) ...
Selecting previously unselected package procps.
Preparing to unpack .../02-procps_2%3a3.3.15-2_amd64.deb ...
Unpacking procps (2:3.3.15-2) ...
Selecting previously unselected package krb5-locales.
Preparing to unpack .../03-krb5-locales_1.17-3+deb10u2_all.deb ...
Unpacking krb5-locales (1.17-3+deb10u2) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../04-libkeyutils1_1.6-6_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.6-6) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../05-libkrb5support0_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../06-libk5crypto3_1.17-3+deb10u2_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../07-libkrb5-3_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../08-libgssapi-krb5-2_1.17-3+deb10u2_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../09-libnghttp2-14_1.36.0-2+deb10u1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../10-libpsl5_0.20.2-2_amd64.deb ...
Unpacking libpsl5:amd64 (0.20.2-2) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../11-librtmp1_2.4+20151223.gitfa8646d.1-2_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Selecting previously unselected package libssh2-1:amd64.
Preparing to unpack .../12-libssh2-1_1.8.0-2.1_amd64.deb ...
Unpacking libssh2-1:amd64 (1.8.0-2.1) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../13-libcurl4_7.64.0-4+deb10u2_amd64.deb ...
Unpacking libcurl4:amd64 (7.64.0-4+deb10u2) ...
Selecting previously unselected package curl.
Preparing to unpack .../14-curl_7.64.0-4+deb10u2_amd64.deb ...
Unpacking curl (7.64.0-4+deb10u2) ...
Selecting previously unselected package libicu63:amd64.
Preparing to unpack .../15-libicu63_63.1-6+deb10u1_amd64.deb ...
Unpacking libicu63:amd64 (63.1-6+deb10u1) ...
Selecting previously unselected package libmozjs-60-0:amd64.
Preparing to unpack .../16-libmozjs-60-0_60.2.3-3_amd64.deb ...
Unpacking libmozjs-60-0:amd64 (60.2.3-3) ...
Selecting previously unselected package couchdb.
Preparing to unpack .../17-couchdb_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package couchdb-dbgsym.
Preparing to unpack .../18-couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package libgpm2:amd64.
Preparing to unpack .../19-libgpm2_1.20.7-5_amd64.deb ...
Unpacking libgpm2:amd64 (1.20.7-5) ...
Selecting previously unselected package psmisc.
Preparing to unpack .../20-psmisc_23.2-1_amd64.deb ...
Unpacking psmisc (23.2-1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../21-publicsuffix_20190415.1030-1_all.deb ...
Unpacking publicsuffix (20190415.1030-1) ...
Setting up libkeyutils1:amd64 (1.6-6) ...
Setting up libpsl5:amd64 (0.20.2-2) ...
Setting up libgpm2:amd64 (1.20.7-5) ...
Setting up psmisc (23.2-1) ...
Setting up libprocps7:amd64 (2:3.3.15-2) ...
Setting up libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Setting up krb5-locales (1.17-3+deb10u2) ...
Setting up libicu63:amd64 (63.1-6+deb10u1) ...
Setting up libkrb5support0:amd64 (1.17-3+deb10u2) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Setting up libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Setting up libk5crypto3:amd64 (1.17-3+deb10u2) ...
Setting up procps (2:3.3.15-2) ...
update-alternatives: using /usr/bin/w.procps to provide /usr/bin/w (w) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/w.1.gz because associated file /usr/share/man/man1/w.procps.1.gz (of link group w) doesn't exist
Setting up libssh2-1:amd64 (1.8.0-2.1) ...
Setting up libkrb5-3:amd64 (1.17-3+deb10u2) ...
Setting up publicsuffix (20190415.1030-1) ...
Setting up libmozjs-60-0:amd64 (60.2.3-3) ...
Setting up libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Setting up libcurl4:amd64 (7.64.0-4+deb10u2) ...
Setting up curl (7.64.0-4+deb10u2) ...
Setting up couchdb (3.1.1-6c2d01a~buster) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Processing triggers for libc-bin (2.28-10) ...
+ rmdir /var/lib/couchdb /var/log/couchdb
+ rm /opt/couchdb/data /opt/couchdb/var/log
+ mkdir -p /opt/couchdb/data /opt/couchdb/var/log
+ chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log
+ chmod 777 /opt/couchdb/data /opt/couchdb/var/log
+ rm /opt/couchdb/etc/default.d/10-filelog.ini
+ find /opt/couchdb ! ( -user couchdb -group couchdb ) -exec chown -f couchdb:couchdb {} +
+ find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 {} +
+ find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 {} +
+ chmod -f 0777 /opt/couchdb/etc/local.d
+ rm -rf /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_InRelease /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/auxfiles /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/deb.debian.org_debian_dists_buster_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_main_binary-amd64_Packages.lz4
Removing intermediate container 6f530f886258
---> eaea07a7886e
Step 13/20 : COPY --chown=couchdb:couchdb 10-docker-default.ini /opt/couchdb/etc/default.d/
---> f1c9d8e86732
Step 14/20 : COPY --chown=couchdb:couchdb vm.args /opt/couchdb/etc/
---> a984087a4ee3
Step 15/20 : COPY docker-entrypoint.sh /usr/local/bin
---> 66cd3ececce1
Step 16/20 : RUN ln -s usr/local/bin/docker-entrypoint.sh /docker-entrypoint.sh # backwards compat
---> Running in ae10e0e02906
Removing intermediate container ae10e0e02906
---> bf8817264923
Step 17/20 : ENTRYPOINT ["tini", "--", "/docker-entrypoint.sh"]
---> Running in 1485a79c319f
Removing intermediate container 1485a79c319f
---> 737d9f7a9b97
Step 18/20 : VOLUME /opt/couchdb/data
---> Running in 6aa4eafbc054
Removing intermediate container 6aa4eafbc054
---> 3f111d557168
Step 19/20 : EXPOSE 5984 4369 9100
---> Running in 07b6a3204c82
Removing intermediate container 07b6a3204c82
---> c70917bea3fd
Step 20/20 : CMD ["/opt/couchdb/bin/couchdb"]
---> Running in d61719ef6dea
Removing intermediate container d61719ef6dea
---> ca3940266314
Successfully built ca3940266314
Successfully tagged couchdb:3.1.1-patch-jan
```
I pushed the image and pulled it to my workstation again to have the same underlying hardware as in the previous test.
I also adjusted the couchdb-test.sh [couchdb-test-validate-patch-3517.zip](https://github.com/apache/couchdb/files/7083304/couchdb-test-validate-patch-3517.zip) script to use the patched image and compare it against the vanilla CouchDB 3.1.1 image.
```bash
$ ./couchdb-test.sh setup
...
inserting docs cycle 2000/2000
{"ok":true,"id":"a479f05c01bf22c5168ee73617783eb7","rev":"1-f7fac9a3f6ce538d67747271336211c4"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784bb6","rev":"1-7e3d2b01acbc416b60e6e5be050dd790"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784e38","rev":"1-c959b425a724ad6346f646d8a63da6cd"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617785d5b","rev":"1-d8bd0d00d0b04b52796cfe73d9a80760"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617786743","rev":"1-24f6bc4fcfdceae7ff2fae47a68fbf42"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617787715","rev":"1-b0dfc903b7f27dc98130c8d51931f5d4"}
$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"84f2eae60ea04fe192284667cb0e4640","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10
real 0m36.391s
user 0m0.014s
sys 0m0.010s
Round: 2/10
real 0m36.426s
user 0m0.009s
sys 0m0.015s
Round: 3/10
real 0m37.448s
user 0m0.015s
sys 0m0.010s
Round: 4/10
real 0m37.182s
user 0m0.010s
sys 0m0.016s
Round: 5/10
real 0m34.802s
user 0m0.013s
sys 0m0.013s
Round: 6/10
real 0m34.877s
user 0m0.009s
sys 0m0.016s
Round: 7/10
real 0m34.204s
user 0m0.007s
sys 0m0.017s
Round: 8/10
real 0m34.092s
user 0m0.009s
sys 0m0.015s
Round: 9/10
real 0m35.571s
user 0m0.014s
sys 0m0.011s
Round: 10/10
real 0m35.790s
user 0m0.010s
sys 0m0.015s
================ Query CouchDB 3.1.1 patch jan ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"6c2d01a","uuid":"7df6b96ce9f321de839f65a9a6f264f3","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3004
Round: 1/10
real 0m11.194s
user 0m0.012s
sys 0m0.010s
Round: 2/10
real 0m11.224s
user 0m0.014s
sys 0m0.011s
Round: 3/10
real 0m10.966s
user 0m0.012s
sys 0m0.012s
Round: 4/10
real 0m11.212s
user 0m0.013s
sys 0m0.012s
Round: 5/10
real 0m10.696s
user 0m0.016s
sys 0m0.009s
Round: 6/10
real 0m10.859s
user 0m0.013s
sys 0m0.012s
Round: 7/10
real 0m11.342s
user 0m0.012s
sys 0m0.013s
Round: 8/10
real 0m11.276s
user 0m0.011s
sys 0m0.013s
Round: 9/10
real 0m11.357s
user 0m0.015s
sys 0m0.009s
Round: 10/10
real 0m11.542s
user 0m0.014s
sys 0m0.010s
```
That's actually great news, we have an average response time from about 11.2s now compared to 35.7s on the vanilla image :-)
If possible, I would vote for including your patch in a new 3.x build, as it would be more than 3 times faster for the same queries with unchanged client code (at least for the given dataset).
Still, nothing seem to beat build-in reducers, but at least it would not be way slower to run customer reducers on CouchDB 3 vs. CouchDB 2 installations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-909124414
Okay, I validated your fix, please find the outcome below:
First, I wanted to create a patch build which is as close as possible to what I have tested before, so I used the same OS as in the container image I used initially:
```bash
$ podman run --rm -it docker.io/library/couchdb:3.1.1 cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
```
To get a build env matching the OS and its required libraries, I had a look into <https://github.com/apache/couchdb-ci> and built the following image on an 18.04.5 LTS amd64 VM with Docker version 20.10.8, build 3967b7d:
```bash
$ git clone https://github.com/apache/couchdb-ci.git
Cloning into 'couchdb-ci'...
remote: Enumerating objects: 1742, done.
remote: Counting objects: 100% (87/87), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 1742 (delta 55), reused 56 (delta 29), pack-reused 1655
Receiving objects: 100% (1742/1742), 10.24 MiB | 16.11 MiB/s, done.
Resolving deltas: 100% (998/998), done.
$ cd couchdb-ci/
$ ./build.sh platform debian-buster
...
Successfully built d1c29b763a2c
Successfully tagged apache/couchdbci-debian:buster-erlang-20.3.8.26-1
```
From there I used CouchDB v3.1.1 and applied your patch within the container:
```bash
$ docker run --name couchdb-3.1.1-fix-validation -it apache/couchdbci-debian:buster-erlang-20.3.8.26-1
jenkins@2665e52069d0:/$ cd ~
jenkins@2665e52069d0:~$ whoami
jenkins
jenkins@2665e52069d0:~$ pwd
/home/jenkins
jenkins@2665e52069d0:~$ git clone --depth=1 -b 3.1.1 https://github.com/apache/couchdb.git
Cloning into 'couchdb'...
remote: Enumerating objects: 1207, done.
remote: Counting objects: 100% (1207/1207), done.
remote: Compressing objects: 100% (1099/1099), done.
remote: Total 1207 (delta 245), reused 380 (delta 55), pack-reused 0
Receiving objects: 100% (1207/1207), 1.80 MiB | 6.14 MiB/s, done.
Resolving deltas: 100% (245/245), done.
Note: checking out 'ce596c65d9d7f0bc5d9937bcaf6253b343015690'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
jenkins@2665e52069d0:~$ # Apply Jan's patch
jenkins@2665e52069d0:~$ cat <<'EOF' > jan.patch
> diff --git a/share/server/60/rewrite_fun.js b/share/server/60/rewrite_fun.js
> index 1b27a9d14..9b85f953f 100644
> --- a/share/server/60/rewrite_fun.js
> +++ b/share/server/60/rewrite_fun.js
> @@ -15,30 +15,39 @@
> //
> // https://github.com/dmunch/couch-chakra/blob/master/js/normalizeFunction.js
>
> +
> +const cache = {}
> function rewriteFunInt(fun) {
> - const ast = esprima.parse(fun);
> - let idx = ast.body.length - 1;
> - let decl = {};
> -
> - // Search for the first FunctionDeclaration beginning from the end
> - do {
> - decl = ast.body[idx--];
> - } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> - idx++;
> -
> - // If we have a function declaration without an Id, wrap it
> - // in an ExpressionStatement and change it into
> - // a FuntionExpression
> - if (decl.type == "FunctionDeclaration" && decl.id == null) {
> - decl.type = "FunctionExpression";
> - ast.body[idx] = {
> - type: "ExpressionStatement",
> - expression: decl
> - };
> - }
> + const crc = crc32_str(fun)
> + if (cache[crc]) {
> + return cache[crc];
> + } else {
> + const ast = esprima.parse(fun);
> + let idx = ast.body.length - 1;
> + let decl = {};
> +
> + // Search for the first FunctionDeclaration beginning from the end
> + do {
> + decl = ast.body[idx--];
> + } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> + idx++;
> +
> + // If we have a function declaration without an Id, wrap it
> + // in an ExpressionStatement and change it into
> + // a FuntionExpression
> + if (decl.type == "FunctionDeclaration" && decl.id == null) {
> + decl.type = "FunctionExpression";
> + ast.body[idx] = {
> + type: "ExpressionStatement",
> + expression: decl
> + };
> + }
>
> - // Generate source from the rewritten AST
> - return escodegen.generate(ast);
> + // Generate source from the rewritten AST
> + const gen = escodegen.generate(ast);
> + cache[crc] = gen;
> + return gen;
> + }
> }
>
>
> @@ -53,4 +62,51 @@ function rewriteFuns(funsJSON) {
> return rewriteFunInt(fun);
> });
> return JSON.stringify(results);
> -}
> \ No newline at end of file
> +}
> +
> +// nicked from https://github.com/SheetJS/js-crc32/blob/master/crc32.js
> +
> +function signed_crc_table() {
> +var c = 0, table = new Array(256);
> +
> +for(var n =0; n != 256; ++n){
> +c = n;
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +table[n] = c;
> +}
> +
> +return typeof Int32Array !== 'undefined' ? new Int32Array(table) : table;
> +}
> +
> +var T = signed_crc_table();
> +
> +function crc32_str(str, seed) {
> +var C = seed ^ -1;
> +for(var i = 0, L=str.length, c, d; i < L;) {
> +c = str.charCodeAt(i++);
> +if(c < 0x80) {
> +C = (C>>>8) ^ T[(C ^ c)&0xFF];
> +} else if(c < 0x800) {
> +C = (C>>>8) ^ T[(C ^ (192|((c>>6)&31)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +} else if(c >= 0xD800 && c < 0xE000) {
> +c = (c&1023)+64; d = str.charCodeAt(i++)&1023;
> +C = (C>>>8) ^ T[(C ^ (240|((c>>8)&7)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>2)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((d>>6)&15)|((c&3)<<4)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(d&63)))&0xFF];
> +} else {
> +C = (C>>>8) ^ T[(C ^ (224|((c>>12)&15)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>6)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +}
> +}
> +return C ^ -1;
> +}
> EOF
jenkins@2665e52069d0:~$ cd couchdb/
jenkins@2665e52069d0:~/couchdb$ git status
Not currently on any branch.
nothing to commit, working tree clean
jenkins@2665e52069d0:~/couchdb$ git checkout -b 3.1.1-patch-jan
Switched to a new branch '3.1.1-patch-jan'
jenkins@2665e52069d0:~/couchdb$ git apply ../jan.patch
jenkins@2665e52069d0:~/couchdb$ git status
On branch 3.1.1-patch-jan
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: share/server/60/rewrite_fun.js
no changes added to commit (use "git add" and/or "git commit -a")
```
Afterwards I compiled the source:
```bash
jenkins@2665e52069d0:~/couchdb$ ./configure
...
==> couchdb (update-deps)
Updating config from {git,"https://github.com/apache/couchdb-config.git",
{tag,"2.1.7"}}
Updating b64url from {git,"https://github.com/apache/couchdb-b64url.git",
{tag,"1.0.2"}}
Updating ets_lru from {git,"https://github.com/apache/couchdb-ets-lru.git",
{tag,"1.1.0"}}
Updating khash from {git,"https://github.com/apache/couchdb-khash.git",
{tag,"1.1.0"}}
Updating snappy from {git,"https://github.com/apache/couchdb-snappy.git",
{tag,"CouchDB-1.0.4"}}
Updating docs from {git,"https://github.com/apache/couchdb-documentation",
{tag,"3.1.1-RC2"}}
Updating fauxton from {git,"https://github.com/apache/couchdb-fauxton",
{tag,"v1.2.6"}}
Updating folsom from {git,"https://github.com/apache/couchdb-folsom.git",
{tag,"CouchDB-0.8.3"}}
Updating hyper from {git,"https://github.com/apache/couchdb-hyper.git",
{tag,"CouchDB-2.2.0-6"}}
Updating ibrowse from {git,"https://github.com/apache/couchdb-ibrowse.git",
{tag,"CouchDB-4.0.1-1"}}
Updating jiffy from {git,"https://github.com/apache/couchdb-jiffy.git",
{tag,"CouchDB-1.0.4-1"}}
Updating mochiweb from {git,"https://github.com/apache/couchdb-mochiweb.git",
{tag,"v2.20.0"}}
Updating meck from {git,"https://github.com/apache/couchdb-meck.git",
{tag,"0.8.8"}}
Updating recon from {git,"https://github.com/apache/couchdb-recon.git",
{tag,"2.5.0"}}
Updating proper from {git,"https://github.com/proper-testing/proper",
{tag,"v1.3"}}
Updating bear from {git,"https://github.com/apache/couchdb-bear.git",
"008f48aff819126e281d5ccae80a258bf9bf9c30"}
You have configured Apache CouchDB, time to relax. Relax.
# Compile
jenkins@2665e52069d0:~/couchdb$ make release
...
The HTML pages are in build/html.
sphinx-build -b man -a -W -n -A local=1 -D latex_elements.papersize=a4 -d build/doctree src/ build/man
Running Sphinx v1.8.4
making output directory...
loading pickled environment... done
building [mo]: all of 0 po files
building [man]: all source files
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
writing... apachecouchdb.1 { intro/index intro/overview intro/why intro/consistency intro/curl intro/security intro/tour intro/api replication/index replication/intro replication/replicator replication/conflicts replication/protocol ddocs/index ddocs/ddocs ddocs/views/index ddocs/views/intro ddocs/views/collation ddocs/views/joins ddocs/views/nosql ddocs/views/pagination ddocs/search best-practices/index best-practices/documents best-practices/forms best-practices/iso-date best-practices/jsdevel best-practices/views best-practices/reverse-proxies install/index install/unix install/windows install/mac install/freebsd install/docker install/snap install/kubernetes install/search install/upgrading install/troubleshooting setup/index setup/single-node setup/cluster config/index config/intro config/couchdb config/cluster config/couch-peruser config/http config/auth config/compaction config/indexbuilds config/ioq config/logging config/replicator config/query-servers config/misc config/r
esharding cluster/index cluster/theory cluster/nodes cluster/databases cluster/sharding cluster/purging maintenance/index maintenance/compaction maintenance/performance maintenance/backups fauxton/index fauxton/install experimental api/index api/basics api/server/index api/server/common api/server/authn api/server/configuration api/database/index api/database/common api/database/bulk-api api/database/find api/database/shard api/database/changes api/database/compact api/database/security api/database/misc api/document/index api/document/common api/document/attachments api/ddoc/index api/ddoc/common api/ddoc/views api/ddoc/search api/ddoc/render api/ddoc/rewrites api/partitioned-dbs api/local json-structure query-server/index query-server/protocol query-server/javascript query-server/erlang partitioned-dbs/index whatsnew/index whatsnew/3.1 whatsnew/3.0 whatsnew/2.3 whatsnew/2.2 whatsnew/2.1 whatsnew/2.0 whatsnew/1.7 whatsnew/1.6 whatsnew/1.5 whatsnew/1.4 whatsnew/1.3 whatsnew/1.2 what
snew/1.1 whatsnew/1.0 whatsnew/0.11 whatsnew/0.10 whatsnew/0.9 whatsnew/0.8 cve/index cve/2010-0009 cve/2010-2234 cve/2010-3854 cve/2012-5641 cve/2012-5649 cve/2012-5650 cve/2014-2668 cve/2017-12635 cve/2017-12636 cve/2018-11769 cve/2018-17188 cve/2018-8007 cve/2020-1955 about contributing }
build succeeded.
The manual pages are in build/man.
make[1]: Leaving directory '/home/jenkins/couchdb/src/docs'
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN: 'generate' command does not apply to directory /home/jenkins/couchdb
... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.
jenkins@2665e52069d0:~/couchdb$ echo $?
0
```
The compilation seem to have worked, so I created deb packages based on that. The `make dist` seemed not to like uncommited changes, so I added your patch as a commit:
```bash
jenkins@2665e52069d0:~/couchdb$ git config --global user.email "couchdb-validation@no-reply.com"
jenkins@2665e52069d0:~/couchdb$ git config --global user.name "CouchDB validation"
jenkins@2665e52069d0:~/couchdb$ git commit -am "fix: Apply caching"
[3.1.1-patch-jan 6c2d01a] fix: Apply caching
1 file changed, 79 insertions(+), 23 deletions(-)
jenkins@2665e52069d0:~/couchdb$ make dist
==> config (compile)
==> b64url (compile)
==> ets_lru (compile)
==> khash (compile)
==> snappy (compile)
==> bear (compile)
==> meck (compile)
==> folsom (compile)
==> hyper (compile)
==> ibrowse (compile)
==> jiffy (compile)
==> mochiweb (compile)
==> recon (compile)
==> proper (compile)
==> couch_epi (compile)
==> couch_log (compile)
==> chttpd (compile)
==> couch (compile)
Compiling priv/couch_js/1.8.5/http.c
Compiling priv/couch_js/1.8.5/main.c
Compiling priv/couch_js/1.8.5/utf8.c
Compiling priv/couch_js/1.8.5/util.c
==> couch_event (compile)
==> mem3 (compile)
==> couch_index (compile)
==> couch_mrview (compile)
==> couch_replicator (compile)
==> couch_plugins (compile)
==> couch_pse_tests (compile)
==> couch_stats (compile)
==> couch_peruser (compile)
==> couch_tests (compile)
==> ddoc_cache (compile)
==> dreyfus (compile)
==> fabric (compile)
==> global_changes (compile)
==> ioq (compile)
==> jwtf (compile)
==> ken (compile)
==> mango (compile)
==> rexi (compile)
==> setup (compile)
==> smoosh (compile)
==> rel (compile)
==> couchdb (compile)
COUCHDB_GIT_SHA: 6c2d01a
COUCHDB_VERSION: 3.1.1-6c2d01a
COUCHDB_VERSION_SUFFIX: 6c2d01a
DIRTY:
IN_RC:
IN_RELEASE:
ON_TAG:
REL_TAG: 3.1.1
SUB_VSN:
Building Apache CouchDB 3.1.1-6c2d01a
fatal: ambiguous argument '6c976bd..HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Done: apache-couchdb-3.1.1-6c2d01a.tar.gz
```
I copied that tar back to the host OS:
```bash
$ docker cp couchdb-3.1.1-fix-validation:/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a.tar.gz .
$ git clone https://github.com/apache/couchdb-pkg.git
Cloning into 'couchdb-pkg'...
remote: Enumerating objects: 723, done.
remote: Counting objects: 100% (94/94), done.
remote: Compressing objects: 100% (70/70), done.
remote: Total 723 (delta 40), reused 57 (delta 20), pack-reused 629
Receiving objects: 100% (723/723), 25.31 MiB | 23.49 MiB/s, done.
Resolving deltas: 100% (335/335), done.
$ cd couchdb-pkg/
```
The assumed image name was `couchdbdev/<osname>-<codename>-erlang-<erlang-version>`, so I retagged my created one before running the packaging job:
```bash
$ docker tag apache/couchdbci-debian:buster-erlang-20.3.8.26-1 couchdbdev/debian-buster-erlang-20.3.8.26-1
$ ERLANGVERSION=20.3.8.26-1 ./build.sh couch debian-buster /root/apache-couchdb-3.1.1-6c2d01a.tar.gz
Using apache-couchdb-3.1.1-6c2d01a.tar.gz to build packages...
mkdir -p ../couchdb
cp apache-couchdb-3.1.1-6c2d01a.tar.gz ../couchdb
cd ../couchdb && tar xfz *.tar.gz
cp debian/control.in debian/control
sed -i 's/%SPIDERMONKEY%/libmozjs-60-0/g' debian/control
sed -i 's/%SPIDERMONKEY_DEV%/libmozjs-60-dev/g' debian/control
echo 'SM_VER = 60' > debian/sm_ver.mk
rm -rf /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/debian
cp -R debian /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dch -v 3.1.1-6c2d01a~buster "Automatically generated package from upstream."
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dpkg-buildpackage -b -us -uc
dpkg-buildpackage: info: source package couchdb
dpkg-buildpackage: info: source version 3.1.1-6c2d01a~buster
dpkg-buildpackage: info: source distribution UNRELEASED
dpkg-buildpackage: info: source changed by "CouchDB Developers" <"dev@couchdb.apache.org">
dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
debian/rules clean
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh clean --with=systemd
debian/rules override_dh_auto_clean
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_clean
make -j1 distclean
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (clean)
==> b64url (clean)
==> ets_lru (clean)
==> khash (clean)
==> snappy (clean)
==> bear (clean)
==> meck (clean)
==> folsom (clean)
==> hyper (clean)
==> ibrowse (clean)
==> jiffy (clean)
==> mochiweb (clean)
==> recon (clean)
==> couch_epi (clean)
==> couch_log (clean)
==> chttpd (clean)
==> couch (clean)
==> couch_event (clean)
==> mem3 (clean)
==> couch_index (clean)
==> couch_mrview (clean)
==> couch_replicator (clean)
==> couch_plugins (clean)
==> couch_pse_tests (clean)
==> couch_stats (clean)
==> couch_peruser (clean)
==> couch_tests (clean)
==> ddoc_cache (clean)
==> dreyfus (clean)
==> fabric (clean)
==> global_changes (clean)
==> ioq (clean)
==> jwtf (clean)
==> ken (clean)
==> mango (clean)
==> rexi (clean)
==> setup (clean)
==> smoosh (clean)
==> rel (clean)
==> apache-couchdb-3.1.1-6c2d01a (clean)
make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
mv src/mango/src/mango_cursor_text.erl.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.erl.nocompile': No such file or directory
make[2]: [debian/rules:21: override_dh_auto_clean] Error 1 (ignored)
mv src/mango/src/mango_cursor_text.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.nocompile': No such file or directory
make[2]: [debian/rules:22: override_dh_auto_clean] Error 1 (ignored)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_clean
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules build
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh build --with=systemd
dh_update_autotools_config
debian/rules override_dh_auto_configure
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
./configure --spidermonkey-version 60
==> configuring couchdb in rel/couchdb.config
You have configured Apache CouchDB, time to relax. Relax.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules override_dh_auto_build
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_build -- release
make -j1 release
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (compile)
Compiled src/config_util.erl
Compiled src/config_writer.erl
Compiled src/config_sup.erl
Compiled src/config_notifier.erl
Compiled src/config_listener_mon.erl
Compiled src/config_listener.erl
Compiled src/config_app.erl
Compiled src/config.erl
==> b64url (compile)
Compiled src/b64url.erl
Compiling c_src/b64url.c
==> ets_lru (compile)
Compiled src/ets_lru.erl
==> khash (compile)
Compiled src/khash.erl
Compiling c_src/hash.c
Compiling c_src/khash.c
==> snappy (compile)
Compiled src/snappy.erl
Compiling c_src/snappy_nif.cc
c_src/snappy_nif.cc: In function 'ERL_NIF_TERM snappy_compress(ErlNifEnv*, int, const ERL_NIF_TERM*)':
c_src/snappy_nif.cc:156:28: warning: catching polymorphic type 'class std::bad_alloc' by value [-Wcatch-value=]
} catch(std::bad_alloc e) {
^
Compiling c_src/snappy/snappy-sinksource.cc
Compiling c_src/snappy/snappy-stubs-internal.cc
Compiling c_src/snappy/snappy.cc
c_src/snappy/snappy.cc: In instantiation of 'bool snappy::SnappyScatteredWriter<Allocator>::AppendFromSelf(size_t, size_t) [with Allocator = snappy::SnappySinkAllocator; size_t = long unsigned int]':
c_src/snappy/snappy.cc:779:13: required from 'void snappy::SnappyDecompressor::DecompressAllTags(Writer*) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>]'
c_src/snappy/snappy.cc:865:3: required from 'bool snappy::InternalUncompressAllTags(snappy::SnappyDecompressor*, Writer*, snappy::uint32) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>; snappy::uint32 = unsigned int]'
c_src/snappy/snappy.cc:1549:78: required from here
c_src/snappy/snappy.cc:1401:21: warning: comparison of integer expressions of different signedness: 'size_t' {aka 'long unsigned int'} and 'long int' [-Wsign-compare]
if (offset - 1u < op_ptr_ - op_base_) {
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
==> bear (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/bear/src/bear.erl:28: Warning: export_all flag enabled - all functions will be exported
Compiled src/bear.erl
==> meck (compile)
Compiled src/meck_util.erl
Compiled src/meck_matcher.erl
Compiled src/meck_ret_spec.erl
Compiled src/meck_history.erl
Compiled src/meck_expect.erl
Compiled src/meck_cover.erl
Compiled src/meck_code.erl
Compiled src/meck_code_gen.erl
Compiled src/meck_proc.erl
Compiled src/meck_args_matcher.erl
Compiled src/meck.erl
==> folsom (compile)
Compiled src/folsom_utils.erl
Compiled src/folsom_sup.erl
Compiled src/folsom_sample_slide_uniform.erl
Compiled src/folsom_sample_uniform.erl
Compiled src/folsom_sample_slide_sup.erl
Compiled src/folsom_sample_slide_server.erl
Compiled src/folsom_sample_slide_sorted.erl
Compiled src/folsom_vm_metrics.erl
Compiled src/folsom_sample_none.erl
Compiled src/folsom_sample_slide.erl
Compiled src/folsom_metrics_spiral.erl
Compiled src/folsom_sample_exdec.erl
Compiled src/folsom_sample.erl
Compiled src/folsom_metrics_meter.erl
Compiled src/folsom_metrics_history.erl
Compiled src/folsom_metrics_meter_reader.erl
Compiled src/folsom_metrics_histogram_ets.erl
Compiled src/folsom_metrics_gauge.erl
Compiled src/folsom_metrics_histogram.erl
Compiled src/folsom_metrics_duration.erl
Compiled src/folsom_metrics_counter.erl
Compiled src/folsom_ewma.erl
Compiled src/folsom_meter_timer_server.erl
Compiled src/folsom.erl
Compiled src/folsom_metrics.erl
Compiled src/folsom_ets.erl
==> hyper (compile)
Compiled src/hyper_register.erl
Compiled src/hyper_carray.erl
Compiled src/hyper_gb.erl
Compiled src/hyper_const.erl
Compiled src/hyper_array.erl
Compiled src/hyper_binary_rle.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:245: Warning: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:288: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:333: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
Compiled src/hyper.erl
Compiled src/hyper_binary.erl
Compiling c_src/hyper_carray.c
==> ibrowse (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_lib.erl:372: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_lib.erl
Compiled src/ibrowse_sup.erl
Compiled src/ibrowse_socks5.erl
Compiled src/ibrowse_lb.erl
Compiled src/ibrowse_app.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:83: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:93: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:171: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:182: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_test.erl
Compiled src/ibrowse.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:428: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:1831: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_http_client.erl
==> jiffy (compile)
Compiling c_src/decoder.c
Compiling c_src/encoder.c
Compiling c_src/jiffy.c
Compiling c_src/termstack.c
Compiling c_src/utf8.c
Compiling c_src/util.c
Compiling c_src/doubles.cc
Compiling c_src/objects.cc
Compiling c_src/double-conversion/bignum-dtoa.cc
Compiling c_src/double-conversion/bignum.cc
Compiling c_src/double-conversion/cached-powers.cc
Compiling c_src/double-conversion/diy-fp.cc
Compiling c_src/double-conversion/double-conversion.cc
Compiling c_src/double-conversion/fast-dtoa.cc
Compiling c_src/double-conversion/fixed-dtoa.cc
Compiling c_src/double-conversion/strtod.cc
Compiled src/jiffy_utf8.erl
Compiled src/jiffy.erl
==> mochiweb (compile)
Compiled src/reloader.erl
Compiled src/mochiweb_websocket.erl
Compiled src/mochiweb_socket.erl
Compiled src/mochiweb_util.erl
Compiled src/mochiweb_session.erl
Compiled src/mochiweb_socket_server.erl
Compiled src/mochiweb_response.erl
Compiled src/mochiweb_mime.erl
Compiled src/mochiweb_multipart.erl
Compiled src/mochiweb_io.erl
Compiled src/mochiweb_http.erl
Compiled src/mochiweb_request.erl
Compiled src/mochiweb_echo.erl
Compiled src/mochiweb_headers.erl
Compiled src/mochiweb_cover.erl
Compiled src/mochiweb_cookies.erl
Compiled src/mochiweb_clock.erl
Compiled src/mochiweb_base64url.erl
Compiled src/mochiweb_acceptor.erl
Compiled src/mochiweb.erl
Compiled src/mochiweb_html.erl
Compiled src/mochiutf8.erl
Compiled src/mochitemp.erl
Compiled src/mochinum.erl
Compiled src/mochilogfile2.erl
Compiled src/mochilists.erl
Compiled src/mochijson.erl
Compiled src/mochihex.erl
Compiled src/mochiglobal.erl
Compiled src/mochifmt_std.erl
Compiled src/mochifmt_records.erl
Compiled src/mochijson2.erl
Compiled src/mochifmt.erl
Compiled src/mochiweb_charref.erl
==> recon (compile)
Compiled src/recon_map.erl
Compiled src/recon_rec.erl
Compiled src/recon_trace.erl
Compiled src/recon_alloc.erl
Compiled src/recon_lib.erl
Compiled src/recon.erl
==> proper (compile)
Compiled src/vararg.erl
Compiled src/proper_target.erl
Compiled src/proper_types.erl
Compiled src/proper_unicode.erl
Compiled src/proper_unused_imports_remover.erl
Compiled src/proper_symb.erl
Compiled src/proper_transformer.erl
Compiled src/proper_shrink.erl
Compiled src/proper_statem.erl
Compiled src/proper_sets.erl
Compiled src/proper_queue.erl
Compiled src/proper_sa.erl
Compiled src/proper_prop_remover.erl
Compiled src/proper_orddict.erl
Compiled src/proper_ordsets.erl
Compiled src/proper_gen.erl
Compiled src/proper_typeserver.erl
Compiled src/proper_gb_trees.erl
Compiled src/proper_gb_sets.erl
Compiled src/proper_gen_next.erl
Compiled src/proper_dict.erl
Compiled src/proper_array.erl
Compiled src/proper_fsm.erl
Compiled src/proper_arith.erl
Compiled src/proper.erl
==> couch_epi (compile)
Compiled src/couch_epi_util.erl
Compiled src/couch_epi_sup.erl
Compiled src/couch_epi_plugin.erl
Compiled src/couch_epi_module_keeper.erl
Compiled src/couch_epi_functions.erl
Compiled src/couch_epi_functions_gen.erl
Compiled src/couch_epi_data_gen.erl
Compiled src/couch_epi_data.erl
Compiled src/couch_epi_codegen.erl
Compiled src/couch_epi_app.erl
Compiled src/couch_epi_codechange_monitor.erl
Compiled src/couch_epi.erl
==> couch_log (compile)
Compiled src/couch_log_writer.erl
Compiled src/couch_log_writer_journald.erl
Compiled src/couch_log_writer_stderr.erl
Compiled src/couch_log_writer_syslog.erl
Compiled src/couch_log_writer_file.erl
Compiled src/couch_log_util.erl
Compiled src/couch_log_sup.erl
Compiled src/couch_log_server.erl
Compiled src/couch_log_monitor.erl
Compiled src/couch_log_trunc_io_fmt.erl
Compiled src/couch_log_error_logger_h.erl
Compiled src/couch_log_config_dyn.erl
Compiled src/couch_log_trunc_io.erl
Compiled src/couch_log_app.erl
Compiled src/couch_log_config.erl
Compiled src/couch_log.erl
Compiled src/couch_log_formatter.erl
==> chttpd (compile)
Compiled src/chttpd.erl
Compiled src/chttpd_xframe_options.erl
Compiled src/chttpd_sup.erl
Compiled src/chttpd_view.erl
Compiled src/chttpd_stats.erl
Compiled src/chttpd_test_util.erl
Compiled src/chttpd_prefer_header.erl
Compiled src/chttpd_rewrite.erl
Compiled src/chttpd_show.erl
Compiled src/chttpd_plugin.erl
Compiled src/chttpd_httpd_handlers.erl
Compiled src/chttpd_handlers.erl
Compiled src/chttpd_node.erl
Compiled src/chttpd_misc.erl
Compiled src/chttpd_epi.erl
Compiled src/chttpd_external.erl
Compiled src/chttpd_cors.erl
Compiled src/chttpd_auth_request.erl
Compiled src/chttpd_auth.erl
Compiled src/chttpd_app.erl
Compiled src/chttpd_auth_cache.erl
Compiled src/chttpd_db.erl
==> couch (compile)
Compiled src/couch_httpd.erl
Compiled src/test_request.erl
Compiled src/couch_work_queue.erl
Compiled src/couch_uuids.erl
Compiled src/couch_users_db.erl
Compiled src/couch_totp.erl
Compiled src/test_util.erl
Compiled src/couch_util.erl
Compiled src/couch_task_status.erl
Compiled src/couch_sup.erl
Compiled src/couch_secondary_sup.erl
Compiled src/couch_rand.erl
Compiled src/couch_stream.erl
Compiled src/couch_proc_manager.erl
Compiled src/couch_query_servers.erl
Compiled src/couch_server.erl
Compiled src/couch_primary_sup.erl
Compiled src/couch_partition.erl
Compiled src/couch_passwords.erl
Compiled src/couch_os_process.erl
Compiled src/couch_lru.erl
Compiled src/couch_multidb_changes.erl
Compiled src/couch_io_logger.erl
Compiled src/couch_native_process.erl
Compiled src/couch_httpd_vhost.erl
Compiled src/couch_key_tree.erl
Compiled src/couch_httpd_rewrite.erl
Compiled src/couch_httpd_handlers.erl
Compiled src/couch_httpd_multipart.erl
Compiled src/couch_httpd_misc_handlers.erl
Compiled src/couch_httpd_external.erl
Compiled src/couch_hotp.erl
Compiled src/couch_hash.erl
Compiled src/couch_flags_config.erl
Compiled src/couch_httpd_auth.erl
Compiled src/couch_flags.erl
Compiled src/couch_event_sup.erl
Compiled src/couch_emsort.erl
Compiled src/couch_ejson_size.erl
Compiled src/couch_file.erl
Compiled src/couch_httpd_db.erl
Compiled src/couch_ejson_compare.erl
Compiled src/couch_drv.erl
Compiled src/couch_debug.erl
Compiled src/couch_doc.erl
Compiled src/couch_db_split.erl
Compiled src/couch_db_plugin.erl
Compiled src/couch_db_epi.erl
Compiled src/couch_db_header.erl
Compiled src/couch_db_updater.erl
Compiled src/couch_compress.erl
Compiled src/couch_db_engine.erl
Compiled src/couch_changes.erl
Compiled src/couch_bt_engine_stream.erl
Compiled src/couch_bt_engine_header.erl
Compiled src/couch_db.erl
Compiled src/couch_btree.erl
Compiled src/couch_base32.erl
Compiled src/couch_bt_engine_compactor.erl
Compiled src/couch_auth_cache.erl
Compiled src/couch_att.erl
Compiled src/couch_bt_engine.erl
Compiled src/couch.erl
Compiled src/couch_app.erl
Compiling priv/couch_js/60/http.cpp
Compiling priv/couch_js/60/main.cpp
Compiling priv/couch_js/60/util.cpp
Compiling priv/icu_driver/couch_icu_driver.c
Compiling priv/couch_ejson_compare/couch_ejson_compare.c
==> couch_event (compile)
Compiled src/couch_event_listener.erl
Compiled src/couch_event_sup2.erl
Compiled src/couch_event_os_listener.erl
Compiled src/couch_event_app.erl
Compiled src/couch_event.erl
Compiled src/couch_event_server.erl
Compiled src/couch_event_listener_mfa.erl
==> mem3 (compile)
Compiled src/mem3_sync_nodes.erl
Compiled src/mem3_sync_security.erl
Compiled src/mem3_sync_event.erl
Compiled src/mem3_sync_event_listener.erl
Compiled src/mem3_sup.erl
Compiled src/mem3_sync.erl
Compiled src/mem3_util.erl
Compiled src/mem3_seeds.erl
Compiled src/mem3_reshard_validate.erl
Compiled src/mem3_reshard_sup.erl
Compiled src/mem3_shards.erl
Compiled src/mem3_rpc.erl
Compiled src/mem3_reshard_job_sup.erl
Compiled src/mem3_reshard_store.erl
Compiled src/mem3_reshard_index.erl
Compiled src/mem3_reshard_httpd.erl
Compiled src/mem3_reshard_job.erl
Compiled src/mem3_reshard_api.erl
Compiled src/mem3_reshard_dbdoc.erl
Compiled src/mem3_plugin_couch_db.erl
Compiled src/mem3_nodes.erl
Compiled src/mem3_httpd_handlers.erl
Compiled src/mem3_reshard.erl
Compiled src/mem3_hash.erl
Compiled src/mem3_epi.erl
Compiled src/mem3_httpd.erl
Compiled src/mem3_rep.erl
Compiled src/mem3_app.erl
Compiled src/mem3_cluster.erl
Compiled src/mem3.erl
==> couch_index (compile)
Compiled src/couch_index_sup.erl
Compiled src/couch_index_util.erl
Compiled src/couch_index_plugin_couch_db.erl
Compiled src/couch_index_updater.erl
Compiled src/couch_index_epi.erl
Compiled src/couch_index_plugin.erl
Compiled src/couch_index_app.erl
Compiled src/couch_index_server.erl
Compiled src/couch_index_compactor.erl
Compiled src/couch_index.erl
==> couch_mrview (compile)
Compiled src/couch_mrview_update_notifier.erl
Compiled src/couch_mrview_updater.erl
Compiled src/couch_mrview_test_util.erl
Compiled src/couch_mrview_show.erl
Compiled src/couch_mrview_util.erl
Compiled src/couch_mrview_index.erl
Compiled src/couch_mrview_compactor.erl
Compiled src/couch_mrview_cleanup.erl
Compiled src/couch_mrview_http.erl
Compiled src/couch_mrview.erl
==> couch_replicator (compile)
Compiled src/couch_replicator_httpc.erl
Compiled src/couch_replicator_auth.erl
Compiled src/couch_replicator_utils.erl
Compiled src/couch_replicator_sup.erl
Compiled src/couch_replicator_stats.erl
Compiled src/couch_replicator_scheduler_sup.erl
Compiled src/json_stream_parse.erl
Compiled src/couch_replicator_worker.erl
Compiled src/couch_replicator_rate_limiter_tables.erl
Compiled src/couch_replicator_rate_limiter.erl
Compiled src/couch_replicator_notifier.erl
Compiled src/couch_replicator_job_sup.erl
Compiled src/couch_replicator_scheduler_job.erl
Compiled src/couch_replicator_scheduler.erl
Compiled src/couch_replicator_ids.erl
Compiled src/couch_replicator_httpd_util.erl
Compiled src/couch_replicator_httpd.erl
Compiled src/couch_replicator_httpc_pool.erl
Compiled src/couch_replicator_filters.erl
Compiled src/couch_replicator_fabric_rpc.erl
Compiled src/couch_replicator_doc_processor_worker.erl
Compiled src/couch_replicator_fabric.erl
Compiled src/couch_replicator_db_changes.erl
Compiled src/couch_replicator_connection.erl
Compiled src/couch_replicator_docs.erl
Compiled src/couch_replicator_changes_reader.erl
Compiled src/couch_replicator_doc_processor.erl
Compiled src/couch_replicator_clustering.erl
Compiled src/couch_replicator_app.erl
Compiled src/couch_replicator_auth_noop.erl
Compiled src/couch_replicator_auth_session.erl
Compiled src/couch_replicator.erl
Compiled src/couch_replicator_api_wrap.erl
==> couch_plugins (compile)
Compiled src/couch_plugins_httpd.erl
Compiled src/couch_plugins.erl
==> couch_pse_tests (compile)
Compiled src/cpse_test_ref_counting.erl
Compiled src/cpse_util.erl
Compiled src/cpse_test_read_write_docs.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/couch_pse_tests/src/cpse_test_purge_seqs.erl:14: Warning: export_all flag enabled - all functions will be exported
Compiled src/cpse_test_purge_seqs.erl
Compiled src/cpse_test_purge_replication.erl
Compiled src/cpse_test_purge_docs.erl
Compiled src/cpse_test_open_close_delete.erl
Compiled src/cpse_test_purge_bad_checkpoints.erl
Compiled src/cpse_test_get_set_props.erl
Compiled src/cpse_test_fold_purge_infos.erl
Compiled src/cpse_test_fold_changes.erl
Compiled src/cpse_test_fold_docs.erl
Compiled src/cpse_test_copy_purge_infos.erl
Compiled src/cpse_gather.erl
Compiled src/cpse_test_attachments.erl
Compiled src/cpse_test_compaction.erl
==> couch_stats (compile)
Compiled src/couch_stats_sup.erl
Compiled src/couch_stats_app.erl
Compiled src/couch_stats_process_tracker.erl
Compiled src/couch_stats.erl
Compiled src/couch_stats_httpd.erl
Compiled src/couch_stats_aggregator.erl
==> couch_peruser (compile)
Compiled src/couch_peruser_app.erl
Compiled src/couch_peruser_sup.erl
Compiled src/couch_peruser.erl
==> couch_tests (compile)
Compiled setups/couch_epi_dispatch.erl
Compiled src/couch_tests_combinatorics.erl
Compiled src/couch_tests.erl
==> ddoc_cache (compile)
Compiled src/ddoc_cache_value.erl
Compiled src/ddoc_cache_sup.erl
Compiled src/ddoc_cache_opener.erl
Compiled src/ddoc_cache_entry_validation_funs.erl
Compiled src/ddoc_cache_entry_ddocid_rev.erl
Compiled src/ddoc_cache_entry_ddocid.erl
Compiled src/ddoc_cache_entry_custom.erl
Compiled src/ddoc_cache_app.erl
Compiled src/ddoc_cache_lru.erl
Compiled src/ddoc_cache.erl
Compiled src/ddoc_cache_entry.erl
==> dreyfus (compile)
Compiled src/dreyfus_sup.erl
Compiled src/dreyfus_plugin_couch_db.erl
Compiled src/dreyfus_rpc.erl
Compiled src/dreyfus_index_updater.erl
Compiled src/dreyfus_util.erl
Compiled src/dreyfus_index_manager.erl
Compiled src/dreyfus_httpd_handlers.erl
Compiled src/dreyfus_index.erl
Compiled src/dreyfus_fabric_search.erl
Compiled src/dreyfus_fabric_info.erl
Compiled src/dreyfus_httpd.erl
Compiled src/dreyfus_fabric_group2.erl
Compiled src/dreyfus_fabric_cleanup.erl
Compiled src/dreyfus_fabric_group1.erl
Compiled src/dreyfus_epi.erl
Compiled src/dreyfus_config.erl
Compiled src/dreyfus_app.erl
Compiled src/dreyfus_fabric.erl
Compiled src/dreyfus_bookmark.erl
Compiled src/clouseau_rpc.erl
==> fabric (compile)
Compiled src/fabric_db_update_listener.erl
Compiled src/fabric_view_reduce.erl
Compiled src/fabric_view_map.erl
Compiled src/fabric_view_all_docs.erl
Compiled src/fabric_view_changes.erl
Compiled src/fabric_view.erl
Compiled src/fabric_streams.erl
Compiled src/fabric_util.erl
Compiled src/fabric_rpc.erl
Compiled src/fabric_ring.erl
Compiled src/fabric_group_info.erl
Compiled src/fabric_doc_update.erl
Compiled src/fabric_doc_purge.erl
Compiled src/fabric_doc_missing_revs.erl
Compiled src/fabric_doc_open_revs.erl
Compiled src/fabric_doc_atts.erl
Compiled src/fabric_doc_open.erl
Compiled src/fabric_dict.erl
Compiled src/fabric_doc_attachments.erl
Compiled src/fabric_design_doc_count.erl
Compiled src/fabric_db_partition_info.erl
Compiled src/fabric_db_meta.erl
Compiled src/fabric_db_doc_count.erl
Compiled src/fabric_db_info.erl
Compiled src/fabric_db_delete.erl
Compiled src/fabric_db_create.erl
Compiled src/fabric.erl
==> global_changes (compile)
Compiled src/global_changes_sup.erl
Compiled src/global_changes_util.erl
Compiled src/global_changes_plugin.erl
Compiled src/global_changes_httpd_handlers.erl
Compiled src/global_changes_listener.erl
Compiled src/global_changes_server.erl
Compiled src/global_changes_epi.erl
Compiled src/global_changes_app.erl
Compiled src/global_changes_httpd.erl
==> ioq (compile)
Compiled src/ioq_app.erl
Compiled src/ioq_sup.erl
Compiled src/ioq.erl
==> jwtf (compile)
Compiled src/jwtf_app.erl
Compiled src/jwtf_sup.erl
Compiled src/jwtf.erl
Compiled src/jwtf_keystore.erl
==> ken (compile)
Compiled src/ken_sup.erl
Compiled src/ken_event_handler.erl
Compiled src/ken_app.erl
Compiled src/ken.erl
Compiled src/ken_server.erl
==> mango (compile)
Compiled src/mango_sup.erl
Compiled src/mango_sort.erl
Compiled src/mango_util.erl
Compiled src/mango_selector_text.erl
Compiled src/mango_selector.erl
Compiled src/mango_opts.erl
Compiled src/mango_json.erl
Compiled src/mango_json_bookmark.erl
Compiled src/mango_native_proc.erl
Compiled src/mango_idx_special.erl
Compiled src/mango_idx_text.erl
Compiled src/mango_httpd_handlers.erl
Compiled src/mango_idx_view.erl
Compiled src/mango_fields.erl
Compiled src/mango_execution_stats.erl
Compiled src/mango_idx.erl
Compiled src/mango_epi.erl
Compiled src/mango_httpd.erl
Compiled src/mango_error.erl
Compiled src/mango_doc.erl
Compiled src/mango_cursor_text.erl
Compiled src/mango_cursor_special.erl
Compiled src/mango_cursor.erl
Compiled src/mango_app.erl
Compiled src/mango_crud.erl
Compiled src/mango_cursor_view.erl
==> rexi (compile)
Compiled src/rexi_server_sup.erl
Compiled src/rexi_sup.erl
Compiled src/rexi_utils.erl
Compiled src/rexi_server_mon.erl
Compiled src/rexi_monitor.erl
Compiled src/rexi_app.erl
Compiled src/rexi_server.erl
Compiled src/rexi_buffer.erl
Compiled src/rexi.erl
==> setup (compile)
Compiled src/setup_sup.erl
Compiled src/setup_httpd_handlers.erl
Compiled src/setup_app.erl
Compiled src/setup_epi.erl
Compiled src/setup_httpd.erl
Compiled src/setup.erl
==> smoosh (compile)
Compiled src/smoosh_sup.erl
Compiled src/smoosh_utils.erl
Compiled src/smoosh_priority_queue.erl
Compiled src/smoosh_app.erl
Compiled src/smoosh_channel.erl
Compiled src/smoosh_server.erl
Compiled src/smoosh.erl
==> rel (compile)
==> apache-couchdb-3.1.1-6c2d01a (compile)
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN: 'generate' command does not apply to directory /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.
make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
rm -rf rel/couchdb/var/log
rm -rf rel/couchdb/data
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules binary
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh binary --with=systemd
dh_testroot
dh_prep
dh_installdirs
dh_install
dh_installdocs
dh_installchangelogs
dh_installman
dh_installdebconf
dh_systemd_enable
dh_installinit
dh_systemd_start
dh_installlogrotate
dh_lintian
dh_perl
dh_link
dh_strip_nondeterminism
dh_compress
dh_fixperms
dh_missing
dh_strip
dh_makeshlibs
debian/rules override_dh_shlibdeps
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_shlibdeps -- --ignore-missing-info -xlibmozjs185-1.0
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_ejson_compare.so debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_icu_driver.so were not linked against libicudata.so.63 (they use none of the library's symbols)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_installdeb
dh_gencontrol
dh_md5sums
debian/rules override_dh_builddeb
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_builddeb -- -Zgzip
dpkg-deb: building package 'couchdb-dbgsym' in '../couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb'.
dpkg-deb: building package 'couchdb' in '../couchdb_3.1.1-6c2d01a~buster_amd64.deb'.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dpkg-genbuildinfo --build=binary
dpkg-genchanges --build=binary >../couchdb_3.1.1-6c2d01a~buster_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
dpkg-source --after-build .
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
dpkg-buildpackage: info: binary-only upload (no source included)
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/.. && lintian --profile couchdb couch*.deb || true
warning: the authors of lintian do not recommend running it with root privileges!
W: couchdb: maintainer-script-should-not-use-recursive-chown-or-chmod postinst:207
N: 3 tags overridden (2 errors, 1 info); 2 unused overrides
chmod a+rwx ../couchdb/couchdb*
mkdir -p pkgs/couch/debian-buster && chmod 777 pkgs/couch/debian-buster
cp ../couchdb/couchdb* pkgs/couch/debian-buster
if [ -f debian/control.bak ]; then mv -f debian/control.bak debian/control; fi
if [ -f rpm/SPECS/couchdb.spec.bak ]; then mv -f rpm/SPECS/couchdb.spec.bak rpm/SPECS/couchdb.spec; fi
rm -rf parts prime stage js/build debian/sm_ver.mk
# packages were created
$ ls -lah pkgs/couch/debian-buster/
total 30M
drwxrwxrwx 2 root root 207 Aug 31 01:58 .
drwxrwxrwx 3 root root 45 Aug 31 01:58 ..
-rwxr-xr-x 1 root root 11K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo
-rwxr-xr-x 1 root root 1.5K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.changes
-rwxr-xr-x 1 root root 30M Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.deb
-rwxr-xr-x 1 root root 474K Aug 31 01:58 couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb
```
Next use https://github.com/apache/couchdb-docker/tree/main/3.1.1 to produce the container images, but adjust them to include the custom couchdb version.
```bash
$ git clone https://github.com/apache/couchdb-docker.git
Cloning into 'couchdb-docker'...
remote: Enumerating objects: 1010, done.
remote: Counting objects: 100% (118/118), done.
remote: Compressing objects: 100% (82/82), done.
remote: Total 1010 (delta 64), reused 64 (delta 32), pack-reused 892
Receiving objects: 100% (1010/1010), 248.11 KiB | 1.19 MiB/s, done.
Resolving deltas: 100% (490/490), done.
$ cd ~/couchdb-docker/3.1.1
$ cp -r ~/couchdb-pkg/pkgs/couch/debian-buster/ .
$ ls debian-buster/
couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo couchdb_3.1.1-6c2d01a~buster_amd64.changes couchdb_3.1.1-6c2d01a~buster_amd64.deb couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb
$ git diff
diff --git a/3.1.1/Dockerfile b/3.1.1/Dockerfile
index 75b4b29..59a90d9 100644
--- a/3.1.1/Dockerfile
+++ b/3.1.1/Dockerfile
@@ -60,6 +60,10 @@ RUN . /etc/os-release; \
echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | \
tee /etc/apt/sources.list.d/couchdb.list >/dev/null
+# I know, we should not include deb packages in images, but it should be good enough to validate the fix
+COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+
# https://github.com/apache/couchdb-pkg/blob/master/debian/README.Debian
RUN set -eux; \
apt-get update; \
@@ -67,7 +71,7 @@ RUN set -eux; \
echo "couchdb couchdb/mode select none" | debconf-set-selections; \
# we DO want recommends this time
DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages \
- couchdb="$COUCHDB_VERSION"~buster \
+ /tmp/packages/* \
; \
# Undo symlinks to /var/log and /var/lib
rmdir /var/lib/couchdb /var/log/couchdb; \
docker build -t "couchdb:3.1.1-patch-jan" .
Sending build context to Docker daemon 31.45MB
Step 1/20 : FROM debian:buster-slim
---> 5c3d57f3e47a
Step 2/20 : LABEL maintainer="CouchDB Developers dev@couchdb.apache.org"
---> Using cache
---> 60f37c14a0e1
Step 3/20 : RUN groupadd -g 5984 -r couchdb && useradd -u 5984 -d /opt/couchdb -g couchdb couchdb
---> Using cache
---> 78202346b500
Step 4/20 : RUN set -ex; apt-get update; apt-get install -y --no-install-recommends apt-transport-https ca-certificates dirmngr gnupg ; rm -rf /var/lib/apt/lists/*
---> Using cache
---> fad0f8a736e0
Step 5/20 : RUN set -eux; apt-get update; apt-get install -y --no-install-recommends gosu tini; rm -rf /var/lib/apt/lists/*; gosu nobody true; tini --version
---> Using cache
---> aca1db0792a3
Step 6/20 : ENV GPG_COUCH_KEY 390EF70BB1EA12B2773962950EE62FB37A00258D
---> Using cache
---> d28f3c244c10
Step 7/20 : RUN set -eux; apt-get update; apt-get install -y curl; export GNUPGHOME="$(mktemp -d)"; curl -fL -o keys.asc https://couchdb.apache.org/repo/keys.asc; gpg --batch --import keys.asc; gpg --batch --export "${GPG_COUCH_KEY}" > /usr/share/keyrings/couchdb-archive-keyring.gpg; command -v gpgconf && gpgconf --kill all || :; rm -rf "$GNUPGHOME"; apt-key list; apt purge -y --autoremove curl; rm -rf /var/lib/apt/lists/*
---> Using cache
---> e2949d6cc031
Step 8/20 : ENV COUCHDB_VERSION 3.1.1
---> Using cache
---> baa8089d1bea
Step 9/20 : RUN . /etc/os-release; echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | tee /etc/apt/sources.list.d/couchdb.list >/dev/null
---> Using cache
---> a7933b88e802
Step 10/20 : COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
---> Using cache
---> ded52d34b729
Step 11/20 : COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
---> Using cache
---> eb9b215cb430
Step 12/20 : RUN set -eux; apt-get update; echo "couchdb couchdb/mode select none" | debconf-set-selections; DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/* ; rmdir /var/lib/couchdb /var/log/couchdb; rm /opt/couchdb/data /opt/couchdb/var/log; mkdir -p /opt/couchdb/data /opt/couchdb/var/log; chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log; chmod 777 /opt/couchdb/data /opt/couchdb/var/log; rm /opt/couchdb/etc/default.d/10-filelog.ini; find /opt/couchdb \! \( -user couchdb -group couchdb \) -exec chown -f couchdb:couchdb '{}' +; find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 '{}' +; find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 '{}' +; chmod -f 0777 /opt/couchdb/etc/local.d; rm -rf /var/lib/apt/lists/*;
---> Running in 6f530f886258
+ apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [302 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 https://apache.jfrog.io/artifactory/couchdb-deb buster InRelease [5151 B]
Get:7 https://apache.jfrog.io/artifactory/couchdb-deb buster/main amd64 Packages [3889 B]
Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages [15.2 kB]
Fetched 8473 kB in 3s (3017 kB/s)
Reading package lists...
+ echo couchdb couchdb/mode select none
+ debconf-set-selections
+ DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2 libicu63 libk5crypto3
libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0 libncurses6
libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps psmisc
publicsuffix
Suggested packages:
gpm krb5-doc krb5-user
The following NEW packages will be installed:
couchdb couchdb-dbgsym curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2
libicu63 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0
libncurses6 libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps
psmisc publicsuffix
0 upgraded, 22 newly installed, 0 to remove and 0 not upgraded.
Need to get 13.5 MB/45.0 MB of archives.
After this operation, 104 MB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security buster/updates/main amd64 krb5-locales all 1.17-3+deb10u2 [95.5 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libncurses6 amd64 6.1+20181013-2+deb10u2 [102 kB]
Get:3 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5support0 amd64 1.17-3+deb10u2 [65.7 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 libk5crypto3 amd64 1.17-3+deb10u2 [122 kB]
Get:5 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5-3 amd64 1.17-3+deb10u2 [369 kB]
Get:6 http://security.debian.org/debian-security buster/updates/main amd64 libgssapi-krb5-2 amd64 1.17-3+deb10u2 [158 kB]
Get:7 http://deb.debian.org/debian buster/main amd64 libprocps7 amd64 2:3.3.15-2 [61.7 kB]
Get:8 http://deb.debian.org/debian buster/main amd64 procps amd64 2:3.3.15-2 [259 kB]
Get:9 http://deb.debian.org/debian buster/main amd64 libkeyutils1 amd64 1.6-6 [15.0 kB]
Get:10 http://deb.debian.org/debian buster/main amd64 libnghttp2-14 amd64 1.36.0-2+deb10u1 [85.0 kB]
Get:11 /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb couchdb amd64 3.1.1-6c2d01a~buster [30.9 MB]
Get:12 /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb couchdb-dbgsym amd64 3.1.1-6c2d01a~buster [485 kB]
Get:13 http://deb.debian.org/debian buster/main amd64 libpsl5 amd64 0.20.2-2 [53.7 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2 [60.5 kB]
Get:15 http://deb.debian.org/debian buster/main amd64 libssh2-1 amd64 1.8.0-2.1 [140 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 libcurl4 amd64 7.64.0-4+deb10u2 [332 kB]
Get:17 http://deb.debian.org/debian buster/main amd64 curl amd64 7.64.0-4+deb10u2 [265 kB]
Get:18 http://deb.debian.org/debian buster/main amd64 libicu63 amd64 63.1-6+deb10u1 [8300 kB]
Get:19 http://deb.debian.org/debian buster/main amd64 libmozjs-60-0 amd64 60.2.3-3 [2785 kB]
Get:20 http://deb.debian.org/debian buster/main amd64 libgpm2 amd64 1.20.7-5 [35.1 kB]
Get:21 http://deb.debian.org/debian buster/main amd64 psmisc amd64 23.2-1 [126 kB]
Get:22 http://deb.debian.org/debian buster/main amd64 publicsuffix all 20190415.1030-1 [116 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 13.5 MB in 1s (12.8 MB/s)
Selecting previously unselected package libncurses6:amd64.
(Reading database ... 7118 files and directories currently installed.)
Preparing to unpack .../00-libncurses6_6.1+20181013-2+deb10u2_amd64.deb ...
Unpacking libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Selecting previously unselected package libprocps7:amd64.
Preparing to unpack .../01-libprocps7_2%3a3.3.15-2_amd64.deb ...
Unpacking libprocps7:amd64 (2:3.3.15-2) ...
Selecting previously unselected package procps.
Preparing to unpack .../02-procps_2%3a3.3.15-2_amd64.deb ...
Unpacking procps (2:3.3.15-2) ...
Selecting previously unselected package krb5-locales.
Preparing to unpack .../03-krb5-locales_1.17-3+deb10u2_all.deb ...
Unpacking krb5-locales (1.17-3+deb10u2) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../04-libkeyutils1_1.6-6_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.6-6) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../05-libkrb5support0_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../06-libk5crypto3_1.17-3+deb10u2_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../07-libkrb5-3_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../08-libgssapi-krb5-2_1.17-3+deb10u2_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../09-libnghttp2-14_1.36.0-2+deb10u1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../10-libpsl5_0.20.2-2_amd64.deb ...
Unpacking libpsl5:amd64 (0.20.2-2) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../11-librtmp1_2.4+20151223.gitfa8646d.1-2_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Selecting previously unselected package libssh2-1:amd64.
Preparing to unpack .../12-libssh2-1_1.8.0-2.1_amd64.deb ...
Unpacking libssh2-1:amd64 (1.8.0-2.1) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../13-libcurl4_7.64.0-4+deb10u2_amd64.deb ...
Unpacking libcurl4:amd64 (7.64.0-4+deb10u2) ...
Selecting previously unselected package curl.
Preparing to unpack .../14-curl_7.64.0-4+deb10u2_amd64.deb ...
Unpacking curl (7.64.0-4+deb10u2) ...
Selecting previously unselected package libicu63:amd64.
Preparing to unpack .../15-libicu63_63.1-6+deb10u1_amd64.deb ...
Unpacking libicu63:amd64 (63.1-6+deb10u1) ...
Selecting previously unselected package libmozjs-60-0:amd64.
Preparing to unpack .../16-libmozjs-60-0_60.2.3-3_amd64.deb ...
Unpacking libmozjs-60-0:amd64 (60.2.3-3) ...
Selecting previously unselected package couchdb.
Preparing to unpack .../17-couchdb_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package couchdb-dbgsym.
Preparing to unpack .../18-couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package libgpm2:amd64.
Preparing to unpack .../19-libgpm2_1.20.7-5_amd64.deb ...
Unpacking libgpm2:amd64 (1.20.7-5) ...
Selecting previously unselected package psmisc.
Preparing to unpack .../20-psmisc_23.2-1_amd64.deb ...
Unpacking psmisc (23.2-1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../21-publicsuffix_20190415.1030-1_all.deb ...
Unpacking publicsuffix (20190415.1030-1) ...
Setting up libkeyutils1:amd64 (1.6-6) ...
Setting up libpsl5:amd64 (0.20.2-2) ...
Setting up libgpm2:amd64 (1.20.7-5) ...
Setting up psmisc (23.2-1) ...
Setting up libprocps7:amd64 (2:3.3.15-2) ...
Setting up libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Setting up krb5-locales (1.17-3+deb10u2) ...
Setting up libicu63:amd64 (63.1-6+deb10u1) ...
Setting up libkrb5support0:amd64 (1.17-3+deb10u2) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Setting up libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Setting up libk5crypto3:amd64 (1.17-3+deb10u2) ...
Setting up procps (2:3.3.15-2) ...
update-alternatives: using /usr/bin/w.procps to provide /usr/bin/w (w) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/w.1.gz because associated file /usr/share/man/man1/w.procps.1.gz (of link group w) doesn't exist
Setting up libssh2-1:amd64 (1.8.0-2.1) ...
Setting up libkrb5-3:amd64 (1.17-3+deb10u2) ...
Setting up publicsuffix (20190415.1030-1) ...
Setting up libmozjs-60-0:amd64 (60.2.3-3) ...
Setting up libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Setting up libcurl4:amd64 (7.64.0-4+deb10u2) ...
Setting up curl (7.64.0-4+deb10u2) ...
Setting up couchdb (3.1.1-6c2d01a~buster) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Processing triggers for libc-bin (2.28-10) ...
+ rmdir /var/lib/couchdb /var/log/couchdb
+ rm /opt/couchdb/data /opt/couchdb/var/log
+ mkdir -p /opt/couchdb/data /opt/couchdb/var/log
+ chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log
+ chmod 777 /opt/couchdb/data /opt/couchdb/var/log
+ rm /opt/couchdb/etc/default.d/10-filelog.ini
+ find /opt/couchdb ! ( -user couchdb -group couchdb ) -exec chown -f couchdb:couchdb {} +
+ find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 {} +
+ find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 {} +
+ chmod -f 0777 /opt/couchdb/etc/local.d
+ rm -rf /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_InRelease /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/auxfiles /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/deb.debian.org_debian_dists_buster_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_main_binary-amd64_Packages.lz4
Removing intermediate container 6f530f886258
---> eaea07a7886e
Step 13/20 : COPY --chown=couchdb:couchdb 10-docker-default.ini /opt/couchdb/etc/default.d/
---> f1c9d8e86732
Step 14/20 : COPY --chown=couchdb:couchdb vm.args /opt/couchdb/etc/
---> a984087a4ee3
Step 15/20 : COPY docker-entrypoint.sh /usr/local/bin
---> 66cd3ececce1
Step 16/20 : RUN ln -s usr/local/bin/docker-entrypoint.sh /docker-entrypoint.sh # backwards compat
---> Running in ae10e0e02906
Removing intermediate container ae10e0e02906
---> bf8817264923
Step 17/20 : ENTRYPOINT ["tini", "--", "/docker-entrypoint.sh"]
---> Running in 1485a79c319f
Removing intermediate container 1485a79c319f
---> 737d9f7a9b97
Step 18/20 : VOLUME /opt/couchdb/data
---> Running in 6aa4eafbc054
Removing intermediate container 6aa4eafbc054
---> 3f111d557168
Step 19/20 : EXPOSE 5984 4369 9100
---> Running in 07b6a3204c82
Removing intermediate container 07b6a3204c82
---> c70917bea3fd
Step 20/20 : CMD ["/opt/couchdb/bin/couchdb"]
---> Running in d61719ef6dea
Removing intermediate container d61719ef6dea
---> ca3940266314
Successfully built ca3940266314
Successfully tagged couchdb:3.1.1-patch-jan
```
I pushed the image and pulled it to my workstation again to have the same underlying hardware as in the previous test.
I also adjusted the couchdb-test.sh [couchdb-test-validate-patch-3517.zip](https://github.com/apache/couchdb/files/7083304/couchdb-test-validate-patch-3517.zip) script to use the patched image and compare it against the vanilla CouchDB 3.1.1 image.
```bash
$ ./couchdb-test.sh setup
...
inserting docs cycle 2000/2000
{"ok":true,"id":"a479f05c01bf22c5168ee73617783eb7","rev":"1-f7fac9a3f6ce538d67747271336211c4"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784bb6","rev":"1-7e3d2b01acbc416b60e6e5be050dd790"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784e38","rev":"1-c959b425a724ad6346f646d8a63da6cd"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617785d5b","rev":"1-d8bd0d00d0b04b52796cfe73d9a80760"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617786743","rev":"1-24f6bc4fcfdceae7ff2fae47a68fbf42"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617787715","rev":"1-b0dfc903b7f27dc98130c8d51931f5d4"}
$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"84f2eae60ea04fe192284667cb0e4640","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10
real 0m36.391s
user 0m0.014s
sys 0m0.010s
Round: 2/10
real 0m36.426s
user 0m0.009s
sys 0m0.015s
Round: 3/10
real 0m37.448s
user 0m0.015s
sys 0m0.010s
Round: 4/10
real 0m37.182s
user 0m0.010s
sys 0m0.016s
Round: 5/10
real 0m34.802s
user 0m0.013s
sys 0m0.013s
Round: 6/10
real 0m34.877s
user 0m0.009s
sys 0m0.016s
Round: 7/10
real 0m34.204s
user 0m0.007s
sys 0m0.017s
Round: 8/10
real 0m34.092s
user 0m0.009s
sys 0m0.015s
Round: 9/10
real 0m35.571s
user 0m0.014s
sys 0m0.011s
Round: 10/10
real 0m35.790s
user 0m0.010s
sys 0m0.015s
================ Query CouchDB 3.1.1 patch jan ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"6c2d01a","uuid":"7df6b96ce9f321de839f65a9a6f264f3","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3004
Round: 1/10
real 0m11.194s
user 0m0.012s
sys 0m0.010s
Round: 2/10
real 0m11.224s
user 0m0.014s
sys 0m0.011s
Round: 3/10
real 0m10.966s
user 0m0.012s
sys 0m0.012s
Round: 4/10
real 0m11.212s
user 0m0.013s
sys 0m0.012s
Round: 5/10
real 0m10.696s
user 0m0.016s
sys 0m0.009s
Round: 6/10
real 0m10.859s
user 0m0.013s
sys 0m0.012s
Round: 7/10
real 0m11.342s
user 0m0.012s
sys 0m0.013s
Round: 8/10
real 0m11.276s
user 0m0.011s
sys 0m0.013s
Round: 9/10
real 0m11.357s
user 0m0.015s
sys 0m0.009s
Round: 10/10
real 0m11.542s
user 0m0.014s
sys 0m0.010s
```
That's actually great news, we have an average response time from about 11.2s now compared to 35.7s on the vanilla image :-)
If possible, I would vote for including your patch in a new 3.x build, as it would be more than 3 times faster for the same queries with unchanged client code (at least for the given dataset).
Still, nothing seem to beat build-in reducers, but at least it would not be way slower to run customer reducers on CouchDB 3 vs. CouchDB 2 installations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-904550385
Thanks @nickva and sorry for my delayed response, it took me a while to gather the right data to check if that change would be feasible for our application code.
TLDR: The proposed solution works and is about 130 times faster on the given dataset, that are great results, thanks so much for your input :-)
## View adjustments
I tried to use the build-in reducers for a small set of documents on CouchDB 3.
For the testing out your proposal, I created 7 documents, all shared the same aDefId to return one single reduced output object. I have one document per execution result and an additional document which executed successful to be able to check the computation.
The original view was using this map function:
```javascript
function (doc) {
if(doc.object === "ainst" && doc.result) {
var subscription = doc.subscription || doc.tenant;
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 1,
execTimeSum : 0,
execTimeCount : 0,
execTimeMin : Infinity,
execTimeMax : 0
}
if (doc.result.status === "successful") {
stats.successfulCount = 1;
if (doc.result.executionTime) {
stats.execTimeSum = doc.result.executionTime;
stats.execTimeCount = 1;
stats.execTimeMin = doc.result.executionTime;
stats.execTimeMax = doc.result.executionTime;
}
} else if (doc.result.status === "failed") {
stats.failedCount = 1;
} else if (doc.result.status === "canceled") {
stats.canceledCount = 1;
} else if (doc.result.status === "executing") {
stats.executingCount = 1;
} else if (doc.result.status === "unsuccessful") {
stats.unsuccessfulCount = 1;
} else if (doc.result.status === "unknown") {
stats.unknownCount = 1;
}
emit([subscription, doc.aDefId], stats);
}
}
```
and the following custom reduce function:
```javascript
function (keys, values, rereduce) {
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 0,
execTimeSum : 0,
execTimeCount : 0,
execTimeMin : Infinity,
execTimeMax : 0
}
for (var index in values) {
stats.successfulCount += values[index].successfulCount;
stats.failedCount += values[index].failedCount;
stats.canceledCount += values[index].canceledCount;
stats.executingCount += values[index].executingCount;
stats.unsuccessfulCount += values[index].unsuccessfulCount;
stats.unknownCount += values[index].unknownCount;
stats.totalCount += values[index].totalCount;
if (values[index].execTimeCount) {
stats.execTimeSum += values[index].execTimeSum;
stats.execTimeCount += values[index].execTimeCount;
stats.execTimeMin = Math.min(stats.execTimeMin, values[index].execTimeMin);
stats.execTimeMax = Math.max(stats.execTimeMax, values[index].execTimeMax);
}
}
return stats;
}
```
The original view returned the following as response which I will use to validate the result of the following adjustments:
```bash
$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo/_view/slow?reduce=true&group=true" | jq "."
{
"rows": [
{
"key": [
"xxx",
"135600b7487db5804c1961408a115e0270e5"
],
"value": {
"successfulCount": 2,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 7,
"execTimeSum": 1557,
"execTimeCount": 2,
"execTimeMin": 200,
"execTimeMax": 1357
}
}
]
}
```
As you mentioned, the build-in _sum reducer can be used to count different executions in a single view.
This is my updated demo-build-in-reduce/count view:
```javascript
function (doc) {
if(doc.object === "ainst" && doc.result) {
var subscription = doc.subscription || doc.tenant;
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 1
}
if (doc.result.status === "successful") {
stats.successfulCount = 1;
} else if (doc.result.status === "failed") {
stats.failedCount = 1;
} else if (doc.result.status === "canceled") {
stats.canceledCount = 1;
} else if (doc.result.status === "executing") {
stats.executingCount = 1;
} else if (doc.result.status === "unsuccessful") {
stats.unsuccessfulCount = 1;
} else if (doc.result.status === "unknown") {
stats.unknownCount = 1;
}
emit([subscription, doc.aDefId], stats);
}
}
```
The reduce function is set to _sum on that view.
When I query that view, I get results as expected for the counts:
```bash
$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo-build-in-reduce/_view/count?reduce=true&group=true" | jq "."
{
"rows": [
{
"key": [
"xxx",
"135600b7487db5804c1961408a115e0270e5"
],
"value": {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 2,
"totalCount": 7,
"unknownCount": 1,
"unsuccessfulCount": 1
}
}
]
}
```
The response looks good, but as expected is lacking the execution time stats. This can be done with a second view which I named demo-build-in-reduce/successfulExecutionTime.
The second view returns the following:
```bash
$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true" | jq "."
{
"rows": [
{
"key": [
"xxx",
"135600b7487db5804c1961408a115e0270e5"
],
"value": {
"sum": 1557,
"count": 2,
"min": 200,
"max": 1357,
"sumsqr": 1881449
}
}
]
}
```
This looks good either, I do not care for the sumsqr, but for sum, count, min and max. The count is also included in the demo-build-in-reduce/count view, which can be used in the application theoretically to render a more high level view of data. The successfulExecutionTime view is providing the other data.
So, your proposal works for the given data model, thank you very much. The downside is that to keep our API compatible, we need to submit two requests concurrently instead of using the single original view. Nevertheless, the performance is way more consistent, let me share a few measurements based on the new views.
## Test setup to measure performance improvements of the new implementation
I used the initial setup script to deploy both CouchDBs and to load data.
Afterwards I injected the design doc into the demo database:
```bash
$ curl -X POST \
"http://admin:password@localhost:3003/demo" \
-H "Content-Type: application/json" \
-d '{
"_id": "_design/demo-build-in-reduce",
"views": {
"count": {
"reduce": "_sum",
"map": "function (doc) {\n if(doc.object === \"ainst\" && doc.result) {\n var subscription = doc.subscription || doc.tenant;\n var stats = {\n successfulCount : 0,\n failedCount : 0,\n canceledCount : 0,\n executingCount : 0,\n unsuccessfulCount : 0,\n unknownCount : 0,\n totalCount : 1\n }\n\n if (doc.result.status === \"successful\") {\n stats.successfulCount = 1;\n } else if (doc.result.status === \"failed\") {\n stats.failedCount = 1;\n } else if (doc.result.status === \"canceled\") {\n stats.canceledCount = 1;\n } else if (doc.result.status === \"executing\") {\n stats.e
xecutingCount = 1;\n } else if (doc.result.status === \"unsuccessful\") {\n stats.unsuccessfulCount = 1;\n } else if (doc.result.status === \"unknown\") {\n stats.unknownCount = 1;\n }\n\n emit([subscription, doc.aDefId], stats);\n }\n }"
},
"successfulExecutionTime": {
"reduce": "_stats",
"map": "function (doc) {\n if(doc.object === \"ainst\" && doc.result && doc.result.status === \"successful\") {\n var subscription = doc.subscription || doc.tenant;\n emit([subscription, doc.aDefId], doc.result.executionTime);\n }\n}"
}
},
"language": "javascript"
}'
{"ok":true,"id":"_design/demo-build-in-reduce","rev":"1-e769ee9d7efdeb5f80075e16a1113968"}
```
I used the following Node.js script to query data and combine them within the application.
[reducers-demo.zip](https://github.com/apache/couchdb/files/7038700/reducers-demo.zip)
```javascript
'use strict';
const parallel = require('async').parallel;
const http = require('http');
const user = 'admin';
const password = 'password';
const port = 3003;
const dbname = 'demo';
let newRequest = (async () => {
console.log("==================== New approach ====================");
for( let i=1; i<=10; i++) {
console.log(`===== Iteration ${i} ======`);
console.time(`Fast iteration ${i} took`);
let responseMapCount = {};
let responseMapSuccessfulExecutionTime = {};
await parallel([
(cb) => {
let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo-build-in-reduce/_view/count?reduce=true&group=true`;
console.log(`Requesting: ${currentUrl}`)
let req = http.get(currentUrl, (res) => {
console.log('statusCode:', res.statusCode);
let body = ""
res.on('data', (d) => {
body += d;
});
res.on('end', function () {
// store rows in map to combine them after http requests are completed
body = JSON.parse(body);
body.rows.forEach(currentRow => {
responseMapCount[currentRow.key.join('-')] = currentRow.value;
});
cb()
});
}).on('error', (e) => {
console.error(e);
cb(e);
});
req.end();
},
(cb) => {
let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true`;
console.log(`Requesting: ${currentUrl}`)
let req = http.get(currentUrl, (res) => {
console.log('statusCode:', res.statusCode);
let body = '';
res.on('data', (d) => {
body += d;
});
res.on('end', function () {
// store rows in map to combine them after http requests are completed
body = JSON.parse(body);
body.rows.forEach(currentRow => {
responseMapSuccessfulExecutionTime[currentRow.key.join('-')] = currentRow.value;
});
cb()
});
}).on('error', (e) => {
console.error(e);
cb(e);
});
req.end();
}
]);
console.log(`Combining results`);
// iterate over responseMapCount, as there is always an entry for every ainst result
let lastMergedKey;
Object.keys(responseMapCount).forEach((currentKey) => {
// check if the aInst result had a successful invocation, merge it back in that case
if(responseMapSuccessfulExecutionTime[currentKey]) {
// merge second result property back into responseMapCount
responseMapCount[currentKey].execTimeSum = responseMapSuccessfulExecutionTime[currentKey].sum;
responseMapCount[currentKey].execTimeMin = responseMapSuccessfulExecutionTime[currentKey].min;
responseMapCount[currentKey].execTimeMax = responseMapSuccessfulExecutionTime[currentKey].max;
}
});
console.log(`Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: ${JSON.stringify(responseMapCount['xxx-100200b7487db5804c1961408a115e0270e5'], null, 4)}`)
console.timeEnd(`Fast iteration ${i} took`);
}
});
let oldRequest = (async () => {
console.log("==================== Old approach ====================");
for( let i=1; i<=10; i++) {
console.log(`===== Iteration ${i} ======`);
console.time(`Slow iteration ${i} took`);
let results = await parallel([
(cb) => {
let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo/_view/slow?reduce=true&group=true`;
console.log(`Requesting: ${currentUrl}`)
let req = http.get(currentUrl, (res) => {
console.log('statusCode:', res.statusCode);
let body = ''
res.on('data', (d) => {
body += d;
});
res.on('end', function () {
body = JSON.parse(body);
cb(null, body);
});
}).on('error', (e) => {
console.error(e);
cb(e);
});
req.end();
}
]);
console.log(`Result for aDefId=100200b7487db5804c1961408a115e0270e5:`);
// the view could have been invoked directly with the key, but all results should be retrieved to measure the performance
// only a single result is picked to compare those are equal in both query approaches.
results[0].rows.forEach((currentRow) => {
if(currentRow.key[1] === '100200b7487db5804c1961408a115e0270e5') {
console.log(JSON.stringify(currentRow.value, null, 4));
}
})
console.timeEnd(`Slow iteration ${i} took`);
}
});
(async () => {
await newRequest();
await oldRequest();
})()
```
The output shows an impressive difference, even if the application logic gets more complicated (full output: [couchdb3-test-output.txt](https://github.com/apache/couchdb/files/7038811/couchdb3-test-output.txt)):
```bash
==================== New approach ====================
===== Iteration 1 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 1 took: 270.495ms
...
==================== Old approach ====================
===== Iteration 1 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 1 took: 34.973s
...
```
## Conclusion
On average (10 iterations), I could get combined results of all records within 274 ms compared to 35714 ms, so the workaround on my given dataset is actually ~ 130 times faster.
I repeated the test for CouchDB 2 as well ([couchdb2-test-output](https://github.com/apache/couchdb/files/7038776/output2.txt)) and got an average of 447 ms with the new approach compared to 6917 ms which is still more than 15 times faster. So independent on the question if we should migrate to CouchDB 3 or not, it might still be a useful optimization also for CouchDB 2 setups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva closed issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
nickva closed issue #3517:
URL: https://github.com/apache/couchdb/issues/3517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-911375056
Perfect, thank you very much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-904741147
Excellent write-up, @konrad-ohms. Thank you for getting back to us and sharing the results. 130x speedup is not something we see every day!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-909124414
Okay, I validated your fix, please find the outcome below:
## Creating a custom build
First, I wanted to create a patch build which is as close as possible to what I have tested before, so I used the same OS as in the container image I used initially:
```bash
$ podman run --rm -it docker.io/library/couchdb:3.1.1 cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
```
To get a build env matching the OS and its required libraries, I had a look into <https://github.com/apache/couchdb-ci> and built the following image on an 18.04.5 LTS amd64 VM with Docker version 20.10.8, build 3967b7d:
```bash
$ git clone https://github.com/apache/couchdb-ci.git
Cloning into 'couchdb-ci'...
remote: Enumerating objects: 1742, done.
remote: Counting objects: 100% (87/87), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 1742 (delta 55), reused 56 (delta 29), pack-reused 1655
Receiving objects: 100% (1742/1742), 10.24 MiB | 16.11 MiB/s, done.
Resolving deltas: 100% (998/998), done.
$ cd couchdb-ci/
$ ./build.sh platform debian-buster
...
Successfully built d1c29b763a2c
Successfully tagged apache/couchdbci-debian:buster-erlang-20.3.8.26-1
```
From there I used CouchDB v3.1.1 and applied your patch within the container:
```bash
$ docker run --name couchdb-3.1.1-fix-validation -it apache/couchdbci-debian:buster-erlang-20.3.8.26-1
jenkins@2665e52069d0:/$ cd ~
jenkins@2665e52069d0:~$ whoami
jenkins
jenkins@2665e52069d0:~$ pwd
/home/jenkins
jenkins@2665e52069d0:~$ git clone --depth=1 -b 3.1.1 https://github.com/apache/couchdb.git
Cloning into 'couchdb'...
remote: Enumerating objects: 1207, done.
remote: Counting objects: 100% (1207/1207), done.
remote: Compressing objects: 100% (1099/1099), done.
remote: Total 1207 (delta 245), reused 380 (delta 55), pack-reused 0
Receiving objects: 100% (1207/1207), 1.80 MiB | 6.14 MiB/s, done.
Resolving deltas: 100% (245/245), done.
Note: checking out 'ce596c65d9d7f0bc5d9937bcaf6253b343015690'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
jenkins@2665e52069d0:~$ # Apply Jan's patch
jenkins@2665e52069d0:~$ cat <<'EOF' > jan.patch
> diff --git a/share/server/60/rewrite_fun.js b/share/server/60/rewrite_fun.js
> index 1b27a9d14..9b85f953f 100644
> --- a/share/server/60/rewrite_fun.js
> +++ b/share/server/60/rewrite_fun.js
> @@ -15,30 +15,39 @@
> //
> // https://github.com/dmunch/couch-chakra/blob/master/js/normalizeFunction.js
>
> +
> +const cache = {}
> function rewriteFunInt(fun) {
> - const ast = esprima.parse(fun);
> - let idx = ast.body.length - 1;
> - let decl = {};
> -
> - // Search for the first FunctionDeclaration beginning from the end
> - do {
> - decl = ast.body[idx--];
> - } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> - idx++;
> -
> - // If we have a function declaration without an Id, wrap it
> - // in an ExpressionStatement and change it into
> - // a FuntionExpression
> - if (decl.type == "FunctionDeclaration" && decl.id == null) {
> - decl.type = "FunctionExpression";
> - ast.body[idx] = {
> - type: "ExpressionStatement",
> - expression: decl
> - };
> - }
> + const crc = crc32_str(fun)
> + if (cache[crc]) {
> + return cache[crc];
> + } else {
> + const ast = esprima.parse(fun);
> + let idx = ast.body.length - 1;
> + let decl = {};
> +
> + // Search for the first FunctionDeclaration beginning from the end
> + do {
> + decl = ast.body[idx--];
> + } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> + idx++;
> +
> + // If we have a function declaration without an Id, wrap it
> + // in an ExpressionStatement and change it into
> + // a FuntionExpression
> + if (decl.type == "FunctionDeclaration" && decl.id == null) {
> + decl.type = "FunctionExpression";
> + ast.body[idx] = {
> + type: "ExpressionStatement",
> + expression: decl
> + };
> + }
>
> - // Generate source from the rewritten AST
> - return escodegen.generate(ast);
> + // Generate source from the rewritten AST
> + const gen = escodegen.generate(ast);
> + cache[crc] = gen;
> + return gen;
> + }
> }
>
>
> @@ -53,4 +62,51 @@ function rewriteFuns(funsJSON) {
> return rewriteFunInt(fun);
> });
> return JSON.stringify(results);
> -}
> \ No newline at end of file
> +}
> +
> +// nicked from https://github.com/SheetJS/js-crc32/blob/master/crc32.js
> +
> +function signed_crc_table() {
> +var c = 0, table = new Array(256);
> +
> +for(var n =0; n != 256; ++n){
> +c = n;
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +table[n] = c;
> +}
> +
> +return typeof Int32Array !== 'undefined' ? new Int32Array(table) : table;
> +}
> +
> +var T = signed_crc_table();
> +
> +function crc32_str(str, seed) {
> +var C = seed ^ -1;
> +for(var i = 0, L=str.length, c, d; i < L;) {
> +c = str.charCodeAt(i++);
> +if(c < 0x80) {
> +C = (C>>>8) ^ T[(C ^ c)&0xFF];
> +} else if(c < 0x800) {
> +C = (C>>>8) ^ T[(C ^ (192|((c>>6)&31)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +} else if(c >= 0xD800 && c < 0xE000) {
> +c = (c&1023)+64; d = str.charCodeAt(i++)&1023;
> +C = (C>>>8) ^ T[(C ^ (240|((c>>8)&7)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>2)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((d>>6)&15)|((c&3)<<4)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(d&63)))&0xFF];
> +} else {
> +C = (C>>>8) ^ T[(C ^ (224|((c>>12)&15)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>6)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +}
> +}
> +return C ^ -1;
> +}
> EOF
jenkins@2665e52069d0:~$ cd couchdb/
jenkins@2665e52069d0:~/couchdb$ git status
Not currently on any branch.
nothing to commit, working tree clean
jenkins@2665e52069d0:~/couchdb$ git checkout -b 3.1.1-patch-jan
Switched to a new branch '3.1.1-patch-jan'
jenkins@2665e52069d0:~/couchdb$ git apply ../jan.patch
jenkins@2665e52069d0:~/couchdb$ git status
On branch 3.1.1-patch-jan
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: share/server/60/rewrite_fun.js
no changes added to commit (use "git add" and/or "git commit -a")
```
Afterwards I compiled the source:
```bash
jenkins@2665e52069d0:~/couchdb$ ./configure
...
==> couchdb (update-deps)
Updating config from {git,"https://github.com/apache/couchdb-config.git",
{tag,"2.1.7"}}
Updating b64url from {git,"https://github.com/apache/couchdb-b64url.git",
{tag,"1.0.2"}}
Updating ets_lru from {git,"https://github.com/apache/couchdb-ets-lru.git",
{tag,"1.1.0"}}
Updating khash from {git,"https://github.com/apache/couchdb-khash.git",
{tag,"1.1.0"}}
Updating snappy from {git,"https://github.com/apache/couchdb-snappy.git",
{tag,"CouchDB-1.0.4"}}
Updating docs from {git,"https://github.com/apache/couchdb-documentation",
{tag,"3.1.1-RC2"}}
Updating fauxton from {git,"https://github.com/apache/couchdb-fauxton",
{tag,"v1.2.6"}}
Updating folsom from {git,"https://github.com/apache/couchdb-folsom.git",
{tag,"CouchDB-0.8.3"}}
Updating hyper from {git,"https://github.com/apache/couchdb-hyper.git",
{tag,"CouchDB-2.2.0-6"}}
Updating ibrowse from {git,"https://github.com/apache/couchdb-ibrowse.git",
{tag,"CouchDB-4.0.1-1"}}
Updating jiffy from {git,"https://github.com/apache/couchdb-jiffy.git",
{tag,"CouchDB-1.0.4-1"}}
Updating mochiweb from {git,"https://github.com/apache/couchdb-mochiweb.git",
{tag,"v2.20.0"}}
Updating meck from {git,"https://github.com/apache/couchdb-meck.git",
{tag,"0.8.8"}}
Updating recon from {git,"https://github.com/apache/couchdb-recon.git",
{tag,"2.5.0"}}
Updating proper from {git,"https://github.com/proper-testing/proper",
{tag,"v1.3"}}
Updating bear from {git,"https://github.com/apache/couchdb-bear.git",
"008f48aff819126e281d5ccae80a258bf9bf9c30"}
You have configured Apache CouchDB, time to relax. Relax.
# Compile
jenkins@2665e52069d0:~/couchdb$ make release
...
The HTML pages are in build/html.
sphinx-build -b man -a -W -n -A local=1 -D latex_elements.papersize=a4 -d build/doctree src/ build/man
Running Sphinx v1.8.4
making output directory...
loading pickled environment... done
building [mo]: all of 0 po files
building [man]: all source files
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
writing... apachecouchdb.1 { intro/index intro/overview intro/why intro/consistency intro/curl intro/security intro/tour intro/api replication/index replication/intro replication/replicator replication/conflicts replication/protocol ddocs/index ddocs/ddocs ddocs/views/index ddocs/views/intro ddocs/views/collation ddocs/views/joins ddocs/views/nosql ddocs/views/pagination ddocs/search best-practices/index best-practices/documents best-practices/forms best-practices/iso-date best-practices/jsdevel best-practices/views best-practices/reverse-proxies install/index install/unix install/windows install/mac install/freebsd install/docker install/snap install/kubernetes install/search install/upgrading install/troubleshooting setup/index setup/single-node setup/cluster config/index config/intro config/couchdb config/cluster config/couch-peruser config/http config/auth config/compaction config/indexbuilds config/ioq config/logging config/replicator config/query-servers config/misc config/r
esharding cluster/index cluster/theory cluster/nodes cluster/databases cluster/sharding cluster/purging maintenance/index maintenance/compaction maintenance/performance maintenance/backups fauxton/index fauxton/install experimental api/index api/basics api/server/index api/server/common api/server/authn api/server/configuration api/database/index api/database/common api/database/bulk-api api/database/find api/database/shard api/database/changes api/database/compact api/database/security api/database/misc api/document/index api/document/common api/document/attachments api/ddoc/index api/ddoc/common api/ddoc/views api/ddoc/search api/ddoc/render api/ddoc/rewrites api/partitioned-dbs api/local json-structure query-server/index query-server/protocol query-server/javascript query-server/erlang partitioned-dbs/index whatsnew/index whatsnew/3.1 whatsnew/3.0 whatsnew/2.3 whatsnew/2.2 whatsnew/2.1 whatsnew/2.0 whatsnew/1.7 whatsnew/1.6 whatsnew/1.5 whatsnew/1.4 whatsnew/1.3 whatsnew/1.2 what
snew/1.1 whatsnew/1.0 whatsnew/0.11 whatsnew/0.10 whatsnew/0.9 whatsnew/0.8 cve/index cve/2010-0009 cve/2010-2234 cve/2010-3854 cve/2012-5641 cve/2012-5649 cve/2012-5650 cve/2014-2668 cve/2017-12635 cve/2017-12636 cve/2018-11769 cve/2018-17188 cve/2018-8007 cve/2020-1955 about contributing }
build succeeded.
The manual pages are in build/man.
make[1]: Leaving directory '/home/jenkins/couchdb/src/docs'
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN: 'generate' command does not apply to directory /home/jenkins/couchdb
... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.
jenkins@2665e52069d0:~/couchdb$ echo $?
0
```
The compilation seem to have worked, so I created deb packages based on that. The `make dist` seemed not to like uncommited changes, so I added your patch as a commit:
```bash
jenkins@2665e52069d0:~/couchdb$ git config --global user.email "couchdb-validation@no-reply.com"
jenkins@2665e52069d0:~/couchdb$ git config --global user.name "CouchDB validation"
jenkins@2665e52069d0:~/couchdb$ git commit -am "fix: Apply caching"
[3.1.1-patch-jan 6c2d01a] fix: Apply caching
1 file changed, 79 insertions(+), 23 deletions(-)
jenkins@2665e52069d0:~/couchdb$ make dist
==> config (compile)
==> b64url (compile)
==> ets_lru (compile)
==> khash (compile)
==> snappy (compile)
==> bear (compile)
==> meck (compile)
==> folsom (compile)
==> hyper (compile)
==> ibrowse (compile)
==> jiffy (compile)
==> mochiweb (compile)
==> recon (compile)
==> proper (compile)
==> couch_epi (compile)
==> couch_log (compile)
==> chttpd (compile)
==> couch (compile)
Compiling priv/couch_js/1.8.5/http.c
Compiling priv/couch_js/1.8.5/main.c
Compiling priv/couch_js/1.8.5/utf8.c
Compiling priv/couch_js/1.8.5/util.c
==> couch_event (compile)
==> mem3 (compile)
==> couch_index (compile)
==> couch_mrview (compile)
==> couch_replicator (compile)
==> couch_plugins (compile)
==> couch_pse_tests (compile)
==> couch_stats (compile)
==> couch_peruser (compile)
==> couch_tests (compile)
==> ddoc_cache (compile)
==> dreyfus (compile)
==> fabric (compile)
==> global_changes (compile)
==> ioq (compile)
==> jwtf (compile)
==> ken (compile)
==> mango (compile)
==> rexi (compile)
==> setup (compile)
==> smoosh (compile)
==> rel (compile)
==> couchdb (compile)
COUCHDB_GIT_SHA: 6c2d01a
COUCHDB_VERSION: 3.1.1-6c2d01a
COUCHDB_VERSION_SUFFIX: 6c2d01a
DIRTY:
IN_RC:
IN_RELEASE:
ON_TAG:
REL_TAG: 3.1.1
SUB_VSN:
Building Apache CouchDB 3.1.1-6c2d01a
fatal: ambiguous argument '6c976bd..HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Done: apache-couchdb-3.1.1-6c2d01a.tar.gz
```
I copied that tar back to the host OS:
```bash
$ docker cp couchdb-3.1.1-fix-validation:/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a.tar.gz .
$ git clone https://github.com/apache/couchdb-pkg.git
Cloning into 'couchdb-pkg'...
remote: Enumerating objects: 723, done.
remote: Counting objects: 100% (94/94), done.
remote: Compressing objects: 100% (70/70), done.
remote: Total 723 (delta 40), reused 57 (delta 20), pack-reused 629
Receiving objects: 100% (723/723), 25.31 MiB | 23.49 MiB/s, done.
Resolving deltas: 100% (335/335), done.
$ cd couchdb-pkg/
```
The assumed image name was `couchdbdev/<osname>-<codename>-erlang-<erlang-version>`, so I retagged my created one before running the packaging job:
```bash
$ docker tag apache/couchdbci-debian:buster-erlang-20.3.8.26-1 couchdbdev/debian-buster-erlang-20.3.8.26-1
$ ERLANGVERSION=20.3.8.26-1 ./build.sh couch debian-buster /root/apache-couchdb-3.1.1-6c2d01a.tar.gz
Using apache-couchdb-3.1.1-6c2d01a.tar.gz to build packages...
mkdir -p ../couchdb
cp apache-couchdb-3.1.1-6c2d01a.tar.gz ../couchdb
cd ../couchdb && tar xfz *.tar.gz
cp debian/control.in debian/control
sed -i 's/%SPIDERMONKEY%/libmozjs-60-0/g' debian/control
sed -i 's/%SPIDERMONKEY_DEV%/libmozjs-60-dev/g' debian/control
echo 'SM_VER = 60' > debian/sm_ver.mk
rm -rf /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/debian
cp -R debian /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dch -v 3.1.1-6c2d01a~buster "Automatically generated package from upstream."
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dpkg-buildpackage -b -us -uc
dpkg-buildpackage: info: source package couchdb
dpkg-buildpackage: info: source version 3.1.1-6c2d01a~buster
dpkg-buildpackage: info: source distribution UNRELEASED
dpkg-buildpackage: info: source changed by "CouchDB Developers" <"dev@couchdb.apache.org">
dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
debian/rules clean
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh clean --with=systemd
debian/rules override_dh_auto_clean
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_clean
make -j1 distclean
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (clean)
==> b64url (clean)
==> ets_lru (clean)
==> khash (clean)
==> snappy (clean)
==> bear (clean)
==> meck (clean)
==> folsom (clean)
==> hyper (clean)
==> ibrowse (clean)
==> jiffy (clean)
==> mochiweb (clean)
==> recon (clean)
==> couch_epi (clean)
==> couch_log (clean)
==> chttpd (clean)
==> couch (clean)
==> couch_event (clean)
==> mem3 (clean)
==> couch_index (clean)
==> couch_mrview (clean)
==> couch_replicator (clean)
==> couch_plugins (clean)
==> couch_pse_tests (clean)
==> couch_stats (clean)
==> couch_peruser (clean)
==> couch_tests (clean)
==> ddoc_cache (clean)
==> dreyfus (clean)
==> fabric (clean)
==> global_changes (clean)
==> ioq (clean)
==> jwtf (clean)
==> ken (clean)
==> mango (clean)
==> rexi (clean)
==> setup (clean)
==> smoosh (clean)
==> rel (clean)
==> apache-couchdb-3.1.1-6c2d01a (clean)
make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
mv src/mango/src/mango_cursor_text.erl.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.erl.nocompile': No such file or directory
make[2]: [debian/rules:21: override_dh_auto_clean] Error 1 (ignored)
mv src/mango/src/mango_cursor_text.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.nocompile': No such file or directory
make[2]: [debian/rules:22: override_dh_auto_clean] Error 1 (ignored)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_clean
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules build
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh build --with=systemd
dh_update_autotools_config
debian/rules override_dh_auto_configure
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
./configure --spidermonkey-version 60
==> configuring couchdb in rel/couchdb.config
You have configured Apache CouchDB, time to relax. Relax.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules override_dh_auto_build
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_build -- release
make -j1 release
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (compile)
Compiled src/config_util.erl
Compiled src/config_writer.erl
Compiled src/config_sup.erl
Compiled src/config_notifier.erl
Compiled src/config_listener_mon.erl
Compiled src/config_listener.erl
Compiled src/config_app.erl
Compiled src/config.erl
==> b64url (compile)
Compiled src/b64url.erl
Compiling c_src/b64url.c
==> ets_lru (compile)
Compiled src/ets_lru.erl
==> khash (compile)
Compiled src/khash.erl
Compiling c_src/hash.c
Compiling c_src/khash.c
==> snappy (compile)
Compiled src/snappy.erl
Compiling c_src/snappy_nif.cc
c_src/snappy_nif.cc: In function 'ERL_NIF_TERM snappy_compress(ErlNifEnv*, int, const ERL_NIF_TERM*)':
c_src/snappy_nif.cc:156:28: warning: catching polymorphic type 'class std::bad_alloc' by value [-Wcatch-value=]
} catch(std::bad_alloc e) {
^
Compiling c_src/snappy/snappy-sinksource.cc
Compiling c_src/snappy/snappy-stubs-internal.cc
Compiling c_src/snappy/snappy.cc
c_src/snappy/snappy.cc: In instantiation of 'bool snappy::SnappyScatteredWriter<Allocator>::AppendFromSelf(size_t, size_t) [with Allocator = snappy::SnappySinkAllocator; size_t = long unsigned int]':
c_src/snappy/snappy.cc:779:13: required from 'void snappy::SnappyDecompressor::DecompressAllTags(Writer*) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>]'
c_src/snappy/snappy.cc:865:3: required from 'bool snappy::InternalUncompressAllTags(snappy::SnappyDecompressor*, Writer*, snappy::uint32) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>; snappy::uint32 = unsigned int]'
c_src/snappy/snappy.cc:1549:78: required from here
c_src/snappy/snappy.cc:1401:21: warning: comparison of integer expressions of different signedness: 'size_t' {aka 'long unsigned int'} and 'long int' [-Wsign-compare]
if (offset - 1u < op_ptr_ - op_base_) {
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
==> bear (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/bear/src/bear.erl:28: Warning: export_all flag enabled - all functions will be exported
Compiled src/bear.erl
==> meck (compile)
Compiled src/meck_util.erl
Compiled src/meck_matcher.erl
Compiled src/meck_ret_spec.erl
Compiled src/meck_history.erl
Compiled src/meck_expect.erl
Compiled src/meck_cover.erl
Compiled src/meck_code.erl
Compiled src/meck_code_gen.erl
Compiled src/meck_proc.erl
Compiled src/meck_args_matcher.erl
Compiled src/meck.erl
==> folsom (compile)
Compiled src/folsom_utils.erl
Compiled src/folsom_sup.erl
Compiled src/folsom_sample_slide_uniform.erl
Compiled src/folsom_sample_uniform.erl
Compiled src/folsom_sample_slide_sup.erl
Compiled src/folsom_sample_slide_server.erl
Compiled src/folsom_sample_slide_sorted.erl
Compiled src/folsom_vm_metrics.erl
Compiled src/folsom_sample_none.erl
Compiled src/folsom_sample_slide.erl
Compiled src/folsom_metrics_spiral.erl
Compiled src/folsom_sample_exdec.erl
Compiled src/folsom_sample.erl
Compiled src/folsom_metrics_meter.erl
Compiled src/folsom_metrics_history.erl
Compiled src/folsom_metrics_meter_reader.erl
Compiled src/folsom_metrics_histogram_ets.erl
Compiled src/folsom_metrics_gauge.erl
Compiled src/folsom_metrics_histogram.erl
Compiled src/folsom_metrics_duration.erl
Compiled src/folsom_metrics_counter.erl
Compiled src/folsom_ewma.erl
Compiled src/folsom_meter_timer_server.erl
Compiled src/folsom.erl
Compiled src/folsom_metrics.erl
Compiled src/folsom_ets.erl
==> hyper (compile)
Compiled src/hyper_register.erl
Compiled src/hyper_carray.erl
Compiled src/hyper_gb.erl
Compiled src/hyper_const.erl
Compiled src/hyper_array.erl
Compiled src/hyper_binary_rle.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:245: Warning: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:288: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:333: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
Compiled src/hyper.erl
Compiled src/hyper_binary.erl
Compiling c_src/hyper_carray.c
==> ibrowse (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_lib.erl:372: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_lib.erl
Compiled src/ibrowse_sup.erl
Compiled src/ibrowse_socks5.erl
Compiled src/ibrowse_lb.erl
Compiled src/ibrowse_app.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:83: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:93: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:171: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:182: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_test.erl
Compiled src/ibrowse.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:428: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:1831: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_http_client.erl
==> jiffy (compile)
Compiling c_src/decoder.c
Compiling c_src/encoder.c
Compiling c_src/jiffy.c
Compiling c_src/termstack.c
Compiling c_src/utf8.c
Compiling c_src/util.c
Compiling c_src/doubles.cc
Compiling c_src/objects.cc
Compiling c_src/double-conversion/bignum-dtoa.cc
Compiling c_src/double-conversion/bignum.cc
Compiling c_src/double-conversion/cached-powers.cc
Compiling c_src/double-conversion/diy-fp.cc
Compiling c_src/double-conversion/double-conversion.cc
Compiling c_src/double-conversion/fast-dtoa.cc
Compiling c_src/double-conversion/fixed-dtoa.cc
Compiling c_src/double-conversion/strtod.cc
Compiled src/jiffy_utf8.erl
Compiled src/jiffy.erl
==> mochiweb (compile)
Compiled src/reloader.erl
Compiled src/mochiweb_websocket.erl
Compiled src/mochiweb_socket.erl
Compiled src/mochiweb_util.erl
Compiled src/mochiweb_session.erl
Compiled src/mochiweb_socket_server.erl
Compiled src/mochiweb_response.erl
Compiled src/mochiweb_mime.erl
Compiled src/mochiweb_multipart.erl
Compiled src/mochiweb_io.erl
Compiled src/mochiweb_http.erl
Compiled src/mochiweb_request.erl
Compiled src/mochiweb_echo.erl
Compiled src/mochiweb_headers.erl
Compiled src/mochiweb_cover.erl
Compiled src/mochiweb_cookies.erl
Compiled src/mochiweb_clock.erl
Compiled src/mochiweb_base64url.erl
Compiled src/mochiweb_acceptor.erl
Compiled src/mochiweb.erl
Compiled src/mochiweb_html.erl
Compiled src/mochiutf8.erl
Compiled src/mochitemp.erl
Compiled src/mochinum.erl
Compiled src/mochilogfile2.erl
Compiled src/mochilists.erl
Compiled src/mochijson.erl
Compiled src/mochihex.erl
Compiled src/mochiglobal.erl
Compiled src/mochifmt_std.erl
Compiled src/mochifmt_records.erl
Compiled src/mochijson2.erl
Compiled src/mochifmt.erl
Compiled src/mochiweb_charref.erl
==> recon (compile)
Compiled src/recon_map.erl
Compiled src/recon_rec.erl
Compiled src/recon_trace.erl
Compiled src/recon_alloc.erl
Compiled src/recon_lib.erl
Compiled src/recon.erl
==> proper (compile)
Compiled src/vararg.erl
Compiled src/proper_target.erl
Compiled src/proper_types.erl
Compiled src/proper_unicode.erl
Compiled src/proper_unused_imports_remover.erl
Compiled src/proper_symb.erl
Compiled src/proper_transformer.erl
Compiled src/proper_shrink.erl
Compiled src/proper_statem.erl
Compiled src/proper_sets.erl
Compiled src/proper_queue.erl
Compiled src/proper_sa.erl
Compiled src/proper_prop_remover.erl
Compiled src/proper_orddict.erl
Compiled src/proper_ordsets.erl
Compiled src/proper_gen.erl
Compiled src/proper_typeserver.erl
Compiled src/proper_gb_trees.erl
Compiled src/proper_gb_sets.erl
Compiled src/proper_gen_next.erl
Compiled src/proper_dict.erl
Compiled src/proper_array.erl
Compiled src/proper_fsm.erl
Compiled src/proper_arith.erl
Compiled src/proper.erl
==> couch_epi (compile)
Compiled src/couch_epi_util.erl
Compiled src/couch_epi_sup.erl
Compiled src/couch_epi_plugin.erl
Compiled src/couch_epi_module_keeper.erl
Compiled src/couch_epi_functions.erl
Compiled src/couch_epi_functions_gen.erl
Compiled src/couch_epi_data_gen.erl
Compiled src/couch_epi_data.erl
Compiled src/couch_epi_codegen.erl
Compiled src/couch_epi_app.erl
Compiled src/couch_epi_codechange_monitor.erl
Compiled src/couch_epi.erl
==> couch_log (compile)
Compiled src/couch_log_writer.erl
Compiled src/couch_log_writer_journald.erl
Compiled src/couch_log_writer_stderr.erl
Compiled src/couch_log_writer_syslog.erl
Compiled src/couch_log_writer_file.erl
Compiled src/couch_log_util.erl
Compiled src/couch_log_sup.erl
Compiled src/couch_log_server.erl
Compiled src/couch_log_monitor.erl
Compiled src/couch_log_trunc_io_fmt.erl
Compiled src/couch_log_error_logger_h.erl
Compiled src/couch_log_config_dyn.erl
Compiled src/couch_log_trunc_io.erl
Compiled src/couch_log_app.erl
Compiled src/couch_log_config.erl
Compiled src/couch_log.erl
Compiled src/couch_log_formatter.erl
==> chttpd (compile)
Compiled src/chttpd.erl
Compiled src/chttpd_xframe_options.erl
Compiled src/chttpd_sup.erl
Compiled src/chttpd_view.erl
Compiled src/chttpd_stats.erl
Compiled src/chttpd_test_util.erl
Compiled src/chttpd_prefer_header.erl
Compiled src/chttpd_rewrite.erl
Compiled src/chttpd_show.erl
Compiled src/chttpd_plugin.erl
Compiled src/chttpd_httpd_handlers.erl
Compiled src/chttpd_handlers.erl
Compiled src/chttpd_node.erl
Compiled src/chttpd_misc.erl
Compiled src/chttpd_epi.erl
Compiled src/chttpd_external.erl
Compiled src/chttpd_cors.erl
Compiled src/chttpd_auth_request.erl
Compiled src/chttpd_auth.erl
Compiled src/chttpd_app.erl
Compiled src/chttpd_auth_cache.erl
Compiled src/chttpd_db.erl
==> couch (compile)
Compiled src/couch_httpd.erl
Compiled src/test_request.erl
Compiled src/couch_work_queue.erl
Compiled src/couch_uuids.erl
Compiled src/couch_users_db.erl
Compiled src/couch_totp.erl
Compiled src/test_util.erl
Compiled src/couch_util.erl
Compiled src/couch_task_status.erl
Compiled src/couch_sup.erl
Compiled src/couch_secondary_sup.erl
Compiled src/couch_rand.erl
Compiled src/couch_stream.erl
Compiled src/couch_proc_manager.erl
Compiled src/couch_query_servers.erl
Compiled src/couch_server.erl
Compiled src/couch_primary_sup.erl
Compiled src/couch_partition.erl
Compiled src/couch_passwords.erl
Compiled src/couch_os_process.erl
Compiled src/couch_lru.erl
Compiled src/couch_multidb_changes.erl
Compiled src/couch_io_logger.erl
Compiled src/couch_native_process.erl
Compiled src/couch_httpd_vhost.erl
Compiled src/couch_key_tree.erl
Compiled src/couch_httpd_rewrite.erl
Compiled src/couch_httpd_handlers.erl
Compiled src/couch_httpd_multipart.erl
Compiled src/couch_httpd_misc_handlers.erl
Compiled src/couch_httpd_external.erl
Compiled src/couch_hotp.erl
Compiled src/couch_hash.erl
Compiled src/couch_flags_config.erl
Compiled src/couch_httpd_auth.erl
Compiled src/couch_flags.erl
Compiled src/couch_event_sup.erl
Compiled src/couch_emsort.erl
Compiled src/couch_ejson_size.erl
Compiled src/couch_file.erl
Compiled src/couch_httpd_db.erl
Compiled src/couch_ejson_compare.erl
Compiled src/couch_drv.erl
Compiled src/couch_debug.erl
Compiled src/couch_doc.erl
Compiled src/couch_db_split.erl
Compiled src/couch_db_plugin.erl
Compiled src/couch_db_epi.erl
Compiled src/couch_db_header.erl
Compiled src/couch_db_updater.erl
Compiled src/couch_compress.erl
Compiled src/couch_db_engine.erl
Compiled src/couch_changes.erl
Compiled src/couch_bt_engine_stream.erl
Compiled src/couch_bt_engine_header.erl
Compiled src/couch_db.erl
Compiled src/couch_btree.erl
Compiled src/couch_base32.erl
Compiled src/couch_bt_engine_compactor.erl
Compiled src/couch_auth_cache.erl
Compiled src/couch_att.erl
Compiled src/couch_bt_engine.erl
Compiled src/couch.erl
Compiled src/couch_app.erl
Compiling priv/couch_js/60/http.cpp
Compiling priv/couch_js/60/main.cpp
Compiling priv/couch_js/60/util.cpp
Compiling priv/icu_driver/couch_icu_driver.c
Compiling priv/couch_ejson_compare/couch_ejson_compare.c
==> couch_event (compile)
Compiled src/couch_event_listener.erl
Compiled src/couch_event_sup2.erl
Compiled src/couch_event_os_listener.erl
Compiled src/couch_event_app.erl
Compiled src/couch_event.erl
Compiled src/couch_event_server.erl
Compiled src/couch_event_listener_mfa.erl
==> mem3 (compile)
Compiled src/mem3_sync_nodes.erl
Compiled src/mem3_sync_security.erl
Compiled src/mem3_sync_event.erl
Compiled src/mem3_sync_event_listener.erl
Compiled src/mem3_sup.erl
Compiled src/mem3_sync.erl
Compiled src/mem3_util.erl
Compiled src/mem3_seeds.erl
Compiled src/mem3_reshard_validate.erl
Compiled src/mem3_reshard_sup.erl
Compiled src/mem3_shards.erl
Compiled src/mem3_rpc.erl
Compiled src/mem3_reshard_job_sup.erl
Compiled src/mem3_reshard_store.erl
Compiled src/mem3_reshard_index.erl
Compiled src/mem3_reshard_httpd.erl
Compiled src/mem3_reshard_job.erl
Compiled src/mem3_reshard_api.erl
Compiled src/mem3_reshard_dbdoc.erl
Compiled src/mem3_plugin_couch_db.erl
Compiled src/mem3_nodes.erl
Compiled src/mem3_httpd_handlers.erl
Compiled src/mem3_reshard.erl
Compiled src/mem3_hash.erl
Compiled src/mem3_epi.erl
Compiled src/mem3_httpd.erl
Compiled src/mem3_rep.erl
Compiled src/mem3_app.erl
Compiled src/mem3_cluster.erl
Compiled src/mem3.erl
==> couch_index (compile)
Compiled src/couch_index_sup.erl
Compiled src/couch_index_util.erl
Compiled src/couch_index_plugin_couch_db.erl
Compiled src/couch_index_updater.erl
Compiled src/couch_index_epi.erl
Compiled src/couch_index_plugin.erl
Compiled src/couch_index_app.erl
Compiled src/couch_index_server.erl
Compiled src/couch_index_compactor.erl
Compiled src/couch_index.erl
==> couch_mrview (compile)
Compiled src/couch_mrview_update_notifier.erl
Compiled src/couch_mrview_updater.erl
Compiled src/couch_mrview_test_util.erl
Compiled src/couch_mrview_show.erl
Compiled src/couch_mrview_util.erl
Compiled src/couch_mrview_index.erl
Compiled src/couch_mrview_compactor.erl
Compiled src/couch_mrview_cleanup.erl
Compiled src/couch_mrview_http.erl
Compiled src/couch_mrview.erl
==> couch_replicator (compile)
Compiled src/couch_replicator_httpc.erl
Compiled src/couch_replicator_auth.erl
Compiled src/couch_replicator_utils.erl
Compiled src/couch_replicator_sup.erl
Compiled src/couch_replicator_stats.erl
Compiled src/couch_replicator_scheduler_sup.erl
Compiled src/json_stream_parse.erl
Compiled src/couch_replicator_worker.erl
Compiled src/couch_replicator_rate_limiter_tables.erl
Compiled src/couch_replicator_rate_limiter.erl
Compiled src/couch_replicator_notifier.erl
Compiled src/couch_replicator_job_sup.erl
Compiled src/couch_replicator_scheduler_job.erl
Compiled src/couch_replicator_scheduler.erl
Compiled src/couch_replicator_ids.erl
Compiled src/couch_replicator_httpd_util.erl
Compiled src/couch_replicator_httpd.erl
Compiled src/couch_replicator_httpc_pool.erl
Compiled src/couch_replicator_filters.erl
Compiled src/couch_replicator_fabric_rpc.erl
Compiled src/couch_replicator_doc_processor_worker.erl
Compiled src/couch_replicator_fabric.erl
Compiled src/couch_replicator_db_changes.erl
Compiled src/couch_replicator_connection.erl
Compiled src/couch_replicator_docs.erl
Compiled src/couch_replicator_changes_reader.erl
Compiled src/couch_replicator_doc_processor.erl
Compiled src/couch_replicator_clustering.erl
Compiled src/couch_replicator_app.erl
Compiled src/couch_replicator_auth_noop.erl
Compiled src/couch_replicator_auth_session.erl
Compiled src/couch_replicator.erl
Compiled src/couch_replicator_api_wrap.erl
==> couch_plugins (compile)
Compiled src/couch_plugins_httpd.erl
Compiled src/couch_plugins.erl
==> couch_pse_tests (compile)
Compiled src/cpse_test_ref_counting.erl
Compiled src/cpse_util.erl
Compiled src/cpse_test_read_write_docs.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/couch_pse_tests/src/cpse_test_purge_seqs.erl:14: Warning: export_all flag enabled - all functions will be exported
Compiled src/cpse_test_purge_seqs.erl
Compiled src/cpse_test_purge_replication.erl
Compiled src/cpse_test_purge_docs.erl
Compiled src/cpse_test_open_close_delete.erl
Compiled src/cpse_test_purge_bad_checkpoints.erl
Compiled src/cpse_test_get_set_props.erl
Compiled src/cpse_test_fold_purge_infos.erl
Compiled src/cpse_test_fold_changes.erl
Compiled src/cpse_test_fold_docs.erl
Compiled src/cpse_test_copy_purge_infos.erl
Compiled src/cpse_gather.erl
Compiled src/cpse_test_attachments.erl
Compiled src/cpse_test_compaction.erl
==> couch_stats (compile)
Compiled src/couch_stats_sup.erl
Compiled src/couch_stats_app.erl
Compiled src/couch_stats_process_tracker.erl
Compiled src/couch_stats.erl
Compiled src/couch_stats_httpd.erl
Compiled src/couch_stats_aggregator.erl
==> couch_peruser (compile)
Compiled src/couch_peruser_app.erl
Compiled src/couch_peruser_sup.erl
Compiled src/couch_peruser.erl
==> couch_tests (compile)
Compiled setups/couch_epi_dispatch.erl
Compiled src/couch_tests_combinatorics.erl
Compiled src/couch_tests.erl
==> ddoc_cache (compile)
Compiled src/ddoc_cache_value.erl
Compiled src/ddoc_cache_sup.erl
Compiled src/ddoc_cache_opener.erl
Compiled src/ddoc_cache_entry_validation_funs.erl
Compiled src/ddoc_cache_entry_ddocid_rev.erl
Compiled src/ddoc_cache_entry_ddocid.erl
Compiled src/ddoc_cache_entry_custom.erl
Compiled src/ddoc_cache_app.erl
Compiled src/ddoc_cache_lru.erl
Compiled src/ddoc_cache.erl
Compiled src/ddoc_cache_entry.erl
==> dreyfus (compile)
Compiled src/dreyfus_sup.erl
Compiled src/dreyfus_plugin_couch_db.erl
Compiled src/dreyfus_rpc.erl
Compiled src/dreyfus_index_updater.erl
Compiled src/dreyfus_util.erl
Compiled src/dreyfus_index_manager.erl
Compiled src/dreyfus_httpd_handlers.erl
Compiled src/dreyfus_index.erl
Compiled src/dreyfus_fabric_search.erl
Compiled src/dreyfus_fabric_info.erl
Compiled src/dreyfus_httpd.erl
Compiled src/dreyfus_fabric_group2.erl
Compiled src/dreyfus_fabric_cleanup.erl
Compiled src/dreyfus_fabric_group1.erl
Compiled src/dreyfus_epi.erl
Compiled src/dreyfus_config.erl
Compiled src/dreyfus_app.erl
Compiled src/dreyfus_fabric.erl
Compiled src/dreyfus_bookmark.erl
Compiled src/clouseau_rpc.erl
==> fabric (compile)
Compiled src/fabric_db_update_listener.erl
Compiled src/fabric_view_reduce.erl
Compiled src/fabric_view_map.erl
Compiled src/fabric_view_all_docs.erl
Compiled src/fabric_view_changes.erl
Compiled src/fabric_view.erl
Compiled src/fabric_streams.erl
Compiled src/fabric_util.erl
Compiled src/fabric_rpc.erl
Compiled src/fabric_ring.erl
Compiled src/fabric_group_info.erl
Compiled src/fabric_doc_update.erl
Compiled src/fabric_doc_purge.erl
Compiled src/fabric_doc_missing_revs.erl
Compiled src/fabric_doc_open_revs.erl
Compiled src/fabric_doc_atts.erl
Compiled src/fabric_doc_open.erl
Compiled src/fabric_dict.erl
Compiled src/fabric_doc_attachments.erl
Compiled src/fabric_design_doc_count.erl
Compiled src/fabric_db_partition_info.erl
Compiled src/fabric_db_meta.erl
Compiled src/fabric_db_doc_count.erl
Compiled src/fabric_db_info.erl
Compiled src/fabric_db_delete.erl
Compiled src/fabric_db_create.erl
Compiled src/fabric.erl
==> global_changes (compile)
Compiled src/global_changes_sup.erl
Compiled src/global_changes_util.erl
Compiled src/global_changes_plugin.erl
Compiled src/global_changes_httpd_handlers.erl
Compiled src/global_changes_listener.erl
Compiled src/global_changes_server.erl
Compiled src/global_changes_epi.erl
Compiled src/global_changes_app.erl
Compiled src/global_changes_httpd.erl
==> ioq (compile)
Compiled src/ioq_app.erl
Compiled src/ioq_sup.erl
Compiled src/ioq.erl
==> jwtf (compile)
Compiled src/jwtf_app.erl
Compiled src/jwtf_sup.erl
Compiled src/jwtf.erl
Compiled src/jwtf_keystore.erl
==> ken (compile)
Compiled src/ken_sup.erl
Compiled src/ken_event_handler.erl
Compiled src/ken_app.erl
Compiled src/ken.erl
Compiled src/ken_server.erl
==> mango (compile)
Compiled src/mango_sup.erl
Compiled src/mango_sort.erl
Compiled src/mango_util.erl
Compiled src/mango_selector_text.erl
Compiled src/mango_selector.erl
Compiled src/mango_opts.erl
Compiled src/mango_json.erl
Compiled src/mango_json_bookmark.erl
Compiled src/mango_native_proc.erl
Compiled src/mango_idx_special.erl
Compiled src/mango_idx_text.erl
Compiled src/mango_httpd_handlers.erl
Compiled src/mango_idx_view.erl
Compiled src/mango_fields.erl
Compiled src/mango_execution_stats.erl
Compiled src/mango_idx.erl
Compiled src/mango_epi.erl
Compiled src/mango_httpd.erl
Compiled src/mango_error.erl
Compiled src/mango_doc.erl
Compiled src/mango_cursor_text.erl
Compiled src/mango_cursor_special.erl
Compiled src/mango_cursor.erl
Compiled src/mango_app.erl
Compiled src/mango_crud.erl
Compiled src/mango_cursor_view.erl
==> rexi (compile)
Compiled src/rexi_server_sup.erl
Compiled src/rexi_sup.erl
Compiled src/rexi_utils.erl
Compiled src/rexi_server_mon.erl
Compiled src/rexi_monitor.erl
Compiled src/rexi_app.erl
Compiled src/rexi_server.erl
Compiled src/rexi_buffer.erl
Compiled src/rexi.erl
==> setup (compile)
Compiled src/setup_sup.erl
Compiled src/setup_httpd_handlers.erl
Compiled src/setup_app.erl
Compiled src/setup_epi.erl
Compiled src/setup_httpd.erl
Compiled src/setup.erl
==> smoosh (compile)
Compiled src/smoosh_sup.erl
Compiled src/smoosh_utils.erl
Compiled src/smoosh_priority_queue.erl
Compiled src/smoosh_app.erl
Compiled src/smoosh_channel.erl
Compiled src/smoosh_server.erl
Compiled src/smoosh.erl
==> rel (compile)
==> apache-couchdb-3.1.1-6c2d01a (compile)
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN: 'generate' command does not apply to directory /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.
make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
rm -rf rel/couchdb/var/log
rm -rf rel/couchdb/data
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules binary
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh binary --with=systemd
dh_testroot
dh_prep
dh_installdirs
dh_install
dh_installdocs
dh_installchangelogs
dh_installman
dh_installdebconf
dh_systemd_enable
dh_installinit
dh_systemd_start
dh_installlogrotate
dh_lintian
dh_perl
dh_link
dh_strip_nondeterminism
dh_compress
dh_fixperms
dh_missing
dh_strip
dh_makeshlibs
debian/rules override_dh_shlibdeps
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_shlibdeps -- --ignore-missing-info -xlibmozjs185-1.0
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_ejson_compare.so debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_icu_driver.so were not linked against libicudata.so.63 (they use none of the library's symbols)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_installdeb
dh_gencontrol
dh_md5sums
debian/rules override_dh_builddeb
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_builddeb -- -Zgzip
dpkg-deb: building package 'couchdb-dbgsym' in '../couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb'.
dpkg-deb: building package 'couchdb' in '../couchdb_3.1.1-6c2d01a~buster_amd64.deb'.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dpkg-genbuildinfo --build=binary
dpkg-genchanges --build=binary >../couchdb_3.1.1-6c2d01a~buster_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
dpkg-source --after-build .
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
dpkg-buildpackage: info: binary-only upload (no source included)
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/.. && lintian --profile couchdb couch*.deb || true
warning: the authors of lintian do not recommend running it with root privileges!
W: couchdb: maintainer-script-should-not-use-recursive-chown-or-chmod postinst:207
N: 3 tags overridden (2 errors, 1 info); 2 unused overrides
chmod a+rwx ../couchdb/couchdb*
mkdir -p pkgs/couch/debian-buster && chmod 777 pkgs/couch/debian-buster
cp ../couchdb/couchdb* pkgs/couch/debian-buster
if [ -f debian/control.bak ]; then mv -f debian/control.bak debian/control; fi
if [ -f rpm/SPECS/couchdb.spec.bak ]; then mv -f rpm/SPECS/couchdb.spec.bak rpm/SPECS/couchdb.spec; fi
rm -rf parts prime stage js/build debian/sm_ver.mk
# packages were created
$ ls -lah pkgs/couch/debian-buster/
total 30M
drwxrwxrwx 2 root root 207 Aug 31 01:58 .
drwxrwxrwx 3 root root 45 Aug 31 01:58 ..
-rwxr-xr-x 1 root root 11K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo
-rwxr-xr-x 1 root root 1.5K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.changes
-rwxr-xr-x 1 root root 30M Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.deb
-rwxr-xr-x 1 root root 474K Aug 31 01:58 couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb
```
Next use https://github.com/apache/couchdb-docker/tree/main/3.1.1 to produce the container images, but adjust them to include the custom couchdb version.
```bash
$ git clone https://github.com/apache/couchdb-docker.git
Cloning into 'couchdb-docker'...
remote: Enumerating objects: 1010, done.
remote: Counting objects: 100% (118/118), done.
remote: Compressing objects: 100% (82/82), done.
remote: Total 1010 (delta 64), reused 64 (delta 32), pack-reused 892
Receiving objects: 100% (1010/1010), 248.11 KiB | 1.19 MiB/s, done.
Resolving deltas: 100% (490/490), done.
$ cd ~/couchdb-docker/3.1.1
$ cp -r ~/couchdb-pkg/pkgs/couch/debian-buster/ .
$ ls debian-buster/
couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo couchdb_3.1.1-6c2d01a~buster_amd64.changes couchdb_3.1.1-6c2d01a~buster_amd64.deb couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb
$ git diff
diff --git a/3.1.1/Dockerfile b/3.1.1/Dockerfile
index 75b4b29..59a90d9 100644
--- a/3.1.1/Dockerfile
+++ b/3.1.1/Dockerfile
@@ -60,6 +60,10 @@ RUN . /etc/os-release; \
echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | \
tee /etc/apt/sources.list.d/couchdb.list >/dev/null
+# I know, we should not include deb packages in images, but it should be good enough to validate the fix
+COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+
# https://github.com/apache/couchdb-pkg/blob/master/debian/README.Debian
RUN set -eux; \
apt-get update; \
@@ -67,7 +71,7 @@ RUN set -eux; \
echo "couchdb couchdb/mode select none" | debconf-set-selections; \
# we DO want recommends this time
DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages \
- couchdb="$COUCHDB_VERSION"~buster \
+ /tmp/packages/* \
; \
# Undo symlinks to /var/log and /var/lib
rmdir /var/lib/couchdb /var/log/couchdb; \
docker build -t "couchdb:3.1.1-patch-jan" .
Sending build context to Docker daemon 31.45MB
Step 1/20 : FROM debian:buster-slim
---> 5c3d57f3e47a
Step 2/20 : LABEL maintainer="CouchDB Developers dev@couchdb.apache.org"
---> Using cache
---> 60f37c14a0e1
Step 3/20 : RUN groupadd -g 5984 -r couchdb && useradd -u 5984 -d /opt/couchdb -g couchdb couchdb
---> Using cache
---> 78202346b500
Step 4/20 : RUN set -ex; apt-get update; apt-get install -y --no-install-recommends apt-transport-https ca-certificates dirmngr gnupg ; rm -rf /var/lib/apt/lists/*
---> Using cache
---> fad0f8a736e0
Step 5/20 : RUN set -eux; apt-get update; apt-get install -y --no-install-recommends gosu tini; rm -rf /var/lib/apt/lists/*; gosu nobody true; tini --version
---> Using cache
---> aca1db0792a3
Step 6/20 : ENV GPG_COUCH_KEY 390EF70BB1EA12B2773962950EE62FB37A00258D
---> Using cache
---> d28f3c244c10
Step 7/20 : RUN set -eux; apt-get update; apt-get install -y curl; export GNUPGHOME="$(mktemp -d)"; curl -fL -o keys.asc https://couchdb.apache.org/repo/keys.asc; gpg --batch --import keys.asc; gpg --batch --export "${GPG_COUCH_KEY}" > /usr/share/keyrings/couchdb-archive-keyring.gpg; command -v gpgconf && gpgconf --kill all || :; rm -rf "$GNUPGHOME"; apt-key list; apt purge -y --autoremove curl; rm -rf /var/lib/apt/lists/*
---> Using cache
---> e2949d6cc031
Step 8/20 : ENV COUCHDB_VERSION 3.1.1
---> Using cache
---> baa8089d1bea
Step 9/20 : RUN . /etc/os-release; echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | tee /etc/apt/sources.list.d/couchdb.list >/dev/null
---> Using cache
---> a7933b88e802
Step 10/20 : COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
---> Using cache
---> ded52d34b729
Step 11/20 : COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
---> Using cache
---> eb9b215cb430
Step 12/20 : RUN set -eux; apt-get update; echo "couchdb couchdb/mode select none" | debconf-set-selections; DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/* ; rmdir /var/lib/couchdb /var/log/couchdb; rm /opt/couchdb/data /opt/couchdb/var/log; mkdir -p /opt/couchdb/data /opt/couchdb/var/log; chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log; chmod 777 /opt/couchdb/data /opt/couchdb/var/log; rm /opt/couchdb/etc/default.d/10-filelog.ini; find /opt/couchdb \! \( -user couchdb -group couchdb \) -exec chown -f couchdb:couchdb '{}' +; find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 '{}' +; find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 '{}' +; chmod -f 0777 /opt/couchdb/etc/local.d; rm -rf /var/lib/apt/lists/*;
---> Running in 6f530f886258
+ apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [302 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 https://apache.jfrog.io/artifactory/couchdb-deb buster InRelease [5151 B]
Get:7 https://apache.jfrog.io/artifactory/couchdb-deb buster/main amd64 Packages [3889 B]
Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages [15.2 kB]
Fetched 8473 kB in 3s (3017 kB/s)
Reading package lists...
+ echo couchdb couchdb/mode select none
+ debconf-set-selections
+ DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2 libicu63 libk5crypto3
libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0 libncurses6
libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps psmisc
publicsuffix
Suggested packages:
gpm krb5-doc krb5-user
The following NEW packages will be installed:
couchdb couchdb-dbgsym curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2
libicu63 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0
libncurses6 libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps
psmisc publicsuffix
0 upgraded, 22 newly installed, 0 to remove and 0 not upgraded.
Need to get 13.5 MB/45.0 MB of archives.
After this operation, 104 MB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security buster/updates/main amd64 krb5-locales all 1.17-3+deb10u2 [95.5 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libncurses6 amd64 6.1+20181013-2+deb10u2 [102 kB]
Get:3 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5support0 amd64 1.17-3+deb10u2 [65.7 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 libk5crypto3 amd64 1.17-3+deb10u2 [122 kB]
Get:5 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5-3 amd64 1.17-3+deb10u2 [369 kB]
Get:6 http://security.debian.org/debian-security buster/updates/main amd64 libgssapi-krb5-2 amd64 1.17-3+deb10u2 [158 kB]
Get:7 http://deb.debian.org/debian buster/main amd64 libprocps7 amd64 2:3.3.15-2 [61.7 kB]
Get:8 http://deb.debian.org/debian buster/main amd64 procps amd64 2:3.3.15-2 [259 kB]
Get:9 http://deb.debian.org/debian buster/main amd64 libkeyutils1 amd64 1.6-6 [15.0 kB]
Get:10 http://deb.debian.org/debian buster/main amd64 libnghttp2-14 amd64 1.36.0-2+deb10u1 [85.0 kB]
Get:11 /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb couchdb amd64 3.1.1-6c2d01a~buster [30.9 MB]
Get:12 /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb couchdb-dbgsym amd64 3.1.1-6c2d01a~buster [485 kB]
Get:13 http://deb.debian.org/debian buster/main amd64 libpsl5 amd64 0.20.2-2 [53.7 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2 [60.5 kB]
Get:15 http://deb.debian.org/debian buster/main amd64 libssh2-1 amd64 1.8.0-2.1 [140 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 libcurl4 amd64 7.64.0-4+deb10u2 [332 kB]
Get:17 http://deb.debian.org/debian buster/main amd64 curl amd64 7.64.0-4+deb10u2 [265 kB]
Get:18 http://deb.debian.org/debian buster/main amd64 libicu63 amd64 63.1-6+deb10u1 [8300 kB]
Get:19 http://deb.debian.org/debian buster/main amd64 libmozjs-60-0 amd64 60.2.3-3 [2785 kB]
Get:20 http://deb.debian.org/debian buster/main amd64 libgpm2 amd64 1.20.7-5 [35.1 kB]
Get:21 http://deb.debian.org/debian buster/main amd64 psmisc amd64 23.2-1 [126 kB]
Get:22 http://deb.debian.org/debian buster/main amd64 publicsuffix all 20190415.1030-1 [116 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 13.5 MB in 1s (12.8 MB/s)
Selecting previously unselected package libncurses6:amd64.
(Reading database ... 7118 files and directories currently installed.)
Preparing to unpack .../00-libncurses6_6.1+20181013-2+deb10u2_amd64.deb ...
Unpacking libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Selecting previously unselected package libprocps7:amd64.
Preparing to unpack .../01-libprocps7_2%3a3.3.15-2_amd64.deb ...
Unpacking libprocps7:amd64 (2:3.3.15-2) ...
Selecting previously unselected package procps.
Preparing to unpack .../02-procps_2%3a3.3.15-2_amd64.deb ...
Unpacking procps (2:3.3.15-2) ...
Selecting previously unselected package krb5-locales.
Preparing to unpack .../03-krb5-locales_1.17-3+deb10u2_all.deb ...
Unpacking krb5-locales (1.17-3+deb10u2) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../04-libkeyutils1_1.6-6_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.6-6) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../05-libkrb5support0_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../06-libk5crypto3_1.17-3+deb10u2_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../07-libkrb5-3_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../08-libgssapi-krb5-2_1.17-3+deb10u2_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../09-libnghttp2-14_1.36.0-2+deb10u1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../10-libpsl5_0.20.2-2_amd64.deb ...
Unpacking libpsl5:amd64 (0.20.2-2) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../11-librtmp1_2.4+20151223.gitfa8646d.1-2_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Selecting previously unselected package libssh2-1:amd64.
Preparing to unpack .../12-libssh2-1_1.8.0-2.1_amd64.deb ...
Unpacking libssh2-1:amd64 (1.8.0-2.1) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../13-libcurl4_7.64.0-4+deb10u2_amd64.deb ...
Unpacking libcurl4:amd64 (7.64.0-4+deb10u2) ...
Selecting previously unselected package curl.
Preparing to unpack .../14-curl_7.64.0-4+deb10u2_amd64.deb ...
Unpacking curl (7.64.0-4+deb10u2) ...
Selecting previously unselected package libicu63:amd64.
Preparing to unpack .../15-libicu63_63.1-6+deb10u1_amd64.deb ...
Unpacking libicu63:amd64 (63.1-6+deb10u1) ...
Selecting previously unselected package libmozjs-60-0:amd64.
Preparing to unpack .../16-libmozjs-60-0_60.2.3-3_amd64.deb ...
Unpacking libmozjs-60-0:amd64 (60.2.3-3) ...
Selecting previously unselected package couchdb.
Preparing to unpack .../17-couchdb_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package couchdb-dbgsym.
Preparing to unpack .../18-couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package libgpm2:amd64.
Preparing to unpack .../19-libgpm2_1.20.7-5_amd64.deb ...
Unpacking libgpm2:amd64 (1.20.7-5) ...
Selecting previously unselected package psmisc.
Preparing to unpack .../20-psmisc_23.2-1_amd64.deb ...
Unpacking psmisc (23.2-1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../21-publicsuffix_20190415.1030-1_all.deb ...
Unpacking publicsuffix (20190415.1030-1) ...
Setting up libkeyutils1:amd64 (1.6-6) ...
Setting up libpsl5:amd64 (0.20.2-2) ...
Setting up libgpm2:amd64 (1.20.7-5) ...
Setting up psmisc (23.2-1) ...
Setting up libprocps7:amd64 (2:3.3.15-2) ...
Setting up libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Setting up krb5-locales (1.17-3+deb10u2) ...
Setting up libicu63:amd64 (63.1-6+deb10u1) ...
Setting up libkrb5support0:amd64 (1.17-3+deb10u2) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Setting up libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Setting up libk5crypto3:amd64 (1.17-3+deb10u2) ...
Setting up procps (2:3.3.15-2) ...
update-alternatives: using /usr/bin/w.procps to provide /usr/bin/w (w) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/w.1.gz because associated file /usr/share/man/man1/w.procps.1.gz (of link group w) doesn't exist
Setting up libssh2-1:amd64 (1.8.0-2.1) ...
Setting up libkrb5-3:amd64 (1.17-3+deb10u2) ...
Setting up publicsuffix (20190415.1030-1) ...
Setting up libmozjs-60-0:amd64 (60.2.3-3) ...
Setting up libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Setting up libcurl4:amd64 (7.64.0-4+deb10u2) ...
Setting up curl (7.64.0-4+deb10u2) ...
Setting up couchdb (3.1.1-6c2d01a~buster) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Processing triggers for libc-bin (2.28-10) ...
+ rmdir /var/lib/couchdb /var/log/couchdb
+ rm /opt/couchdb/data /opt/couchdb/var/log
+ mkdir -p /opt/couchdb/data /opt/couchdb/var/log
+ chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log
+ chmod 777 /opt/couchdb/data /opt/couchdb/var/log
+ rm /opt/couchdb/etc/default.d/10-filelog.ini
+ find /opt/couchdb ! ( -user couchdb -group couchdb ) -exec chown -f couchdb:couchdb {} +
+ find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 {} +
+ find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 {} +
+ chmod -f 0777 /opt/couchdb/etc/local.d
+ rm -rf /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_InRelease /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/auxfiles /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/deb.debian.org_debian_dists_buster_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_main_binary-amd64_Packages.lz4
Removing intermediate container 6f530f886258
---> eaea07a7886e
Step 13/20 : COPY --chown=couchdb:couchdb 10-docker-default.ini /opt/couchdb/etc/default.d/
---> f1c9d8e86732
Step 14/20 : COPY --chown=couchdb:couchdb vm.args /opt/couchdb/etc/
---> a984087a4ee3
Step 15/20 : COPY docker-entrypoint.sh /usr/local/bin
---> 66cd3ececce1
Step 16/20 : RUN ln -s usr/local/bin/docker-entrypoint.sh /docker-entrypoint.sh # backwards compat
---> Running in ae10e0e02906
Removing intermediate container ae10e0e02906
---> bf8817264923
Step 17/20 : ENTRYPOINT ["tini", "--", "/docker-entrypoint.sh"]
---> Running in 1485a79c319f
Removing intermediate container 1485a79c319f
---> 737d9f7a9b97
Step 18/20 : VOLUME /opt/couchdb/data
---> Running in 6aa4eafbc054
Removing intermediate container 6aa4eafbc054
---> 3f111d557168
Step 19/20 : EXPOSE 5984 4369 9100
---> Running in 07b6a3204c82
Removing intermediate container 07b6a3204c82
---> c70917bea3fd
Step 20/20 : CMD ["/opt/couchdb/bin/couchdb"]
---> Running in d61719ef6dea
Removing intermediate container d61719ef6dea
---> ca3940266314
Successfully built ca3940266314
Successfully tagged couchdb:3.1.1-patch-jan
```
I pushed the image and pulled it to my workstation again to have the same underlying hardware as in the previous test.
## Results
I adjusted the couchdb-test.sh [couchdb-test-validate-patch-3517.zip](https://github.com/apache/couchdb/files/7083304/couchdb-test-validate-patch-3517.zip) script to use the patched image and compare it against the vanilla CouchDB 3.1.1 image.
```bash
$ ./couchdb-test.sh setup
...
inserting docs cycle 2000/2000
{"ok":true,"id":"a479f05c01bf22c5168ee73617783eb7","rev":"1-f7fac9a3f6ce538d67747271336211c4"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784bb6","rev":"1-7e3d2b01acbc416b60e6e5be050dd790"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784e38","rev":"1-c959b425a724ad6346f646d8a63da6cd"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617785d5b","rev":"1-d8bd0d00d0b04b52796cfe73d9a80760"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617786743","rev":"1-24f6bc4fcfdceae7ff2fae47a68fbf42"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617787715","rev":"1-b0dfc903b7f27dc98130c8d51931f5d4"}
$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"84f2eae60ea04fe192284667cb0e4640","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10
real 0m36.391s
user 0m0.014s
sys 0m0.010s
Round: 2/10
real 0m36.426s
user 0m0.009s
sys 0m0.015s
Round: 3/10
real 0m37.448s
user 0m0.015s
sys 0m0.010s
Round: 4/10
real 0m37.182s
user 0m0.010s
sys 0m0.016s
Round: 5/10
real 0m34.802s
user 0m0.013s
sys 0m0.013s
Round: 6/10
real 0m34.877s
user 0m0.009s
sys 0m0.016s
Round: 7/10
real 0m34.204s
user 0m0.007s
sys 0m0.017s
Round: 8/10
real 0m34.092s
user 0m0.009s
sys 0m0.015s
Round: 9/10
real 0m35.571s
user 0m0.014s
sys 0m0.011s
Round: 10/10
real 0m35.790s
user 0m0.010s
sys 0m0.015s
================ Query CouchDB 3.1.1 patch jan ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"6c2d01a","uuid":"7df6b96ce9f321de839f65a9a6f264f3","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3004
Round: 1/10
real 0m11.194s
user 0m0.012s
sys 0m0.010s
Round: 2/10
real 0m11.224s
user 0m0.014s
sys 0m0.011s
Round: 3/10
real 0m10.966s
user 0m0.012s
sys 0m0.012s
Round: 4/10
real 0m11.212s
user 0m0.013s
sys 0m0.012s
Round: 5/10
real 0m10.696s
user 0m0.016s
sys 0m0.009s
Round: 6/10
real 0m10.859s
user 0m0.013s
sys 0m0.012s
Round: 7/10
real 0m11.342s
user 0m0.012s
sys 0m0.013s
Round: 8/10
real 0m11.276s
user 0m0.011s
sys 0m0.013s
Round: 9/10
real 0m11.357s
user 0m0.015s
sys 0m0.009s
Round: 10/10
real 0m11.542s
user 0m0.014s
sys 0m0.010s
```
That's actually great news, we have an average response time from about 11.2s now compared to 35.7s on the vanilla image :-)
If possible, I would vote for including your patch in a new 3.x build, as it would be more than 3 times faster for the same queries with unchanged client code (at least for the given dataset).
Still, nothing seem to beat build-in reducers, but at least it would not be way slower to run customer reducers on CouchDB 3 vs. CouchDB 2 installations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-874168606
Thanks @nickva for investigating the issue.
I tried to adjust the function from
```
setupDbs() {
COUCHDB_PORT=${1}
waitForDb "${COUCHDB_PORT}"
SYSTEM_DBS=( _users _replicator _global_changes demo )
for CURRENT_DB in "${SYSTEM_DBS[@]}"
do
echo "Creating ${CURRENT_DB}"
curl -X PUT "http://${COUCHDB_USER}:${COUCHDB_PASSWORD}@127.0.0.1:${COUCHDB_PORT}/${CURRENT_DB}"
done
}
```
to
```
setupDbs() {
COUCHDB_PORT=${1}
waitForDb "${COUCHDB_PORT}"
SYSTEM_DBS=( _users _replicator _global_changes demo )
for CURRENT_DB in "${SYSTEM_DBS[@]}"
do
echo "Creating ${CURRENT_DB}"
curl -X PUT "http://${COUCHDB_USER}:${COUCHDB_PASSWORD}@127.0.0.1:${COUCHDB_PORT}/${CURRENT_DB}?q=8"
done
}
```
Unfortunately that did not solve the performance degradation of CouchDB 3.1.1 compared to v2.3.1.
I checked that the databases are using the same number of shards:
CouchDB 2:
```
ohmsk:~$ curl -u "xxx:xxx" http://localhost:3002/demo/
{"db_name":"demo","purge_seq":"0-g1AAAAEzeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgYg9f____lZiQx41R2AqLuPW11SApBMqgcqImTWA4hZeNQlKYDMsifCrAUQs_bjVpfIkCSP3zIHkGXxEDVZAMg5YvY","update_seq":"12001-g1AAAAFLeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjMlMiTJ____PyuJgYH1Cw41SQpAMskeqoyNFZcyB5CyeJhpD3EpSwApq4cpO4hDWR4LkGRoAFJAlfPBStfjVboAonQ_WOl-vEoPQJTex-triNIHEKUQtz7JAgDbh2kI","sizes":{"file":11425440,"external":11923785,"active":11287083},"other":{"data_size":11923785},"doc_del_count":0,"doc_count":12001,"disk_size":11425440,"disk_format_version":7,"data_size":11287083,"compact_running":false,"cluster":{"q":8,"n":1,"w":1,"r":1},"instance_start_time":"0"}
ohmsk:~$ curl -u "xxx:xxx" http://localhost:3003/demo/
```
CouchDB 3:
```
{"db_name":"demo","purge_seq":"0-g1AAAAEzeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgdA6j8QZCUy4FV3AKLuPiF1CyDq9hNS1wBRNx-3uqQEIJlUj9dtSQ4gNfH41SiA1NjjVZPIkCQPUZAFAN6AYvY","update_seq":"12001-g1AAAAFLeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgdA6j8QZCUxMLDew6v0AETpfbDSg3iVLoAo3Q9W-gKv0gaI0vlgpa9wKE1KAJJJ9TB33sGlzAGkLB6mDKdpCiBl9jBll3AoS2RIkoepuZEFAAsmagc","sizes":{"file":14468808,"external":9785687,"active":11343864},"props":{},"doc_del_count":0,"doc_count":12001,"disk_format_version":8,"compact_running":false,"cluster":{"q":8,"n":1,"w":1,"r":1},"instance_start_time":"0"}
```
Both state:
```
"cluster":{"q":8,"n":1,"w":1,"r":1}
```
After injecting the data I waited multiple minutes before running the queries:
```
ohmsk:~/Downloads/couchdb$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"5584fb6a7a94ff8ab44bbee93823352d","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3002
Round: 1/10
real 0m8.534s
user 0m0.017s
sys 0m0.013s
Round: 2/10
real 0m5.050s
user 0m0.012s
sys 0m0.010s
Round: 3/10
real 0m4.677s
user 0m0.010s
sys 0m0.011s
Round: 4/10
real 0m4.820s
user 0m0.013s
sys 0m0.009s
Round: 5/10
real 0m4.798s
user 0m0.011s
sys 0m0.011s
Round: 6/10
real 0m5.093s
user 0m0.013s
sys 0m0.009s
Round: 7/10
real 0m5.285s
user 0m0.015s
sys 0m0.007s
Round: 8/10
real 0m5.627s
user 0m0.010s
sys 0m0.014s
Round: 9/10
real 0m5.459s
user 0m0.013s
sys 0m0.010s
Round: 10/10
real 0m5.589s
user 0m0.009s
sys 0m0.015s
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"6445c39f1422b264fbf7837dbb055bb9","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10
real 0m53.993s
user 0m0.014s
sys 0m0.011s
Round: 2/10
real 0m53.707s
user 0m0.016s
sys 0m0.009s
Round: 3/10
real 0m56.130s
user 0m0.011s
sys 0m0.016s
Round: 4/10
real 0m56.537s
user 0m0.012s
sys 0m0.014s
Round: 5/10
real 0m55.933s
user 0m0.012s
sys 0m0.013s
Round: 6/10
real 0m52.887s
user 0m0.013s
sys 0m0.014s
Round: 7/10
real 1m1.758s
user 0m0.012s
sys 0m0.012s
Round: 8/10
real 0m52.182s
user 0m0.010s
sys 0m0.018s
Round: 9/10
real 0m53.072s
user 0m0.011s
sys 0m0.016s
Round: 10/10
real 0m53.720s
user 0m0.014s
sys 0m0.013s
```
I am not sure if the default log level does not indicate the auto-builder and auto-compactor actions, but at least the container logs are not reflecting such actions:
```
$ podman logs -f couchdb3
...
[notice] 2021-07-05T12:34:42.581798Z nonode@nohost <0.19894.3> 14ea633501 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 19
[notice] 2021-07-05T12:34:42.618414Z nonode@nohost <0.19902.3> 96877188fc 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 22
[notice] 2021-07-05T12:34:42.650066Z nonode@nohost <0.19910.3> c2fe735612 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 20
[info] 2021-07-05T12:35:10.329291Z nonode@nohost <0.3652.0> -------- Starting index update for db: shards/40000000-5fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:11.749276Z nonode@nohost <0.3652.0> -------- Index update finished for db: shards/40000000-5fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:11.936249Z nonode@nohost <0.3778.0> -------- Starting index update for db: shards/00000000-1fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:13.008010Z nonode@nohost <0.3778.0> -------- Index update finished for db: shards/00000000-1fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:13.291704Z nonode@nohost <0.3817.0> -------- Starting index update for db: shards/e0000000-ffffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:14.564826Z nonode@nohost <0.3817.0> -------- Index update finished for db: shards/e0000000-ffffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.314002Z nonode@nohost <0.3907.0> -------- Starting index update for db: shards/20000000-3fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.373015Z nonode@nohost <0.3975.0> -------- Starting index update for db: shards/60000000-7fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.510076Z nonode@nohost <0.3621.0> -------- Starting index update for db: shards/c0000000-dfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.010943Z nonode@nohost <0.4169.0> -------- Starting index update for db: shards/a0000000-bfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.654203Z nonode@nohost <0.3975.0> -------- Index update finished for db: shards/60000000-7fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.677526Z nonode@nohost <0.3907.0> -------- Index update finished for db: shards/20000000-3fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.788062Z nonode@nohost <0.3621.0> -------- Index update finished for db: shards/c0000000-dfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:17.306988Z nonode@nohost <0.4169.0> -------- Index update finished for db: shards/a0000000-bfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:18.494925Z nonode@nohost <0.3686.0> -------- Starting index update for db: shards/80000000-9fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:19.388880Z nonode@nohost <0.3686.0> -------- Index update finished for db: shards/80000000-9fffffff/demo.1625487748 idx: _design/demo
[notice] 2021-07-05T14:32:23.887866Z nonode@nohost <0.13346.5> 61a2177aae 127.0.0.1:3003 10.0.2.100 admin GET / 200 ok 9
[notice] 2021-07-05T14:33:17.881591Z nonode@nohost <0.13347.5> 39c531c67e 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53986
[notice] 2021-07-05T14:34:11.588538Z nonode@nohost <0.13889.5> d3cc5e0fa8 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53700
[notice] 2021-07-05T14:35:07.718592Z nonode@nohost <0.14346.5> 05b6354796 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 56122
[notice] 2021-07-05T14:36:04.255503Z nonode@nohost <0.14842.5> 1535baf654 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 56529
[notice] 2021-07-05T14:37:00.187963Z nonode@nohost <0.15334.5> 00a9c90ecf 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 55926
[notice] 2021-07-05T14:37:53.074959Z nonode@nohost <0.15818.5> e48cbf750a 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 52880
[notice] 2021-07-05T14:38:54.833050Z nonode@nohost <0.16272.5> 07b2646c16 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 61751
[notice] 2021-07-05T14:39:47.015319Z nonode@nohost <0.16808.5> a26c18d19d 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 52175
[notice] 2021-07-05T14:40:40.086929Z nonode@nohost <0.17255.5> 5267817a12 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53065
[notice] 2021-07-05T14:41:33.807566Z nonode@nohost <0.17723.5> 21e5c4941c 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53713
```
Compared to couchdb2:
```
$ podman logs couchdb2
...
[notice] 2021-07-05T12:27:09.139663Z nonode@nohost <0.23495.2> 714b3e7a37 127.0.0.1:3002 10.0.2.100 admin POST /demo 201 ok 30
[notice] 2021-07-05T12:27:09.165180Z nonode@nohost <0.23502.2> 9a12b27d94 127.0.0.1:3002 10.0.2.100 admin POST /demo 201 ok 18
[info] 2021-07-05T13:22:11.293974Z nonode@nohost <0.12126.4> -------- Starting compaction for db "shards/20000000-3fffffff/demo.1625487744" at 1541
[info] 2021-07-05T13:22:32.858211Z nonode@nohost <0.12139.4> -------- Starting compaction for db "shards/00000000-1fffffff/demo.1625487744" at 1524
[info] 2021-07-05T13:22:54.395354Z nonode@nohost <0.12142.4> -------- Starting compaction for db "shards/40000000-5fffffff/demo.1625487744" at 1505
[info] 2021-07-05T13:23:15.835989Z nonode@nohost <0.12145.4> -------- Starting compaction for db "shards/60000000-7fffffff/demo.1625487744" at 1473
[info] 2021-07-05T13:23:37.277484Z nonode@nohost <0.12151.4> -------- Starting compaction for db "shards/a0000000-bfffffff/demo.1625487744" at 1471
[info] 2021-07-05T13:23:58.739812Z nonode@nohost <0.12157.4> -------- Starting compaction for db "shards/e0000000-ffffffff/demo.1625487744" at 1508
[info] 2021-07-05T13:24:20.179296Z nonode@nohost <0.12148.4> -------- Starting compaction for db "shards/80000000-9fffffff/demo.1625487744" at 1455
[info] 2021-07-05T13:24:41.714250Z nonode@nohost <0.12154.4> -------- Starting compaction for db "shards/c0000000-dfffffff/demo.1625487744" at 1524
[info] 2021-07-05T13:24:54.231782Z nonode@nohost <0.12450.4> -------- Starting compaction for db "shards/c0000000-dfffffff/_global_changes.1625487744" at 554
[notice] 2021-07-05T14:31:28.936974Z nonode@nohost <0.15912.6> 312602f8c0 127.0.0.1:3002 10.0.2.100 admin GET / 200 ok 2
[info] 2021-07-05T14:31:28.989213Z nonode@nohost <0.15983.6> -------- Starting index update for db: shards/00000000-1fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.989257Z nonode@nohost <0.15984.6> -------- Starting index update for db: shards/20000000-3fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990506Z nonode@nohost <0.16010.6> -------- Starting index update for db: shards/40000000-5fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990507Z nonode@nohost <0.16011.6> -------- Starting index update for db: shards/a0000000-bfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990573Z nonode@nohost <0.16012.6> -------- Starting index update for db: shards/60000000-7fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990650Z nonode@nohost <0.16016.6> -------- Starting index update for db: shards/80000000-9fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990715Z nonode@nohost <0.16020.6> -------- Starting index update for db: shards/c0000000-dfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.991237Z nonode@nohost <0.16025.6> -------- Starting index update for db: shards/e0000000-ffffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.578442Z nonode@nohost <0.16012.6> -------- Index update finished for db: shards/60000000-7fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.592533Z nonode@nohost <0.16025.6> -------- Index update finished for db: shards/e0000000-ffffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.624614Z nonode@nohost <0.16011.6> -------- Index update finished for db: shards/a0000000-bfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.630679Z nonode@nohost <0.16010.6> -------- Index update finished for db: shards/40000000-5fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.641088Z nonode@nohost <0.16016.6> -------- Index update finished for db: shards/80000000-9fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.682452Z nonode@nohost <0.15984.6> -------- Index update finished for db: shards/20000000-3fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.729299Z nonode@nohost <0.16020.6> -------- Index update finished for db: shards/c0000000-dfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.746659Z nonode@nohost <0.15983.6> -------- Index update finished for db: shards/00000000-1fffffff/demo.1625487744 idx: _design/demo
[notice] 2021-07-05T14:31:37.472657Z nonode@nohost <0.15913.6> 3b98ddbf30 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 8516
[notice] 2021-07-05T14:31:42.522912Z nonode@nohost <0.16231.6> 139c6ac71b 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5042
[notice] 2021-07-05T14:31:47.199986Z nonode@nohost <0.16315.6> 63625be9aa 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4671
[notice] 2021-07-05T14:31:52.020073Z nonode@nohost <0.16417.6> 06140e6373 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4813
[notice] 2021-07-05T14:31:56.818616Z nonode@nohost <0.16495.6> 73aef1519b 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4791
[notice] 2021-07-05T14:32:01.911785Z nonode@nohost <0.16590.6> e7371ec8db 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5086
[notice] 2021-07-05T14:32:07.196507Z nonode@nohost <0.16725.6> 19201fdd74 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5278
[notice] 2021-07-05T14:32:12.823661Z nonode@nohost <0.16833.6> 1cfc7a7f8a 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5618
[notice] 2021-07-05T14:32:18.282645Z nonode@nohost <0.16924.6> a70a1462dc 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5453
[notice] 2021-07-05T14:32:23.871410Z nonode@nohost <0.17021.6> 8364bc2506 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5581
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-906670629
Thank you @janl for your code change. I will try to compare the performance of your change within the test suite above, I try to fit that in next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-890372099
I confirmed that the JS engine is implicated. I ran 2.3.1 and 3.x latest and 2.3.1 on Ubuntu 20.04. For 3.x I installed the libmozjs185 couchdb-build package for Ubuntu 18.04 and ran with q=8 to minimize difference and got faster performance number than 2.3.1 even.
Packages I downloaded and installed manually:
```
couch-libmozjs185-1.0_1.8.5-1.0.0+couch-2_bionic_amd64.deb
couch-libmozjs185-dev_1.8.5-1.0.0+couch-2_bionic_amd64.deb
libffi6_3.2.1-8_amd64.deb
libffi-dev_3.2.1-8_amd64.deb
```
Configure 3.x to build with version 1.8.5
```
./configure --spidermonkey-version 1.8.5 --dev
```
Test was modified to not set up docker container.
```
./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"fake_uuid_for_dev","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15994
Round: 1/10
real 0m13.022s
user 0m0.023s
sys 0m0.023s
Round: 2/10
real 0m7.234s
user 0m0.020s
sys 0m0.008s
Round: 3/10
real 0m6.715s
user 0m0.026s
sys 0m0.000s
Round: 4/10
real 0m7.021s
user 0m0.018s
sys 0m0.009s
Round: 5/10
real 0m7.159s
user 0m0.005s
sys 0m0.021s
Round: 6/10
real 0m7.318s
user 0m0.014s
sys 0m0.014s
Round: 7/10
real 0m7.464s
user 0m0.007s
sys 0m0.021s
Round: 8/10
real 0m7.175s
user 0m0.016s
sys 0m0.011s
Round: 9/10
real 0m7.096s
user 0m0.004s
sys 0m0.023s
Round: 10/10
real 0m7.235s
user 0m0.006s
sys 0m0.022s
```
```
q=8
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1-ba63878","git_sha":"ba63878","uuid":"fake_uuid_for_dev","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15984
Round: 1/10
real 0m5.979s
user 0m0.017s
sys 0m0.010s
Round: 2/10
real 0m6.757s
user 0m0.013s
sys 0m0.013s
Round: 3/10
real 0m7.112s
user 0m0.012s
sys 0m0.016s
Round: 4/10
real 0m7.359s
user 0m0.017s
sys 0m0.011s
Round: 5/10
real 0m7.210s
user 0m0.017s
sys 0m0.013s
Round: 6/10
real 0m7.233s
user 0m0.008s
sys 0m0.021s
Round: 7/10
real 0m6.284s
user 0m0.028s
sys 0m0.000s
Round: 8/10
real 0m7.101s
user 0m0.020s
sys 0m0.008s
Round: 9/10
real 0m6.943s
user 0m0.022s
sys 0m0.006s
Round: 10/10
real 0m6.736s
user 0m0.016s
sys 0m0.011s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-873022044
3.x decreased the default sharding factor (Q) from 8 to 2. I noticed your test script creates dbs with a default sharding factor 2 and before it was 8, and that might impact indexing times. Try explicitly passing `?q=8` on db creation.
Also 3.x has an index auto-builder and auto-compactor and those might be working the background while the test run and consume some resources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-909124414
Okay, I validated your fix, please find the outcome below:
## Creating a custom build
First, I wanted to create a patch build which is as close as possible to what I have tested before, so I used the same OS as in the container image I used initially:
```bash
$ podman run --rm -it docker.io/library/couchdb:3.1.1 cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
```
To get a build env matching the OS and its required libraries, I had a look into <https://github.com/apache/couchdb-ci> and built the following image on an 18.04.5 LTS amd64 VM with Docker version 20.10.8, build 3967b7d:
```bash
$ git clone https://github.com/apache/couchdb-ci.git
Cloning into 'couchdb-ci'...
remote: Enumerating objects: 1742, done.
remote: Counting objects: 100% (87/87), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 1742 (delta 55), reused 56 (delta 29), pack-reused 1655
Receiving objects: 100% (1742/1742), 10.24 MiB | 16.11 MiB/s, done.
Resolving deltas: 100% (998/998), done.
$ cd couchdb-ci/
$ ./build.sh platform debian-buster
...
Successfully built d1c29b763a2c
Successfully tagged apache/couchdbci-debian:buster-erlang-20.3.8.26-1
```
From there I used CouchDB v3.1.1 and applied your patch within the container:
```bash
$ docker run --name couchdb-3.1.1-fix-validation -it apache/couchdbci-debian:buster-erlang-20.3.8.26-1
jenkins@2665e52069d0:/$ cd ~
jenkins@2665e52069d0:~$ whoami
jenkins
jenkins@2665e52069d0:~$ pwd
/home/jenkins
jenkins@2665e52069d0:~$ git clone --depth=1 -b 3.1.1 https://github.com/apache/couchdb.git
Cloning into 'couchdb'...
remote: Enumerating objects: 1207, done.
remote: Counting objects: 100% (1207/1207), done.
remote: Compressing objects: 100% (1099/1099), done.
remote: Total 1207 (delta 245), reused 380 (delta 55), pack-reused 0
Receiving objects: 100% (1207/1207), 1.80 MiB | 6.14 MiB/s, done.
Resolving deltas: 100% (245/245), done.
Note: checking out 'ce596c65d9d7f0bc5d9937bcaf6253b343015690'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
jenkins@2665e52069d0:~$ # Apply Jan's patch
jenkins@2665e52069d0:~$ cat <<'EOF' > jan.patch
> diff --git a/share/server/60/rewrite_fun.js b/share/server/60/rewrite_fun.js
> index 1b27a9d14..9b85f953f 100644
> --- a/share/server/60/rewrite_fun.js
> +++ b/share/server/60/rewrite_fun.js
> @@ -15,30 +15,39 @@
> //
> // https://github.com/dmunch/couch-chakra/blob/master/js/normalizeFunction.js
>
> +
> +const cache = {}
> function rewriteFunInt(fun) {
> - const ast = esprima.parse(fun);
> - let idx = ast.body.length - 1;
> - let decl = {};
> -
> - // Search for the first FunctionDeclaration beginning from the end
> - do {
> - decl = ast.body[idx--];
> - } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> - idx++;
> -
> - // If we have a function declaration without an Id, wrap it
> - // in an ExpressionStatement and change it into
> - // a FuntionExpression
> - if (decl.type == "FunctionDeclaration" && decl.id == null) {
> - decl.type = "FunctionExpression";
> - ast.body[idx] = {
> - type: "ExpressionStatement",
> - expression: decl
> - };
> - }
> + const crc = crc32_str(fun)
> + if (cache[crc]) {
> + return cache[crc];
> + } else {
> + const ast = esprima.parse(fun);
> + let idx = ast.body.length - 1;
> + let decl = {};
> +
> + // Search for the first FunctionDeclaration beginning from the end
> + do {
> + decl = ast.body[idx--];
> + } while (idx >= 0 && decl.type !== "FunctionDeclaration");
> + idx++;
> +
> + // If we have a function declaration without an Id, wrap it
> + // in an ExpressionStatement and change it into
> + // a FuntionExpression
> + if (decl.type == "FunctionDeclaration" && decl.id == null) {
> + decl.type = "FunctionExpression";
> + ast.body[idx] = {
> + type: "ExpressionStatement",
> + expression: decl
> + };
> + }
>
> - // Generate source from the rewritten AST
> - return escodegen.generate(ast);
> + // Generate source from the rewritten AST
> + const gen = escodegen.generate(ast);
> + cache[crc] = gen;
> + return gen;
> + }
> }
>
>
> @@ -53,4 +62,51 @@ function rewriteFuns(funsJSON) {
> return rewriteFunInt(fun);
> });
> return JSON.stringify(results);
> -}
> \ No newline at end of file
> +}
> +
> +// nicked from https://github.com/SheetJS/js-crc32/blob/master/crc32.js
> +
> +function signed_crc_table() {
> +var c = 0, table = new Array(256);
> +
> +for(var n =0; n != 256; ++n){
> +c = n;
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +c = ((c&1) ? (-306674912 ^ (c >>> 1)) : (c >>> 1));
> +table[n] = c;
> +}
> +
> +return typeof Int32Array !== 'undefined' ? new Int32Array(table) : table;
> +}
> +
> +var T = signed_crc_table();
> +
> +function crc32_str(str, seed) {
> +var C = seed ^ -1;
> +for(var i = 0, L=str.length, c, d; i < L;) {
> +c = str.charCodeAt(i++);
> +if(c < 0x80) {
> +C = (C>>>8) ^ T[(C ^ c)&0xFF];
> +} else if(c < 0x800) {
> +C = (C>>>8) ^ T[(C ^ (192|((c>>6)&31)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +} else if(c >= 0xD800 && c < 0xE000) {
> +c = (c&1023)+64; d = str.charCodeAt(i++)&1023;
> +C = (C>>>8) ^ T[(C ^ (240|((c>>8)&7)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>2)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((d>>6)&15)|((c&3)<<4)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(d&63)))&0xFF];
> +} else {
> +C = (C>>>8) ^ T[(C ^ (224|((c>>12)&15)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|((c>>6)&63)))&0xFF];
> +C = (C>>>8) ^ T[(C ^ (128|(c&63)))&0xFF];
> +}
> +}
> +return C ^ -1;
> +}
> EOF
jenkins@2665e52069d0:~$ cd couchdb/
jenkins@2665e52069d0:~/couchdb$ git status
Not currently on any branch.
nothing to commit, working tree clean
jenkins@2665e52069d0:~/couchdb$ git checkout -b 3.1.1-patch-jan
Switched to a new branch '3.1.1-patch-jan'
jenkins@2665e52069d0:~/couchdb$ git apply ../jan.patch
jenkins@2665e52069d0:~/couchdb$ git status
On branch 3.1.1-patch-jan
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: share/server/60/rewrite_fun.js
no changes added to commit (use "git add" and/or "git commit -a")
```
Afterwards I compiled the source:
```bash
jenkins@2665e52069d0:~/couchdb$ ./configure
...
==> couchdb (update-deps)
Updating config from {git,"https://github.com/apache/couchdb-config.git",
{tag,"2.1.7"}}
Updating b64url from {git,"https://github.com/apache/couchdb-b64url.git",
{tag,"1.0.2"}}
Updating ets_lru from {git,"https://github.com/apache/couchdb-ets-lru.git",
{tag,"1.1.0"}}
Updating khash from {git,"https://github.com/apache/couchdb-khash.git",
{tag,"1.1.0"}}
Updating snappy from {git,"https://github.com/apache/couchdb-snappy.git",
{tag,"CouchDB-1.0.4"}}
Updating docs from {git,"https://github.com/apache/couchdb-documentation",
{tag,"3.1.1-RC2"}}
Updating fauxton from {git,"https://github.com/apache/couchdb-fauxton",
{tag,"v1.2.6"}}
Updating folsom from {git,"https://github.com/apache/couchdb-folsom.git",
{tag,"CouchDB-0.8.3"}}
Updating hyper from {git,"https://github.com/apache/couchdb-hyper.git",
{tag,"CouchDB-2.2.0-6"}}
Updating ibrowse from {git,"https://github.com/apache/couchdb-ibrowse.git",
{tag,"CouchDB-4.0.1-1"}}
Updating jiffy from {git,"https://github.com/apache/couchdb-jiffy.git",
{tag,"CouchDB-1.0.4-1"}}
Updating mochiweb from {git,"https://github.com/apache/couchdb-mochiweb.git",
{tag,"v2.20.0"}}
Updating meck from {git,"https://github.com/apache/couchdb-meck.git",
{tag,"0.8.8"}}
Updating recon from {git,"https://github.com/apache/couchdb-recon.git",
{tag,"2.5.0"}}
Updating proper from {git,"https://github.com/proper-testing/proper",
{tag,"v1.3"}}
Updating bear from {git,"https://github.com/apache/couchdb-bear.git",
"008f48aff819126e281d5ccae80a258bf9bf9c30"}
You have configured Apache CouchDB, time to relax. Relax.
# Compile
jenkins@2665e52069d0:~/couchdb$ make release
...
The HTML pages are in build/html.
sphinx-build -b man -a -W -n -A local=1 -D latex_elements.papersize=a4 -d build/doctree src/ build/man
Running Sphinx v1.8.4
making output directory...
loading pickled environment... done
building [mo]: all of 0 po files
building [man]: all source files
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
writing... apachecouchdb.1 { intro/index intro/overview intro/why intro/consistency intro/curl intro/security intro/tour intro/api replication/index replication/intro replication/replicator replication/conflicts replication/protocol ddocs/index ddocs/ddocs ddocs/views/index ddocs/views/intro ddocs/views/collation ddocs/views/joins ddocs/views/nosql ddocs/views/pagination ddocs/search best-practices/index best-practices/documents best-practices/forms best-practices/iso-date best-practices/jsdevel best-practices/views best-practices/reverse-proxies install/index install/unix install/windows install/mac install/freebsd install/docker install/snap install/kubernetes install/search install/upgrading install/troubleshooting setup/index setup/single-node setup/cluster config/index config/intro config/couchdb config/cluster config/couch-peruser config/http config/auth config/compaction config/indexbuilds config/ioq config/logging config/replicator config/query-servers config/misc config/r
esharding cluster/index cluster/theory cluster/nodes cluster/databases cluster/sharding cluster/purging maintenance/index maintenance/compaction maintenance/performance maintenance/backups fauxton/index fauxton/install experimental api/index api/basics api/server/index api/server/common api/server/authn api/server/configuration api/database/index api/database/common api/database/bulk-api api/database/find api/database/shard api/database/changes api/database/compact api/database/security api/database/misc api/document/index api/document/common api/document/attachments api/ddoc/index api/ddoc/common api/ddoc/views api/ddoc/search api/ddoc/render api/ddoc/rewrites api/partitioned-dbs api/local json-structure query-server/index query-server/protocol query-server/javascript query-server/erlang partitioned-dbs/index whatsnew/index whatsnew/3.1 whatsnew/3.0 whatsnew/2.3 whatsnew/2.2 whatsnew/2.1 whatsnew/2.0 whatsnew/1.7 whatsnew/1.6 whatsnew/1.5 whatsnew/1.4 whatsnew/1.3 whatsnew/1.2 what
snew/1.1 whatsnew/1.0 whatsnew/0.11 whatsnew/0.10 whatsnew/0.9 whatsnew/0.8 cve/index cve/2010-0009 cve/2010-2234 cve/2010-3854 cve/2012-5641 cve/2012-5649 cve/2012-5650 cve/2014-2668 cve/2017-12635 cve/2017-12636 cve/2018-11769 cve/2018-17188 cve/2018-8007 cve/2020-1955 about contributing }
build succeeded.
The manual pages are in build/man.
make[1]: Leaving directory '/home/jenkins/couchdb/src/docs'
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN: 'generate' command does not apply to directory /home/jenkins/couchdb
... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.
jenkins@2665e52069d0:~/couchdb$ echo $?
0
```
The compilation seem to have worked, so I created deb packages based on that. The `make dist` seemed not to like uncommited changes, so I added your patch as a commit:
```bash
jenkins@2665e52069d0:~/couchdb$ git config --global user.email "couchdb-validation@no-reply.com"
jenkins@2665e52069d0:~/couchdb$ git config --global user.name "CouchDB validation"
jenkins@2665e52069d0:~/couchdb$ git commit -am "fix: Apply caching"
[3.1.1-patch-jan 6c2d01a] fix: Apply caching
1 file changed, 79 insertions(+), 23 deletions(-)
jenkins@2665e52069d0:~/couchdb$ make dist
==> config (compile)
==> b64url (compile)
==> ets_lru (compile)
==> khash (compile)
==> snappy (compile)
==> bear (compile)
==> meck (compile)
==> folsom (compile)
==> hyper (compile)
==> ibrowse (compile)
==> jiffy (compile)
==> mochiweb (compile)
==> recon (compile)
==> proper (compile)
==> couch_epi (compile)
==> couch_log (compile)
==> chttpd (compile)
==> couch (compile)
Compiling priv/couch_js/1.8.5/http.c
Compiling priv/couch_js/1.8.5/main.c
Compiling priv/couch_js/1.8.5/utf8.c
Compiling priv/couch_js/1.8.5/util.c
==> couch_event (compile)
==> mem3 (compile)
==> couch_index (compile)
==> couch_mrview (compile)
==> couch_replicator (compile)
==> couch_plugins (compile)
==> couch_pse_tests (compile)
==> couch_stats (compile)
==> couch_peruser (compile)
==> couch_tests (compile)
==> ddoc_cache (compile)
==> dreyfus (compile)
==> fabric (compile)
==> global_changes (compile)
==> ioq (compile)
==> jwtf (compile)
==> ken (compile)
==> mango (compile)
==> rexi (compile)
==> setup (compile)
==> smoosh (compile)
==> rel (compile)
==> couchdb (compile)
COUCHDB_GIT_SHA: 6c2d01a
COUCHDB_VERSION: 3.1.1-6c2d01a
COUCHDB_VERSION_SUFFIX: 6c2d01a
DIRTY:
IN_RC:
IN_RELEASE:
ON_TAG:
REL_TAG: 3.1.1
SUB_VSN:
Building Apache CouchDB 3.1.1-6c2d01a
fatal: ambiguous argument '6c976bd..HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Done: apache-couchdb-3.1.1-6c2d01a.tar.gz
```
I copied that tar back to the host OS:
```bash
$ docker cp couchdb-3.1.1-fix-validation:/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a.tar.gz .
$ git clone https://github.com/apache/couchdb-pkg.git
Cloning into 'couchdb-pkg'...
remote: Enumerating objects: 723, done.
remote: Counting objects: 100% (94/94), done.
remote: Compressing objects: 100% (70/70), done.
remote: Total 723 (delta 40), reused 57 (delta 20), pack-reused 629
Receiving objects: 100% (723/723), 25.31 MiB | 23.49 MiB/s, done.
Resolving deltas: 100% (335/335), done.
$ cd couchdb-pkg/
```
The assumed image name was `couchdbdev/<osname>-<codename>-erlang-<erlang-version>`, so I retagged my created one before running the packaging job:
```bash
$ docker tag apache/couchdbci-debian:buster-erlang-20.3.8.26-1 couchdbdev/debian-buster-erlang-20.3.8.26-1
$ ERLANGVERSION=20.3.8.26-1 ./build.sh couch debian-buster /root/apache-couchdb-3.1.1-6c2d01a.tar.gz
Using apache-couchdb-3.1.1-6c2d01a.tar.gz to build packages...
mkdir -p ../couchdb
cp apache-couchdb-3.1.1-6c2d01a.tar.gz ../couchdb
cd ../couchdb && tar xfz *.tar.gz
cp debian/control.in debian/control
sed -i 's/%SPIDERMONKEY%/libmozjs-60-0/g' debian/control
sed -i 's/%SPIDERMONKEY_DEV%/libmozjs-60-dev/g' debian/control
echo 'SM_VER = 60' > debian/sm_ver.mk
rm -rf /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/debian
cp -R debian /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dch -v 3.1.1-6c2d01a~buster "Automatically generated package from upstream."
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a && dpkg-buildpackage -b -us -uc
dpkg-buildpackage: info: source package couchdb
dpkg-buildpackage: info: source version 3.1.1-6c2d01a~buster
dpkg-buildpackage: info: source distribution UNRELEASED
dpkg-buildpackage: info: source changed by "CouchDB Developers" <"dev@couchdb.apache.org">
dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
debian/rules clean
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh clean --with=systemd
debian/rules override_dh_auto_clean
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_clean
make -j1 distclean
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (clean)
==> b64url (clean)
==> ets_lru (clean)
==> khash (clean)
==> snappy (clean)
==> bear (clean)
==> meck (clean)
==> folsom (clean)
==> hyper (clean)
==> ibrowse (clean)
==> jiffy (clean)
==> mochiweb (clean)
==> recon (clean)
==> couch_epi (clean)
==> couch_log (clean)
==> chttpd (clean)
==> couch (clean)
==> couch_event (clean)
==> mem3 (clean)
==> couch_index (clean)
==> couch_mrview (clean)
==> couch_replicator (clean)
==> couch_plugins (clean)
==> couch_pse_tests (clean)
==> couch_stats (clean)
==> couch_peruser (clean)
==> couch_tests (clean)
==> ddoc_cache (clean)
==> dreyfus (clean)
==> fabric (clean)
==> global_changes (clean)
==> ioq (clean)
==> jwtf (clean)
==> ken (clean)
==> mango (clean)
==> rexi (clean)
==> setup (clean)
==> smoosh (clean)
==> rel (clean)
==> apache-couchdb-3.1.1-6c2d01a (clean)
make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
mv src/mango/src/mango_cursor_text.erl.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.erl.nocompile': No such file or directory
make[2]: [debian/rules:21: override_dh_auto_clean] Error 1 (ignored)
mv src/mango/src/mango_cursor_text.nocompile src/mango/src/mango_cursor_text.erl
mv: cannot stat 'src/mango/src/mango_cursor_text.nocompile': No such file or directory
make[2]: [debian/rules:22: override_dh_auto_clean] Error 1 (ignored)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_clean
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules build
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh build --with=systemd
dh_update_autotools_config
debian/rules override_dh_auto_configure
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
./configure --spidermonkey-version 60
==> configuring couchdb in rel/couchdb.config
You have configured Apache CouchDB, time to relax. Relax.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules override_dh_auto_build
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_auto_build -- release
make -j1 release
make[3]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
==> config (compile)
Compiled src/config_util.erl
Compiled src/config_writer.erl
Compiled src/config_sup.erl
Compiled src/config_notifier.erl
Compiled src/config_listener_mon.erl
Compiled src/config_listener.erl
Compiled src/config_app.erl
Compiled src/config.erl
==> b64url (compile)
Compiled src/b64url.erl
Compiling c_src/b64url.c
==> ets_lru (compile)
Compiled src/ets_lru.erl
==> khash (compile)
Compiled src/khash.erl
Compiling c_src/hash.c
Compiling c_src/khash.c
==> snappy (compile)
Compiled src/snappy.erl
Compiling c_src/snappy_nif.cc
c_src/snappy_nif.cc: In function 'ERL_NIF_TERM snappy_compress(ErlNifEnv*, int, const ERL_NIF_TERM*)':
c_src/snappy_nif.cc:156:28: warning: catching polymorphic type 'class std::bad_alloc' by value [-Wcatch-value=]
} catch(std::bad_alloc e) {
^
Compiling c_src/snappy/snappy-sinksource.cc
Compiling c_src/snappy/snappy-stubs-internal.cc
Compiling c_src/snappy/snappy.cc
c_src/snappy/snappy.cc: In instantiation of 'bool snappy::SnappyScatteredWriter<Allocator>::AppendFromSelf(size_t, size_t) [with Allocator = snappy::SnappySinkAllocator; size_t = long unsigned int]':
c_src/snappy/snappy.cc:779:13: required from 'void snappy::SnappyDecompressor::DecompressAllTags(Writer*) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>]'
c_src/snappy/snappy.cc:865:3: required from 'bool snappy::InternalUncompressAllTags(snappy::SnappyDecompressor*, Writer*, snappy::uint32) [with Writer = snappy::SnappyScatteredWriter<snappy::SnappySinkAllocator>; snappy::uint32 = unsigned int]'
c_src/snappy/snappy.cc:1549:78: required from here
c_src/snappy/snappy.cc:1401:21: warning: comparison of integer expressions of different signedness: 'size_t' {aka 'long unsigned int'} and 'long int' [-Wsign-compare]
if (offset - 1u < op_ptr_ - op_base_) {
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
==> bear (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/bear/src/bear.erl:28: Warning: export_all flag enabled - all functions will be exported
Compiled src/bear.erl
==> meck (compile)
Compiled src/meck_util.erl
Compiled src/meck_matcher.erl
Compiled src/meck_ret_spec.erl
Compiled src/meck_history.erl
Compiled src/meck_expect.erl
Compiled src/meck_cover.erl
Compiled src/meck_code.erl
Compiled src/meck_code_gen.erl
Compiled src/meck_proc.erl
Compiled src/meck_args_matcher.erl
Compiled src/meck.erl
==> folsom (compile)
Compiled src/folsom_utils.erl
Compiled src/folsom_sup.erl
Compiled src/folsom_sample_slide_uniform.erl
Compiled src/folsom_sample_uniform.erl
Compiled src/folsom_sample_slide_sup.erl
Compiled src/folsom_sample_slide_server.erl
Compiled src/folsom_sample_slide_sorted.erl
Compiled src/folsom_vm_metrics.erl
Compiled src/folsom_sample_none.erl
Compiled src/folsom_sample_slide.erl
Compiled src/folsom_metrics_spiral.erl
Compiled src/folsom_sample_exdec.erl
Compiled src/folsom_sample.erl
Compiled src/folsom_metrics_meter.erl
Compiled src/folsom_metrics_history.erl
Compiled src/folsom_metrics_meter_reader.erl
Compiled src/folsom_metrics_histogram_ets.erl
Compiled src/folsom_metrics_gauge.erl
Compiled src/folsom_metrics_histogram.erl
Compiled src/folsom_metrics_duration.erl
Compiled src/folsom_metrics_counter.erl
Compiled src/folsom_ewma.erl
Compiled src/folsom_meter_timer_server.erl
Compiled src/folsom.erl
Compiled src/folsom_metrics.erl
Compiled src/folsom_ets.erl
==> hyper (compile)
Compiled src/hyper_register.erl
Compiled src/hyper_carray.erl
Compiled src/hyper_gb.erl
Compiled src/hyper_const.erl
Compiled src/hyper_array.erl
Compiled src/hyper_binary_rle.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:245: Warning: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:288: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/hyper/src/hyper.erl:333: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
Compiled src/hyper.erl
Compiled src/hyper_binary.erl
Compiling c_src/hyper_carray.c
==> ibrowse (compile)
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_lib.erl:372: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_lib.erl
Compiled src/ibrowse_sup.erl
Compiled src/ibrowse_socks5.erl
Compiled src/ibrowse_lb.erl
Compiled src/ibrowse_app.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:83: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:93: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:171: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_test.erl:182: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_test.erl
Compiled src/ibrowse.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:428: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/ibrowse/src/ibrowse_http_client.erl:1831: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled src/ibrowse_http_client.erl
==> jiffy (compile)
Compiling c_src/decoder.c
Compiling c_src/encoder.c
Compiling c_src/jiffy.c
Compiling c_src/termstack.c
Compiling c_src/utf8.c
Compiling c_src/util.c
Compiling c_src/doubles.cc
Compiling c_src/objects.cc
Compiling c_src/double-conversion/bignum-dtoa.cc
Compiling c_src/double-conversion/bignum.cc
Compiling c_src/double-conversion/cached-powers.cc
Compiling c_src/double-conversion/diy-fp.cc
Compiling c_src/double-conversion/double-conversion.cc
Compiling c_src/double-conversion/fast-dtoa.cc
Compiling c_src/double-conversion/fixed-dtoa.cc
Compiling c_src/double-conversion/strtod.cc
Compiled src/jiffy_utf8.erl
Compiled src/jiffy.erl
==> mochiweb (compile)
Compiled src/reloader.erl
Compiled src/mochiweb_websocket.erl
Compiled src/mochiweb_socket.erl
Compiled src/mochiweb_util.erl
Compiled src/mochiweb_session.erl
Compiled src/mochiweb_socket_server.erl
Compiled src/mochiweb_response.erl
Compiled src/mochiweb_mime.erl
Compiled src/mochiweb_multipart.erl
Compiled src/mochiweb_io.erl
Compiled src/mochiweb_http.erl
Compiled src/mochiweb_request.erl
Compiled src/mochiweb_echo.erl
Compiled src/mochiweb_headers.erl
Compiled src/mochiweb_cover.erl
Compiled src/mochiweb_cookies.erl
Compiled src/mochiweb_clock.erl
Compiled src/mochiweb_base64url.erl
Compiled src/mochiweb_acceptor.erl
Compiled src/mochiweb.erl
Compiled src/mochiweb_html.erl
Compiled src/mochiutf8.erl
Compiled src/mochitemp.erl
Compiled src/mochinum.erl
Compiled src/mochilogfile2.erl
Compiled src/mochilists.erl
Compiled src/mochijson.erl
Compiled src/mochihex.erl
Compiled src/mochiglobal.erl
Compiled src/mochifmt_std.erl
Compiled src/mochifmt_records.erl
Compiled src/mochijson2.erl
Compiled src/mochifmt.erl
Compiled src/mochiweb_charref.erl
==> recon (compile)
Compiled src/recon_map.erl
Compiled src/recon_rec.erl
Compiled src/recon_trace.erl
Compiled src/recon_alloc.erl
Compiled src/recon_lib.erl
Compiled src/recon.erl
==> proper (compile)
Compiled src/vararg.erl
Compiled src/proper_target.erl
Compiled src/proper_types.erl
Compiled src/proper_unicode.erl
Compiled src/proper_unused_imports_remover.erl
Compiled src/proper_symb.erl
Compiled src/proper_transformer.erl
Compiled src/proper_shrink.erl
Compiled src/proper_statem.erl
Compiled src/proper_sets.erl
Compiled src/proper_queue.erl
Compiled src/proper_sa.erl
Compiled src/proper_prop_remover.erl
Compiled src/proper_orddict.erl
Compiled src/proper_ordsets.erl
Compiled src/proper_gen.erl
Compiled src/proper_typeserver.erl
Compiled src/proper_gb_trees.erl
Compiled src/proper_gb_sets.erl
Compiled src/proper_gen_next.erl
Compiled src/proper_dict.erl
Compiled src/proper_array.erl
Compiled src/proper_fsm.erl
Compiled src/proper_arith.erl
Compiled src/proper.erl
==> couch_epi (compile)
Compiled src/couch_epi_util.erl
Compiled src/couch_epi_sup.erl
Compiled src/couch_epi_plugin.erl
Compiled src/couch_epi_module_keeper.erl
Compiled src/couch_epi_functions.erl
Compiled src/couch_epi_functions_gen.erl
Compiled src/couch_epi_data_gen.erl
Compiled src/couch_epi_data.erl
Compiled src/couch_epi_codegen.erl
Compiled src/couch_epi_app.erl
Compiled src/couch_epi_codechange_monitor.erl
Compiled src/couch_epi.erl
==> couch_log (compile)
Compiled src/couch_log_writer.erl
Compiled src/couch_log_writer_journald.erl
Compiled src/couch_log_writer_stderr.erl
Compiled src/couch_log_writer_syslog.erl
Compiled src/couch_log_writer_file.erl
Compiled src/couch_log_util.erl
Compiled src/couch_log_sup.erl
Compiled src/couch_log_server.erl
Compiled src/couch_log_monitor.erl
Compiled src/couch_log_trunc_io_fmt.erl
Compiled src/couch_log_error_logger_h.erl
Compiled src/couch_log_config_dyn.erl
Compiled src/couch_log_trunc_io.erl
Compiled src/couch_log_app.erl
Compiled src/couch_log_config.erl
Compiled src/couch_log.erl
Compiled src/couch_log_formatter.erl
==> chttpd (compile)
Compiled src/chttpd.erl
Compiled src/chttpd_xframe_options.erl
Compiled src/chttpd_sup.erl
Compiled src/chttpd_view.erl
Compiled src/chttpd_stats.erl
Compiled src/chttpd_test_util.erl
Compiled src/chttpd_prefer_header.erl
Compiled src/chttpd_rewrite.erl
Compiled src/chttpd_show.erl
Compiled src/chttpd_plugin.erl
Compiled src/chttpd_httpd_handlers.erl
Compiled src/chttpd_handlers.erl
Compiled src/chttpd_node.erl
Compiled src/chttpd_misc.erl
Compiled src/chttpd_epi.erl
Compiled src/chttpd_external.erl
Compiled src/chttpd_cors.erl
Compiled src/chttpd_auth_request.erl
Compiled src/chttpd_auth.erl
Compiled src/chttpd_app.erl
Compiled src/chttpd_auth_cache.erl
Compiled src/chttpd_db.erl
==> couch (compile)
Compiled src/couch_httpd.erl
Compiled src/test_request.erl
Compiled src/couch_work_queue.erl
Compiled src/couch_uuids.erl
Compiled src/couch_users_db.erl
Compiled src/couch_totp.erl
Compiled src/test_util.erl
Compiled src/couch_util.erl
Compiled src/couch_task_status.erl
Compiled src/couch_sup.erl
Compiled src/couch_secondary_sup.erl
Compiled src/couch_rand.erl
Compiled src/couch_stream.erl
Compiled src/couch_proc_manager.erl
Compiled src/couch_query_servers.erl
Compiled src/couch_server.erl
Compiled src/couch_primary_sup.erl
Compiled src/couch_partition.erl
Compiled src/couch_passwords.erl
Compiled src/couch_os_process.erl
Compiled src/couch_lru.erl
Compiled src/couch_multidb_changes.erl
Compiled src/couch_io_logger.erl
Compiled src/couch_native_process.erl
Compiled src/couch_httpd_vhost.erl
Compiled src/couch_key_tree.erl
Compiled src/couch_httpd_rewrite.erl
Compiled src/couch_httpd_handlers.erl
Compiled src/couch_httpd_multipart.erl
Compiled src/couch_httpd_misc_handlers.erl
Compiled src/couch_httpd_external.erl
Compiled src/couch_hotp.erl
Compiled src/couch_hash.erl
Compiled src/couch_flags_config.erl
Compiled src/couch_httpd_auth.erl
Compiled src/couch_flags.erl
Compiled src/couch_event_sup.erl
Compiled src/couch_emsort.erl
Compiled src/couch_ejson_size.erl
Compiled src/couch_file.erl
Compiled src/couch_httpd_db.erl
Compiled src/couch_ejson_compare.erl
Compiled src/couch_drv.erl
Compiled src/couch_debug.erl
Compiled src/couch_doc.erl
Compiled src/couch_db_split.erl
Compiled src/couch_db_plugin.erl
Compiled src/couch_db_epi.erl
Compiled src/couch_db_header.erl
Compiled src/couch_db_updater.erl
Compiled src/couch_compress.erl
Compiled src/couch_db_engine.erl
Compiled src/couch_changes.erl
Compiled src/couch_bt_engine_stream.erl
Compiled src/couch_bt_engine_header.erl
Compiled src/couch_db.erl
Compiled src/couch_btree.erl
Compiled src/couch_base32.erl
Compiled src/couch_bt_engine_compactor.erl
Compiled src/couch_auth_cache.erl
Compiled src/couch_att.erl
Compiled src/couch_bt_engine.erl
Compiled src/couch.erl
Compiled src/couch_app.erl
Compiling priv/couch_js/60/http.cpp
Compiling priv/couch_js/60/main.cpp
Compiling priv/couch_js/60/util.cpp
Compiling priv/icu_driver/couch_icu_driver.c
Compiling priv/couch_ejson_compare/couch_ejson_compare.c
==> couch_event (compile)
Compiled src/couch_event_listener.erl
Compiled src/couch_event_sup2.erl
Compiled src/couch_event_os_listener.erl
Compiled src/couch_event_app.erl
Compiled src/couch_event.erl
Compiled src/couch_event_server.erl
Compiled src/couch_event_listener_mfa.erl
==> mem3 (compile)
Compiled src/mem3_sync_nodes.erl
Compiled src/mem3_sync_security.erl
Compiled src/mem3_sync_event.erl
Compiled src/mem3_sync_event_listener.erl
Compiled src/mem3_sup.erl
Compiled src/mem3_sync.erl
Compiled src/mem3_util.erl
Compiled src/mem3_seeds.erl
Compiled src/mem3_reshard_validate.erl
Compiled src/mem3_reshard_sup.erl
Compiled src/mem3_shards.erl
Compiled src/mem3_rpc.erl
Compiled src/mem3_reshard_job_sup.erl
Compiled src/mem3_reshard_store.erl
Compiled src/mem3_reshard_index.erl
Compiled src/mem3_reshard_httpd.erl
Compiled src/mem3_reshard_job.erl
Compiled src/mem3_reshard_api.erl
Compiled src/mem3_reshard_dbdoc.erl
Compiled src/mem3_plugin_couch_db.erl
Compiled src/mem3_nodes.erl
Compiled src/mem3_httpd_handlers.erl
Compiled src/mem3_reshard.erl
Compiled src/mem3_hash.erl
Compiled src/mem3_epi.erl
Compiled src/mem3_httpd.erl
Compiled src/mem3_rep.erl
Compiled src/mem3_app.erl
Compiled src/mem3_cluster.erl
Compiled src/mem3.erl
==> couch_index (compile)
Compiled src/couch_index_sup.erl
Compiled src/couch_index_util.erl
Compiled src/couch_index_plugin_couch_db.erl
Compiled src/couch_index_updater.erl
Compiled src/couch_index_epi.erl
Compiled src/couch_index_plugin.erl
Compiled src/couch_index_app.erl
Compiled src/couch_index_server.erl
Compiled src/couch_index_compactor.erl
Compiled src/couch_index.erl
==> couch_mrview (compile)
Compiled src/couch_mrview_update_notifier.erl
Compiled src/couch_mrview_updater.erl
Compiled src/couch_mrview_test_util.erl
Compiled src/couch_mrview_show.erl
Compiled src/couch_mrview_util.erl
Compiled src/couch_mrview_index.erl
Compiled src/couch_mrview_compactor.erl
Compiled src/couch_mrview_cleanup.erl
Compiled src/couch_mrview_http.erl
Compiled src/couch_mrview.erl
==> couch_replicator (compile)
Compiled src/couch_replicator_httpc.erl
Compiled src/couch_replicator_auth.erl
Compiled src/couch_replicator_utils.erl
Compiled src/couch_replicator_sup.erl
Compiled src/couch_replicator_stats.erl
Compiled src/couch_replicator_scheduler_sup.erl
Compiled src/json_stream_parse.erl
Compiled src/couch_replicator_worker.erl
Compiled src/couch_replicator_rate_limiter_tables.erl
Compiled src/couch_replicator_rate_limiter.erl
Compiled src/couch_replicator_notifier.erl
Compiled src/couch_replicator_job_sup.erl
Compiled src/couch_replicator_scheduler_job.erl
Compiled src/couch_replicator_scheduler.erl
Compiled src/couch_replicator_ids.erl
Compiled src/couch_replicator_httpd_util.erl
Compiled src/couch_replicator_httpd.erl
Compiled src/couch_replicator_httpc_pool.erl
Compiled src/couch_replicator_filters.erl
Compiled src/couch_replicator_fabric_rpc.erl
Compiled src/couch_replicator_doc_processor_worker.erl
Compiled src/couch_replicator_fabric.erl
Compiled src/couch_replicator_db_changes.erl
Compiled src/couch_replicator_connection.erl
Compiled src/couch_replicator_docs.erl
Compiled src/couch_replicator_changes_reader.erl
Compiled src/couch_replicator_doc_processor.erl
Compiled src/couch_replicator_clustering.erl
Compiled src/couch_replicator_app.erl
Compiled src/couch_replicator_auth_noop.erl
Compiled src/couch_replicator_auth_session.erl
Compiled src/couch_replicator.erl
Compiled src/couch_replicator_api_wrap.erl
==> couch_plugins (compile)
Compiled src/couch_plugins_httpd.erl
Compiled src/couch_plugins.erl
==> couch_pse_tests (compile)
Compiled src/cpse_test_ref_counting.erl
Compiled src/cpse_util.erl
Compiled src/cpse_test_read_write_docs.erl
/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/src/couch_pse_tests/src/cpse_test_purge_seqs.erl:14: Warning: export_all flag enabled - all functions will be exported
Compiled src/cpse_test_purge_seqs.erl
Compiled src/cpse_test_purge_replication.erl
Compiled src/cpse_test_purge_docs.erl
Compiled src/cpse_test_open_close_delete.erl
Compiled src/cpse_test_purge_bad_checkpoints.erl
Compiled src/cpse_test_get_set_props.erl
Compiled src/cpse_test_fold_purge_infos.erl
Compiled src/cpse_test_fold_changes.erl
Compiled src/cpse_test_fold_docs.erl
Compiled src/cpse_test_copy_purge_infos.erl
Compiled src/cpse_gather.erl
Compiled src/cpse_test_attachments.erl
Compiled src/cpse_test_compaction.erl
==> couch_stats (compile)
Compiled src/couch_stats_sup.erl
Compiled src/couch_stats_app.erl
Compiled src/couch_stats_process_tracker.erl
Compiled src/couch_stats.erl
Compiled src/couch_stats_httpd.erl
Compiled src/couch_stats_aggregator.erl
==> couch_peruser (compile)
Compiled src/couch_peruser_app.erl
Compiled src/couch_peruser_sup.erl
Compiled src/couch_peruser.erl
==> couch_tests (compile)
Compiled setups/couch_epi_dispatch.erl
Compiled src/couch_tests_combinatorics.erl
Compiled src/couch_tests.erl
==> ddoc_cache (compile)
Compiled src/ddoc_cache_value.erl
Compiled src/ddoc_cache_sup.erl
Compiled src/ddoc_cache_opener.erl
Compiled src/ddoc_cache_entry_validation_funs.erl
Compiled src/ddoc_cache_entry_ddocid_rev.erl
Compiled src/ddoc_cache_entry_ddocid.erl
Compiled src/ddoc_cache_entry_custom.erl
Compiled src/ddoc_cache_app.erl
Compiled src/ddoc_cache_lru.erl
Compiled src/ddoc_cache.erl
Compiled src/ddoc_cache_entry.erl
==> dreyfus (compile)
Compiled src/dreyfus_sup.erl
Compiled src/dreyfus_plugin_couch_db.erl
Compiled src/dreyfus_rpc.erl
Compiled src/dreyfus_index_updater.erl
Compiled src/dreyfus_util.erl
Compiled src/dreyfus_index_manager.erl
Compiled src/dreyfus_httpd_handlers.erl
Compiled src/dreyfus_index.erl
Compiled src/dreyfus_fabric_search.erl
Compiled src/dreyfus_fabric_info.erl
Compiled src/dreyfus_httpd.erl
Compiled src/dreyfus_fabric_group2.erl
Compiled src/dreyfus_fabric_cleanup.erl
Compiled src/dreyfus_fabric_group1.erl
Compiled src/dreyfus_epi.erl
Compiled src/dreyfus_config.erl
Compiled src/dreyfus_app.erl
Compiled src/dreyfus_fabric.erl
Compiled src/dreyfus_bookmark.erl
Compiled src/clouseau_rpc.erl
==> fabric (compile)
Compiled src/fabric_db_update_listener.erl
Compiled src/fabric_view_reduce.erl
Compiled src/fabric_view_map.erl
Compiled src/fabric_view_all_docs.erl
Compiled src/fabric_view_changes.erl
Compiled src/fabric_view.erl
Compiled src/fabric_streams.erl
Compiled src/fabric_util.erl
Compiled src/fabric_rpc.erl
Compiled src/fabric_ring.erl
Compiled src/fabric_group_info.erl
Compiled src/fabric_doc_update.erl
Compiled src/fabric_doc_purge.erl
Compiled src/fabric_doc_missing_revs.erl
Compiled src/fabric_doc_open_revs.erl
Compiled src/fabric_doc_atts.erl
Compiled src/fabric_doc_open.erl
Compiled src/fabric_dict.erl
Compiled src/fabric_doc_attachments.erl
Compiled src/fabric_design_doc_count.erl
Compiled src/fabric_db_partition_info.erl
Compiled src/fabric_db_meta.erl
Compiled src/fabric_db_doc_count.erl
Compiled src/fabric_db_info.erl
Compiled src/fabric_db_delete.erl
Compiled src/fabric_db_create.erl
Compiled src/fabric.erl
==> global_changes (compile)
Compiled src/global_changes_sup.erl
Compiled src/global_changes_util.erl
Compiled src/global_changes_plugin.erl
Compiled src/global_changes_httpd_handlers.erl
Compiled src/global_changes_listener.erl
Compiled src/global_changes_server.erl
Compiled src/global_changes_epi.erl
Compiled src/global_changes_app.erl
Compiled src/global_changes_httpd.erl
==> ioq (compile)
Compiled src/ioq_app.erl
Compiled src/ioq_sup.erl
Compiled src/ioq.erl
==> jwtf (compile)
Compiled src/jwtf_app.erl
Compiled src/jwtf_sup.erl
Compiled src/jwtf.erl
Compiled src/jwtf_keystore.erl
==> ken (compile)
Compiled src/ken_sup.erl
Compiled src/ken_event_handler.erl
Compiled src/ken_app.erl
Compiled src/ken.erl
Compiled src/ken_server.erl
==> mango (compile)
Compiled src/mango_sup.erl
Compiled src/mango_sort.erl
Compiled src/mango_util.erl
Compiled src/mango_selector_text.erl
Compiled src/mango_selector.erl
Compiled src/mango_opts.erl
Compiled src/mango_json.erl
Compiled src/mango_json_bookmark.erl
Compiled src/mango_native_proc.erl
Compiled src/mango_idx_special.erl
Compiled src/mango_idx_text.erl
Compiled src/mango_httpd_handlers.erl
Compiled src/mango_idx_view.erl
Compiled src/mango_fields.erl
Compiled src/mango_execution_stats.erl
Compiled src/mango_idx.erl
Compiled src/mango_epi.erl
Compiled src/mango_httpd.erl
Compiled src/mango_error.erl
Compiled src/mango_doc.erl
Compiled src/mango_cursor_text.erl
Compiled src/mango_cursor_special.erl
Compiled src/mango_cursor.erl
Compiled src/mango_app.erl
Compiled src/mango_crud.erl
Compiled src/mango_cursor_view.erl
==> rexi (compile)
Compiled src/rexi_server_sup.erl
Compiled src/rexi_sup.erl
Compiled src/rexi_utils.erl
Compiled src/rexi_server_mon.erl
Compiled src/rexi_monitor.erl
Compiled src/rexi_app.erl
Compiled src/rexi_server.erl
Compiled src/rexi_buffer.erl
Compiled src/rexi.erl
==> setup (compile)
Compiled src/setup_sup.erl
Compiled src/setup_httpd_handlers.erl
Compiled src/setup_app.erl
Compiled src/setup_epi.erl
Compiled src/setup_httpd.erl
Compiled src/setup.erl
==> smoosh (compile)
Compiled src/smoosh_sup.erl
Compiled src/smoosh_utils.erl
Compiled src/smoosh_priority_queue.erl
Compiled src/smoosh_app.erl
Compiled src/smoosh_channel.erl
Compiled src/smoosh_server.erl
Compiled src/smoosh.erl
==> rel (compile)
==> apache-couchdb-3.1.1-6c2d01a (compile)
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN: 'generate' command does not apply to directory /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a
... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.
make[3]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
rm -rf rel/couchdb/var/log
rm -rf rel/couchdb/data
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
debian/rules binary
make[1]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh binary --with=systemd
dh_testroot
dh_prep
dh_installdirs
dh_install
dh_installdocs
dh_installchangelogs
dh_installman
dh_installdebconf
dh_systemd_enable
dh_installinit
dh_systemd_start
dh_installlogrotate
dh_lintian
dh_perl
dh_link
dh_strip_nondeterminism
dh_compress
dh_fixperms
dh_missing
dh_strip
dh_makeshlibs
debian/rules override_dh_shlibdeps
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_shlibdeps -- --ignore-missing-info -xlibmozjs185-1.0
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/otp_test_engine.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto_callback.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: not a dynamic object
objdump: debian/couchdb/opt/couchdb/lib/crypto-4.2.2.4/priv/obj/crypto.o: invalid operation
dpkg-shlibdeps: warning: couldn't parse dynamic symbol definition: no symbols
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_ejson_compare.so debian/couchdb/opt/couchdb/lib/couch-3.1.1-6c2d01a/priv/couch_icu_driver.so were not linked against libicudata.so.63 (they use none of the library's symbols)
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_installdeb
dh_gencontrol
dh_md5sums
debian/rules override_dh_builddeb
make[2]: Entering directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dh_builddeb -- -Zgzip
dpkg-deb: building package 'couchdb-dbgsym' in '../couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb'.
dpkg-deb: building package 'couchdb' in '../couchdb_3.1.1-6c2d01a~buster_amd64.deb'.
make[2]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
make[1]: Leaving directory '/home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a'
dpkg-genbuildinfo --build=binary
dpkg-genchanges --build=binary >../couchdb_3.1.1-6c2d01a~buster_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
dpkg-source --after-build .
dpkg-source: info: using options from apache-couchdb-3.1.1-6c2d01a/debian/source/options: --compression=bzip2 --compression-level=9
dpkg-buildpackage: info: binary-only upload (no source included)
cd /home/jenkins/couchdb/apache-couchdb-3.1.1-6c2d01a/.. && lintian --profile couchdb couch*.deb || true
warning: the authors of lintian do not recommend running it with root privileges!
W: couchdb: maintainer-script-should-not-use-recursive-chown-or-chmod postinst:207
N: 3 tags overridden (2 errors, 1 info); 2 unused overrides
chmod a+rwx ../couchdb/couchdb*
mkdir -p pkgs/couch/debian-buster && chmod 777 pkgs/couch/debian-buster
cp ../couchdb/couchdb* pkgs/couch/debian-buster
if [ -f debian/control.bak ]; then mv -f debian/control.bak debian/control; fi
if [ -f rpm/SPECS/couchdb.spec.bak ]; then mv -f rpm/SPECS/couchdb.spec.bak rpm/SPECS/couchdb.spec; fi
rm -rf parts prime stage js/build debian/sm_ver.mk
# packages were created
$ ls -lah pkgs/couch/debian-buster/
total 30M
drwxrwxrwx 2 root root 207 Aug 31 01:58 .
drwxrwxrwx 3 root root 45 Aug 31 01:58 ..
-rwxr-xr-x 1 root root 11K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo
-rwxr-xr-x 1 root root 1.5K Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.changes
-rwxr-xr-x 1 root root 30M Aug 31 01:58 couchdb_3.1.1-6c2d01a~buster_amd64.deb
-rwxr-xr-x 1 root root 474K Aug 31 01:58 couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb
```
Next use https://github.com/apache/couchdb-docker/tree/main/3.1.1 to produce the container images, but adjust them to include the custom couchdb version.
```bash
$ git clone https://github.com/apache/couchdb-docker.git
Cloning into 'couchdb-docker'...
remote: Enumerating objects: 1010, done.
remote: Counting objects: 100% (118/118), done.
remote: Compressing objects: 100% (82/82), done.
remote: Total 1010 (delta 64), reused 64 (delta 32), pack-reused 892
Receiving objects: 100% (1010/1010), 248.11 KiB | 1.19 MiB/s, done.
Resolving deltas: 100% (490/490), done.
$ cd ~/couchdb-docker/3.1.1
$ cp -r ~/couchdb-pkg/pkgs/couch/debian-buster/ .
$ ls debian-buster/
couchdb_3.1.1-6c2d01a~buster_amd64.buildinfo couchdb_3.1.1-6c2d01a~buster_amd64.changes couchdb_3.1.1-6c2d01a~buster_amd64.deb couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb
$ git diff
diff --git a/3.1.1/Dockerfile b/3.1.1/Dockerfile
index 75b4b29..59a90d9 100644
--- a/3.1.1/Dockerfile
+++ b/3.1.1/Dockerfile
@@ -60,6 +60,10 @@ RUN . /etc/os-release; \
echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | \
tee /etc/apt/sources.list.d/couchdb.list >/dev/null
+# I know, we should not include deb packages in images, but it should be good enough to validate the fix
+COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
+
# https://github.com/apache/couchdb-pkg/blob/master/debian/README.Debian
RUN set -eux; \
apt-get update; \
@@ -67,7 +71,7 @@ RUN set -eux; \
echo "couchdb couchdb/mode select none" | debconf-set-selections; \
# we DO want recommends this time
DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages \
- couchdb="$COUCHDB_VERSION"~buster \
+ /tmp/packages/* \
; \
# Undo symlinks to /var/log and /var/lib
rmdir /var/lib/couchdb /var/log/couchdb; \
docker build -t "couchdb:3.1.1-patch-jan" .
Sending build context to Docker daemon 31.45MB
Step 1/20 : FROM debian:buster-slim
---> 5c3d57f3e47a
Step 2/20 : LABEL maintainer="CouchDB Developers dev@couchdb.apache.org"
---> Using cache
---> 60f37c14a0e1
Step 3/20 : RUN groupadd -g 5984 -r couchdb && useradd -u 5984 -d /opt/couchdb -g couchdb couchdb
---> Using cache
---> 78202346b500
Step 4/20 : RUN set -ex; apt-get update; apt-get install -y --no-install-recommends apt-transport-https ca-certificates dirmngr gnupg ; rm -rf /var/lib/apt/lists/*
---> Using cache
---> fad0f8a736e0
Step 5/20 : RUN set -eux; apt-get update; apt-get install -y --no-install-recommends gosu tini; rm -rf /var/lib/apt/lists/*; gosu nobody true; tini --version
---> Using cache
---> aca1db0792a3
Step 6/20 : ENV GPG_COUCH_KEY 390EF70BB1EA12B2773962950EE62FB37A00258D
---> Using cache
---> d28f3c244c10
Step 7/20 : RUN set -eux; apt-get update; apt-get install -y curl; export GNUPGHOME="$(mktemp -d)"; curl -fL -o keys.asc https://couchdb.apache.org/repo/keys.asc; gpg --batch --import keys.asc; gpg --batch --export "${GPG_COUCH_KEY}" > /usr/share/keyrings/couchdb-archive-keyring.gpg; command -v gpgconf && gpgconf --kill all || :; rm -rf "$GNUPGHOME"; apt-key list; apt purge -y --autoremove curl; rm -rf /var/lib/apt/lists/*
---> Using cache
---> e2949d6cc031
Step 8/20 : ENV COUCHDB_VERSION 3.1.1
---> Using cache
---> baa8089d1bea
Step 9/20 : RUN . /etc/os-release; echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | tee /etc/apt/sources.list.d/couchdb.list >/dev/null
---> Using cache
---> a7933b88e802
Step 10/20 : COPY debian-buster/couchdb_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
---> Using cache
---> ded52d34b729
Step 11/20 : COPY debian-buster/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/
---> Using cache
---> eb9b215cb430
Step 12/20 : RUN set -eux; apt-get update; echo "couchdb couchdb/mode select none" | debconf-set-selections; DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/* ; rmdir /var/lib/couchdb /var/log/couchdb; rm /opt/couchdb/data /opt/couchdb/var/log; mkdir -p /opt/couchdb/data /opt/couchdb/var/log; chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log; chmod 777 /opt/couchdb/data /opt/couchdb/var/log; rm /opt/couchdb/etc/default.d/10-filelog.ini; find /opt/couchdb \! \( -user couchdb -group couchdb \) -exec chown -f couchdb:couchdb '{}' +; find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 '{}' +; find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 '{}' +; chmod -f 0777 /opt/couchdb/etc/local.d; rm -rf /var/lib/apt/lists/*;
---> Running in 6f530f886258
+ apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [302 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 https://apache.jfrog.io/artifactory/couchdb-deb buster InRelease [5151 B]
Get:7 https://apache.jfrog.io/artifactory/couchdb-deb buster/main amd64 Packages [3889 B]
Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages [15.2 kB]
Fetched 8473 kB in 3s (3017 kB/s)
Reading package lists...
+ echo couchdb couchdb/mode select none
+ debconf-set-selections
+ DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-downgrades --allow-remove-essential --allow-change-held-packages /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2 libicu63 libk5crypto3
libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0 libncurses6
libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps psmisc
publicsuffix
Suggested packages:
gpm krb5-doc krb5-user
The following NEW packages will be installed:
couchdb couchdb-dbgsym curl krb5-locales libcurl4 libgpm2 libgssapi-krb5-2
libicu63 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libmozjs-60-0
libncurses6 libnghttp2-14 libprocps7 libpsl5 librtmp1 libssh2-1 procps
psmisc publicsuffix
0 upgraded, 22 newly installed, 0 to remove and 0 not upgraded.
Need to get 13.5 MB/45.0 MB of archives.
After this operation, 104 MB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security buster/updates/main amd64 krb5-locales all 1.17-3+deb10u2 [95.5 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libncurses6 amd64 6.1+20181013-2+deb10u2 [102 kB]
Get:3 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5support0 amd64 1.17-3+deb10u2 [65.7 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 libk5crypto3 amd64 1.17-3+deb10u2 [122 kB]
Get:5 http://security.debian.org/debian-security buster/updates/main amd64 libkrb5-3 amd64 1.17-3+deb10u2 [369 kB]
Get:6 http://security.debian.org/debian-security buster/updates/main amd64 libgssapi-krb5-2 amd64 1.17-3+deb10u2 [158 kB]
Get:7 http://deb.debian.org/debian buster/main amd64 libprocps7 amd64 2:3.3.15-2 [61.7 kB]
Get:8 http://deb.debian.org/debian buster/main amd64 procps amd64 2:3.3.15-2 [259 kB]
Get:9 http://deb.debian.org/debian buster/main amd64 libkeyutils1 amd64 1.6-6 [15.0 kB]
Get:10 http://deb.debian.org/debian buster/main amd64 libnghttp2-14 amd64 1.36.0-2+deb10u1 [85.0 kB]
Get:11 /tmp/packages/couchdb_3.1.1-6c2d01a~buster_amd64.deb couchdb amd64 3.1.1-6c2d01a~buster [30.9 MB]
Get:12 /tmp/packages/couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb couchdb-dbgsym amd64 3.1.1-6c2d01a~buster [485 kB]
Get:13 http://deb.debian.org/debian buster/main amd64 libpsl5 amd64 0.20.2-2 [53.7 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2 [60.5 kB]
Get:15 http://deb.debian.org/debian buster/main amd64 libssh2-1 amd64 1.8.0-2.1 [140 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 libcurl4 amd64 7.64.0-4+deb10u2 [332 kB]
Get:17 http://deb.debian.org/debian buster/main amd64 curl amd64 7.64.0-4+deb10u2 [265 kB]
Get:18 http://deb.debian.org/debian buster/main amd64 libicu63 amd64 63.1-6+deb10u1 [8300 kB]
Get:19 http://deb.debian.org/debian buster/main amd64 libmozjs-60-0 amd64 60.2.3-3 [2785 kB]
Get:20 http://deb.debian.org/debian buster/main amd64 libgpm2 amd64 1.20.7-5 [35.1 kB]
Get:21 http://deb.debian.org/debian buster/main amd64 psmisc amd64 23.2-1 [126 kB]
Get:22 http://deb.debian.org/debian buster/main amd64 publicsuffix all 20190415.1030-1 [116 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 13.5 MB in 1s (12.8 MB/s)
Selecting previously unselected package libncurses6:amd64.
(Reading database ... 7118 files and directories currently installed.)
Preparing to unpack .../00-libncurses6_6.1+20181013-2+deb10u2_amd64.deb ...
Unpacking libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Selecting previously unselected package libprocps7:amd64.
Preparing to unpack .../01-libprocps7_2%3a3.3.15-2_amd64.deb ...
Unpacking libprocps7:amd64 (2:3.3.15-2) ...
Selecting previously unselected package procps.
Preparing to unpack .../02-procps_2%3a3.3.15-2_amd64.deb ...
Unpacking procps (2:3.3.15-2) ...
Selecting previously unselected package krb5-locales.
Preparing to unpack .../03-krb5-locales_1.17-3+deb10u2_all.deb ...
Unpacking krb5-locales (1.17-3+deb10u2) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../04-libkeyutils1_1.6-6_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.6-6) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../05-libkrb5support0_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../06-libk5crypto3_1.17-3+deb10u2_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../07-libkrb5-3_1.17-3+deb10u2_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../08-libgssapi-krb5-2_1.17-3+deb10u2_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../09-libnghttp2-14_1.36.0-2+deb10u1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../10-libpsl5_0.20.2-2_amd64.deb ...
Unpacking libpsl5:amd64 (0.20.2-2) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../11-librtmp1_2.4+20151223.gitfa8646d.1-2_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Selecting previously unselected package libssh2-1:amd64.
Preparing to unpack .../12-libssh2-1_1.8.0-2.1_amd64.deb ...
Unpacking libssh2-1:amd64 (1.8.0-2.1) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../13-libcurl4_7.64.0-4+deb10u2_amd64.deb ...
Unpacking libcurl4:amd64 (7.64.0-4+deb10u2) ...
Selecting previously unselected package curl.
Preparing to unpack .../14-curl_7.64.0-4+deb10u2_amd64.deb ...
Unpacking curl (7.64.0-4+deb10u2) ...
Selecting previously unselected package libicu63:amd64.
Preparing to unpack .../15-libicu63_63.1-6+deb10u1_amd64.deb ...
Unpacking libicu63:amd64 (63.1-6+deb10u1) ...
Selecting previously unselected package libmozjs-60-0:amd64.
Preparing to unpack .../16-libmozjs-60-0_60.2.3-3_amd64.deb ...
Unpacking libmozjs-60-0:amd64 (60.2.3-3) ...
Selecting previously unselected package couchdb.
Preparing to unpack .../17-couchdb_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package couchdb-dbgsym.
Preparing to unpack .../18-couchdb-dbgsym_3.1.1-6c2d01a~buster_amd64.deb ...
Unpacking couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Selecting previously unselected package libgpm2:amd64.
Preparing to unpack .../19-libgpm2_1.20.7-5_amd64.deb ...
Unpacking libgpm2:amd64 (1.20.7-5) ...
Selecting previously unselected package psmisc.
Preparing to unpack .../20-psmisc_23.2-1_amd64.deb ...
Unpacking psmisc (23.2-1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../21-publicsuffix_20190415.1030-1_all.deb ...
Unpacking publicsuffix (20190415.1030-1) ...
Setting up libkeyutils1:amd64 (1.6-6) ...
Setting up libpsl5:amd64 (0.20.2-2) ...
Setting up libgpm2:amd64 (1.20.7-5) ...
Setting up psmisc (23.2-1) ...
Setting up libprocps7:amd64 (2:3.3.15-2) ...
Setting up libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Setting up krb5-locales (1.17-3+deb10u2) ...
Setting up libicu63:amd64 (63.1-6+deb10u1) ...
Setting up libkrb5support0:amd64 (1.17-3+deb10u2) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Setting up libncurses6:amd64 (6.1+20181013-2+deb10u2) ...
Setting up libk5crypto3:amd64 (1.17-3+deb10u2) ...
Setting up procps (2:3.3.15-2) ...
update-alternatives: using /usr/bin/w.procps to provide /usr/bin/w (w) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/w.1.gz because associated file /usr/share/man/man1/w.procps.1.gz (of link group w) doesn't exist
Setting up libssh2-1:amd64 (1.8.0-2.1) ...
Setting up libkrb5-3:amd64 (1.17-3+deb10u2) ...
Setting up publicsuffix (20190415.1030-1) ...
Setting up libmozjs-60-0:amd64 (60.2.3-3) ...
Setting up libgssapi-krb5-2:amd64 (1.17-3+deb10u2) ...
Setting up libcurl4:amd64 (7.64.0-4+deb10u2) ...
Setting up curl (7.64.0-4+deb10u2) ...
Setting up couchdb (3.1.1-6c2d01a~buster) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up couchdb-dbgsym (3.1.1-6c2d01a~buster) ...
Processing triggers for libc-bin (2.28-10) ...
+ rmdir /var/lib/couchdb /var/log/couchdb
+ rm /opt/couchdb/data /opt/couchdb/var/log
+ mkdir -p /opt/couchdb/data /opt/couchdb/var/log
+ chown couchdb:couchdb /opt/couchdb/data /opt/couchdb/var/log
+ chmod 777 /opt/couchdb/data /opt/couchdb/var/log
+ rm /opt/couchdb/etc/default.d/10-filelog.ini
+ find /opt/couchdb ! ( -user couchdb -group couchdb ) -exec chown -f couchdb:couchdb {} +
+ find /opt/couchdb/etc -type d ! -perm 0755 -exec chmod -f 0755 {} +
+ find /opt/couchdb/etc -type f ! -perm 0644 -exec chmod -f 0644 {} +
+ chmod -f 0777 /opt/couchdb/etc/local.d
+ rm -rf /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_InRelease /var/lib/apt/lists/apache.jfrog.io_artifactory_couchdb-deb_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/auxfiles /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster-updates_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/deb.debian.org_debian_dists_buster_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_main_binary-amd64_Packages.lz4
Removing intermediate container 6f530f886258
---> eaea07a7886e
Step 13/20 : COPY --chown=couchdb:couchdb 10-docker-default.ini /opt/couchdb/etc/default.d/
---> f1c9d8e86732
Step 14/20 : COPY --chown=couchdb:couchdb vm.args /opt/couchdb/etc/
---> a984087a4ee3
Step 15/20 : COPY docker-entrypoint.sh /usr/local/bin
---> 66cd3ececce1
Step 16/20 : RUN ln -s usr/local/bin/docker-entrypoint.sh /docker-entrypoint.sh # backwards compat
---> Running in ae10e0e02906
Removing intermediate container ae10e0e02906
---> bf8817264923
Step 17/20 : ENTRYPOINT ["tini", "--", "/docker-entrypoint.sh"]
---> Running in 1485a79c319f
Removing intermediate container 1485a79c319f
---> 737d9f7a9b97
Step 18/20 : VOLUME /opt/couchdb/data
---> Running in 6aa4eafbc054
Removing intermediate container 6aa4eafbc054
---> 3f111d557168
Step 19/20 : EXPOSE 5984 4369 9100
---> Running in 07b6a3204c82
Removing intermediate container 07b6a3204c82
---> c70917bea3fd
Step 20/20 : CMD ["/opt/couchdb/bin/couchdb"]
---> Running in d61719ef6dea
Removing intermediate container d61719ef6dea
---> ca3940266314
Successfully built ca3940266314
Successfully tagged couchdb:3.1.1-patch-jan
```
I pushed the image and pulled it to my workstation again to have the same underlying hardware as in the previous test.
## Results
I adjusted the couchdb-test.sh [couchdb-test-validate-patch-3517.zip](https://github.com/apache/couchdb/files/7083304/couchdb-test-validate-patch-3517.zip) script to use the patched image and compare it against the vanilla CouchDB 3.1.1 image.
```bash
$ ./couchdb-test.sh setup
...
inserting docs cycle 2000/2000
{"ok":true,"id":"a479f05c01bf22c5168ee73617783eb7","rev":"1-f7fac9a3f6ce538d67747271336211c4"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784bb6","rev":"1-7e3d2b01acbc416b60e6e5be050dd790"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617784e38","rev":"1-c959b425a724ad6346f646d8a63da6cd"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617785d5b","rev":"1-d8bd0d00d0b04b52796cfe73d9a80760"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617786743","rev":"1-24f6bc4fcfdceae7ff2fae47a68fbf42"}
{"ok":true,"id":"a479f05c01bf22c5168ee73617787715","rev":"1-b0dfc903b7f27dc98130c8d51931f5d4"}
$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"84f2eae60ea04fe192284667cb0e4640","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10
real 0m36.391s
user 0m0.014s
sys 0m0.010s
Round: 2/10
real 0m36.426s
user 0m0.009s
sys 0m0.015s
Round: 3/10
real 0m37.448s
user 0m0.015s
sys 0m0.010s
Round: 4/10
real 0m37.182s
user 0m0.010s
sys 0m0.016s
Round: 5/10
real 0m34.802s
user 0m0.013s
sys 0m0.013s
Round: 6/10
real 0m34.877s
user 0m0.009s
sys 0m0.016s
Round: 7/10
real 0m34.204s
user 0m0.007s
sys 0m0.017s
Round: 8/10
real 0m34.092s
user 0m0.009s
sys 0m0.015s
Round: 9/10
real 0m35.571s
user 0m0.014s
sys 0m0.011s
Round: 10/10
real 0m35.790s
user 0m0.010s
sys 0m0.015s
================ Query CouchDB 3.1.1 patch jan ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"6c2d01a","uuid":"7df6b96ce9f321de839f65a9a6f264f3","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3004
Round: 1/10
real 0m11.194s
user 0m0.012s
sys 0m0.010s
Round: 2/10
real 0m11.224s
user 0m0.014s
sys 0m0.011s
Round: 3/10
real 0m10.966s
user 0m0.012s
sys 0m0.012s
Round: 4/10
real 0m11.212s
user 0m0.013s
sys 0m0.012s
Round: 5/10
real 0m10.696s
user 0m0.016s
sys 0m0.009s
Round: 6/10
real 0m10.859s
user 0m0.013s
sys 0m0.012s
Round: 7/10
real 0m11.342s
user 0m0.012s
sys 0m0.013s
Round: 8/10
real 0m11.276s
user 0m0.011s
sys 0m0.013s
Round: 9/10
real 0m11.357s
user 0m0.015s
sys 0m0.009s
Round: 10/10
real 0m11.542s
user 0m0.014s
sys 0m0.010s
```
That's actually great news, we have an average response time from about 11.2s now compared to 35.7s on the vanilla image :-)
If possible, I would vote for including your patch in a new 3.x build, as it would be more than 3 times faster for the same queries with unchanged client code (at least for the given dataset).
Still, nothing seem to beat build-in reducers, but at least it would not be way slower to run customer reducers on CouchDB 3 vs. CouchDB 2 installations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-904550385
Thanks @nickva and sorry for my delayed response, it took me a while to gather the right data to check if that change would be feasible for our application code.
TLDR: The proposed solution works and is about 130 times faster on the given dataset, that are great results, thanks so much for your input :-)
## View adjustments
I tried to use the build-in reducers for a small set of documents on CouchDB 3.
For the testing out your proposal, I created 7 documents, all shared the same aDefId to return one single reduced output object. I have one document per execution result and an additional document which executed successful to be able to check the computation.
The original view was using this map function:
```javascript
function (doc) {
if(doc.object === "ainst" && doc.result) {
var subscription = doc.subscription || doc.tenant;
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 1,
execTimeSum : 0,
execTimeCount : 0,
execTimeMin : Infinity,
execTimeMax : 0
}
if (doc.result.status === "successful") {
stats.successfulCount = 1;
if (doc.result.executionTime) {
stats.execTimeSum = doc.result.executionTime;
stats.execTimeCount = 1;
stats.execTimeMin = doc.result.executionTime;
stats.execTimeMax = doc.result.executionTime;
}
} else if (doc.result.status === "failed") {
stats.failedCount = 1;
} else if (doc.result.status === "canceled") {
stats.canceledCount = 1;
} else if (doc.result.status === "executing") {
stats.executingCount = 1;
} else if (doc.result.status === "unsuccessful") {
stats.unsuccessfulCount = 1;
} else if (doc.result.status === "unknown") {
stats.unknownCount = 1;
}
emit([subscription, doc.aDefId], stats);
}
}
```
and the following custom reduce function:
```javascript
function (keys, values, rereduce) {
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 0,
execTimeSum : 0,
execTimeCount : 0,
execTimeMin : Infinity,
execTimeMax : 0
}
for (var index in values) {
stats.successfulCount += values[index].successfulCount;
stats.failedCount += values[index].failedCount;
stats.canceledCount += values[index].canceledCount;
stats.executingCount += values[index].executingCount;
stats.unsuccessfulCount += values[index].unsuccessfulCount;
stats.unknownCount += values[index].unknownCount;
stats.totalCount += values[index].totalCount;
if (values[index].execTimeCount) {
stats.execTimeSum += values[index].execTimeSum;
stats.execTimeCount += values[index].execTimeCount;
stats.execTimeMin = Math.min(stats.execTimeMin, values[index].execTimeMin);
stats.execTimeMax = Math.max(stats.execTimeMax, values[index].execTimeMax);
}
}
return stats;
}
```
The original view returned the following as response which I will use to validate the result of the following adjustments:
```bash
$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo/_view/slow?reduce=true&group=true" | jq "."
{
"rows": [
{
"key": [
"xxx",
"135600b7487db5804c1961408a115e0270e5"
],
"value": {
"successfulCount": 2,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 7,
"execTimeSum": 1557,
"execTimeCount": 2,
"execTimeMin": 200,
"execTimeMax": 1357
}
}
]
}
```
As you mentioned, the build-in _sum reducer can be used to count different executions in a single view.
This is my updated demo-build-in-reduce/count view:
```javascript
function (doc) {
if(doc.object === "ainst" && doc.result) {
var subscription = doc.subscription || doc.tenant;
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 1
}
if (doc.result.status === "successful") {
stats.successfulCount = 1;
} else if (doc.result.status === "failed") {
stats.failedCount = 1;
} else if (doc.result.status === "canceled") {
stats.canceledCount = 1;
} else if (doc.result.status === "executing") {
stats.executingCount = 1;
} else if (doc.result.status === "unsuccessful") {
stats.unsuccessfulCount = 1;
} else if (doc.result.status === "unknown") {
stats.unknownCount = 1;
}
emit([subscription, doc.aDefId], stats);
}
}
```
The reduce function is set to _sum on that view.
When I query that view, I get results as expected for the counts:
```bash
$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo-build-in-reduce/_view/count?reduce=true&group=true" | jq "."
{
"rows": [
{
"key": [
"xxx",
"135600b7487db5804c1961408a115e0270e5"
],
"value": {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 2,
"totalCount": 7,
"unknownCount": 1,
"unsuccessfulCount": 1
}
}
]
}
```
The response looks good, but as expected is lacking the execution time stats. This can be done with a second view which I named demo-build-in-reduce/successfulExecutionTime.
The second view returns the following:
```bash
$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true" | jq "."
{
"rows": [
{
"key": [
"xxx",
"135600b7487db5804c1961408a115e0270e5"
],
"value": {
"sum": 1557,
"count": 2,
"min": 200,
"max": 1357,
"sumsqr": 1881449
}
}
]
}
```
This looks good either, I do not care for the sumsqr, but for sum, count, min and max. The count is also included in the demo-build-in-reduce/count view, which can be used in the application theoretically to render a more high level view of data. The successfulExecutionTime view is providing the other data.
So, your proposal works for the given data model, thank you very much. The downside is that to keep our API compatible, we need to submit two requests concurrently instead of using the single original view. Nevertheless, the performance is way more consistent, let me share a few measurements based on the new views.
## Test setup to measure performance improvements of the new implementation
I used the initial setup script to deploy both CouchDBs and to load data.
Afterwards I injected the design doc into the demo database:
```bash
$ curl -X POST \
"http://admin:password@localhost:3003/demo" \
-H "Content-Type: application/json" \
-d '{
"_id": "_design/demo-build-in-reduce",
"views": {
"count": {
"reduce": "_sum",
"map": "function (doc) {\n if(doc.object === \"ainst\" && doc.result) {\n var subscription = doc.subscription || doc.tenant;\n var stats = {\n successfulCount : 0,\n failedCount : 0,\n canceledCount : 0,\n executingCount : 0,\n unsuccessfulCount : 0,\n unknownCount : 0,\n totalCount : 1\n }\n\n if (doc.result.status === \"successful\") {\n stats.successfulCount = 1;\n } else if (doc.result.status === \"failed\") {\n stats.failedCount = 1;\n } else if (doc.result.status === \"canceled\") {\n stats.canceledCount = 1;\n } else if (doc.result.status === \"executing\") {\n stats.e
xecutingCount = 1;\n } else if (doc.result.status === \"unsuccessful\") {\n stats.unsuccessfulCount = 1;\n } else if (doc.result.status === \"unknown\") {\n stats.unknownCount = 1;\n }\n\n emit([subscription, doc.aDefId], stats);\n }\n }"
},
"successfulExecutionTime": {
"reduce": "_stats",
"map": "function (doc) {\n if(doc.object === \"ainst\" && doc.result && doc.result.status === \"successful\") {\n var subscription = doc.subscription || doc.tenant;\n emit([subscription, doc.aDefId], doc.result.executionTime);\n }\n}"
}
},
"language": "javascript"
}'
{"ok":true,"id":"_design/demo-build-in-reduce","rev":"1-e769ee9d7efdeb5f80075e16a1113968"}
```
I used the following Node.js script to query data and combine them within the application.
[reducers-demo.zip](https://github.com/apache/couchdb/files/7038700/reducers-demo.zip)
```javascript
'use strict';
const parallel = require('async').parallel;
const http = require('http');
const user = 'admin';
const password = 'password';
const port = 3003;
const dbname = 'demo';
let newRequest = (async () => {
console.log("==================== New approach ====================");
for( let i=1; i<=10; i++) {
console.log(`===== Iteration ${i} ======`);
console.time(`Fast iteration ${i} took`);
let responseMapCount = {};
let responseMapSuccessfulExecutionTime = {};
await parallel([
(cb) => {
let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo-build-in-reduce/_view/count?reduce=true&group=true`;
console.log(`Requesting: ${currentUrl}`)
let req = http.get(currentUrl, (res) => {
console.log('statusCode:', res.statusCode);
let body = ""
res.on('data', (d) => {
body += d;
});
res.on('end', function () {
// store rows in map to combine them after http requests are completed
body = JSON.parse(body);
body.rows.forEach(currentRow => {
responseMapCount[currentRow.key.join('-')] = currentRow.value;
});
cb()
});
}).on('error', (e) => {
console.error(e);
cb(e);
});
req.end();
},
(cb) => {
let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true`;
console.log(`Requesting: ${currentUrl}`)
let req = http.get(currentUrl, (res) => {
console.log('statusCode:', res.statusCode);
let body = '';
res.on('data', (d) => {
body += d;
});
res.on('end', function () {
// store rows in map to combine them after http requests are completed
body = JSON.parse(body);
body.rows.forEach(currentRow => {
responseMapSuccessfulExecutionTime[currentRow.key.join('-')] = currentRow.value;
});
cb()
});
}).on('error', (e) => {
console.error(e);
cb(e);
});
req.end();
}
]);
console.log(`Combining results`);
// iterate over responseMapCount, as there is always an entry for every ainst result
let lastMergedKey;
Object.keys(responseMapCount).forEach((currentKey) => {
// check if the aInst result had a successful invocation, merge it back in that case
if(responseMapSuccessfulExecutionTime[currentKey]) {
// merge second result property back into responseMapCount
responseMapCount[currentKey].execTimeSum = responseMapSuccessfulExecutionTime[currentKey].sum;
responseMapCount[currentKey].execTimeMin = responseMapSuccessfulExecutionTime[currentKey].min;
responseMapCount[currentKey].execTimeMax = responseMapSuccessfulExecutionTime[currentKey].max;
}
});
console.log(`Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: ${JSON.stringify(responseMapCount['xxx-100200b7487db5804c1961408a115e0270e5'], null, 4)}`)
console.timeEnd(`Fast iteration ${i} took`);
}
});
let oldRequest = (async () => {
console.log("==================== Old approach ====================");
for( let i=1; i<=10; i++) {
console.log(`===== Iteration ${i} ======`);
console.time(`Slow iteration ${i} took`);
let results = await parallel([
(cb) => {
let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo/_view/slow?reduce=true&group=true`;
console.log(`Requesting: ${currentUrl}`)
let req = http.get(currentUrl, (res) => {
console.log('statusCode:', res.statusCode);
let body = ''
res.on('data', (d) => {
body += d;
});
res.on('end', function () {
body = JSON.parse(body);
cb(null, body);
});
}).on('error', (e) => {
console.error(e);
cb(e);
});
req.end();
}
]);
console.log(`Result for aDefId=100200b7487db5804c1961408a115e0270e5:`);
// the view could have been invoked directly with the key, but all results should be retrieved to measure the performance
// only a single result is picked to compare those are equal in both query approaches.
results[0].rows.forEach((currentRow) => {
if(currentRow.key[1] === '100200b7487db5804c1961408a115e0270e5') {
console.log(JSON.stringify(currentRow.value, null, 4));
}
})
console.timeEnd(`Slow iteration ${i} took`);
}
});
(async () => {
await newRequest();
await oldRequest();
})()
```
The output shows an impressive difference, even if the application logic gets more complicated:
```bash
==================== New approach ====================
===== Iteration 1 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 1 took: 270.495ms
===== Iteration 2 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 2 took: 262.454ms
===== Iteration 3 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 3 took: 261.309ms
===== Iteration 4 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 4 took: 271.7ms
===== Iteration 5 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 5 took: 280.972ms
===== Iteration 6 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 6 took: 263.916ms
===== Iteration 7 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 7 took: 291.969ms
===== Iteration 8 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 8 took: 308.858ms
===== Iteration 9 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 9 took: 267.336ms
===== Iteration 10 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 10 took: 259.752ms
==================== Old approach ====================
===== Iteration 1 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 1 took: 34.973s
===== Iteration 2 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 2 took: 35.636s
===== Iteration 3 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 3 took: 35.189s
===== Iteration 4 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 4 took: 37.665s
===== Iteration 5 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 5 took: 37.669s
===== Iteration 6 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 6 took: 35.679s
===== Iteration 7 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 7 took: 35.368s
===== Iteration 8 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 8 took: 34.711s
===== Iteration 9 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 9 took: 34.831s
===== Iteration 10 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 10 took: 35.418s
```
## Conclusion
On average (10 iterations), I could get combined results of all records within 274 ms compared to 35714 ms, so the workaround on my given dataset is actually ~ 130 times faster.
I repeated the test for CouchDB 2 as well and got an average of 447 ms with the new approach compared to 6917 ms which is still more than 15 times faster. So independent on the question if we should migrate to CouchDB 3 or not, it might still be a useful optimization also for CouchDB 2 setups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-891897388
Discussing this on couchdb-dev channel, CouchDB with JS runtimes later than mozjs 60 need to re-write their design doc functions [1]. Also, unlike pure map views, where the JS runtime gets to process (learn) the function first, then only handle the keys and values [2], custom reduce functions have the function body sent with each query server command. Therefore, any overhead in that JS code re-writing is amplified for the custom reducers.
Looking back at the reduce function, I wonder if it would be possible to rewrite it using two _sum[3] and _stats[4] built-in reducers, which should not suffer from the same rewriting overhead.
For example, the _sum reducer can be applied to objects, such as:
```
"views":{"v1":{ "map": "function(doc){ emit(null, {x:doc.x, y:doc.y}) };", "reduce":"_sum"}}
...
{"x": 1, "y": 3}
{"x": 2, "y": 4}
```
And would emit:
```
{"key":null,"value":{"x":3,"y":7}}
```
That could be used for "unsuccessful" results. For the "successful" case, execution stats could be accumulated using _stats [3]
[1] https://github.com/apache/couchdb/blob/0059b8f90e58e10b199a4b768a06a762d12a30d3/share/server/60/rewrite_fun.js#L29-L38
[2] https://docs.couchdb.org/en/stable/query-server/protocol.html?highlight=add_fun#add-fun
[3] https://docs.couchdb.org/en/stable/ddocs/ddocs.html#_sum
[4] https://docs.couchdb.org/en/stable/ddocs/ddocs.html#_stats
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] AltenTMAOrion commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
AltenTMAOrion commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-988617458
Hello,
Do you have any ideas on a date of integration for this patch?
We are stuck on couchdb 2.3.1 because of the performance on the map/ reduce design, as our application is mainly based on this feature
Thank
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-870453565
Friendly reminder, if anyone could have a look, please?
It is blocking us to upgrade to CouchDB v3 at the moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-874308851
Thanks for trying with q=8, @konrad-ohms. The JS engine could be responsible as you indicated then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
nickva edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-890372099
I confirmed that the JS engine is implicated. I ran 2.3.1 and 3.x latest on Ubuntu 20.04. For 3.x I installed the libmozjs185 couchdb-build package for Ubuntu 18.04 on my Ubuntu 20.04 system and ran with q=8 to minimize difference and similar performance numbers as 2.3.1.
Packages I downloaded and installed manually:
```
couch-libmozjs185-1.0_1.8.5-1.0.0+couch-2_bionic_amd64.deb
couch-libmozjs185-dev_1.8.5-1.0.0+couch-2_bionic_amd64.deb
libffi6_3.2.1-8_amd64.deb
libffi-dev_3.2.1-8_amd64.deb
```
(libffi6 was needed as a dependency of couch-libmozjs185)
Configure 3.x to build with version 1.8.5
```
./configure --spidermonkey-version 1.8.5 --dev
```
Test script was modified to not set up docker containers, just run locally with 2.3.1 and 3.1.1 running on different ports.
```
./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"fake_uuid_for_dev","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15994
Round: 1/10
real 0m13.022s
user 0m0.023s
sys 0m0.023s
Round: 2/10
real 0m7.234s
user 0m0.020s
sys 0m0.008s
Round: 3/10
real 0m6.715s
user 0m0.026s
sys 0m0.000s
Round: 4/10
real 0m7.021s
user 0m0.018s
sys 0m0.009s
Round: 5/10
real 0m7.159s
user 0m0.005s
sys 0m0.021s
Round: 6/10
real 0m7.318s
user 0m0.014s
sys 0m0.014s
Round: 7/10
real 0m7.464s
user 0m0.007s
sys 0m0.021s
Round: 8/10
real 0m7.175s
user 0m0.016s
sys 0m0.011s
Round: 9/10
real 0m7.096s
user 0m0.004s
sys 0m0.023s
Round: 10/10
real 0m7.235s
user 0m0.006s
sys 0m0.022s
```
```
q=8
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1-ba63878","git_sha":"ba63878","uuid":"fake_uuid_for_dev","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15984
Round: 1/10
real 0m5.979s
user 0m0.017s
sys 0m0.010s
Round: 2/10
real 0m6.757s
user 0m0.013s
sys 0m0.013s
Round: 3/10
real 0m7.112s
user 0m0.012s
sys 0m0.016s
Round: 4/10
real 0m7.359s
user 0m0.017s
sys 0m0.011s
Round: 5/10
real 0m7.210s
user 0m0.017s
sys 0m0.013s
Round: 6/10
real 0m7.233s
user 0m0.008s
sys 0m0.021s
Round: 7/10
real 0m6.284s
user 0m0.028s
sys 0m0.000s
Round: 8/10
real 0m7.101s
user 0m0.020s
sys 0m0.008s
Round: 9/10
real 0m6.943s
user 0m0.022s
sys 0m0.006s
Round: 10/10
real 0m6.736s
user 0m0.016s
sys 0m0.011s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-874168606
Thanks @nickva for investigating the issue.
I tried to adjust the function from
```
setupDbs() {
COUCHDB_PORT=${1}
waitForDb "${COUCHDB_PORT}"
SYSTEM_DBS=( _users _replicator _global_changes demo )
for CURRENT_DB in "${SYSTEM_DBS[@]}"
do
echo "Creating ${CURRENT_DB}"
curl -X PUT "http://${COUCHDB_USER}:${COUCHDB_PASSWORD}@127.0.0.1:${COUCHDB_PORT}/${CURRENT_DB}"
done
}
```
to
```
setupDbs() {
COUCHDB_PORT=${1}
waitForDb "${COUCHDB_PORT}"
SYSTEM_DBS=( _users _replicator _global_changes demo )
for CURRENT_DB in "${SYSTEM_DBS[@]}"
do
echo "Creating ${CURRENT_DB}"
curl -X PUT "http://${COUCHDB_USER}:${COUCHDB_PASSWORD}@127.0.0.1:${COUCHDB_PORT}/${CURRENT_DB}?q=8"
done
}
```
Unfortunately that did not solve the performance degradation of CouchDB 3.1.1 compared to v2.3.1.
I checked that the databases are using the same number of shards:
CouchDB 2:
```
ohmsk:~$ curl -u "xxx:xxx" http://localhost:3002/demo/
{"db_name":"demo","purge_seq":"0-g1AAAAEzeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgYg9f____lZiQx41R2AqLuPW11SApBMqgcqImTWA4hZeNQlKYDMsifCrAUQs_bjVpfIkCSP3zIHkGXxEDVZAMg5YvY","update_seq":"12001-g1AAAAFLeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjMlMiTJ____PyuJgYH1Cw41SQpAMskeqoyNFZcyB5CyeJhpD3EpSwApq4cpO4hDWR4LkGRoAFJAlfPBStfjVboAonQ_WOl-vEoPQJTex-triNIHEKUQtz7JAgDbh2kI","sizes":{"file":11425440,"external":11923785,"active":11287083},"other":{"data_size":11923785},"doc_del_count":0,"doc_count":12001,"disk_size":11425440,"disk_format_version":7,"data_size":11287083,"compact_running":false,"cluster":{"q":8,"n":1,"w":1,"r":1},"instance_start_time":"0"}
ohmsk:~$ curl -u "xxx:xxx" http://localhost:3003/demo/
```
CouchDB 3:
```
{"db_name":"demo","purge_seq":"0-g1AAAAEzeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgdA6j8QZCUy4FV3AKLuPiF1CyDq9hNS1wBRNx-3uqQEIJlUj9dtSQ4gNfH41SiA1NjjVZPIkCQPUZAFAN6AYvY","update_seq":"12001-g1AAAAFLeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgdA6j8QZCUxMLDew6v0AETpfbDSg3iVLoAo3Q9W-gKv0gaI0vlgpa9wKE1KAJJJ9TB33sGlzAGkLB6mDKdpCiBl9jBll3AoS2RIkoepuZEFAAsmagc","sizes":{"file":14468808,"external":9785687,"active":11343864},"props":{},"doc_del_count":0,"doc_count":12001,"disk_format_version":8,"compact_running":false,"cluster":{"q":8,"n":1,"w":1,"r":1},"instance_start_time":"0"}
```
Both state:
```
"cluster":{"q":8,"n":1,"w":1,"r":1}
```
After injecting the data I waited multiple minutes before running the queries:
```
ohmsk:~/Downloads/couchdb$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"5584fb6a7a94ff8ab44bbee93823352d","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3002
Round: 1/10
real 0m8.534s
user 0m0.017s
sys 0m0.013s
Round: 2/10
real 0m5.050s
user 0m0.012s
sys 0m0.010s
Round: 3/10
real 0m4.677s
user 0m0.010s
sys 0m0.011s
Round: 4/10
real 0m4.820s
user 0m0.013s
sys 0m0.009s
Round: 5/10
real 0m4.798s
user 0m0.011s
sys 0m0.011s
Round: 6/10
real 0m5.093s
user 0m0.013s
sys 0m0.009s
Round: 7/10
real 0m5.285s
user 0m0.015s
sys 0m0.007s
Round: 8/10
real 0m5.627s
user 0m0.010s
sys 0m0.014s
Round: 9/10
real 0m5.459s
user 0m0.013s
sys 0m0.010s
Round: 10/10
real 0m5.589s
user 0m0.009s
sys 0m0.015s
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"6445c39f1422b264fbf7837dbb055bb9","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10
real 0m53.993s
user 0m0.014s
sys 0m0.011s
Round: 2/10
real 0m53.707s
user 0m0.016s
sys 0m0.009s
Round: 3/10
real 0m56.130s
user 0m0.011s
sys 0m0.016s
Round: 4/10
real 0m56.537s
user 0m0.012s
sys 0m0.014s
Round: 5/10
real 0m55.933s
user 0m0.012s
sys 0m0.013s
Round: 6/10
real 0m52.887s
user 0m0.013s
sys 0m0.014s
Round: 7/10
real 1m1.758s
user 0m0.012s
sys 0m0.012s
Round: 8/10
real 0m52.182s
user 0m0.010s
sys 0m0.018s
Round: 9/10
real 0m53.072s
user 0m0.011s
sys 0m0.016s
Round: 10/10
real 0m53.720s
user 0m0.014s
sys 0m0.013s
```
I am not sure if the default log level does not indicate the auto-builder and auto-compactor actions, but at least the container logs are not reflecting such actions:
```
$ podman logs -f couchdb3
...
[notice] 2021-07-05T12:34:42.581798Z nonode@nohost <0.19894.3> 14ea633501 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 19
[notice] 2021-07-05T12:34:42.618414Z nonode@nohost <0.19902.3> 96877188fc 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 22
[notice] 2021-07-05T12:34:42.650066Z nonode@nohost <0.19910.3> c2fe735612 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 20
[info] 2021-07-05T12:35:10.329291Z nonode@nohost <0.3652.0> -------- Starting index update for db: shards/40000000-5fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:11.749276Z nonode@nohost <0.3652.0> -------- Index update finished for db: shards/40000000-5fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:11.936249Z nonode@nohost <0.3778.0> -------- Starting index update for db: shards/00000000-1fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:13.008010Z nonode@nohost <0.3778.0> -------- Index update finished for db: shards/00000000-1fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:13.291704Z nonode@nohost <0.3817.0> -------- Starting index update for db: shards/e0000000-ffffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:14.564826Z nonode@nohost <0.3817.0> -------- Index update finished for db: shards/e0000000-ffffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.314002Z nonode@nohost <0.3907.0> -------- Starting index update for db: shards/20000000-3fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.373015Z nonode@nohost <0.3975.0> -------- Starting index update for db: shards/60000000-7fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.510076Z nonode@nohost <0.3621.0> -------- Starting index update for db: shards/c0000000-dfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.010943Z nonode@nohost <0.4169.0> -------- Starting index update for db: shards/a0000000-bfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.654203Z nonode@nohost <0.3975.0> -------- Index update finished for db: shards/60000000-7fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.677526Z nonode@nohost <0.3907.0> -------- Index update finished for db: shards/20000000-3fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.788062Z nonode@nohost <0.3621.0> -------- Index update finished for db: shards/c0000000-dfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:17.306988Z nonode@nohost <0.4169.0> -------- Index update finished for db: shards/a0000000-bfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:18.494925Z nonode@nohost <0.3686.0> -------- Starting index update for db: shards/80000000-9fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:19.388880Z nonode@nohost <0.3686.0> -------- Index update finished for db: shards/80000000-9fffffff/demo.1625487748 idx: _design/demo
[notice] 2021-07-05T14:32:23.887866Z nonode@nohost <0.13346.5> 61a2177aae 127.0.0.1:3003 10.0.2.100 admin GET / 200 ok 9
[notice] 2021-07-05T14:33:17.881591Z nonode@nohost <0.13347.5> 39c531c67e 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53986
[notice] 2021-07-05T14:34:11.588538Z nonode@nohost <0.13889.5> d3cc5e0fa8 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53700
[notice] 2021-07-05T14:35:07.718592Z nonode@nohost <0.14346.5> 05b6354796 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 56122
[notice] 2021-07-05T14:36:04.255503Z nonode@nohost <0.14842.5> 1535baf654 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 56529
[notice] 2021-07-05T14:37:00.187963Z nonode@nohost <0.15334.5> 00a9c90ecf 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 55926
[notice] 2021-07-05T14:37:53.074959Z nonode@nohost <0.15818.5> e48cbf750a 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 52880
[notice] 2021-07-05T14:38:54.833050Z nonode@nohost <0.16272.5> 07b2646c16 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 61751
[notice] 2021-07-05T14:39:47.015319Z nonode@nohost <0.16808.5> a26c18d19d 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 52175
[notice] 2021-07-05T14:40:40.086929Z nonode@nohost <0.17255.5> 5267817a12 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53065
[notice] 2021-07-05T14:41:33.807566Z nonode@nohost <0.17723.5> 21e5c4941c 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53713
```
Compared to couchdb2:
```
$ podman logs couchdb2
...
[notice] 2021-07-05T12:27:09.139663Z nonode@nohost <0.23495.2> 714b3e7a37 127.0.0.1:3002 10.0.2.100 admin POST /demo 201 ok 30
[notice] 2021-07-05T12:27:09.165180Z nonode@nohost <0.23502.2> 9a12b27d94 127.0.0.1:3002 10.0.2.100 admin POST /demo 201 ok 18
[info] 2021-07-05T13:22:11.293974Z nonode@nohost <0.12126.4> -------- Starting compaction for db "shards/20000000-3fffffff/demo.1625487744" at 1541
[info] 2021-07-05T13:22:32.858211Z nonode@nohost <0.12139.4> -------- Starting compaction for db "shards/00000000-1fffffff/demo.1625487744" at 1524
[info] 2021-07-05T13:22:54.395354Z nonode@nohost <0.12142.4> -------- Starting compaction for db "shards/40000000-5fffffff/demo.1625487744" at 1505
[info] 2021-07-05T13:23:15.835989Z nonode@nohost <0.12145.4> -------- Starting compaction for db "shards/60000000-7fffffff/demo.1625487744" at 1473
[info] 2021-07-05T13:23:37.277484Z nonode@nohost <0.12151.4> -------- Starting compaction for db "shards/a0000000-bfffffff/demo.1625487744" at 1471
[info] 2021-07-05T13:23:58.739812Z nonode@nohost <0.12157.4> -------- Starting compaction for db "shards/e0000000-ffffffff/demo.1625487744" at 1508
[info] 2021-07-05T13:24:20.179296Z nonode@nohost <0.12148.4> -------- Starting compaction for db "shards/80000000-9fffffff/demo.1625487744" at 1455
[info] 2021-07-05T13:24:41.714250Z nonode@nohost <0.12154.4> -------- Starting compaction for db "shards/c0000000-dfffffff/demo.1625487744" at 1524
[info] 2021-07-05T13:24:54.231782Z nonode@nohost <0.12450.4> -------- Starting compaction for db "shards/c0000000-dfffffff/_global_changes.1625487744" at 554
[notice] 2021-07-05T14:31:28.936974Z nonode@nohost <0.15912.6> 312602f8c0 127.0.0.1:3002 10.0.2.100 admin GET / 200 ok 2
[info] 2021-07-05T14:31:28.989213Z nonode@nohost <0.15983.6> -------- Starting index update for db: shards/00000000-1fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.989257Z nonode@nohost <0.15984.6> -------- Starting index update for db: shards/20000000-3fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990506Z nonode@nohost <0.16010.6> -------- Starting index update for db: shards/40000000-5fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990507Z nonode@nohost <0.16011.6> -------- Starting index update for db: shards/a0000000-bfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990573Z nonode@nohost <0.16012.6> -------- Starting index update for db: shards/60000000-7fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990650Z nonode@nohost <0.16016.6> -------- Starting index update for db: shards/80000000-9fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990715Z nonode@nohost <0.16020.6> -------- Starting index update for db: shards/c0000000-dfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.991237Z nonode@nohost <0.16025.6> -------- Starting index update for db: shards/e0000000-ffffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.578442Z nonode@nohost <0.16012.6> -------- Index update finished for db: shards/60000000-7fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.592533Z nonode@nohost <0.16025.6> -------- Index update finished for db: shards/e0000000-ffffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.624614Z nonode@nohost <0.16011.6> -------- Index update finished for db: shards/a0000000-bfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.630679Z nonode@nohost <0.16010.6> -------- Index update finished for db: shards/40000000-5fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.641088Z nonode@nohost <0.16016.6> -------- Index update finished for db: shards/80000000-9fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.682452Z nonode@nohost <0.15984.6> -------- Index update finished for db: shards/20000000-3fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.729299Z nonode@nohost <0.16020.6> -------- Index update finished for db: shards/c0000000-dfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.746659Z nonode@nohost <0.15983.6> -------- Index update finished for db: shards/00000000-1fffffff/demo.1625487744 idx: _design/demo
[notice] 2021-07-05T14:31:37.472657Z nonode@nohost <0.15913.6> 3b98ddbf30 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 8516
[notice] 2021-07-05T14:31:42.522912Z nonode@nohost <0.16231.6> 139c6ac71b 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5042
[notice] 2021-07-05T14:31:47.199986Z nonode@nohost <0.16315.6> 63625be9aa 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4671
[notice] 2021-07-05T14:31:52.020073Z nonode@nohost <0.16417.6> 06140e6373 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4813
[notice] 2021-07-05T14:31:56.818616Z nonode@nohost <0.16495.6> 73aef1519b 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4791
[notice] 2021-07-05T14:32:01.911785Z nonode@nohost <0.16590.6> e7371ec8db 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5086
[notice] 2021-07-05T14:32:07.196507Z nonode@nohost <0.16725.6> 19201fdd74 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5278
[notice] 2021-07-05T14:32:12.823661Z nonode@nohost <0.16833.6> 1cfc7a7f8a 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5618
[notice] 2021-07-05T14:32:18.282645Z nonode@nohost <0.16924.6> a70a1462dc 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5453
[notice] 2021-07-05T14:32:23.871410Z nonode@nohost <0.17021.6> 8364bc2506 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5581
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-874168606
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] janl commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
janl commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-914376860
@konrad-ohms would you be able to once more test this PR: https://github.com/apache/couchdb/pull/3728 — it uses SHA-256 over crc32, and thus is production ready, we just wanna make sure this still speeds you up :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] janl commented on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
janl commented on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-919304082
Thanks for the update! We expected a reduction in performance, but this is quite a jump. Maybe we can think about making this configurable, most folks only ever run trusted design docs anyway and a simpler and faster hashing function might do the trick.
After all, this might not make it into 3.2, but we will remain on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
nickva edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-890372099
I confirmed that the JS engine is implicated. I ran 2.3.1 and 3.x latest on Ubuntu 20.04. For 3.x I installed the libmozjs185 couchdb-build package for Ubuntu 18.04 on my Ubuntu 20.04 system and ran with q=8 to minimize difference and similar performance numbers as 2.3.1.
Packages I downloaded and installed manually:
```
couch-libmozjs185-1.0_1.8.5-1.0.0+couch-2_bionic_amd64.deb
couch-libmozjs185-dev_1.8.5-1.0.0+couch-2_bionic_amd64.deb
libffi6_3.2.1-8_amd64.deb
libffi-dev_3.2.1-8_amd64.deb
```
(libffi6 was needed as a dependency of couch-libmozjs185)
Configure 3.x to build with version 1.8.5
```
./configure --spidermonkey-version 1.8.5 --dev
```
Test script was modified to not set up docker containers, just run locally with 2.3.1 and 3.1.1 running on different ports.
```
./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"fake_uuid_for_dev","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15994
Round: 1/10
real 0m13.022s
user 0m0.023s
sys 0m0.023s
Round: 2/10
real 0m7.234s
user 0m0.020s
sys 0m0.008s
Round: 3/10
real 0m6.715s
user 0m0.026s
sys 0m0.000s
Round: 4/10
real 0m7.021s
user 0m0.018s
sys 0m0.009s
Round: 5/10
real 0m7.159s
user 0m0.005s
sys 0m0.021s
Round: 6/10
real 0m7.318s
user 0m0.014s
sys 0m0.014s
Round: 7/10
real 0m7.464s
user 0m0.007s
sys 0m0.021s
Round: 8/10
real 0m7.175s
user 0m0.016s
sys 0m0.011s
Round: 9/10
real 0m7.096s
user 0m0.004s
sys 0m0.023s
Round: 10/10
real 0m7.235s
user 0m0.006s
sys 0m0.022s
```
```
q=8
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1-ba63878","git_sha":"ba63878","uuid":"fake_uuid_for_dev","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15984
Round: 1/10
real 0m5.979s
user 0m0.017s
sys 0m0.010s
Round: 2/10
real 0m6.757s
user 0m0.013s
sys 0m0.013s
Round: 3/10
real 0m7.112s
user 0m0.012s
sys 0m0.016s
Round: 4/10
real 0m7.359s
user 0m0.017s
sys 0m0.011s
Round: 5/10
real 0m7.210s
user 0m0.017s
sys 0m0.013s
Round: 6/10
real 0m7.233s
user 0m0.008s
sys 0m0.021s
Round: 7/10
real 0m6.284s
user 0m0.028s
sys 0m0.000s
Round: 8/10
real 0m7.101s
user 0m0.020s
sys 0m0.008s
Round: 9/10
real 0m6.943s
user 0m0.022s
sys 0m0.006s
Round: 10/10
real 0m6.736s
user 0m0.016s
sys 0m0.011s
```
With default mozjs68 library on Ubuntu I was getting around 30 seconds
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-904550385
Thanks @nickva and sorry for my delayed response, it took me a while to gather the right data to check if that change would be feasible for our application code.
TLDR: The proposed solution works and is about 130 times faster on the given dataset, that are great results, thanks so much for your input :-)
## View adjustments
I tried to use the build-in reducers for a small set of documents on CouchDB 3.
For the testing out your proposal, I created 7 documents, all shared the same aDefId to return one single reduced output object. I have one document per execution result and an additional document which executed successful to be able to check the computation.
The original view was using this map function:
```javascript
function (doc) {
if(doc.object === "ainst" && doc.result) {
var subscription = doc.subscription || doc.tenant;
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 1,
execTimeSum : 0,
execTimeCount : 0,
execTimeMin : Infinity,
execTimeMax : 0
}
if (doc.result.status === "successful") {
stats.successfulCount = 1;
if (doc.result.executionTime) {
stats.execTimeSum = doc.result.executionTime;
stats.execTimeCount = 1;
stats.execTimeMin = doc.result.executionTime;
stats.execTimeMax = doc.result.executionTime;
}
} else if (doc.result.status === "failed") {
stats.failedCount = 1;
} else if (doc.result.status === "canceled") {
stats.canceledCount = 1;
} else if (doc.result.status === "executing") {
stats.executingCount = 1;
} else if (doc.result.status === "unsuccessful") {
stats.unsuccessfulCount = 1;
} else if (doc.result.status === "unknown") {
stats.unknownCount = 1;
}
emit([subscription, doc.aDefId], stats);
}
}
```
and the following custom reduce function:
```javascript
function (keys, values, rereduce) {
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 0,
execTimeSum : 0,
execTimeCount : 0,
execTimeMin : Infinity,
execTimeMax : 0
}
for (var index in values) {
stats.successfulCount += values[index].successfulCount;
stats.failedCount += values[index].failedCount;
stats.canceledCount += values[index].canceledCount;
stats.executingCount += values[index].executingCount;
stats.unsuccessfulCount += values[index].unsuccessfulCount;
stats.unknownCount += values[index].unknownCount;
stats.totalCount += values[index].totalCount;
if (values[index].execTimeCount) {
stats.execTimeSum += values[index].execTimeSum;
stats.execTimeCount += values[index].execTimeCount;
stats.execTimeMin = Math.min(stats.execTimeMin, values[index].execTimeMin);
stats.execTimeMax = Math.max(stats.execTimeMax, values[index].execTimeMax);
}
}
return stats;
}
```
The original view returned the following as response which I will use to validate the result of the following adjustments:
```bash
$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo/_view/slow?reduce=true&group=true" | jq "."
{
"rows": [
{
"key": [
"xxx",
"135600b7487db5804c1961408a115e0270e5"
],
"value": {
"successfulCount": 2,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 7,
"execTimeSum": 1557,
"execTimeCount": 2,
"execTimeMin": 200,
"execTimeMax": 1357
}
}
]
}
```
As you mentioned, the build-in _sum reducer can be used to count different executions in a single view.
This is my updated demo-build-in-reduce/count view:
```javascript
function (doc) {
if(doc.object === "ainst" && doc.result) {
var subscription = doc.subscription || doc.tenant;
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 1
}
if (doc.result.status === "successful") {
stats.successfulCount = 1;
} else if (doc.result.status === "failed") {
stats.failedCount = 1;
} else if (doc.result.status === "canceled") {
stats.canceledCount = 1;
} else if (doc.result.status === "executing") {
stats.executingCount = 1;
} else if (doc.result.status === "unsuccessful") {
stats.unsuccessfulCount = 1;
} else if (doc.result.status === "unknown") {
stats.unknownCount = 1;
}
emit([subscription, doc.aDefId], stats);
}
}
```
The reduce function is set to _sum on that view.
When I query that view, I get results as expected for the counts:
```bash
$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo-build-in-reduce/_view/count?reduce=true&group=true" | jq "."
{
"rows": [
{
"key": [
"xxx",
"135600b7487db5804c1961408a115e0270e5"
],
"value": {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 2,
"totalCount": 7,
"unknownCount": 1,
"unsuccessfulCount": 1
}
}
]
}
```
The response looks good, but as expected is lacking the execution time stats. This can be done with a second view which I named demo-build-in-reduce/successfulExecutionTime.
The second view returns the following:
```bash
$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true" | jq "."
{
"rows": [
{
"key": [
"xxx",
"135600b7487db5804c1961408a115e0270e5"
],
"value": {
"sum": 1557,
"count": 2,
"min": 200,
"max": 1357,
"sumsqr": 1881449
}
}
]
}
```
This looks good either, I do not care for the sumsqr, but for sum, count, min and max. The count is also included in the demo-build-in-reduce/count view, which can be used in the application theoretically to render a more high level view of data. The successfulExecutionTime view is providing the other data.
So, your proposal works for the given data model, thank you very much. The downside is that to keep our API compatible, we need to submit two requests concurrently instead of using the single original view. Nevertheless, the performance is way more consistent, let me share a few measurements based on the new views.
## Test setup to measure performance improvements of the new implementation
I used the initial setup script to deploy both CouchDBs and to load data.
Afterwards I injected the design doc into the demo database:
```bash
$ curl -X POST \
"http://admin:password@localhost:3003/demo" \
-H "Content-Type: application/json" \
-d '{
"_id": "_design/demo-build-in-reduce",
"views": {
"count": {
"reduce": "_sum",
"map": "function (doc) {\n if(doc.object === \"ainst\" && doc.result) {\n var subscription = doc.subscription || doc.tenant;\n var stats = {\n successfulCount : 0,\n failedCount : 0,\n canceledCount : 0,\n executingCount : 0,\n unsuccessfulCount : 0,\n unknownCount : 0,\n totalCount : 1\n }\n\n if (doc.result.status === \"successful\") {\n stats.successfulCount = 1;\n } else if (doc.result.status === \"failed\") {\n stats.failedCount = 1;\n } else if (doc.result.status === \"canceled\") {\n stats.canceledCount = 1;\n } else if (doc.result.status === \"executing\") {\n stats.e
xecutingCount = 1;\n } else if (doc.result.status === \"unsuccessful\") {\n stats.unsuccessfulCount = 1;\n } else if (doc.result.status === \"unknown\") {\n stats.unknownCount = 1;\n }\n\n emit([subscription, doc.aDefId], stats);\n }\n }"
},
"successfulExecutionTime": {
"reduce": "_stats",
"map": "function (doc) {\n if(doc.object === \"ainst\" && doc.result && doc.result.status === \"successful\") {\n var subscription = doc.subscription || doc.tenant;\n emit([subscription, doc.aDefId], doc.result.executionTime);\n }\n}"
}
},
"language": "javascript"
}'
{"ok":true,"id":"_design/demo-build-in-reduce","rev":"1-e769ee9d7efdeb5f80075e16a1113968"}
```
I used the following Node.js script to query data and combine them within the application.
[reducers-demo.zip](https://github.com/apache/couchdb/files/7038700/reducers-demo.zip)
```javascript
'use strict';
const parallel = require('async').parallel;
const http = require('http');
const user = 'admin';
const password = 'password';
const port = 3003;
const dbname = 'demo';
let newRequest = (async () => {
console.log("==================== New approach ====================");
for( let i=1; i<=10; i++) {
console.log(`===== Iteration ${i} ======`);
console.time(`Fast iteration ${i} took`);
let responseMapCount = {};
let responseMapSuccessfulExecutionTime = {};
await parallel([
(cb) => {
let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo-build-in-reduce/_view/count?reduce=true&group=true`;
console.log(`Requesting: ${currentUrl}`)
let req = http.get(currentUrl, (res) => {
console.log('statusCode:', res.statusCode);
let body = ""
res.on('data', (d) => {
body += d;
});
res.on('end', function () {
// store rows in map to combine them after http requests are completed
body = JSON.parse(body);
body.rows.forEach(currentRow => {
responseMapCount[currentRow.key.join('-')] = currentRow.value;
});
cb()
});
}).on('error', (e) => {
console.error(e);
cb(e);
});
req.end();
},
(cb) => {
let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true`;
console.log(`Requesting: ${currentUrl}`)
let req = http.get(currentUrl, (res) => {
console.log('statusCode:', res.statusCode);
let body = '';
res.on('data', (d) => {
body += d;
});
res.on('end', function () {
// store rows in map to combine them after http requests are completed
body = JSON.parse(body);
body.rows.forEach(currentRow => {
responseMapSuccessfulExecutionTime[currentRow.key.join('-')] = currentRow.value;
});
cb()
});
}).on('error', (e) => {
console.error(e);
cb(e);
});
req.end();
}
]);
console.log(`Combining results`);
// iterate over responseMapCount, as there is always an entry for every ainst result
let lastMergedKey;
Object.keys(responseMapCount).forEach((currentKey) => {
// check if the aInst result had a successful invocation, merge it back in that case
if(responseMapSuccessfulExecutionTime[currentKey]) {
// merge second result property back into responseMapCount
responseMapCount[currentKey].execTimeSum = responseMapSuccessfulExecutionTime[currentKey].sum;
responseMapCount[currentKey].execTimeMin = responseMapSuccessfulExecutionTime[currentKey].min;
responseMapCount[currentKey].execTimeMax = responseMapSuccessfulExecutionTime[currentKey].max;
}
});
console.log(`Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: ${JSON.stringify(responseMapCount['xxx-100200b7487db5804c1961408a115e0270e5'], null, 4)}`)
console.timeEnd(`Fast iteration ${i} took`);
}
});
let oldRequest = (async () => {
console.log("==================== Old approach ====================");
for( let i=1; i<=10; i++) {
console.log(`===== Iteration ${i} ======`);
console.time(`Slow iteration ${i} took`);
let results = await parallel([
(cb) => {
let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo/_view/slow?reduce=true&group=true`;
console.log(`Requesting: ${currentUrl}`)
let req = http.get(currentUrl, (res) => {
console.log('statusCode:', res.statusCode);
let body = ''
res.on('data', (d) => {
body += d;
});
res.on('end', function () {
body = JSON.parse(body);
cb(null, body);
});
}).on('error', (e) => {
console.error(e);
cb(e);
});
req.end();
}
]);
console.log(`Result for aDefId=100200b7487db5804c1961408a115e0270e5:`);
// the view could have been invoked directly with the key, but all results should be retrieved to measure the performance
// only a single result is picked to compare those are equal in both query approaches.
results[0].rows.forEach((currentRow) => {
if(currentRow.key[1] === '100200b7487db5804c1961408a115e0270e5') {
console.log(JSON.stringify(currentRow.value, null, 4));
}
})
console.timeEnd(`Slow iteration ${i} took`);
}
});
(async () => {
await newRequest();
await oldRequest();
})()
```
The output shows an impressive difference, even if the application logic gets more complicated:
```bash
==================== New approach ====================
===== Iteration 1 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 1 took: 270.495ms
===== Iteration 2 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 2 took: 262.454ms
===== Iteration 3 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 3 took: 261.309ms
===== Iteration 4 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 4 took: 271.7ms
===== Iteration 5 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 5 took: 280.972ms
===== Iteration 6 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 6 took: 263.916ms
===== Iteration 7 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 7 took: 291.969ms
===== Iteration 8 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 8 took: 308.858ms
===== Iteration 9 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 9 took: 267.336ms
===== Iteration 10 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 10 took: 259.752ms
==================== Old approach ====================
===== Iteration 1 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 1 took: 34.973s
===== Iteration 2 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 2 took: 35.636s
===== Iteration 3 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 3 took: 35.189s
===== Iteration 4 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 4 took: 37.665s
===== Iteration 5 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 5 took: 37.669s
===== Iteration 6 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 6 took: 35.679s
===== Iteration 7 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 7 took: 35.368s
===== Iteration 8 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 8 took: 34.711s
===== Iteration 9 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 9 took: 34.831s
===== Iteration 10 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 10 took: 35.418s
```
## Conclusion
On average (10 iterations), I could get combined results of all records within 274 ms compared to 35714 ms, so the workaround on my given dataset is actually ~ 130 times faster.
I repeated the test for CouchDB 2 as well ([couchdb2-test-output](https://github.com/apache/couchdb/files/7038776/output2.txt)) and got an average of 447 ms with the new approach compared to 6917 ms which is still more than 15 times faster. So independent on the question if we should migrate to CouchDB 3 or not, it might still be a useful optimization also for CouchDB 2 setups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-874168606
Thanks @nickva for investigating the issue.
I tried to adjust the function from
```
setupDbs() {
COUCHDB_PORT=${1}
waitForDb "${COUCHDB_PORT}"
SYSTEM_DBS=( _users _replicator _global_changes demo )
for CURRENT_DB in "${SYSTEM_DBS[@]}"
do
echo "Creating ${CURRENT_DB}"
curl -X PUT "http://${COUCHDB_USER}:${COUCHDB_PASSWORD}@127.0.0.1:${COUCHDB_PORT}/${CURRENT_DB}"
done
}
```
to
```
setupDbs() {
COUCHDB_PORT=${1}
waitForDb "${COUCHDB_PORT}"
SYSTEM_DBS=( _users _replicator _global_changes demo )
for CURRENT_DB in "${SYSTEM_DBS[@]}"
do
echo "Creating ${CURRENT_DB}"
curl -X PUT "http://${COUCHDB_USER}:${COUCHDB_PASSWORD}@127.0.0.1:${COUCHDB_PORT}/${CURRENT_DB}?q=8"
done
}
```
Unfortunately that did not solve the performance degradation of CouchDB 3.1.1 compared to v2.3.1.
I checked that the databases are using the same number of shards:
CouchDB 2:
```
$ curl -u "xxx:xxx" http://localhost:3002/demo/
{"db_name":"demo","purge_seq":"0-g1AAAAEzeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgYg9f____lZiQx41R2AqLuPW11SApBMqgcqImTWA4hZeNQlKYDMsifCrAUQs_bjVpfIkCSP3zIHkGXxEDVZAMg5YvY","update_seq":"12001-g1AAAAFLeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjMlMiTJ____PyuJgYH1Cw41SQpAMskeqoyNFZcyB5CyeJhpD3EpSwApq4cpO4hDWR4LkGRoAFJAlfPBStfjVboAonQ_WOl-vEoPQJTex-triNIHEKUQtz7JAgDbh2kI","sizes":{"file":11425440,"external":11923785,"active":11287083},"other":{"data_size":11923785},"doc_del_count":0,"doc_count":12001,"disk_size":11425440,"disk_format_version":7,"data_size":11287083,"compact_running":false,"cluster":{"q":8,"n":1,"w":1,"r":1},"instance_start_time":"0"}
```
CouchDB 3:
```
$ curl -u "xxx:xxx" http://localhost:3003/demo/
{"db_name":"demo","purge_seq":"0-g1AAAAEzeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgdA6j8QZCUy4FV3AKLuPiF1CyDq9hNS1wBRNx-3uqQEIJlUj9dtSQ4gNfH41SiA1NjjVZPIkCQPUZAFAN6AYvY","update_seq":"12001-g1AAAAFLeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgdA6j8QZCUxMLDew6v0AETpfbDSg3iVLoAo3Q9W-gKv0gaI0vlgpa9wKE1KAJJJ9TB33sGlzAGkLB6mDKdpCiBl9jBll3AoS2RIkoepuZEFAAsmagc","sizes":{"file":14468808,"external":9785687,"active":11343864},"props":{},"doc_del_count":0,"doc_count":12001,"disk_format_version":8,"compact_running":false,"cluster":{"q":8,"n":1,"w":1,"r":1},"instance_start_time":"0"}
```
Both state:
```
"cluster":{"q":8,"n":1,"w":1,"r":1}
```
After injecting the data I waited multiple minutes before running the queries:
```
ohmsk:~/Downloads/couchdb$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"5584fb6a7a94ff8ab44bbee93823352d","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3002
Round: 1/10
real 0m8.534s
user 0m0.017s
sys 0m0.013s
Round: 2/10
real 0m5.050s
user 0m0.012s
sys 0m0.010s
Round: 3/10
real 0m4.677s
user 0m0.010s
sys 0m0.011s
Round: 4/10
real 0m4.820s
user 0m0.013s
sys 0m0.009s
Round: 5/10
real 0m4.798s
user 0m0.011s
sys 0m0.011s
Round: 6/10
real 0m5.093s
user 0m0.013s
sys 0m0.009s
Round: 7/10
real 0m5.285s
user 0m0.015s
sys 0m0.007s
Round: 8/10
real 0m5.627s
user 0m0.010s
sys 0m0.014s
Round: 9/10
real 0m5.459s
user 0m0.013s
sys 0m0.010s
Round: 10/10
real 0m5.589s
user 0m0.009s
sys 0m0.015s
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"6445c39f1422b264fbf7837dbb055bb9","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10
real 0m53.993s
user 0m0.014s
sys 0m0.011s
Round: 2/10
real 0m53.707s
user 0m0.016s
sys 0m0.009s
Round: 3/10
real 0m56.130s
user 0m0.011s
sys 0m0.016s
Round: 4/10
real 0m56.537s
user 0m0.012s
sys 0m0.014s
Round: 5/10
real 0m55.933s
user 0m0.012s
sys 0m0.013s
Round: 6/10
real 0m52.887s
user 0m0.013s
sys 0m0.014s
Round: 7/10
real 1m1.758s
user 0m0.012s
sys 0m0.012s
Round: 8/10
real 0m52.182s
user 0m0.010s
sys 0m0.018s
Round: 9/10
real 0m53.072s
user 0m0.011s
sys 0m0.016s
Round: 10/10
real 0m53.720s
user 0m0.014s
sys 0m0.013s
```
I am not sure if the default log level does not indicate the auto-builder and auto-compactor actions, but at least the container logs are not reflecting such actions:
```
$ podman logs -f couchdb3
...
[notice] 2021-07-05T12:34:42.581798Z nonode@nohost <0.19894.3> 14ea633501 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 19
[notice] 2021-07-05T12:34:42.618414Z nonode@nohost <0.19902.3> 96877188fc 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 22
[notice] 2021-07-05T12:34:42.650066Z nonode@nohost <0.19910.3> c2fe735612 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 20
[info] 2021-07-05T12:35:10.329291Z nonode@nohost <0.3652.0> -------- Starting index update for db: shards/40000000-5fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:11.749276Z nonode@nohost <0.3652.0> -------- Index update finished for db: shards/40000000-5fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:11.936249Z nonode@nohost <0.3778.0> -------- Starting index update for db: shards/00000000-1fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:13.008010Z nonode@nohost <0.3778.0> -------- Index update finished for db: shards/00000000-1fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:13.291704Z nonode@nohost <0.3817.0> -------- Starting index update for db: shards/e0000000-ffffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:14.564826Z nonode@nohost <0.3817.0> -------- Index update finished for db: shards/e0000000-ffffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.314002Z nonode@nohost <0.3907.0> -------- Starting index update for db: shards/20000000-3fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.373015Z nonode@nohost <0.3975.0> -------- Starting index update for db: shards/60000000-7fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.510076Z nonode@nohost <0.3621.0> -------- Starting index update for db: shards/c0000000-dfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.010943Z nonode@nohost <0.4169.0> -------- Starting index update for db: shards/a0000000-bfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.654203Z nonode@nohost <0.3975.0> -------- Index update finished for db: shards/60000000-7fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.677526Z nonode@nohost <0.3907.0> -------- Index update finished for db: shards/20000000-3fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.788062Z nonode@nohost <0.3621.0> -------- Index update finished for db: shards/c0000000-dfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:17.306988Z nonode@nohost <0.4169.0> -------- Index update finished for db: shards/a0000000-bfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:18.494925Z nonode@nohost <0.3686.0> -------- Starting index update for db: shards/80000000-9fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:19.388880Z nonode@nohost <0.3686.0> -------- Index update finished for db: shards/80000000-9fffffff/demo.1625487748 idx: _design/demo
[notice] 2021-07-05T14:32:23.887866Z nonode@nohost <0.13346.5> 61a2177aae 127.0.0.1:3003 10.0.2.100 admin GET / 200 ok 9
[notice] 2021-07-05T14:33:17.881591Z nonode@nohost <0.13347.5> 39c531c67e 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53986
[notice] 2021-07-05T14:34:11.588538Z nonode@nohost <0.13889.5> d3cc5e0fa8 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53700
[notice] 2021-07-05T14:35:07.718592Z nonode@nohost <0.14346.5> 05b6354796 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 56122
[notice] 2021-07-05T14:36:04.255503Z nonode@nohost <0.14842.5> 1535baf654 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 56529
[notice] 2021-07-05T14:37:00.187963Z nonode@nohost <0.15334.5> 00a9c90ecf 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 55926
[notice] 2021-07-05T14:37:53.074959Z nonode@nohost <0.15818.5> e48cbf750a 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 52880
[notice] 2021-07-05T14:38:54.833050Z nonode@nohost <0.16272.5> 07b2646c16 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 61751
[notice] 2021-07-05T14:39:47.015319Z nonode@nohost <0.16808.5> a26c18d19d 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 52175
[notice] 2021-07-05T14:40:40.086929Z nonode@nohost <0.17255.5> 5267817a12 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53065
[notice] 2021-07-05T14:41:33.807566Z nonode@nohost <0.17723.5> 21e5c4941c 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53713
```
Compared to couchdb2:
```
$ podman logs couchdb2
...
[notice] 2021-07-05T12:27:09.139663Z nonode@nohost <0.23495.2> 714b3e7a37 127.0.0.1:3002 10.0.2.100 admin POST /demo 201 ok 30
[notice] 2021-07-05T12:27:09.165180Z nonode@nohost <0.23502.2> 9a12b27d94 127.0.0.1:3002 10.0.2.100 admin POST /demo 201 ok 18
[info] 2021-07-05T13:22:11.293974Z nonode@nohost <0.12126.4> -------- Starting compaction for db "shards/20000000-3fffffff/demo.1625487744" at 1541
[info] 2021-07-05T13:22:32.858211Z nonode@nohost <0.12139.4> -------- Starting compaction for db "shards/00000000-1fffffff/demo.1625487744" at 1524
[info] 2021-07-05T13:22:54.395354Z nonode@nohost <0.12142.4> -------- Starting compaction for db "shards/40000000-5fffffff/demo.1625487744" at 1505
[info] 2021-07-05T13:23:15.835989Z nonode@nohost <0.12145.4> -------- Starting compaction for db "shards/60000000-7fffffff/demo.1625487744" at 1473
[info] 2021-07-05T13:23:37.277484Z nonode@nohost <0.12151.4> -------- Starting compaction for db "shards/a0000000-bfffffff/demo.1625487744" at 1471
[info] 2021-07-05T13:23:58.739812Z nonode@nohost <0.12157.4> -------- Starting compaction for db "shards/e0000000-ffffffff/demo.1625487744" at 1508
[info] 2021-07-05T13:24:20.179296Z nonode@nohost <0.12148.4> -------- Starting compaction for db "shards/80000000-9fffffff/demo.1625487744" at 1455
[info] 2021-07-05T13:24:41.714250Z nonode@nohost <0.12154.4> -------- Starting compaction for db "shards/c0000000-dfffffff/demo.1625487744" at 1524
[info] 2021-07-05T13:24:54.231782Z nonode@nohost <0.12450.4> -------- Starting compaction for db "shards/c0000000-dfffffff/_global_changes.1625487744" at 554
[notice] 2021-07-05T14:31:28.936974Z nonode@nohost <0.15912.6> 312602f8c0 127.0.0.1:3002 10.0.2.100 admin GET / 200 ok 2
[info] 2021-07-05T14:31:28.989213Z nonode@nohost <0.15983.6> -------- Starting index update for db: shards/00000000-1fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.989257Z nonode@nohost <0.15984.6> -------- Starting index update for db: shards/20000000-3fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990506Z nonode@nohost <0.16010.6> -------- Starting index update for db: shards/40000000-5fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990507Z nonode@nohost <0.16011.6> -------- Starting index update for db: shards/a0000000-bfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990573Z nonode@nohost <0.16012.6> -------- Starting index update for db: shards/60000000-7fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990650Z nonode@nohost <0.16016.6> -------- Starting index update for db: shards/80000000-9fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990715Z nonode@nohost <0.16020.6> -------- Starting index update for db: shards/c0000000-dfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.991237Z nonode@nohost <0.16025.6> -------- Starting index update for db: shards/e0000000-ffffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.578442Z nonode@nohost <0.16012.6> -------- Index update finished for db: shards/60000000-7fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.592533Z nonode@nohost <0.16025.6> -------- Index update finished for db: shards/e0000000-ffffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.624614Z nonode@nohost <0.16011.6> -------- Index update finished for db: shards/a0000000-bfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.630679Z nonode@nohost <0.16010.6> -------- Index update finished for db: shards/40000000-5fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.641088Z nonode@nohost <0.16016.6> -------- Index update finished for db: shards/80000000-9fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.682452Z nonode@nohost <0.15984.6> -------- Index update finished for db: shards/20000000-3fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.729299Z nonode@nohost <0.16020.6> -------- Index update finished for db: shards/c0000000-dfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.746659Z nonode@nohost <0.15983.6> -------- Index update finished for db: shards/00000000-1fffffff/demo.1625487744 idx: _design/demo
[notice] 2021-07-05T14:31:37.472657Z nonode@nohost <0.15913.6> 3b98ddbf30 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 8516
[notice] 2021-07-05T14:31:42.522912Z nonode@nohost <0.16231.6> 139c6ac71b 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5042
[notice] 2021-07-05T14:31:47.199986Z nonode@nohost <0.16315.6> 63625be9aa 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4671
[notice] 2021-07-05T14:31:52.020073Z nonode@nohost <0.16417.6> 06140e6373 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4813
[notice] 2021-07-05T14:31:56.818616Z nonode@nohost <0.16495.6> 73aef1519b 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4791
[notice] 2021-07-05T14:32:01.911785Z nonode@nohost <0.16590.6> e7371ec8db 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5086
[notice] 2021-07-05T14:32:07.196507Z nonode@nohost <0.16725.6> 19201fdd74 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5278
[notice] 2021-07-05T14:32:12.823661Z nonode@nohost <0.16833.6> 1cfc7a7f8a 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5618
[notice] 2021-07-05T14:32:18.282645Z nonode@nohost <0.16924.6> a70a1462dc 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5453
[notice] 2021-07-05T14:32:23.871410Z nonode@nohost <0.17021.6> 8364bc2506 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5581
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
nickva edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-890372099
I confirmed that the JS engine is implicated. I ran 2.3.1 and 3.x latest on Ubuntu 20.04. For 3.x I installed the libmozjs185 couchdb-build package for Ubuntu 18.04 and ran with q=8 to minimize difference and similar performance numbers as 2.3.1.
Packages I downloaded and installed manually:
```
couch-libmozjs185-1.0_1.8.5-1.0.0+couch-2_bionic_amd64.deb
couch-libmozjs185-dev_1.8.5-1.0.0+couch-2_bionic_amd64.deb
libffi6_3.2.1-8_amd64.deb
libffi-dev_3.2.1-8_amd64.deb
```
Configure 3.x to build with version 1.8.5
```
./configure --spidermonkey-version 1.8.5 --dev
```
Test script was modified to not set up docker containers, just run locally with 2.3.1 and 3.1.1 running on different ports.
```
./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"fake_uuid_for_dev","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15994
Round: 1/10
real 0m13.022s
user 0m0.023s
sys 0m0.023s
Round: 2/10
real 0m7.234s
user 0m0.020s
sys 0m0.008s
Round: 3/10
real 0m6.715s
user 0m0.026s
sys 0m0.000s
Round: 4/10
real 0m7.021s
user 0m0.018s
sys 0m0.009s
Round: 5/10
real 0m7.159s
user 0m0.005s
sys 0m0.021s
Round: 6/10
real 0m7.318s
user 0m0.014s
sys 0m0.014s
Round: 7/10
real 0m7.464s
user 0m0.007s
sys 0m0.021s
Round: 8/10
real 0m7.175s
user 0m0.016s
sys 0m0.011s
Round: 9/10
real 0m7.096s
user 0m0.004s
sys 0m0.023s
Round: 10/10
real 0m7.235s
user 0m0.006s
sys 0m0.022s
```
```
q=8
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1-ba63878","git_sha":"ba63878","uuid":"fake_uuid_for_dev","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15984
Round: 1/10
real 0m5.979s
user 0m0.017s
sys 0m0.010s
Round: 2/10
real 0m6.757s
user 0m0.013s
sys 0m0.013s
Round: 3/10
real 0m7.112s
user 0m0.012s
sys 0m0.016s
Round: 4/10
real 0m7.359s
user 0m0.017s
sys 0m0.011s
Round: 5/10
real 0m7.210s
user 0m0.017s
sys 0m0.013s
Round: 6/10
real 0m7.233s
user 0m0.008s
sys 0m0.021s
Round: 7/10
real 0m6.284s
user 0m0.028s
sys 0m0.000s
Round: 8/10
real 0m7.101s
user 0m0.020s
sys 0m0.008s
Round: 9/10
real 0m6.943s
user 0m0.022s
sys 0m0.006s
Round: 10/10
real 0m6.736s
user 0m0.016s
sys 0m0.011s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-904550385
Thanks @nickva and sorry for my delayed response, it took me a while to gather the right data to check if that change would be feasible for our application code.
TLDR: The proposed solution works and is about 130 times faster on the given dataset, that are great results, thanks so much for your input :-)
## View adjustments
I tried to use the build-in reducers for a small set of documents on CouchDB 3.
For the testing out your proposal, I created 7 documents, all shared the same aDefId to return one single reduced output object. I have one document per execution result and an additional document which executed successful to be able to check the computation.
The original view was using this map function:
```javascript
function (doc) {
if(doc.object === "ainst" && doc.result) {
var subscription = doc.subscription || doc.tenant;
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 1,
execTimeSum : 0,
execTimeCount : 0,
execTimeMin : Infinity,
execTimeMax : 0
}
if (doc.result.status === "successful") {
stats.successfulCount = 1;
if (doc.result.executionTime) {
stats.execTimeSum = doc.result.executionTime;
stats.execTimeCount = 1;
stats.execTimeMin = doc.result.executionTime;
stats.execTimeMax = doc.result.executionTime;
}
} else if (doc.result.status === "failed") {
stats.failedCount = 1;
} else if (doc.result.status === "canceled") {
stats.canceledCount = 1;
} else if (doc.result.status === "executing") {
stats.executingCount = 1;
} else if (doc.result.status === "unsuccessful") {
stats.unsuccessfulCount = 1;
} else if (doc.result.status === "unknown") {
stats.unknownCount = 1;
}
emit([subscription, doc.aDefId], stats);
}
}
```
and the following custom reduce function:
```javascript
function (keys, values, rereduce) {
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 0,
execTimeSum : 0,
execTimeCount : 0,
execTimeMin : Infinity,
execTimeMax : 0
}
for (var index in values) {
stats.successfulCount += values[index].successfulCount;
stats.failedCount += values[index].failedCount;
stats.canceledCount += values[index].canceledCount;
stats.executingCount += values[index].executingCount;
stats.unsuccessfulCount += values[index].unsuccessfulCount;
stats.unknownCount += values[index].unknownCount;
stats.totalCount += values[index].totalCount;
if (values[index].execTimeCount) {
stats.execTimeSum += values[index].execTimeSum;
stats.execTimeCount += values[index].execTimeCount;
stats.execTimeMin = Math.min(stats.execTimeMin, values[index].execTimeMin);
stats.execTimeMax = Math.max(stats.execTimeMax, values[index].execTimeMax);
}
}
return stats;
}
```
The original view returned the following as response which I will use to validate the result of the following adjustments:
```bash
$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo/_view/slow?reduce=true&group=true" | jq "."
{
"rows": [
{
"key": [
"xxx",
"135600b7487db5804c1961408a115e0270e5"
],
"value": {
"successfulCount": 2,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 7,
"execTimeSum": 1557,
"execTimeCount": 2,
"execTimeMin": 200,
"execTimeMax": 1357
}
}
]
}
```
As you mentioned, the build-in _sum reducer can be used to count different executions in a single view.
This is my updated demo-build-in-reduce/count view:
```javascript
function (doc) {
if(doc.object === "ainst" && doc.result) {
var subscription = doc.subscription || doc.tenant;
var stats = {
successfulCount : 0,
failedCount : 0,
canceledCount : 0,
executingCount : 0,
unsuccessfulCount : 0,
unknownCount : 0,
totalCount : 1
}
if (doc.result.status === "successful") {
stats.successfulCount = 1;
} else if (doc.result.status === "failed") {
stats.failedCount = 1;
} else if (doc.result.status === "canceled") {
stats.canceledCount = 1;
} else if (doc.result.status === "executing") {
stats.executingCount = 1;
} else if (doc.result.status === "unsuccessful") {
stats.unsuccessfulCount = 1;
} else if (doc.result.status === "unknown") {
stats.unknownCount = 1;
}
emit([subscription, doc.aDefId], stats);
}
}
```
The reduce function is set to _sum on that view.
When I query that view, I get results as expected for the counts:
```bash
$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo-build-in-reduce/_view/count?reduce=true&group=true" | jq "."
{
"rows": [
{
"key": [
"xxx",
"135600b7487db5804c1961408a115e0270e5"
],
"value": {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 2,
"totalCount": 7,
"unknownCount": 1,
"unsuccessfulCount": 1
}
}
]
}
```
The response looks good, but as expected is lacking the execution time stats. This can be done with a second view which I named demo-build-in-reduce/successfulExecutionTime.
The second view returns the following:
```bash
$ curl -ksu "admin:password" "http://localhost:3003/demo2/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true" | jq "."
{
"rows": [
{
"key": [
"xxx",
"135600b7487db5804c1961408a115e0270e5"
],
"value": {
"sum": 1557,
"count": 2,
"min": 200,
"max": 1357,
"sumsqr": 1881449
}
}
]
}
```
This looks good either, I do not care for the sumsqr, but for sum, count, min and max. The count is also included in the demo-build-in-reduce/count view, which can be used in the application theoretically to render a more high level view of data. The successfulExecutionTime view is providing the other data.
So, your proposal works for the given data model, thank you very much. The downside is that to keep our API compatible, we need to submit two requests concurrently instead of using the single original view. Nevertheless, the performance is way more consistent, let me share a few measurements based on the new views.
## Test setup to measure performance improvements of the new implementation
I used the initial setup script to deploy both CouchDBs and to load data.
Afterwards I injected the design doc into the demo database:
```bash
$ curl -X POST \
"http://admin:password@localhost:3003/demo" \
-H "Content-Type: application/json" \
-d '{
"_id": "_design/demo-build-in-reduce",
"views": {
"count": {
"reduce": "_sum",
"map": "function (doc) {\n if(doc.object === \"ainst\" && doc.result) {\n var subscription = doc.subscription || doc.tenant;\n var stats = {\n successfulCount : 0,\n failedCount : 0,\n canceledCount : 0,\n executingCount : 0,\n unsuccessfulCount : 0,\n unknownCount : 0,\n totalCount : 1\n }\n\n if (doc.result.status === \"successful\") {\n stats.successfulCount = 1;\n } else if (doc.result.status === \"failed\") {\n stats.failedCount = 1;\n } else if (doc.result.status === \"canceled\") {\n stats.canceledCount = 1;\n } else if (doc.result.status === \"executing\") {\n stats.e
xecutingCount = 1;\n } else if (doc.result.status === \"unsuccessful\") {\n stats.unsuccessfulCount = 1;\n } else if (doc.result.status === \"unknown\") {\n stats.unknownCount = 1;\n }\n\n emit([subscription, doc.aDefId], stats);\n }\n }"
},
"successfulExecutionTime": {
"reduce": "_stats",
"map": "function (doc) {\n if(doc.object === \"ainst\" && doc.result && doc.result.status === \"successful\") {\n var subscription = doc.subscription || doc.tenant;\n emit([subscription, doc.aDefId], doc.result.executionTime);\n }\n}"
}
},
"language": "javascript"
}'
{"ok":true,"id":"_design/demo-build-in-reduce","rev":"1-e769ee9d7efdeb5f80075e16a1113968"}
```
I used the following Node.js script to query data and combine them within the application.
[reducers-demo.zip](https://github.com/apache/couchdb/files/7038700/reducers-demo.zip)
```javascript
'use strict';
const parallel = require('async').parallel;
const http = require('http');
const user = 'admin';
const password = 'password';
const port = 3003;
const dbname = 'demo';
let newRequest = (async () => {
console.log("==================== New approach ====================");
for( let i=1; i<=10; i++) {
console.log(`===== Iteration ${i} ======`);
console.time(`Fast iteration ${i} took`);
let responseMapCount = {};
let responseMapSuccessfulExecutionTime = {};
await parallel([
(cb) => {
let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo-build-in-reduce/_view/count?reduce=true&group=true`;
console.log(`Requesting: ${currentUrl}`)
let req = http.get(currentUrl, (res) => {
console.log('statusCode:', res.statusCode);
let body = ""
res.on('data', (d) => {
body += d;
});
res.on('end', function () {
// store rows in map to combine them after http requests are completed
body = JSON.parse(body);
body.rows.forEach(currentRow => {
responseMapCount[currentRow.key.join('-')] = currentRow.value;
});
cb()
});
}).on('error', (e) => {
console.error(e);
cb(e);
});
req.end();
},
(cb) => {
let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true`;
console.log(`Requesting: ${currentUrl}`)
let req = http.get(currentUrl, (res) => {
console.log('statusCode:', res.statusCode);
let body = '';
res.on('data', (d) => {
body += d;
});
res.on('end', function () {
// store rows in map to combine them after http requests are completed
body = JSON.parse(body);
body.rows.forEach(currentRow => {
responseMapSuccessfulExecutionTime[currentRow.key.join('-')] = currentRow.value;
});
cb()
});
}).on('error', (e) => {
console.error(e);
cb(e);
});
req.end();
}
]);
console.log(`Combining results`);
// iterate over responseMapCount, as there is always an entry for every ainst result
let lastMergedKey;
Object.keys(responseMapCount).forEach((currentKey) => {
// check if the aInst result had a successful invocation, merge it back in that case
if(responseMapSuccessfulExecutionTime[currentKey]) {
// merge second result property back into responseMapCount
responseMapCount[currentKey].execTimeSum = responseMapSuccessfulExecutionTime[currentKey].sum;
responseMapCount[currentKey].execTimeMin = responseMapSuccessfulExecutionTime[currentKey].min;
responseMapCount[currentKey].execTimeMax = responseMapSuccessfulExecutionTime[currentKey].max;
}
});
console.log(`Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: ${JSON.stringify(responseMapCount['xxx-100200b7487db5804c1961408a115e0270e5'], null, 4)}`)
console.timeEnd(`Fast iteration ${i} took`);
}
});
let oldRequest = (async () => {
console.log("==================== Old approach ====================");
for( let i=1; i<=10; i++) {
console.log(`===== Iteration ${i} ======`);
console.time(`Slow iteration ${i} took`);
let results = await parallel([
(cb) => {
let currentUrl = `http://${user}:${password}@localhost:${port}/${dbname}/_design/demo/_view/slow?reduce=true&group=true`;
console.log(`Requesting: ${currentUrl}`)
let req = http.get(currentUrl, (res) => {
console.log('statusCode:', res.statusCode);
let body = ''
res.on('data', (d) => {
body += d;
});
res.on('end', function () {
body = JSON.parse(body);
cb(null, body);
});
}).on('error', (e) => {
console.error(e);
cb(e);
});
req.end();
}
]);
console.log(`Result for aDefId=100200b7487db5804c1961408a115e0270e5:`);
// the view could have been invoked directly with the key, but all results should be retrieved to measure the performance
// only a single result is picked to compare those are equal in both query approaches.
results[0].rows.forEach((currentRow) => {
if(currentRow.key[1] === '100200b7487db5804c1961408a115e0270e5') {
console.log(JSON.stringify(currentRow.value, null, 4));
}
})
console.timeEnd(`Slow iteration ${i} took`);
}
});
(async () => {
await newRequest();
await oldRequest();
})()
```
The output shows an impressive difference, even if the application logic gets more complicated:
```bash
==================== New approach ====================
===== Iteration 1 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 1 took: 270.495ms
===== Iteration 2 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 2 took: 262.454ms
===== Iteration 3 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 3 took: 261.309ms
===== Iteration 4 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 4 took: 271.7ms
===== Iteration 5 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 5 took: 280.972ms
===== Iteration 6 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 6 took: 263.916ms
===== Iteration 7 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 7 took: 291.969ms
===== Iteration 8 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 8 took: 308.858ms
===== Iteration 9 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 9 took: 267.336ms
===== Iteration 10 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/count?reduce=true&group=true
Requesting: http://admin:password@localhost:3003/demo/_design/demo-build-in-reduce/_view/successfulExecutionTime?reduce=true&group=true
statusCode: 200
statusCode: 200
Combining results
Example combined result for aDefId=100200b7487db5804c1961408a115e0270e5: {
"canceledCount": 1,
"executingCount": 1,
"failedCount": 1,
"successfulCount": 1,
"totalCount": 6,
"unknownCount": 1,
"unsuccessfulCount": 1,
"execTimeSum": 1002,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Fast iteration 10 took: 259.752ms
==================== Old approach ====================
===== Iteration 1 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 1 took: 34.973s
===== Iteration 2 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 2 took: 35.636s
===== Iteration 3 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 3 took: 35.189s
===== Iteration 4 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 4 took: 37.665s
===== Iteration 5 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 5 took: 37.669s
===== Iteration 6 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 6 took: 35.679s
===== Iteration 7 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 7 took: 35.368s
===== Iteration 8 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 8 took: 34.711s
===== Iteration 9 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 9 took: 34.831s
===== Iteration 10 ======
Requesting: http://admin:password@localhost:3003/demo/_design/demo/_view/slow?reduce=true&group=true
statusCode: 200
Result for aDefId=100200b7487db5804c1961408a115e0270e5:
{
"successfulCount": 1,
"failedCount": 1,
"canceledCount": 1,
"executingCount": 1,
"unsuccessfulCount": 1,
"unknownCount": 1,
"totalCount": 6,
"execTimeSum": 1002,
"execTimeCount": 1,
"execTimeMin": 1002,
"execTimeMax": 1002
}
Slow iteration 10 took: 35.418s
```
## Conclusion
On average (10 iterations), I could get combined results of all records within 274 ms compared to 35714 ms, so the workaround on my given dataset is actually ~ 130 times faster.
I repeated the test for CouchDB 2 as well ([output2.txt](https://github.com/apache/couchdb/files/7038776/output2.txt)) and got an average of 447 ms with the new approach compared to 6917 ms which is still more than 15 times faster. So independent on the question if we should migrate to CouchDB 3 or not, it might still be a useful optimization also for CouchDB 2 setups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] konrad-ohms edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
konrad-ohms edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-874168606
Thanks @nickva for investigating the issue.
I tried to adjust the function from
```
setupDbs() {
COUCHDB_PORT=${1}
waitForDb "${COUCHDB_PORT}"
SYSTEM_DBS=( _users _replicator _global_changes demo )
for CURRENT_DB in "${SYSTEM_DBS[@]}"
do
echo "Creating ${CURRENT_DB}"
curl -X PUT "http://${COUCHDB_USER}:${COUCHDB_PASSWORD}@127.0.0.1:${COUCHDB_PORT}/${CURRENT_DB}"
done
}
```
to
```
setupDbs() {
COUCHDB_PORT=${1}
waitForDb "${COUCHDB_PORT}"
SYSTEM_DBS=( _users _replicator _global_changes demo )
for CURRENT_DB in "${SYSTEM_DBS[@]}"
do
echo "Creating ${CURRENT_DB}"
curl -X PUT "http://${COUCHDB_USER}:${COUCHDB_PASSWORD}@127.0.0.1:${COUCHDB_PORT}/${CURRENT_DB}?q=8"
done
}
```
Unfortunately that did not solve the performance degradation of CouchDB 3.1.1 compared to v2.3.1.
I checked that the databases are using the same number of shards:
CouchDB 2:
```
$ curl -u "xxx:xxx" http://localhost:3002/demo/
{"db_name":"demo","purge_seq":"0-g1AAAAEzeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgYg9f____lZiQx41R2AqLuPW11SApBMqgcqImTWA4hZeNQlKYDMsifCrAUQs_bjVpfIkCSP3zIHkGXxEDVZAMg5YvY","update_seq":"12001-g1AAAAFLeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjMlMiTJ____PyuJgYH1Cw41SQpAMskeqoyNFZcyB5CyeJhpD3EpSwApq4cpO4hDWR4LkGRoAFJAlfPBStfjVboAonQ_WOl-vEoPQJTex-triNIHEKUQtz7JAgDbh2kI","sizes":{"file":11425440,"external":11923785,"active":11287083},"other":{"data_size":11923785},"doc_del_count":0,"doc_count":12001,"disk_size":11425440,"disk_format_version":7,"data_size":11287083,"compact_running":false,"cluster":{"q":8,"n":1,"w":1,"r":1},"instance_start_time":"0"}
```
CouchDB 3:
```
$ curl -u "xxx:xxx" http://localhost:3003/demo/
{"db_name":"demo","purge_seq":"0-g1AAAAEzeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgdA6j8QZCUy4FV3AKLuPiF1CyDq9hNS1wBRNx-3uqQEIJlUj9dtSQ4gNfH41SiA1NjjVZPIkCQPUZAFAN6AYvY","update_seq":"12001-g1AAAAFLeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgdA6j8QZCUxMLDew6v0AETpfbDSg3iVLoAo3Q9W-gKv0gaI0vlgpa9wKE1KAJJJ9TB33sGlzAGkLB6mDKdpCiBl9jBll3AoS2RIkoepuZEFAAsmagc","sizes":{"file":14468808,"external":9785687,"active":11343864},"props":{},"doc_del_count":0,"doc_count":12001,"disk_format_version":8,"compact_running":false,"cluster":{"q":8,"n":1,"w":1,"r":1},"instance_start_time":"0"}
```
Both state:
```
"cluster":{"q":8,"n":1,"w":1,"r":1}
```
After injecting the data I waited multiple minutes before running the queries:
```
$ ./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"5584fb6a7a94ff8ab44bbee93823352d","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3002
Round: 1/10
real 0m8.534s
user 0m0.017s
sys 0m0.013s
Round: 2/10
real 0m5.050s
user 0m0.012s
sys 0m0.010s
Round: 3/10
real 0m4.677s
user 0m0.010s
sys 0m0.011s
Round: 4/10
real 0m4.820s
user 0m0.013s
sys 0m0.009s
Round: 5/10
real 0m4.798s
user 0m0.011s
sys 0m0.011s
Round: 6/10
real 0m5.093s
user 0m0.013s
sys 0m0.009s
Round: 7/10
real 0m5.285s
user 0m0.015s
sys 0m0.007s
Round: 8/10
real 0m5.627s
user 0m0.010s
sys 0m0.014s
Round: 9/10
real 0m5.459s
user 0m0.013s
sys 0m0.010s
Round: 10/10
real 0m5.589s
user 0m0.009s
sys 0m0.015s
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"6445c39f1422b264fbf7837dbb055bb9","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 3003
Round: 1/10
real 0m53.993s
user 0m0.014s
sys 0m0.011s
Round: 2/10
real 0m53.707s
user 0m0.016s
sys 0m0.009s
Round: 3/10
real 0m56.130s
user 0m0.011s
sys 0m0.016s
Round: 4/10
real 0m56.537s
user 0m0.012s
sys 0m0.014s
Round: 5/10
real 0m55.933s
user 0m0.012s
sys 0m0.013s
Round: 6/10
real 0m52.887s
user 0m0.013s
sys 0m0.014s
Round: 7/10
real 1m1.758s
user 0m0.012s
sys 0m0.012s
Round: 8/10
real 0m52.182s
user 0m0.010s
sys 0m0.018s
Round: 9/10
real 0m53.072s
user 0m0.011s
sys 0m0.016s
Round: 10/10
real 0m53.720s
user 0m0.014s
sys 0m0.013s
```
I am not sure if the default log level does not indicate the auto-builder and auto-compactor actions, but at least the container logs are not reflecting such actions:
```
$ podman logs -f couchdb3
...
[notice] 2021-07-05T12:34:42.581798Z nonode@nohost <0.19894.3> 14ea633501 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 19
[notice] 2021-07-05T12:34:42.618414Z nonode@nohost <0.19902.3> 96877188fc 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 22
[notice] 2021-07-05T12:34:42.650066Z nonode@nohost <0.19910.3> c2fe735612 127.0.0.1:3003 10.0.2.100 admin POST /demo 201 ok 20
[info] 2021-07-05T12:35:10.329291Z nonode@nohost <0.3652.0> -------- Starting index update for db: shards/40000000-5fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:11.749276Z nonode@nohost <0.3652.0> -------- Index update finished for db: shards/40000000-5fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:11.936249Z nonode@nohost <0.3778.0> -------- Starting index update for db: shards/00000000-1fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:13.008010Z nonode@nohost <0.3778.0> -------- Index update finished for db: shards/00000000-1fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:13.291704Z nonode@nohost <0.3817.0> -------- Starting index update for db: shards/e0000000-ffffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:14.564826Z nonode@nohost <0.3817.0> -------- Index update finished for db: shards/e0000000-ffffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.314002Z nonode@nohost <0.3907.0> -------- Starting index update for db: shards/20000000-3fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.373015Z nonode@nohost <0.3975.0> -------- Starting index update for db: shards/60000000-7fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:15.510076Z nonode@nohost <0.3621.0> -------- Starting index update for db: shards/c0000000-dfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.010943Z nonode@nohost <0.4169.0> -------- Starting index update for db: shards/a0000000-bfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.654203Z nonode@nohost <0.3975.0> -------- Index update finished for db: shards/60000000-7fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.677526Z nonode@nohost <0.3907.0> -------- Index update finished for db: shards/20000000-3fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:16.788062Z nonode@nohost <0.3621.0> -------- Index update finished for db: shards/c0000000-dfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:17.306988Z nonode@nohost <0.4169.0> -------- Index update finished for db: shards/a0000000-bfffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:18.494925Z nonode@nohost <0.3686.0> -------- Starting index update for db: shards/80000000-9fffffff/demo.1625487748 idx: _design/demo
[info] 2021-07-05T12:35:19.388880Z nonode@nohost <0.3686.0> -------- Index update finished for db: shards/80000000-9fffffff/demo.1625487748 idx: _design/demo
[notice] 2021-07-05T14:32:23.887866Z nonode@nohost <0.13346.5> 61a2177aae 127.0.0.1:3003 10.0.2.100 admin GET / 200 ok 9
[notice] 2021-07-05T14:33:17.881591Z nonode@nohost <0.13347.5> 39c531c67e 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53986
[notice] 2021-07-05T14:34:11.588538Z nonode@nohost <0.13889.5> d3cc5e0fa8 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53700
[notice] 2021-07-05T14:35:07.718592Z nonode@nohost <0.14346.5> 05b6354796 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 56122
[notice] 2021-07-05T14:36:04.255503Z nonode@nohost <0.14842.5> 1535baf654 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 56529
[notice] 2021-07-05T14:37:00.187963Z nonode@nohost <0.15334.5> 00a9c90ecf 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 55926
[notice] 2021-07-05T14:37:53.074959Z nonode@nohost <0.15818.5> e48cbf750a 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 52880
[notice] 2021-07-05T14:38:54.833050Z nonode@nohost <0.16272.5> 07b2646c16 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 61751
[notice] 2021-07-05T14:39:47.015319Z nonode@nohost <0.16808.5> a26c18d19d 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 52175
[notice] 2021-07-05T14:40:40.086929Z nonode@nohost <0.17255.5> 5267817a12 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53065
[notice] 2021-07-05T14:41:33.807566Z nonode@nohost <0.17723.5> 21e5c4941c 127.0.0.1:3003 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 53713
```
Compared to couchdb2:
```
$ podman logs couchdb2
...
[notice] 2021-07-05T12:27:09.139663Z nonode@nohost <0.23495.2> 714b3e7a37 127.0.0.1:3002 10.0.2.100 admin POST /demo 201 ok 30
[notice] 2021-07-05T12:27:09.165180Z nonode@nohost <0.23502.2> 9a12b27d94 127.0.0.1:3002 10.0.2.100 admin POST /demo 201 ok 18
[info] 2021-07-05T13:22:11.293974Z nonode@nohost <0.12126.4> -------- Starting compaction for db "shards/20000000-3fffffff/demo.1625487744" at 1541
[info] 2021-07-05T13:22:32.858211Z nonode@nohost <0.12139.4> -------- Starting compaction for db "shards/00000000-1fffffff/demo.1625487744" at 1524
[info] 2021-07-05T13:22:54.395354Z nonode@nohost <0.12142.4> -------- Starting compaction for db "shards/40000000-5fffffff/demo.1625487744" at 1505
[info] 2021-07-05T13:23:15.835989Z nonode@nohost <0.12145.4> -------- Starting compaction for db "shards/60000000-7fffffff/demo.1625487744" at 1473
[info] 2021-07-05T13:23:37.277484Z nonode@nohost <0.12151.4> -------- Starting compaction for db "shards/a0000000-bfffffff/demo.1625487744" at 1471
[info] 2021-07-05T13:23:58.739812Z nonode@nohost <0.12157.4> -------- Starting compaction for db "shards/e0000000-ffffffff/demo.1625487744" at 1508
[info] 2021-07-05T13:24:20.179296Z nonode@nohost <0.12148.4> -------- Starting compaction for db "shards/80000000-9fffffff/demo.1625487744" at 1455
[info] 2021-07-05T13:24:41.714250Z nonode@nohost <0.12154.4> -------- Starting compaction for db "shards/c0000000-dfffffff/demo.1625487744" at 1524
[info] 2021-07-05T13:24:54.231782Z nonode@nohost <0.12450.4> -------- Starting compaction for db "shards/c0000000-dfffffff/_global_changes.1625487744" at 554
[notice] 2021-07-05T14:31:28.936974Z nonode@nohost <0.15912.6> 312602f8c0 127.0.0.1:3002 10.0.2.100 admin GET / 200 ok 2
[info] 2021-07-05T14:31:28.989213Z nonode@nohost <0.15983.6> -------- Starting index update for db: shards/00000000-1fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.989257Z nonode@nohost <0.15984.6> -------- Starting index update for db: shards/20000000-3fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990506Z nonode@nohost <0.16010.6> -------- Starting index update for db: shards/40000000-5fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990507Z nonode@nohost <0.16011.6> -------- Starting index update for db: shards/a0000000-bfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990573Z nonode@nohost <0.16012.6> -------- Starting index update for db: shards/60000000-7fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990650Z nonode@nohost <0.16016.6> -------- Starting index update for db: shards/80000000-9fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.990715Z nonode@nohost <0.16020.6> -------- Starting index update for db: shards/c0000000-dfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:28.991237Z nonode@nohost <0.16025.6> -------- Starting index update for db: shards/e0000000-ffffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.578442Z nonode@nohost <0.16012.6> -------- Index update finished for db: shards/60000000-7fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.592533Z nonode@nohost <0.16025.6> -------- Index update finished for db: shards/e0000000-ffffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.624614Z nonode@nohost <0.16011.6> -------- Index update finished for db: shards/a0000000-bfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.630679Z nonode@nohost <0.16010.6> -------- Index update finished for db: shards/40000000-5fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.641088Z nonode@nohost <0.16016.6> -------- Index update finished for db: shards/80000000-9fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.682452Z nonode@nohost <0.15984.6> -------- Index update finished for db: shards/20000000-3fffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.729299Z nonode@nohost <0.16020.6> -------- Index update finished for db: shards/c0000000-dfffffff/demo.1625487744 idx: _design/demo
[info] 2021-07-05T14:31:32.746659Z nonode@nohost <0.15983.6> -------- Index update finished for db: shards/00000000-1fffffff/demo.1625487744 idx: _design/demo
[notice] 2021-07-05T14:31:37.472657Z nonode@nohost <0.15913.6> 3b98ddbf30 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 8516
[notice] 2021-07-05T14:31:42.522912Z nonode@nohost <0.16231.6> 139c6ac71b 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5042
[notice] 2021-07-05T14:31:47.199986Z nonode@nohost <0.16315.6> 63625be9aa 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4671
[notice] 2021-07-05T14:31:52.020073Z nonode@nohost <0.16417.6> 06140e6373 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4813
[notice] 2021-07-05T14:31:56.818616Z nonode@nohost <0.16495.6> 73aef1519b 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 4791
[notice] 2021-07-05T14:32:01.911785Z nonode@nohost <0.16590.6> e7371ec8db 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5086
[notice] 2021-07-05T14:32:07.196507Z nonode@nohost <0.16725.6> 19201fdd74 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5278
[notice] 2021-07-05T14:32:12.823661Z nonode@nohost <0.16833.6> 1cfc7a7f8a 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5618
[notice] 2021-07-05T14:32:18.282645Z nonode@nohost <0.16924.6> a70a1462dc 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5453
[notice] 2021-07-05T14:32:23.871410Z nonode@nohost <0.17021.6> 8364bc2506 127.0.0.1:3002 10.0.2.100 admin GET /demo/_design/demo/_view/slow?reduce=true&group=true 200 ok 5581
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva edited a comment on issue #3517: Performance regression on CouchDB v3 while using custom reduce function
Posted by GitBox <gi...@apache.org>.
nickva edited a comment on issue #3517:
URL: https://github.com/apache/couchdb/issues/3517#issuecomment-890372099
I confirmed that the JS engine is implicated. I ran 2.3.1 and 3.x latest on Ubuntu 20.04. For 3.x I installed the libmozjs185 couchdb-build package for Ubuntu 18.04 and ran with q=8 to minimize difference and similar performance numbers as 2.3.1.
Packages I downloaded and installed manually:
```
couch-libmozjs185-1.0_1.8.5-1.0.0+couch-2_bionic_amd64.deb
couch-libmozjs185-dev_1.8.5-1.0.0+couch-2_bionic_amd64.deb
libffi6_3.2.1-8_amd64.deb
libffi-dev_3.2.1-8_amd64.deb
```
Configure 3.x to build with version 1.8.5
```
./configure --spidermonkey-version 1.8.5 --dev
```
Test was modified to not set up docker container.
```
./couchdb-test.sh query
CouchDB performance regression test script
Assuming that setup is already complete
================ Query CouchDB 2.3.1 ================
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"fake_uuid_for_dev","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15994
Round: 1/10
real 0m13.022s
user 0m0.023s
sys 0m0.023s
Round: 2/10
real 0m7.234s
user 0m0.020s
sys 0m0.008s
Round: 3/10
real 0m6.715s
user 0m0.026s
sys 0m0.000s
Round: 4/10
real 0m7.021s
user 0m0.018s
sys 0m0.009s
Round: 5/10
real 0m7.159s
user 0m0.005s
sys 0m0.021s
Round: 6/10
real 0m7.318s
user 0m0.014s
sys 0m0.014s
Round: 7/10
real 0m7.464s
user 0m0.007s
sys 0m0.021s
Round: 8/10
real 0m7.175s
user 0m0.016s
sys 0m0.011s
Round: 9/10
real 0m7.096s
user 0m0.004s
sys 0m0.023s
Round: 10/10
real 0m7.235s
user 0m0.006s
sys 0m0.022s
```
```
q=8
================ Query CouchDB 3.1.1 ================
{"couchdb":"Welcome","version":"3.1.1-ba63878","git_sha":"ba63878","uuid":"fake_uuid_for_dev","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Query Database views on port 15984
Round: 1/10
real 0m5.979s
user 0m0.017s
sys 0m0.010s
Round: 2/10
real 0m6.757s
user 0m0.013s
sys 0m0.013s
Round: 3/10
real 0m7.112s
user 0m0.012s
sys 0m0.016s
Round: 4/10
real 0m7.359s
user 0m0.017s
sys 0m0.011s
Round: 5/10
real 0m7.210s
user 0m0.017s
sys 0m0.013s
Round: 6/10
real 0m7.233s
user 0m0.008s
sys 0m0.021s
Round: 7/10
real 0m6.284s
user 0m0.028s
sys 0m0.000s
Round: 8/10
real 0m7.101s
user 0m0.020s
sys 0m0.008s
Round: 9/10
real 0m6.943s
user 0m0.022s
sys 0m0.006s
Round: 10/10
real 0m6.736s
user 0m0.016s
sys 0m0.011s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org