You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2018/11/02 19:41:20 UTC
[GitHub] wohali closed pull request #27: vm.args is now editable -- enabling
clusters; snap set works for name and cookie
wohali closed pull request #27: vm.args is now editable -- enabling clusters; snap set works for name and cookie
URL: https://github.com/apache/couchdb-pkg/pull/27
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.gitignore b/.gitignore
index a178ae5..30fd9cf 100644
--- a/.gitignore
+++ b/.gitignore
@@ -16,7 +16,7 @@ repo/dists
repo/pool
pkgs/
couchdb/
-couchdb_2.0_amd64.snap
+couchdb_*.snap
parts/
prime/
snap/.snapcraft/
diff --git a/README.md b/README.md
index 0454cbf..71df666 100644
--- a/README.md
+++ b/README.md
@@ -55,3 +55,4 @@ with issue reporting or contributing to the upkeep of this project.
[1]: http://mail-archives.apache.org/mod_mbox/couchdb-user/
[2]: http://mail-archives.apache.org/mod_mbox/couchdb-dev/
[3]: https://github.com/apache/couchdb/blob/master/CONTRIBUTING.md
+
diff --git a/snap/HOWTO.md b/snap/HOWTO.md
new file mode 100644
index 0000000..21124a5
--- /dev/null
+++ b/snap/HOWTO.md
@@ -0,0 +1,109 @@
+# HOW TO install a cluster using snap
+
+# Create three machines
+
+In the instruction below, we are going to set up a three -- the miniumn number needed to gain performace improvement -- Couch cluster database. In this potted example we will be using LXD.
+
+We launch a new container and install couchdb on one machine
+
+1. localhost> `lxc launch ubuntu:18.04 couchdb-c1`
+1. localhost> `lxc exec couchdb-c1 bash`
+1. couchdb-c1> `apt update`
+1. couchdb-c1> `snap install couchdb`
+1. couchdb-c1> `logout`
+
+Here we use LXD copy function to speed up the test
+```
+lxc copy couchdb-c1 couchdb-c2
+lxc copy couchdb-c1 couchdb-c3
+lxc copy couchdb-c1 cdb-backup
+lxc start couchdb-c2
+lxc start couchdb-c3
+lxc start cdb-backup
+```
+
+# Configure CouchDB (using the snap tool)
+
+We are going to need the IP addresses. You can find them here.
+```
+lxc list
+```
+
+Now lets use the snap configuration tool to set the configuration files.
+```
+lxc exec couchdb-c1 snap set couchdb name=couchdb@10.210.199.199 setcookie=monster admin=password bind-address=0.0.0.0
+lxc exec couchdb-c2 snap set couchdb name=couchdb@10.210.199.254 setcookie=monster admin=password bind-address=0.0.0.0
+lxc exec couchdb-c3 snap set couchdb name=couchdb@10.210.199.24 setcookie=monster admin=password bind-address=0.0.0.0
+```
+The backup machine we will leave as a single instance and no sharding.
+```
+lxc exec cdb-backup snap set couchdb name=couchdb@127.0.0.1 setcookie=monster admin=password bind-address=0.0.0.0 n=1 q=1
+```
+
+The snap must be restarted for the new configurations to take affect.
+```
+lxc exec couchdb-c1 snap restart couchdb
+lxc exec couchdb-c2 snap restart couchdb
+lxc exec couchdb-c3 snap restart couchdb
+lxc exec cdb-backup snap restart couchdb
+```
+The configuration files are stored here.
+```
+lxc exec cdb-backup cat /var/snap/couchdb/current/etc/vm.args
+lxc exec cdb-backup cat /var/snap/couchdb/current/etc/local.d/*
+```
+Any changes to couchdb from the http configutation tool are made here
+```
+lxc exec cdb-backup cat /var/snap/couchdb/current/etc/local.d/local.ini
+```
+
+# Configure CouchDB Cluster (using the http interface)
+
+Now we set up the cluster via the http front-end. This only needs to be run once on the first machine. The last command syncs with the other nodes and creates the standard databases.
+```
+curl -X POST -H "Content-Type: application/json" http://admin:password@10.210.199.199:5984/_cluster_setup -d '{"action": "add_node", "host":"10.210.199.254", "port": "5984", "username": "admin", "password":"password"}'
+curl -X POST -H "Content-Type: application/json" http://admin:password@10.210.199.199:5984/_cluster_setup -d '{"action": "add_node", "host":"10.210.199.24", "port": "5984", "username": "admin", "password":"password"}'
+curl -X POST -H "Content-Type: application/json" http://admin:password@10.210.199.199:5984/_cluster_setup -d '{"action": "finish_cluster"}'
+```
+Now we have a functioning three node cluster.
+
+# An Example Database
+
+Let's create an example database ...
+```
+curl -X PUT http://admin:password@10.210.199.199:5984/example
+curl -X PUT http://admin:password@10.210.199.199:5984/example/aaa -d '{"test":1}' -H "Content-Type: application/json"
+curl -X PUT http://admin:password@10.210.199.199:5984/example/aab -d '{"test":2}' -H "Content-Type: application/json"
+curl -X PUT http://admin:password@10.210.199.199:5984/example/aac -d '{"test":3}' -H "Content-Type: application/json"
+```
+... And see that it is sync'd accross the three nodes.
+```
+curl -X GET http://admin:password@10.210.199.199:5984/example/_all_docs
+curl -X GET http://admin:password@10.210.199.254:5984/example/_all_docs
+curl -X GET http://admin:password@10.210.199.24:5984/example/_all_docs
+```
+# Backing Up CouchDB
+
+Our back up server is on 10.210.199.242. We will manually replicate this from one (anyone) of the nodes.
+```
+curl -X POST http://admin:password@10.210.199.242:5984/_replicate -d '{"source":"http://10.210.199.199:5984/example", "target":"example", "continuous":false,"create_target":true}' -H "Content-Type: application/json"
+curl -X GET http://admin:password@10.210.199.242:5984/example/_all_docs
+```
+The data store for the clusters nodes are sharded
+```
+lxc exec couchdb-c1 ls /var/snap/couchdb/common/2.x/data/shards/
+```
+
+The backup database is a single file.
+```
+lxc exec cdb-backup ls /var/snap/couchdb/common/2.x/data/shards/00000000-ffffffff/
+```
+
+# Monitoring CouchDB
+
+The logs, by default, are captured by journald
+```
+lxc exec couchdb-c1 bash
+journalctl -u snap.couchdb -f
+```
+
diff --git a/snap/README.md b/snap/README.md
new file mode 100644
index 0000000..65ce54e
--- /dev/null
+++ b/snap/README.md
@@ -0,0 +1,61 @@
+# Building snaps
+
+## Prerequisites
+
+CouchDB requires Ubuntu 16.04. If building on 18.04, then LXD might be useful.
+
+1. `lxc launch ubuntu:16.04 couchdb-pkg`
+1. `lxc exec couchdb-pkg bash`
+1. `sudo apt update`
+1. `sudo apt install snapd snapcraft`
+
+1. `git clone https://github.com/couchdb/couchdb-pkg.git`
+1. `cd couchdb-pkg`
+
+## How to do it
+
+1. Edit `snap/snapcraft.yaml` to point to the correct tag (e.g. `2.2.0`)
+1. `snapcraft`
+
+# Snap Instalation
+
+You may need to pull the LXD file to the host system.
+
+ $ lxc file pull couchdb-pkg/root/couchdb-pkg/couchdb_2.2.0_amd64.snap /tmp/couchdb_2.2.0_amd64.snap
+
+The self crafted snap will need to be installed in devmode
+
+ $ sudo snap install /tmp/couchdb_2.2.0_amd64.snap --devmode
+
+# Snap Configuration
+
+There are two levels of erlang and couchdb configuration hierarchy.
+
+The default layer is stored in /snap/couchdb/current/rel/couchdb/etc/ and is read only.
+The user override layer, is stored in /var/snap/couchdb/current/etc/ and is writable.
+Within this second layer, configurations are set with the local.d directory (one file
+per section) or the local.ini (co-mingled). The "snap set" command works with the
+former (local.d) and couchdb http configuration overwrites the latter (local.ini).
+Entries in local.ini supersede those in the local.d directory.
+
+The name of the erlang process and the security cookie used is set in vm.args file.
+This can be set through the snap native configuration. For example, when setting up
+a cluster over several machines the convention is to set the erlang
+name to couchdb@your.ip.address. Both erlang and couchdb configuration changes can be
+made at the same time.
+
+ $ sudo snap set couchdb name=couchdb@216.3.128.12 setcookie=cutter admin=Be1stDB bind-address=0.0.0.0
+
+Snap set variable can not contain underscore character, but any dashes are converted to underscore when
+writing to file. Wrap double quotes around any bracets and avoid spaces.
+
+ $ sudo snap set couchdb delayed-commits=true erlang="{couch_native_process,start_link,[]}"
+
+Snap Native Configuration changes only come into effect after a restart
+
+ $ sudo snap restart couchdb
+
+# Example Cluster
+
+See the HOWTO.md file to see an example of a three node cluster.
+
diff --git a/snap/couchdb.ini b/snap/couchdb.ini
new file mode 100644
index 0000000..a384ff3
--- /dev/null
+++ b/snap/couchdb.ini
@@ -0,0 +1,3 @@
+[couchdb]
+database_dir = /var/snap/couchdb/common/data
+view_index_dir = /var/snap/couchdb/common/data
diff --git a/snap/meta/hooks/configure b/snap/meta/hooks/configure
new file mode 100755
index 0000000..8c2b1aa
--- /dev/null
+++ b/snap/meta/hooks/configure
@@ -0,0 +1,178 @@
+#!/bin/sh
+
+set -e
+
+VM_ARGS=$SNAP_DATA/etc/vm.args
+LOCAL_DIR=$SNAP_DATA/etc/local.d
+
+
+## add or replace for the vm.arg file
+_modify_vm_args() {
+ opt=$1
+ value="$2"
+ replace_line="-$opt $value"
+ if $(grep -q "^-$opt " $VM_ARGS); then
+ sed "s/^-$opt .*/$replace_line/" $VM_ARGS 2>/dev/null >${VM_ARGS}.new
+ mv -f ${VM_ARGS}.new $VM_ARGS 2>/dev/null
+ else
+ echo $replace_line >> $VM_ARGS
+ fi
+}
+
+_modify_ini_file() {
+ section=$1
+ opt=`echo $2 | tr "-" "_"`
+ value="$3"
+ config_file=${LOCAL_DIR}/${section}.ini
+ if [ ! -e ${config_file} ]; then
+ echo "[${section}]" > $config_file
+ fi
+ replace_line="$opt=$value"
+ if $(grep -q "^$opt=" $config_file); then
+ sed "s/^$opt=.*/$replace_line/" $config_file 2>/dev/null >${config_file}.new
+ mv -f ${config_file}.new ${config_file} 2>/dev/null
+ else
+ echo $replace_line >> $config_file
+ fi
+}
+
+# The vm_args file can only be changed from the filesystem
+# configutaion vm.args file
+
+VM_ARGS_OPTIONS="name setcookie"
+for key in $VM_ARGS_OPTIONS
+do
+ val=$(snapctl get $key)
+ if [ ! -z "$val" ]; then
+ _modify_vm_args $key $val
+ sleep 0.125
+ fi
+done
+
+# The following list is either those fields that are whitelisted but
+# useful to modifiy before first run; or those fields blacklisted
+# The snap set command modifies the files in local.d; any changes
+# via the URL are reflected in local.ini
+
+# Special Cases
+
+# local.d/admins.ini
+passwd=$(snapctl get admin)
+if [ ! -z "$passwd" ]; then
+ _modify_ini_file admins admin $passwd
+ chmod 600 $SNAP_DATA/etc/local.d/admins.ini
+ sleep 0.125
+fi
+
+# local.d/ssl.ini
+port=$(snapctl get ssl-port)
+if [ ! -z "$port" ]; then
+ _modify_ini_file ssl port $port
+ sleep 0.125
+fi
+
+# local.d/httpd.ini
+port=$(snapctl get httpd-port)
+if [ ! -z "$port" ]; then
+ _modify_ini_file httpd port $port
+ sleep 0.125
+fi
+
+# local.d/chttpd.ini
+port=$(snapctl get chttpd-port)
+if [ ! -z "$port" ]; then
+ _modify_ini_file chttpd port $port
+ sleep 0.125
+fi
+
+# Generic Cases
+
+# local.d/chttpd.ini
+CHTTPD_OPTIONS="port bind-address require-valid-user"
+for key in $CHTTPD_OPTIONS
+do
+ val=$(snapctl get $key)
+ if [ ! -z "$val" ]; then
+ _modify_ini_file chttpd $key $val
+ sleep 0.125
+ fi
+done
+
+# local.d/cluster.ini
+CLUSTER_OPTIONS="n q"
+for key in $CLUSTER_OPTIONS
+do
+ val=$(snapctl get $key)
+ if [ ! -z "$val" ]; then
+ _modify_ini_file cluster $key $val
+ sleep 0.125
+ fi
+done
+
+# local.d/compaction_daemon.ini
+COMPACTION_DAEMON_OPTIONS="check-interval"
+for key in $COMPACTION_DAEMON_OPTIONS
+do
+ val=$(snapctl get $key)
+ if [ ! -z "$val" ]; then
+ _modify_ini_file compaction_daemon $key $val
+ sleep 0.125
+ fi
+done
+
+# local.d/couchdb.ini
+COUCHDB_OPTIONS="database-dir view-index-dir delayed-commits"
+for key in $COUCHDB_OPTIONS
+do
+ val=$(snapctl get $key)
+ if [ ! -z "$val" ]; then
+ _modify_ini_file couchdb $key $val
+ sleep 0.125
+ fi
+done
+
+# local.d/log.ini
+LOG_OPTIONS="writer file level"
+for key in $LOG_OPTIONS
+do
+ val=$(snapctl get $key)
+ if [ ! -z "$val" ]; then
+ _modify_ini_file log $key $val
+ sleep 0.125
+ fi
+done
+
+# local.d/native_query_servers.ini
+NATIVE_QUERY_SERVERS_OPTIONS="query erlang"
+for key in $NATIVE_QUERY_SERVERS_OPTIONS
+do
+ val=$(snapctl get $key)
+ if [ ! -z "$val" ]; then
+ _modify_ini_file native_query_servers $key $val
+ sleep 0.125
+ fi
+done
+
+# local.d/couch_peruser.ini
+COUCH_PERUSER_OPTIONS="database-prefix delete-dbs enable"
+for key in $COUCH_PERUSER_OPTIONS
+do
+ val=$(snapctl get $key)
+ if [ ! -z "$val" ]; then
+ _modify_ini_file couch_peruser $key $val
+ sleep 0.125
+ fi
+done
+
+# local.d/uuids.ini
+UUIDS_OPTIONS="algorithm max-count"
+for key in $UUIDS_OPTIONS
+do
+ val=$(snapctl get $key)
+ if [ ! -z "$val" ]; then
+ _modify_ini_file uuids $key $val
+ sleep 0.125
+ fi
+done
+
+
diff --git a/snap/meta/hooks/install b/snap/meta/hooks/install
new file mode 100755
index 0000000..eb7a541
--- /dev/null
+++ b/snap/meta/hooks/install
@@ -0,0 +1,9 @@
+#!/bin/sh
+
+mkdir -p ${SNAP_DATA}/etc/local.d
+
+cp ${SNAP}/rel/couchdb/etc/vm.args ${SNAP_DATA}/etc/vm.args
+
+cp ${SNAP}/rel/couchdb/etc/local.d/*.ini ${SNAP_DATA}/etc/local.d
+
+
diff --git a/snap/snap.ini b/snap/snap.ini
deleted file mode 100644
index 3170cb1..0000000
--- a/snap/snap.ini
+++ /dev/null
@@ -1,10 +0,0 @@
-; Snap-specific Configuration Settings
-
-[couchdb]
-database_dir = /var/snap/couchdb/common/data
-view_index_dir = /var/snap/couchdb/common/data
-
-[log]
-writer = file
-file = /var/snap/couchdb/current/couchdb.log
-;level = info
diff --git a/snap/snap_run b/snap/snap_run
index 4a43867..5fa783a 100755
--- a/snap/snap_run
+++ b/snap/snap_run
@@ -15,9 +15,22 @@
# configuration files so that couchdb will run inside a Snap package's confinement
export HOME=$SNAP_DATA
-export ERL_FLAGS="-couch_ini ${SNAP}/rel/couchdb/etc/default.ini ${SNAP}/rel/couchdb/etc/snap.ini ${SNAP_DATA}/local.ini"
-if [ ! -e ${SNAP_DATA}/local.ini ]; then
- touch ${SNAP_DATA}/local.ini
+export COUCHDB_ARGS_FILE=${SNAP_DATA}/etc/vm.args
+export ERL_FLAGS="-couch_ini ${SNAP}/rel/couchdb/etc/default.ini ${SNAP_DATA}/etc/local.d ${SNAP_DATA}/etc/local.ini"
+
+mkdir -p ${SNAP_DATA}/etc
+
+if [ ! -e ${SNAP_DATA}/etc/vm.args ]; then
+ cp ${SNAP}/rel/couchdb/etc/vm.args ${SNAP_DATA}/etc/vm.args
+fi
+
+if [ ! -d ${SNAP_DATA}/etc/local.d ]; then
+ mkdir ${SNAP_DATA}/etc/local.d
+ cp ${SNAP}/rel/couchdb/etc/local.d/*.ini ${SNAP_DATA}/etc/local.d
+fi
+
+if [ ! -e ${SNAP_DATA}/etc/local.ini ]; then
+ touch ${SNAP_DATA}/etc/local.ini
fi
exec ${SNAP}/rel/couchdb/bin/couchdb
diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml
index 40ca112..8dd7dea 100644
--- a/snap/snapcraft.yaml
+++ b/snap/snapcraft.yaml
@@ -1,5 +1,5 @@
name: couchdb
-version: 2.0
+version: 2.2.0
summary: Document based database
description: CouchDB is a database that completely embraces the web. Store your data with JSON documents. Access your documents and query your indexes with your web browser, via HTTP. Index, combine, and transform your documents with JavaScript.
confinement: strict
@@ -9,20 +9,19 @@ apps:
server:
daemon: simple
command: rel/couchdb/bin/snap_run
- plugs: [network-bind, process-control, mount-observe, browser-support]
+ plugs: [network-bind, process-control]
couchdb:
command: rel/couchdb/bin/snap_run
- plugs: [network-bind]
+ plugs: [network-bind, mount-observe, browser-support]
parts:
couchdb:
plugin: make
- #source: http://www-us.apache.org/dist/couchdb/source/2.0.0/apache-couchdb-2.0.0.tar.gz
- source: https://github.com/apache/couchdb.git
- source-type: git
- source-tag: 2.1.0
- prepare: |
+ source: http://www-us.apache.org/dist/couchdb/source/2.2.0/apache-couchdb-2.2.0.tar.gz
+ #source: https://github.com/apache/couchdb.git
+ #source-type: git
+ #source-tag: 2.2.0
+ override-build: |
./configure --disable-docs
- build: |
make release
cp -ra ./rel $SNAPCRAFT_PART_INSTALL
cp -ra ./bin $SNAPCRAFT_PART_INSTALL
@@ -59,26 +58,17 @@ parts:
plugin: dump
source: ./snap/
organize:
- snap.ini: rel/couchdb/etc/snap.ini
+ couchdb.ini: rel/couchdb/etc/local.d/couchdb.ini
snap_run: rel/couchdb/bin/snap_run
- erlang:
+ packages:
plugin: nil
stage-packages:
- - erlang-base
- - erlang-crypto
- - erlang-eunit
- - erlang-inets
- - erlang-os-mon
- - erlang-public-key
- - erlang-ssl
- - erlang-syntax-tools
- - erlang-tools
- - erlang-xmerl
- - libc6
- - libcurl3
- - libgcc1
- libicu55
- libmozjs185-1.0
- - libstdc++6
-
+ filesets:
+ exclusion:
+ - "-usr/share/doc/*"
+ - "-usr/share/man/*"
+ prime:
+ - "$exclusion"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services