[GitHub] [couchdb-helm] willholley edited a comment on issue #40: Cluster auto-scaling best practices

[GitBox <gi...@apache.org>]

willholley edited a comment on issue #40:
URL: https://github.com/apache/couchdb-helm/issues/40#issuecomment-693410905


   @tudordumitriu I think this is outside the scope of the Helm chart, though I can see how it's related. I would be extremely wary of using an autoscaler to automate database cluster scaling - the chance of data loss is high and the timescales and nuances involved in moving shards around likely require manual supervision anyway.
   
   In the general case, this is a difficult problem, which is probably why there are no community tools to address it (regardless of Kubernetes). One tool you could look at is [couchdb-admin](https://github.com/cabify/couchdb-admin) from Cabify. I haven't used it personally but it looks to automate at least some of the management tasks. 
   
   Unfortunately, the process for moving shards described in [the docs](https://docs.couchdb.org/en/stable/cluster/sharding.html#moving-a-shard) is tricky in Kubernetes because not many storage backends support `ReadWriteMany` [AccessModes](https://docs.couchdb.org/en/stable/cluster/sharding.html#moving-a-shard). You could try exposing port 22 between the CouchDB pods and using SCP. You're likely better off using CouchDB internal replication to create additional shard replicas instead of moving files/indexes directly, but it's a slower process.
   
   Adding a node would require something like:
   
    * Join the new node to the cluster
    * Put the new node in [maintenance mode](https://docs.couchdb.org/en/stable/config/couchdb.html#couchdb/maintenance_mode).
    * For each database in the cluster:
      * figure out the new shard map to ensure even spread of shards and replicas across machines/availability zones. For your new node, add the desired shard ranges/replicas to each shard map. This temporarily increases the number of replicas of each shard. For large numbers of databases/shards, you might need to serialize this process.
        * This should result in CouchDB internal replication creating and populating the database shards on the new node. When design documents are replicated, indexing jobs will trigger on the new node to create the required indexes. For large shards, this may take some time (hours/days).
       * Monitor the internal replication backlog and indexing process to observe when the replication and indexing process is complete. I'm not sure if metrics for these are exposed by `_stats` or whether you'd need to parse the log output (e.g. look for `[warning] <date> <existing node> <pid> -------- mem3_sync shards/<shard range>/<db name>.<epoch> <new node> {pending_changes,<value>}`) to determine the internal replication backlog. Indexing backlog can be queried using the `_active_tasks` endpoint.
      * If/when internal replication and indexing is up to date, take the node out of [maintenance mode](https://docs.couchdb.org/en/stable/config/couchdb.html#couchdb/maintenance_mode).
      * Update the shard map to remove the original replicas that are now redundant
   
   Removing a node from the cluster would be a similar process, updating the shard map to ensure enough shard replicas exist on the remaining nodes before removing it.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org