Chewbranca's Opinionated Vision for Apache CouchDB and Cloudant Dbcore

[Russell Branca <ru...@chewbranca.com>]

Hi all,

Chewbranca here. I've been fairly quiet for the last few years for a number of
reasons, but this year I've been becoming more involved again in the CouchDB
community and various Cloudant communities. I put together this writeup a while
ago to detail out what my personal views are on what I believe to be some of the
highest priority focus points for CouchDB/Cloudant and targeted approaches
for tackling these issues. Full disclosure, these are my own opinions discussing
my views on direction for Cloudant and CouchDB; most of the views here are
shared by a wider audience, but I want to be clear this is not a formal proposal
from Cloudant but rather the musings of a long standing Cloudant and CouchDB
member with plenty of strong opinions.

The first draft of this document was done as an internal writeup with the
intention of then creating a modified version that defines a set of core
"stakeholders" such that the various tasks defined demonstrate value to the core
stakeholders, but in particular, that all of this functionality is defined and
executed in a way to bring clear value to the Apache CouchDB project and Apache
CouchDB community. I strongly believe the majority of this functionality should
be in the core CouchDB database itself and I want to ensure that it is done in a
way to be clearly beneficial to _all_ stakeholders involved. I started adding in
the discussion of target audience and stakeholders, but the breakout details for
each of the discussion points below is incomplete.

Forgive the lack of polish in this writeup, it's still a draft, but I've already
sat on this longer than I wanted, and with the upcoming CouchDB developer
meeting tomorrow, I felt it was better to get it out now in a rough form than
worry about a completed writeup. We can always rip various pieces out and create
more specific and complete writeups. I've got some additional writeups and ideas
to get out to the mailing list as well, but this is a good starting point.

A quick note on naming, "Dbcore" is the name of the version of CouchDB that
Cloudant runs in production. Many years ago this was an actual fork, but since
the 2.x merge we're essentially running CouchDB upstream with additional
libraries added on for Cloudant specific functionality, like IAM auth and what
not. The majority of the fun stuff was open sourced and incorporated into the
3.x codebase. This document switches between Dbcore and CouchDB in various
areas, but again, this is more of a draft than a finished document.

Thanks for your time and happy hacking!


-Russell "Chewbranca" Branca


==============================================================================


# Chewbranca's Immediate Vision for Cloudant Dbcore and Apache CouchDB


Welcome to my vision of where to go with development and functionality of
Cloudant's Dbcore and Apache CouchDB, for the immediate future in 2022/2023.
This is an opinionated list of the things that I think are critically important
and also readily achievable. We may or may not get to the things in this list,
but here it is.

There's much to do, much that could be done, and endless possibilities. This
vision is targeted at the here and now, focused on immediately prevalent issues,
constraints, and challenges, skewed towards heavy hitters in terms of
time/resource investment to dividends paid. Not everything here is easy and
simple, but everything here is cherry picked because of its significant impact,
and also selected because it *needs* to be here.

This vision is also not exhaustive, to neither Cloudant nor CouchDB. I make no
claims that there isn't other important things to do, rather that the following
list of things *is* of substantial importance and provides a path to paradigm
shift the current state of Cloudant and CouchDB.


## Target Audience and Stakeholders

I want this writeup to target a few different core groups of people and
stakeholders to help understand motivations and help define scope. In
particular, I'm targeting three core groups of people:

  * CouchDB users/operators
  * Cloudant customers
  * Cloudant operators

As it stands today, these groups have very distinct interactions with the
database, and while there will also be distinctions between users of the system
and operators of the system, I think there's far more overlap between the three
groups than is currently represented. In particular, all three of these groups
need visibility into what the system is doing, what are the most active
databases and documents, and what impact requests have on the system. Both users
and operators need to have this visibility, and the more we can do to make it
easier for everyone to see into the activity of the system, the better.

The distinciton between Cloudant operators and CouchDB operators is considerably
different as the tooling is fundamentally different, or rather heavily lacking
in CouchDB and attached at the periphery in Cloudant; which is not ideal nor is
sustainable in my opinion.

I think we can greatly improve the operability and visibility of the system for
all three groups of people by streamlining the tooling and building core
enhancements into CouchDB. Much of this writeup is geared towards cherry picked
improvements I think can have a significant impact. In the sections below, I've
elaborated on what benefit each of the different groups receives from the
changes and improvements, in order to help scope and justify my recommendations,
but also to make sure that the core stakeholders all benefit from these
improvements, and that we can minimize the differences of interactions with the
system between the groups.


Core goals of CouchDB users/operators:

  * Stable, reliable, performant, operable, and introspectable database system
  * Ability to pin point expensive requests
  * Visibility into what operations are utilizing resources
  * Easily scalable without hassle and complex re-architecting
  * Easy to operate, but also easy to know what to operate


Core goals of Cloudant customers:

  * Stable, reliable, performant, and introspectable database system
  * Ability to pin point expensive requests
  * Visibility into what operations are utilizing resources
  * Easily scalable without hassle and complex re-architecting


Core goals of Cloudant operators:

  * Stable, reliable, performant, operable, and introspectable database system
  * Ability to pin point expensive requests
  * Visibility into what operations are utilizing resources
  * Easily scalable without hassle and complex re-architecting
  * Easy to operate, but also easy to know what to operate


## Core Facets


All items below are directly focused on one or more of the following facets:

  * Scalability
  * Performance
  * Reliability
  * Robustness
  * Operability
  * Introspection
  * Predictability

Let's provide a quick description of what each of those means to me, what I see
are the core problems, and what are the core goals.


### Scalability


CouchDB is a durable distributed document store that can scale horizontally with
the number of nodes but also vertically with the volume of computation and
storage per node. CouchDB is managed by end users themselves, but knowing how
and when to scale a CouchDB cluster is still a dark art.

Cloudant is a "Cloud" database, part of IBM Cloud, that is powered by CouchDB.
Magical scaling is a primary expectation of modern Cloud computing, regardless
of the validity of that expectation, especially when taking into consideration
physical properties of shuffling bits around at the speed of light.

To me, our immediate concerns around scalabiity are being able to scale user
applications nearly linearly as we introduce more hardware, doing this
automagically for them, not needing operator intervention to do so, and
minimizing the ways customers can sabotage themselves in this endeavor. We
should be able to scale data volume and query computation independently of each,
allowing for a wider variety of workloads.

Core issues:

  * Q scales single shard operations ~linearly, but inversely for aggregate
    operations. This is the fundamental scaling constraint of Cloudant/CouchDB
    and desperately needs to be solved.
  * IOQ2 has problematic workflows to address before it can be the default
  * Re-sharding is a manual process, should not necessitate ops involvement
  * Storage volume entirely too coupled to node count
    - we need to be able to add additional hard drives independently of the
      amount of computational power in the cluster
    - Storage vs CPU are different axis and should be decoupled

Core Goals:

  * Decouple Q from computational complexity of aggregate queries
    - especially view queries, game changer. TXE delivered on this, now Dbcore
      needs to as well
    - See my "On the Scalability of CouchDB and Dbcore" writeup
      - NOTE: I'll post this soon
  * Ship `couch_file` cache, great overall, but also helps decouple Q
  * Ship IOQ2 fixes; enable IOQ2 by default
    - IOQ2 pid per shard fixes primary issues, but opens other questions around
      how we do rate limiting overall
    - Remove Clouseau requests from IOQ2
  * Scale core storage volume and computational capacity independently
  * Support multiple Dbcore volume mounts, drop use of raid 0
    - Have a volume mount per drive, rather than a raid 0 stripe on all drives
    - Balance database creation across all drives, rebalance during compaction

Core stakeholder benefits:

  * CouchDB users/operators:
    - more scalable and performant CouchDB system
    - eliminates many core scalabiliy constraints
    - easier to run servers with multiple drives
    - more predictable scaling without needing lots of databases

  * Cloudant users:
    - more scalable and performant CouchDB/Cloudant service
    - eliminates many core scalabiliy constraints
    - more predictable scaling without needing lots of databases

  * Cloudant operators:
    - easier to run servers with multiple drives
    - more predictably scalable service, easier to operate


### Performance


Performance can always be better, and how fast/slow we go heavily dictates the
markets and use cases we can serve. The vision here is less focused on making
things go faster, and more focused on preventing things from going slower. In
addition to not going slower, we should be able to better understand the
performance of the system and the _why_ of its performance.

Core issues:

  * Computational complexity of aggregate operations scales with Q
    - This *MUST* go away, full stop.
    - Eg view query performance needs to be dependent on rows queried, not
      overall size of the database or number of shards
    - CouchDB 4.x/TXE solved this, Dbcore needs to catch up
  * Every cluster is a performance snowflake; as is every request!
    - We need bounded performance, can't have individual requests invoke
      millions of operations, especially when that happens invisibily
    - Lack of cost accounting and visibility, can't even find the bad/expensive
      requests. This needs to be exposed to both ops and customers
  * Lack of back pressure results in unpredicatably "bad" behavior and
    performance. Back pressure forces overload errors back to the client, but
    does so in predictable and performant manners and allows us to cap unbounded
    queues and growth
  * IOQ2 noisy shards issues

Core goals:

  - See scalability core goal for "Decouple Q from computational complexity..."
  - No more unbounded growth!!!
  - Introduce back pressure
    - IOQ read timeouts, bubble back to caller
    - Throw 503 errors *IMMEDIATELY* rather than eventual 5xx errors after
      unbounded growth tanks the system
  - Eliminate "bad requests"; no "million dollar" requests
  - Extend CCM rate limits further into Dbcore
  - Fix IOQ2 to avoid noisy neighbors and can enable it by default
  - Potentially add rexi/fabric client/server rate limited queues
  - Introduce couch_file cache as a way to utilize heavy Erlang parallelism

Core stakeholder benefits:

  * CouchDB users/operators:
    - predictable performance predictable overload
    - minimize unknown/unexpected behavior
    - make it easy to diagnose performance and overload issues
    - no unbounded requests

  * Cloudant users:

  * Cloudant operators:


### Reliability & Robustness


Before addressing these individually, I want to briefly cover both to clarify
the distinctions between the two that I'm highlighting here, as they're quite
similar. From the Google Oxford Dictionary definitions, we have:

  * Reliability: the quality of being trustworthy or of performing consistently
    well.
  * Robustness: the quality or condition of being strong and in good condition.

For the purposes here, I'm looking at "reliability" as Dbcore's ability to
provide predictable and consistent performance, on demand; and "robustness" as
Dbcore's ability to continue to operate in the face of innovative internal
failure scenarios and adverse external stimuli induced by customers. (For
simplicity in both of these definitions, I'm assuming non malicious load from
3rd parties and that the results delivered are "correct").

I want to take a step back and commend the various Cloudant employees and
CouchDB committers over the years, Dbcore/CouchDB is a pretty *damn* reliable
and robust database! Great job folks, Dbcore/CouchDB have been battle hardened
over many years of service and care, and we owe it to ourselves to remember that
its put in work day in day out for well over a decade. Huzzah and
congratulations!


### Reliability


Again, I'm using the definition of "Reliability" as "the quality of being
trustworthy or of performing consistently well".

Dbcore/CouchDB is a pretty damn reliable system... until it's not. Under normal
operating conditions the system is very reliable and performant, with minimal
issues arising. Unfortunately, shit inevitably always goes wrong, and that's a
major reason why customers pay us to provide the Cloudant service, so that when
things inevitably go wrong, we're there to clean up the mess. How we choose to
respond to issues and how we choose to have the database respond to issues
largely dictates our day to day operational overhead and also dictates what
types of service impairment that are exposed to the customer. With dbcore, for
better or worse, we strive to *ALWAYS* be present and available, to always
return _something_, and to always save _something_ when data is written.

There are many advantages to this approach, but we err far too heavily on the
side of _always_ trying to perform the actions that have been asked of the
system, rather than establishing healthy boundaries and being able to say "no".
We allow the system to be stretched to its limits, beyond, and then even further
until it snaps violently. It's a classic example of trying to take on too much
work until the limits are surpassed and everything comes crashing down.

Fundamentally, we cannot continue to allow unbounded growth of pending work to
go into the system, and we need to introduce backpressure allowing us to say no
to additional work when overloaded. This is absolutely essential to achieving
consistent and trustworthy performance. A system that continues to take on more
work while progressively responding slower until it's so far backlogged that it
topples over and drops everything on the floor is *not* behaving in a
trustworthy manner, and is *not* performing consistently well.

As a service, saying "no" and throwing `503 Service Unavailable` errors are
often viewed as abject failure, but, all services inevitably fail, the question
is whether or not those systems fail in a trustworthy and consistently
performing way. Running reliably on a good day is easy, failing reliably on a
bad day is the true measure of the health and reliability of a service.

Core issues:

  * Unbounded queue growth
    - can't answer every request in the world, need to be able to say "no"
  * During overload, system performs progressively worse, progressively more
    inconsistently, and progressively less trustworthy
  * Need to choose reliability instead of a "never say no" mentality

Core goals:

  * Introduce backpressure and load shedding
  * Embrace 503 errors
  * `infinity` is not real, stop pretending it's an ok duration for requests
    - IOQ timeouts of infinity need to go away
    - Load shed IOQ work that has timed out
  * Stop RPC work when coordinator request no longer present
  * Eliminate unbounded queue growth
  * Eliminate unbounded request duration, magnitude, and impact
    - Eg `/_changes` with a custom Javascript filter from `?since=0`


### Robustness


Again, I'm using the definition of "Robustness" as "the quality or condition of
being strong and in good condition."

Dbcore/CouchDB is a pretty damn robust system... until it's not. Under normal
operating conditions the system is very strong and resilient, with minimal
issues arising. Unfortunately, shit inevitably always goes wrong. Failures are
mostly well isolated in the system, in large part thanks to Erlang's OTP system
we make extensive use of, but also because we have spent a *lot* of human hours
diagnosing, debugging, fixing, and improving the system. That said, I think we
have improvements to make on our modus operandi in terms of what our catalysts
are for finding and fixing bugs. To provide a consistently strong and in good
condition service, we need to be aggressive with eliminating known defects,
minimizing the introduction of new defects, and auditing the system regularly to
keep ourselves honest.

I think a lot of our Robustness issues are shared with the Reliability category
in terms of allowing the system to get backed up with unbounded growth until it
crumbles, so I won't reiterate those here. Instead, for Robustness, I want to
focus on how we approach finding and fixing bugs.

Right now, we have an abundance of "bug reports" in our internal logging
systems, Dialyzer, xref, and other tools. These are factual bugs that we ignore
by letting the OTP system restart crashed processes, or by triggering full
dbcore reboots. We ignore these errors unless they're sufficiently service
impacting or a customer yells loudly enough. Many of these bugs have isolated
impact, but a number of them end up tanking larger parts of the supervision
hierarchies, and I suspect are responsible for a lot of service "blips" we
encounter. On the more extreme side, we have bugs like the `all_dbs_active`
errors that render the system in an unknown impaired state, with partial and
sporadic functionality.

We need to fix bugs we're aware of, eliminate the excessive quantity of logs
generated by these logs, and improve the service by eliminating various known
sharp edges. Most of these bugs are readily accessible to be fixed, and
essentially have bug reports provided for us that we ignore. Some of these bugs
are more complicated to resolve than others, but there's considerable low
hanging fruit here to fix for large impact.

The `all_dbs_active` bug is an interesting case study here. The fundamental
issue is that opening a database handle inside CouchDB/dbcore can sometimes
return the error `all_dbs_active`, and most callers are not prepared to handle
that error, as it's not a common error and we currently do not ensure all errors
that can arise are properly handled. The major problem here is that any
subsystems unequipped to handle this error at best crash, or at worst, partially
fail in mysterious ways rendering various subsystems unresponsive or behaving in
unexpected ways. What's especially interesting in this case study is that I was
able to add a three line Dialyzer patch codifying the `all_dbs_active` error
return type and immediately it found sixteen instances of the unhandled errors!
This was a fantastic result and makes me incredibly excited by the prospect of
more heavily utilizing Dialyzer, which will deliver us bug reports with minimal
investment.

I think fixing known bugs, reducing the volume of bug logs, and leaning on
Dialyzer (and xref) to provide some powerful static analysis allowing us to
track down existing bugs, minimize new bugs, and have more confidence in the
codebase will go a long ways towards improving the Robustness of CouchDB as a
whole, but also the dbcore service.

Fixing random small bugs is not glorious, but I really do think it's one of the
most important aspects of this entire writeup.

Core issues:

  * Bug reports in logs are ignored
  * Bug reports in Dialyzer are ignored
  * Bug reports in xref are ignored
  * Brutal bugs like `all_dbs_active` exist and their extent/impact are unknown

Core goals:

  * Audit dbcore logs regularly and get all bugs into backlog to be fixed
  * Establish a culture of checking dbcore logs and keeping them minimal
  * Revamp usage of Dialyzer:
    - Greatly extend spec coverage of the code base
    - Fix bugs Dialyzer finds
    - Establish a culture of fixing new Dialyzer bugs found
  * Make use of xref
    - this is a no brainer, great tool, easy, and just delivers bug reports
  * Fix `all_dbs_active` error
    - Calling it out individually here as it's gnarly and exemplary of long
      standing bugs with massive service impact. This bug affects nearly all
      core subsystems of CouchDB/dbcore


### Operability & Introspection


Similarly to Reliability and Robustness, there is a lot of overlap between
Operability and Introspection. For the purposes here, I'm focusing on
Introspection as our ability to understand what is happening in the cluster, and
Operability as our ability to interact with and modify the underlying state of
the cluster. These two have considerable overlap but have potentially different
audiences.


### Operability


My number one priority for Operability is to make it easier for Cloudant
operators and CouchDB users to operate clusters by simplifying what needs to be
done and reducing the amount of things that need human intervention. This is not
an exhaustive list, rather a cherry picked set of things I think are high
priority, and as such, there are two core focuses here.

First off is resharding. This *needs* to become a first class construct of
CouchDB/dbcore that is completely hands off for operators. I think we should
take it a step further and automate the resharding logic based on write volume
to the individual shards, but that requires disconnecting Q from the
computational complexity of view queries. So as a first step, we should
internalize resharding into CouchDB such that it's self managing and interacted
with by a jobs API similar to the replicator jobs system (although ideally
without direct database operations). Making it seamless so that operators do not
need to be involved outside of initiating the jobs is essential. Additionally,
this functionality should also include rebalancing, as the two are very similar
in functionality and if thought through properly could share the majority of
logic. Both of these items should also be done in a way that can be readily
regulated, allowing for safe migrations with minimal impacts on production
workloads.

The next up is making core operational tooling a first class construct of
CouchDB. Cloudant independently deploys operational tooling alongside Dbcore so
that it can be deployed outside of normal release processes as operational
issues arise. This is always done with the intent of moving it into CouchDB
core, but in practice it has a tendency to stay to the side, rather than
becoming core functionality of CouchDB that all users can benefit from. We need
to make the operational tooling a core part of CouchDB so that it's maintained,
improved, documented, etc, along with the core database. There's also many
improvements that can be made here, but getting operational tooling as a core
construct of the database is the first step.

Core issues:

  * resharding/rebalancing, while necessary and standard operations, are very
    complicated, error prone, and labor intensive
  * Lack of internalized operational tooling
  * Insufficient introspection to properly understand system usage and
    saturation

Core goals:

  * Managed resharding/rebalancing engine
    - ideally automatic based on write volume in conjunction with sorted key
      order view sharding
  * First class operational capabilities in CouchDB/dbcore
    - migrate Cloudant secondary opts tooling into core database
    - establish culture of operational tooling in core database
    - we can and should do better on this front
  * Increased introspection capabilities to properly understand system usage,
    saturation, and to faciliate finding hot spots


### Introspection


Visibility is critical to understanding system usage, saturation, and overall
behavior. CouchDB has a variety of introspection capabilities, and combined with
the Erlang VM you can technically see whatever you want, but that doesn't make
it easy to see what's going on. My number one issue with CouchDB/dbcore
visibility is that we can't see the things we don't know about! With the large
variety of API capabilities and interactions within the core database, there is
much we're not aware of the underlying cost. For instance, doing a filtered
changes feed from the beginning of a database's changes seq with a moderately
large limit and a filter function that returns a small subset of the data is a
singular http request, it's logged as a singular http request, it creates
individual Erlang processes for the http request handler and the rpc handlers
per shard, but it can potentially manifest as doing an entire database scan that
runs every document through Javascript engine filters, to return less than the
limit of rows, and is damn hard to find if you're not looking for it.

CouchDB has a widely diverse API in terms of underlying resource cost of
satisfying the request. Individual doc lookups load a single doc from `N` shard
replicas, whereas filtered changes feeds can perform an entire database scan and
Javascript invocation for every doc in the system across all shard ranges, and
then views land somewhere in the middle where they're directly proportional to
the number of shards in the database. This leaves the conundrum of: do you scale
your database for single doc reads, or for view queries? In the current
scalability model of CouchDB, you must increase the number of shards to increase
the capabilities of CouchDB to utilize available CPU cores. Single doc
operations scale beautifully with the number of shards, for a single database or
many, whereas aggregate operations like view queries scale inversely with the
number of shards in the database, requiring a larger number of small databases
for horizontal scalability.

CouchDB 4.x fixed this by way of key ordered sharding that results in only
needing to interact with the appropriate shards for the query, and we can do
that on the 3.x codebase line as well, as I've written up elsewhere. But this is
the `Introspection` section of this vision, not performance, so the relevant
point here is that the wildy different consumption of resources per request type
is *challenging* to introspect and understand, and that needs to change. We need
to introduce cost accounting to how CouchDB utilizes resources both as a
function of incoming http requests and internal background jobs.

Luckily, I think getting this data and exposing it is not only one of the easiet
tasks on this list, but also one of the most impactful. Understanding what the
system is doing, where it's consuming resources, and why, is absolutely critical
for anyone interacting with CouchDB, from users to operators. And even to
developers, because without properly visibility into resource consumption, it's
challenging at best to understand what the expensive operations are. From my
analysis, view query performance computational complexity is roughly `O(Q^2 *
log(Q))`, and I've been vocal about that for a long time now, but things like
innocuous changes requests consuming massive resources continues to catch me off
guard because we can't see the data on _what_ is expensive without studying
particular areas in depth.

This is a solvable problem, we need to begin cost accounting database operations
for number of rpc calls, number of nodes involved, number of shards involved,
number of b-tree lookups, number of IO requests, quantity of data, number of JS
invocations, number of mergesorts, rev count of docs involved, magnitude of
requests in terms of docs processed and docs returned,  number of Erlang
messages sent, etc, and then we need to log this data in a way that is
introspectable retroactively and also made readily available for real time
visibility in Fauxton and various metric engines. Knowing the most expensive
requests and where the system is spending it's time is useful for _everyone_ who
interacts with CouchDB/Cloudant.

We have obvious scalability bottlenecks that can be fixed and targeted, but
until we get proper visibility into the cost of various aspects of the system we
won't know about the non obvious areas we need to fix, and we won't be able to
easily see _where_ those bottlenecks are being pushed. It's one thing to know
that you're overloading the JS engine, it's another thing to know why and where
that's happening. Similarly, it's one thing to know document updates are slow,
it's another thing to understand that it's because of that one heavily
conflicted document tanking performance.

Core issues:

  * Incredibly difficult to find low quantity but high impact requests
  * Lack of database resource accounting means we don't know what the system is
    doing at any given time or why, making it difficult to operate, difficult to
    scale and capacity plan, and very difficult for customers to understand the
    impact of their workload on the underlying system

Core goals:

  * Introduce cost accounting
    - Track resource consumption of all http requests and internal traffic
    - Log this data in an auditable way
    - Expose internal realtime tooling for operators to directly visualize
      current activity on the cluster
    - Expose this functionality to Fauxton so CouchDB and Cloudant users can
      utilize this info
    - Per database API endpoints making it easy to get isolated stats
  * Use cost accounting as a way to understand what the expensive requests are
    and make informed decisions about areas to fix to reduce said cost
  * Reduce log volume as detailed above, making logs more readily useful and
    more indicative of relevant/new issues


### Predictability


The most "predictable" thing about CouchDB and dbcore right now is that the
database will do it's best to handle whatever load is thrown at the system,
regardless of its ability to handle said load, which leads to unpredictable
behavior and fireworks. While it's noble to attempt to handle _all_ requests
thrown at the system, reality is that there are finite limits to what the
database can handle, and ignoring that reality leads to unpredicatable ways in
which the system can fail and when. This is not a good user experience for
anyone, and makes it very challenging to understand _why_ the system fails, when
it _will_ fail, and what the upper bounds are. Waiting for the system to trigger
timeouts and/or get OOM killed is ignoring the problem rather than a direct
approach to handle overload and introducing backpressure.

In my experience, the current approach makes it very difficult to understand and
predict how the system will behave when it aproaches saturation limits and
overload, which results in users being perplexed as to why things went bad, and
leads to operators going on wild goose chases to understand why particular
subsystems did or did not fail when they're all emblematic of the same overload
failure scenarios.

Bottom line is that in my opinion we should strive to throw a predictable
overoad error allowing the system to continue operations in a predictable manner
rather than allowing unbounded growth of queues until overload causes timeouts
and/or full system failure.


Core issues:

  * Unpredictable failure modes, but predictably inevitable
  * Unbounded growth leads to degraded performance and eventual total failure
  * Failures arise somewhat arbitrariliy as a function of system saturation,
    making it difficult for operators to understand and near
impossible for users

Core goals:

  * Introduce backpressure to deliver predictable errors at overload rather than
    unexpected total system failures and/or excessively long timeouts
  * Expose backpressure in a manner that makes it easy for the underlying user
    to understand what limits they tripped
  * No more unbounded growth
  * No more infinite timeout requests


## Core Audiences and Stakeholders


I intend to expand this section more to define a few stakeholders along the
lines of Cloudant users, CouchDB users, Cloudant operators, Cloudant/CouchDB
developers. I think this is useful in general, but in particular for this
writeup, I think the majority of the items here require core changes to CouchDB
and should not be done by way of plugins, so we need to ensure they're
beneficial to not just Cloudant but also all CouchDB users. A lot of these items
are self evident, like fixing bugs is obviously beneficial, but others are more
focused on hardening the core service, and for those, I think we need to make
these changes in ways that make it much easier for not just Cloudant operators
and Cloudant customers to gain visibility, insights, and operability in the
system, but for anyone using CouchDB to benefit from this visibility and ease of
use. Making it easier for CouchDB users to understand what the database is
doing, where it's spending resources, what problems arise, and being able to
take action to diagnose and alleviate issues is not only incredibly valuable to
the CouchDB community as a whole, but IMO would also go a long ways in making it
easier for non core Cloudant/CouchDB developers to understand and operate the
database as it's far too complex and hidden currently.

The initial version of this writeup is for internal Cloudant audiences, but most
everything here is directly applicable to CouchDB and as such I intend to make a
modified version of this list to introduce to the CouchDB mailing list in the
near future to be part of the CouchDB 3.x vs 4.x discussions.


## Other Areas of Interest

There are a handful of other areas of interest in Cloudant and CouchDB that are
pain points and/or need to be addressed. Again, this list was never meant to be
exhaustive, and rather tightly focused on things I believe we *can* and *should*
deliver soon, but for the sake of visibility I'll mention a few here. Also, a
purpose of this writeup is to connect various threads of communication I've had
with various overlapping groups of people into one place, so a few of these
items below are outlined below as further avenues of discussion.

  * Conflicts...
    - Yeap this is a gnarly one, nope I don't have an immediate simple solution
      in mind so it's not listed in the above list
  * CouchDB/Cloudant "Metal"
    - CouchDB/Cloudant "Metal" is the codename I've been bouncing around for the
      idea of a hardened CouchDB service built from the ground up to only focus
      on a subset of functionality in the current API that we can scale
      aggressively and predictably.
    - Due to time constraints I have not outlined this further here, but this is
      an intriguing option that maps well to both CouchDB users and Cloudant
      services billing by request/usage volume
    - The key thing is starting with a subset of core functionality we take
      confidence in our ability to scale, and then go from there, only adding
      additional layers to the stack we can scale properly
    - I think the best way we can encourage users to move off of expensive
      CouchDB/Cloudant API operations is by incentivizing them to use more
      scalable operations that utilize less hardware resources and then provide
      clear introspection capabilities demonstrating the impact of the slower
      and more costly API operations utilized
  * FDB style Namespacing
    - If we adopt the approach outlined in my CouchDB/Dbcore
scalability writeup, I
      believe we could readily adopt a Namespace model like FDB utilizes, making
      a lot of the notions about databases and shards and on disk files go away.
      This would work fine for views, but Search would be trickier, although
      that's the case for Search in that scalability approach in general, I
      suspect we'll need Search to be a standalone service long term
  * Reintroduce Spans/Opentracing/whatever this thing is called
    - I really like the Span system we were putting into place allowing us to
      understand performance and throughput and various "spans" of the stack.
      Hopefully things have stabalized and we can use this someday
  * Disaggregated/heterogeneous CouchDB/Cloudant Architecture
    - CouchdB/Dbcore is a distributed mostly stateless http app stack talking to
      a distributed database app all coupled into one system. This has served us
      well and has a number of advantages, but I think long term, especially for
      a CouchDB/Cloudant Metal like system, we need to properly isolate the
      storage layer allowing us to scale that independently from the http stack,
      the indexing stack, replication stack, etc.
  * New features in BEAM, and all the various BEAM VM args
    - Tons of low hanging fruit on the VM args options, things like starting
      process heap size, or memory carriers, let alone all the various IO tuning
      knobs are a wealth of potential
    - We're also leap frogging up to OTP 23, there's been considerable
      advancement in OTP over the years, what can we incorporate?
  * Consolidated Cloudant and CouchDB auth systems
    - Cloudant's auth system (not the IAM one) and CouchDB's auth system have a
      lot of overlap but neither are full replacements for the other and both
      have functionality desired in the other, we should merge these down to a
      singular more capable auth system with proper RBAC
  * Consistent and Simple dev environment
    - Enough with having this be a mess. We need a simple consistent dev
      environment. I don't believe the argument around having folks use
      different setups to test things has panned out well, but it has certainly
      wasted much time and I don't want to keep dealing with this. Obviously
      still contentious and has some issues, I personally want a reliable and
      maintained Vagrant VM setup and provisioner that can easily bootstrap all
      the things. I extracted the core setup I used a few years back:
      https://github.com/chewbranca/hackcouch


## Next Steps

This is currently an initial version of this document and I'm getting it out to
a wider audience to get more feedback sooner rather than later, especially with
the CouchDB 3.x vs 4.x discussion happening now. I intend to continue refining
this document a bit, and making a version we can share publicly. The other
primary next step for myself is I intend to detail out the various `Core Goals`
bullet points as actionable items in distinct tickets, with better definition of
goals and scope for that particular item. We can then
discuss/edit/approve/delete those items, but I think having separate tickets
will simplify the discussion around many of these issues and help us see how to
achieve them.